Rule engine-based edge fusion intelligent monitoring method and device, equipment and storage medium

Abstract

The invention discloses an edge fusion intelligent monitoring method, device and equipment based on a rule engine and a storage medium, and relates to the technical field of edge computing monitoring, and the method comprises the steps: accessing environment sensor data through various hardware interfaces, and generating sensor data resource information according to the environment sensor data; acquiring video stream data of the camera device, and performing image recognition analysis on the video stream data to generate image analysis resource information; a logic incidence relation between input resources and output resources is configured in the rule engine, the input resources comprise sensor data resource information and image analysis resource information, and the output resources comprise control instruction resource information; and performing fusion calculation on the sensor data resource information and the image analysis resource information according to the logic association relationship, generating control instruction resource information, and sending the control instruction resource information to the target equipment. According to the method, the adaptive control of the external equipment is realized in cooperation with the multi-source heterogeneous data.

Description

Technical Field

[0001] The present application relates to the field of edge computing monitoring technology, and in particular to edge fusion intelligent monitoring methods, devices, equipment and storage media based on a rule engine. Background Art

[0002] The demand for equipment monitoring and management is growing in scenarios such as industrial production, special operating environments, and traditional power plant rooms. These scenarios require not only real-time monitoring of equipment operating status but also awareness of the surrounding environment to ensure safe and efficient operations. For example, in industrial crane operations, real-time monitoring of the crane's operating status and surrounding personnel activity is required to prevent accidents. Furthermore, the advancement of industrial automation and intelligentization is placing higher demands on the flexibility and real-time performance of monitoring systems.

[0003] Currently, traditional data collection devices typically collect various sensor data through protocols or other means, perform simple processing on this data, conduct alarm analysis based on preconfigured parameters, and then send the data directly to the platform. Gateway devices support connecting live video data to the network through network interfaces, but these devices have limited computing power and cannot recognize and process image data at the edge. They can only forward video data through routing to the platform for unified analysis and processing.

[0004] However, traditional data acquisition devices lack the computing power to perform image recognition and data fusion processing at the edge, resulting in poor real-time data processing. Furthermore, custom modifications are often required to meet the fusion requirements of different scenarios, resulting in long development cycles and a high workload. This customized development approach struggles to adapt to changing application scenarios and reduces the flexibility of device reuse. Therefore, how to coordinate multi-source heterogeneous data to achieve adaptive control of external devices has become a pressing issue.

[0005] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0006] The purpose of this application is to provide an edge fusion intelligent monitoring method, device, equipment and storage medium based on a rule engine, aiming to solve the technical problem of how to coordinate multi-source heterogeneous data to achieve adaptive control of external devices.

[0007] To achieve the above objectives, this application proposes an edge fusion intelligent monitoring method based on a rule engine, the method comprising:

[0008] Accessing environmental sensor data through a variety of hardware interfaces and generating sensor data resource information based on the environmental sensor data;

[0009] Acquire video stream data from a camera device, and perform image recognition analysis on the video stream data to generate image analysis resource information;

[0010] Configuring a logical association relationship between input resources and output resources in a rule engine, wherein the input resources include the sensor data resource information and the image analysis resource information, and the output resources include the control instruction resource information;

[0011] The sensor data resource information and the image analysis resource information are fused and calculated according to the logical association relationship to generate the control instruction resource information, and the control instruction resource information is sent to the target device.

[0012] In one embodiment, the steps of acquiring video stream data from a camera device, performing image recognition analysis on the video stream data, and generating image analysis resource information include:

[0013] Obtain video stream data collected by the camera device through the Ethernet interface;

[0014] Performing human figure detection analysis on the video stream data to obtain image analysis data including detection time, detection area, and number of human figures;

[0015] The image analysis data is associated with a device identifier of the camera device to generate image analysis resource information.

[0016] In one embodiment, the step of performing human figure detection analysis on the video stream data to obtain image analysis data including detection time, detection area, and number of human figures includes:

[0017] Decoding the video stream data, extracting a single frame image and adjusting it to a preset resolution;

[0018] Input the adjusted single-frame image into the preset deep learning human figure detection model to obtain the human figure bounding box coordinates and confidence level;

[0019] generating detection area polygon data according to the humanoid bounding box coordinates, wherein the polygon data is composed of a bounding box vertex coordinate sequence;

[0020] Counting the number of human figures in the adjusted single-frame image whose confidence level is greater than a preset confidence threshold, and recording the current system time as the detection time;

[0021] The detection area polygon data, the number of human figures and the detection time are structurally packaged to obtain image analysis data.

[0022] In one embodiment, the input resources include the sensor data resource information and the image analysis resource information, the output resources include control instruction resource information, and the step of configuring the logical association between the input resources and the output resources in the rule engine includes:

[0023] Loading the sensor data resource information and the image analysis resource information into a rule engine;

[0024] associating the crane operation status in the sensor data resource information with the human figure detection result in the image analysis resource information as input resources;

[0025] Associate the alarm control instructions and the crane start and stop control instructions as output resources;

[0026] The logical association relationship between the input resource and the output resource is configured such that when the crane operation state is started and the human figure detection result exists, the alarm control instruction and the crane start-stop control instruction are triggered.

[0027] In one embodiment, the step of performing a fusion calculation on the sensor data resource information and the image analysis resource information according to the logical association relationship to generate the control instruction resource information, and sending the control instruction resource information to the target device includes:

[0028] Obtaining the crane operation status in the sensor data resource information and the human figure detection result in the image analysis resource information;

[0029] Performing a logical operation on the crane operation state and the human figure detection result according to the logical association relationship to obtain an operation result;

[0030] When the operation result is true, an alarm instruction and a driving stop instruction are generated;

[0031] Encapsulating the alarm instruction and the driving stop instruction into the control instruction resource information;

[0032] The control instruction resource information is sent to the target device.

[0033] In one embodiment, when the calculation result is true, the step of generating an alarm instruction and a driving stop instruction includes:

[0034] When the operation result is true, the protocol interface of the dynamic environment service module is called to generate a switch control signal matching the alarm;

[0035] Call the industrial bus control interface to generate the crane's shutdown protocol instructions;

[0036] Performing transaction association on the switch control signal and the shutdown protocol instruction to generate an associated instruction set;

[0037] A timestamp and a scene identifier are added to the associated instruction set to generate an alarm instruction and a driving stop instruction.

[0038] In one embodiment, the step of accessing environmental sensor data through multiple hardware interfaces and generating sensor data resource information based on the environmental sensor data includes:

[0039] Connect to various types of sensors through industrial bus interface and analog signal interface;

[0040] Parsing the raw data transmitted by the sensor and decoding the raw data according to the protocol types of the industrial bus interface and the analog signal interface;

[0041] Convert the decoded data into a preset unified data format and store it in the dynamic environment business module;

[0042] According to the mapping relationship between the sensor data, device identifier and resource identifier in the dynamic environment business module, sensor data resource information including device type, data attribute and interface type is generated.

[0043] In addition, to achieve the above objectives, the present application also proposes an edge fusion intelligent monitoring device based on a rule engine, the device comprising:

[0044] A data access module, configured to access environmental sensor data through a variety of hardware interfaces and generate sensor data resource information based on the environmental sensor data;

[0045] An image analysis module is used to obtain video stream data from a camera device, perform image recognition analysis on the video stream data, and generate image analysis resource information;

[0046] A rule configuration module, configured to configure a logical association relationship between input resources and output resources in a rule engine, wherein the input resources include the sensor data resource information and the image analysis resource information, and the output resources include control instruction resource information;

[0047] A fusion control module is used to perform fusion calculation on the sensor data resource information and the image analysis resource information according to the logical association relationship, generate the control instruction resource information, and send the control instruction resource information to the target device.

[0048] In addition, to achieve the above-mentioned purpose, the present application also proposes an edge fusion intelligent monitoring device based on a rule engine, the device including: a memory, a processor, and a computer program stored on the memory and runnable on the processor, the computer program being configured to implement the steps of the edge fusion intelligent monitoring method based on the rule engine as described above.

[0049] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the edge fusion intelligent monitoring method based on the rule engine as described above are implemented.

[0050] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the edge fusion intelligent monitoring method based on the rule engine as described above.

[0051] One or more technical solutions proposed in this application have at least the following technical effects:

[0052] First, the edge fusion gateway device accesses environmental sensor data through multiple hardware interfaces and generates sensor data resource information based on this data. This process enables standardized processing of different types of sensor data, providing a foundation for subsequent fusion calculations. Supporting multiple hardware interfaces also enables the device to flexibly adapt to different application scenarios. Next, the gateway device acquires video stream data from the camera device and performs image recognition analysis to generate image analysis resource information. This step enhances the device's perception capabilities, enabling the gateway device to perceive dynamic information within the monitored area, such as human activity, in real time. Then, a logical association between input resources and output resources is configured in the rule engine. The input resources include sensor data resource information and image analysis resource information, while the output resources include control instruction resource information. In this way, the gateway device can quickly make decisions based on pre-set logic. Finally, the sensor data resource information and image analysis resource information are fused and calculated based on the logical association to generate control instruction resource information, which is then sent to the target device. This application achieves adaptive control of external devices by collaborating with multi-source heterogeneous data, ensuring that corresponding control actions can be triggered in a timely manner under specific conditions, thereby improving the overall performance and security of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0055] Figure 1 A flowchart of the first embodiment of the edge fusion intelligent monitoring method based on a rule engine of this application is provided;

[0056] Figure 2 A schematic diagram of the logic flow of the fusion gateway rule engine in a driving scenario provided in Example 1 of the edge fusion intelligent monitoring method based on a rule engine of this application;

[0057] Figure 3 A schematic diagram of the fusion gateway data flow in a driving scenario provided in Example 1 of the edge fusion intelligent monitoring method based on a rule engine of this application;

[0058] Figure 4 A flowchart of the second embodiment of the edge fusion intelligent monitoring method based on a rule engine of this application is provided;

[0059] Figure 5 This is a schematic diagram of the module structure of the edge fusion intelligent monitoring device based on the rule engine according to an embodiment of the present application;

[0060] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the edge fusion intelligent monitoring method based on the rule engine in the embodiment of the present application.

[0061] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0062] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0063] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0064] In scenarios such as industrial production, special operating environments, and traditional dynamic environment rooms, there is a growing demand for real-time monitoring of equipment operating status and surrounding environments to ensure operational safety and efficiency. For example, industrial crane operation scenarios require real-time monitoring of crane operation and surrounding personnel activities to prevent accidents. Traditional data acquisition equipment collects sensor data through protocols, performs simple processing, and then generates alarm analysis and sends it to the platform. Gateway devices forward the video data to the platform for unified processing. However, traditional equipment has limited computing power and is unable to perform image recognition and data fusion processing at the edge, resulting in poor real-time performance. In addition, customized modifications are required to meet the fusion function requirements of different scenarios, resulting in a long development cycle and a large workload. It is difficult to adapt to changing scenarios and reduces the flexibility of equipment reuse.

[0065] The main solution of the embodiment of this application is to access environmental sensor data through multiple hardware interfaces and generate standardized sensor data resource information, while simultaneously obtaining video stream data from camera devices for image recognition analysis to generate image analysis resource information. The logical relationship between input resources (sensor data and image analysis information) and output resources (control instructions) is configured in the rule engine. Based on these relationships, the data is fused and calculated to generate control instructions, which are ultimately sent to the target device to achieve intelligent control.

[0066] It should be noted that the execution subject of the embodiments of the present application can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of implementing the above functions, an edge convergence gateway device, etc. The following uses an edge convergence gateway device as an example to illustrate this embodiment and the following embodiments.

[0067] Based on this, the embodiment of the present application provides an edge fusion intelligent monitoring method based on a rule engine, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the edge fusion intelligent monitoring method based on the rule engine of this application.

[0068] In this embodiment, the edge fusion intelligent monitoring method based on the rule engine includes steps S10 to S40:

[0069] Step S10: access environmental sensor data through various hardware interfaces, and generate sensor data resource information according to the environmental sensor data.

[0070] It should be noted that multiple hardware interfaces refer to the different types of physical connection interfaces supported by edge converged gateway devices, which are used to connect to various external devices and sensors. These interfaces include but are not limited to industrial 485 buses, Ethernet interfaces, serial ports, and analog data output interfaces. Through these interfaces, edge converged gateways can connect to various sensors such as temperature sensors, humidity sensors, pressure sensors, cameras, crane operating status sensors, and smoke sensors, enabling monitoring and data collection of various environments and device statuses.

[0071] Environmental sensor data refers to various physical quantities or status information about the surrounding environment collected by sensors, including parameters such as temperature, humidity, pressure, light intensity, gas concentration, and vibration. In scenarios such as industrial production sites and dynamic environment rooms, environmental sensor data can reflect the real-time status of the equipment's operating environment, providing basic data for monitoring and analysis. For example, in an industrial crane operation scenario, environmental sensor data can include the crane's operating status data, including its start and stop status, location information, and speed. This data is crucial for real-time monitoring and safety control.

[0072] Sensor data resource information refers to processed and converted sensor data. This data is encapsulated into resource information for further analysis and control, enabling integration and logical judgment within the rule engine. For example, temperature data collected by a temperature sensor can be processed and converted into resource information representing the temperature status, such as "temperature too high" or "temperature normal." This resource information can serve as input to the rule engine, triggering corresponding control actions or alarms, thereby enabling intelligent monitoring and management.

[0073] As an example, the steps of accessing environmental sensor data through multiple hardware interfaces and generating sensor data resource information based on the environmental sensor data include: accessing multiple types of sensors through an industrial bus interface and an analog signal interface; parsing the original data transmitted by the sensor, and decoding the original data according to the protocol type of the industrial bus interface and the analog signal interface; converting the decoded data into a preset unified data format and storing it in a dynamic environment business module; generating sensor data resource information including device type, data attributes and interface type based on the mapping relationship between sensor data, device identifier and resource identifier in the dynamic environment business module.

[0074] An industrial bus interface is a standardized interface used for industrial automation and device communication, allowing data to be transmitted between devices over a shared communication line. In this embodiment, the industrial bus interface is used to connect various industrial sensors and devices, such as a crane's operating status sensor.

[0075] An analog signal interface is an interface used to transmit continuously changing signals and is typically used to connect analog sensors such as temperature sensors and pressure sensors. These sensors output continuous voltage or current signals and need to be connected to the edge fusion gateway through an analog signal interface.

[0076] The protocol type refers to the communication protocol followed by the communication interface, which specifies the format, rate, verification method, etc. of data transmission.

[0077] The pre-set unified data format is a standardized data representation method used to convert sensor data from different sources into a consistent format. In this embodiment, the unified data format includes three key components: data value, timestamp, and device identifier. This format enables unified management and processing of different types of sensor data, facilitating fusion analysis within the rule engine.

[0078] The dynamic environment business module is a functional module within the edge convergence gateway, responsible for processing business logic related to dynamic environment monitoring. It receives and stores sensor data converted into a unified data format. The dynamic environment business module is also responsible for analyzing the data according to preset rules and logic, generating corresponding resource information for subsequent control and alarming.

[0079] The device identifier is a code or label used to uniquely identify a device connected to the edge convergence gateway. In this embodiment, each connected sensor or device has a unique device identifier to distinguish different devices and data sources.

[0080] A resource identifier is a code or label used to uniquely identify a sensor data resource. In this embodiment, the resource identifier is used to distinguish different types of sensor data and their corresponding business logic. For example, data from a temperature sensor and data from a pressure sensor can be assigned different resource identifiers to facilitate targeted processing within the rules engine.

[0081] A mapping relationship refers to the correspondence between device identifiers and resource identifiers. In this embodiment, this mapping relationship is used to associate specific devices with corresponding sensor data resources. Through this mapping, the dynamic environment business module can accurately convert collected sensor data into resource information and process it in the rules engine. This mapping relationship ensures data accuracy and consistency.

[0082] The device type refers to the type or category of devices connected to the edge convergence gateway. In this embodiment, device types can include temperature sensors, pressure sensors, cameras, alarms, and so on. The device type describes the function and purpose of the device, helping the system correctly process and analyze data from different devices.

[0083] Data attributes refer to specific characteristics or properties of sensor data, such as the unit, range, and accuracy. In this embodiment, data attributes describe the detailed information of sensor data, enabling accurate analysis and processing within the rule engine. For example, attributes for temperature data might include degrees Celsius or Fahrenheit, and attributes for pressure data might include Pascals or pounds-force per square inch.

[0084] The interface type refers to the type of interface used for communication between the device and the edge convergence gateway. In this embodiment, the interface type includes industrial bus interface, analog signal interface, Ethernet interface, etc.

[0085] First, the edge convergence gateway device connects to various sensors via industrial bus and analog signal interfaces, ensuring that all different types of data can be collected and transmitted to the gateway device. Second, after receiving the raw data from the sensors, the gateway device decodes the data according to the protocol type followed by the interface. This converts the raw data in various formats into a format that the gateway can understand, ensuring data accuracy and consistency. The decoded data is then converted to a pre-set, unified data format and stored in the dynamic environment business module. This unified data format facilitates subsequent data processing and analysis, while the dynamic environment business module is responsible for managing and initially processing this data. Finally, based on the mapping between sensor data, device identifiers, and resource identifiers in the dynamic environment business module, sensor data resource information is generated. This process associates specific sensor data with corresponding identifiers, generating resource information containing device type, data attributes, and interface type for further fusion processing and logical judgment in the rule engine, enabling intelligent monitoring and management.

[0086] Step S20: acquiring video stream data from a camera device, performing image recognition analysis on the video stream data, and generating image analysis resource information.

[0087] It should be noted that video stream data refers to a continuous sequence of images captured and transmitted in real time by a camera. In this embodiment, the camera is connected to the edge fusion gateway via an Ethernet interface and transmits the captured scene to the gateway device in the form of a video stream. This video stream data contains dynamic information in the scene, such as human activity and object movement.

[0088] Image analysis resource information refers to the result information generated by analyzing video stream data using an image recognition algorithm. For example, a human figure detection algorithm can detect whether there are human figures in the video stream and generate corresponding detection results.

[0089] It can be understood that, first, the edge fusion gateway device establishes a connection with the camera device via an Ethernet interface, receiving the video stream data transmitted by the camera device in real time, ensuring that it can obtain continuous image information within the monitored area. The gateway device then inputs the received video stream data into its built-in AI function module. Using a pre-installed image recognition algorithm, it analyzes and processes each frame in the video stream, identifies specific targets or features in the image, and encapsulates these recognition results into image analysis resource information for subsequent further fusion processing and logical judgment, realizing intelligent monitoring and control.

[0090] Step S30: configuring a logical association relationship between input resources and output resources in a rule engine, wherein the input resources include the sensor data resource information and the image analysis resource information, and the output resources include control instruction resource information.

[0091] It's important to note that the rule engine is a core functional module within the edge convergence gateway, allowing users to define and manage business logic in a rule-driven manner. In this embodiment, the rule engine uses predefined semantic modules to write business rules, separating business decision logic from application code. This allows for the modification and maintenance of business rules without relying on developers.

[0092] Input resources refer to various data sources received during the rule engine processing process. These data sources provide the rule engine with input information required for decision-making. In this embodiment, the input resources mainly include sensor data resource information and image analysis resource information.

[0093] Output resources refer to control instructions or other response information generated by the rule engine after processing the input resources and preset rules. In this embodiment, the output resources mainly include control instruction resource information.

[0094] A logical association relationship refers to an association relationship established through logical conditions between input resources and output resources in a rule engine. In this embodiment, the logical association relationship is implemented through the interface operation or script definition of the rule engine. Users can configure the logical relationship between input resources (such as logical operations such as AND, OR, and NOT) according to specific needs and associate it with the output resources. For example, when the image analysis results in the input resource show that a human figure is detected and the sensor data resource information shows that the device is in operation, the rule engine can generate corresponding control instruction resource information through the preset logical association relationship to trigger the alarm.

[0095] Control instruction resource information refers to the specific control instructions generated by the rule engine based on preset rules. These instructions are used to operate and control devices or systems. They are part of the output resource and include specific instructions for controlling device behavior, such as start, stop, and alarm. These instructions are generated based on sensor data and image analysis results from the input resources, ensuring the accuracy and timeliness of control actions.

[0096] As an example, the input resources include the sensor data resource information and the image analysis resource information, and the output resources include control instruction resource information. The step of configuring the logical association relationship between the input resources and the output resources in the rule engine includes: loading the sensor data resource information and the image analysis resource information in the rule engine; associating the crane operation status in the sensor data resource information and the human detection result in the image analysis resource information as input resources; associating the alarm control instruction and the crane start and stop control instruction as output resources; and configuring the logical association relationship between the input resources and the output resources to trigger the alarm control instruction and the crane start and stop control instruction when the crane operation status is started and the human detection result exists.

[0097] Crane operating status refers to information collected by sensors regarding the crane's current operating status. In this embodiment, this status information is part of the sensor data resource information and is connected to the edge fusion gateway via an industrial bus interface or analog signal interface. For example, sensors may detect whether the crane's motor is running or whether the crane is moving. This information is converted into resource information and used as one of the input resources of the rule engine.

[0098] Human detection results refer to information obtained through image recognition analysis by cameras regarding the presence of human figures within the monitored area. This result is part of the image analysis resource information and is generated by the AI ​​function module of the edge fusion gateway. For example, if the camera's image recognition algorithm detects the presence of a human figure within the monitored area, it will generate a detection result containing information such as the human figure's location and number.

[0099] Alarm control instructions refer to instructions generated by the rule engine based on preset logic to control the action of the alarm. This instruction is part of the output resource and is used to trigger the alarm to sound an alarm.

[0100] The crane start / stop control instruction is an instruction generated by the rule engine according to the preset logic to control the start or stop of the crane. In this embodiment, this instruction is part of the output resource and is used to control the operating state of the crane.

[0101] First, in the rule engine's configuration interface or script, load the sensor data resource information and image analysis resource information as available input resources. This allows the rule engine to recognize and use this data to perform subsequent logical judgments. Second, through the rule engine's association function, explicitly set the crane's operating status in the sensor data resource information and the human detection results in the image analysis resource information as input resources. This allows the rule engine to simultaneously monitor two key factors: whether the crane is operating and whether a human figure is present in the monitoring area. Then, similarly in the rule engine, set the alarm control instructions and crane start / stop control instructions as output resources. This allows the rule engine to control the alarm and crane through these instructions when specific conditions are met. Finally, configure the logical association relationship so that when the crane's operating status is "start" and a human figure detection result is present, the alarm control instructions and crane start / stop control instructions are triggered. This ensures that if the crane is operating and a human figure is detected, an alarm is automatically issued and the crane stops, effectively preventing safety accidents.

[0102] Please refer to Figure 2 , Figure 2 The following is a logical flow diagram of the fusion gateway rule engine in the driving scenario provided in Example 1 of the edge fusion intelligent monitoring method based on the rule engine of this application. First, the dynamic environment module is used to detect whether the driving is in the start state, and then the AI ​​module performs human detection and analysis on the video stream data. If the driving is in the start state, the system will further check whether there is anyone. If someone is detected, the dynamic environment module will trigger the alarm and send a driving stop instruction to ensure safety. At the same time, the AI ​​module may perform the storage operation of the warning picture. If the driving is not started or no one is detected, no action is taken and the system remains in the monitoring state. This process automatically realizes the monitoring of personnel safety in the driving scenario, and improves the monitoring efficiency and response speed by flexibly configuring the logical relationship of input and output resources through the rule engine.

[0103] Step S40: performing fusion calculation on the sensor data resource information and the image analysis resource information according to the logical association relationship, generating the control instruction resource information, and sending the control instruction resource information to the target device.

[0104] It should be noted that the target device refers to the device that needs to receive and execute the control instruction resource information in the system. In this embodiment, these target devices mainly include alarms and crane control systems.

[0105] As an example, the steps of fusing the sensor data resource information and the image analysis resource information according to the logical association relationship to generate the control instruction resource information, and sending the control instruction resource information to the target device include: obtaining the crane operation status in the sensor data resource information and the human shape detection result in the image analysis resource information; performing logical operations on the crane operation status and the human shape detection result according to the logical association relationship to obtain operation results; when the operation result is true, generating an alarm instruction and a driving stop instruction; encapsulating the alarm instruction and the driving stop instruction into the control instruction resource information; and sending the control instruction resource information to the target device.

[0106] The calculation result refers to the judgment result obtained by the rule engine after performing logical operations on the input resources based on the preset logical associations. In this embodiment, the logical operations are typically performed based on Boolean logic (such as AND, OR, and NOT). For example, if the logical association is configured as "the crane operating status is started and a human detection result exists," if both conditions are met, the calculation result is True; otherwise, it is False.

[0107] The alarm instruction refers to an instruction generated by the rule engine to trigger the alarm action when the operation result is true. In this embodiment, the specific content of the alarm instruction may include information such as the alarm start signal, alarm level, and alarm type.

[0108] The crane stop instruction is an instruction generated by the rule engine to control the crane to stop running when the operation result is true. In this embodiment, the specific content of the crane stop instruction may include information such as the type of stop signal and the urgency of the stop.

[0109] First, the edge fusion gateway retrieves the crane's operating status data from its internal sensor data resources and extracts human detection results from its image analysis resources. This is to obtain key information about the crane and personnel status in the monitoring scenario. Second, the edge fusion gateway performs logical operations on the crane's operating status data and the human detection results based on the logical relationships pre-set in the rule engine. For example, it checks whether the crane is operating and whether the human detection result indicates the presence of a human. If both conditions are met, the logical operation evaluates to true, indicating that the scenario meets the pre-set safety risk criteria. Then, if the logical operation evaluates to true, the edge fusion gateway generates an alarm command and a stop command. These commands, based on pre-set safety policies, ensure swift action when potential hazards are detected. The edge fusion gateway then integrates and encapsulates the alarm and stop commands into control command resource information. This process formats the commands into a standard format that can be recognized and executed by the target device. Finally, the edge fusion gateway device sends the encapsulated control instruction resource information to the target device, such as the alarm and crane control system, through its communication interface to trigger the alarm and stop the crane operation, thereby achieving timely response and control of potential safety risks.

[0110] Please refer to Figure 3 , Figure 3 The schematic diagram of the fusion gateway data flow in the driving scenario provided by the first embodiment of the edge fusion intelligent monitoring method based on the rule engine of this application is as follows: first, the external camera captures the images of the driving area, and these images are processed by the AI ​​function module to realize functions such as human detection. At the same time, the external sensor obtains the driving status signal, and these signals are sent to the dynamic environment module for analysis. Then, the AI ​​function module and the dynamic environment module pass the processed data to the rule engine module. The rule engine module performs fusion calculations on these data according to the preset logical association relationship and generates corresponding control instruction resource information. Subsequently, these control instructions are sent to the target device to perform specific actions. For example, the AI ​​function module may be responsible for recording the alarm image, and the dynamic environment module may trigger the alarm or control the driving status to implement safety control measures such as stopping the vehicle. The entire process automatically realizes the monitoring and response of personnel safety in the driving scenario. The logical relationship between the input and output resources is flexibly configured by the rule engine, which improves the monitoring efficiency and response speed.

[0111] As an example, when the operation result is true, the step of generating an alarm instruction and a driving stop instruction includes: when the operation result is true, calling the protocol interface of the dynamic environment business module to generate a switch control signal that matches the alarm; calling the industrial bus control interface to generate a stop protocol instruction for the crane operation; transactionally associating the switch control signal with the stop protocol instruction to generate an associated instruction set; adding a timestamp and a scene identifier to the associated instruction set to generate an alarm instruction and a driving stop instruction.

[0112] The protocol interface refers to the standardized interface in the dynamic environment business module used to communicate with external devices. It supports specific communication protocols, enabling the edge fusion gateway to interact with and control devices such as alarms.

[0113] The switch control signal is a simple control signal, which is usually used to control the switch state of a device. In this embodiment, the switch control signal is used to control the on or off of an alarm.

[0114] The industrial bus control interface refers to a standardized interface used for equipment control in industrial environments. It supports the industrial bus protocol, enabling the edge fusion gateway to communicate with and control industrial equipment.

[0115] The stop protocol instruction is a control instruction that complies with the industrial bus protocol and is used to control the crane to stop running.

[0116] The associated instruction set is an instruction set that combines multiple related control instructions. In this embodiment, the associated instruction set combines the switch control signal and the shutdown protocol instruction to ensure that the crane can stop running when the alarm is triggered.

[0117] The scene identifier is a unique identifier for identifying a specific monitoring scene or event. In this embodiment, the scene identifier is used to mark the monitoring scene corresponding to the alarm instruction and the driving stop instruction, such as the scene of "crane running and human figure detected".

[0118] First, when the calculation result is true, the edge fusion gateway device generates a digital control signal matching the alarm through the protocol interface of the dynamic environment service module according to the preset protocol specifications. Specifically, the gateway device queries the alarm's protocol requirements, such as voltage level and signal format, and then generates the corresponding digital signal according to these requirements, ensuring that the alarm can correctly identify and execute the alarm triggering action. Second, the gateway device calls the industrial bus control interface and generates the crane's shutdown protocol instruction based on the industrial bus protocol. This process involves querying the specific format and instruction code of the crane control protocol, and then generating the shutdown instruction according to these format and code, ensuring that the crane accurately receives and executes the shutdown command. The generated digital control signal and the shutdown protocol instruction are then transactionally linked, combining the two instructions into a linked instruction set. A transaction management mechanism ensures that the two instructions are executed simultaneously, avoiding instruction conflicts or incorrect execution order, thereby ensuring the coordinated execution of the alarm and shutdown actions. Finally, a timestamp and scene identifier are added to the associated instruction set. The timestamp records the specific time when the instruction is generated, and the scene identifier identifies the specific monitoring scene corresponding to the instruction set. After completing these operations, the final alarm instruction and driving stop instruction are generated and sent to the target device for execution, ensuring that safety measures can be triggered quickly and accurately in specific scenarios.

[0119] This embodiment provides a rule-engine-based edge fusion intelligent monitoring method. First, an edge fusion gateway device accesses environmental sensor data through multiple hardware interfaces and generates sensor data resource information based on this data. This process standardizes the processing of different types of sensor data, providing a foundation for subsequent fusion calculations. Supporting multiple hardware interfaces also enables the device to flexibly adapt to different application scenarios. Next, the gateway device acquires video stream data from the camera device and performs image recognition analysis to generate image analysis resource information. This step enhances the device's perception capabilities, enabling the gateway device to perceive dynamic information within the monitoring area, such as human activity, in real time. Then, a logical association between input resources and output resources is configured in the rule engine. The input resources include sensor data resource information and image analysis resource information, while the output resources include control instruction resource information. This approach enables the gateway device to quickly make decisions based on pre-set logic. Finally, the sensor data resource information and image analysis resource information are fused and calculated based on the logical associations to generate control instruction resource information, which is then sent to the target device. This embodiment leverages multi-source heterogeneous data to achieve adaptive control of external devices, ensuring that corresponding control actions can be triggered promptly under specific conditions, thereby improving the overall performance and security of the system.

[0120] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 4 , Figure 4 This is a flow chart of the second embodiment of the edge fusion intelligent monitoring method based on the rule engine of the present application. Step S20 of the edge fusion intelligent monitoring method based on the rule engine includes steps S21 to S23:

[0121] Step S21: obtaining video stream data collected by a camera device through an Ethernet interface.

[0122] It should be noted that the Ethernet interface refers to the network interface on the edge fusion gateway device used to connect to the camera device. It supports the Ethernet communication protocol and can receive video stream data transmitted by the camera device through the network.

[0123] As you can understand, the edge fusion gateway device first establishes a network connection with the camera device through its Ethernet interface, ensuring a smooth communication link between the two. The camera device then collects video stream data from the monitored area and sends this data in the form of data packets to the gateway device via the Ethernet interface. After receiving these data packets, the gateway device's Ethernet interface reassembles them into complete video stream data for subsequent image recognition analysis.

[0124] Step S22: performing human figure detection analysis on the video stream data to obtain image analysis data including detection time, detection area, and number of human figures.

[0125] It should be noted that the detection area refers to the specific range or location in the video stream data where the AI ​​algorithm module detects human figures. In this embodiment, the detection area can be defined by preset parameters, such as specifying a specific part of the video image (such as a specific location or area in the monitoring area) as the detection target.

[0126] The number of humanoids refers to the number of humanoid targets detected by the AI ​​algorithm module within the detection area.

[0127] Image analysis data refers to the result data generated by the AI ​​algorithm module after performing human figure detection analysis on video stream data. In this embodiment, the image analysis data includes key information such as detection time, detection area, and number of human figures. This data is encapsulated in a structured format to facilitate subsequent processing and analysis. For example, the detection time is used to record the specific moment of detection, the detection area identifies the specific location of the detection, and the number of human figures provides information on the number of people in the area.

[0128] As an example, the step of performing human figure detection analysis on the video stream data to obtain image analysis data including detection time, detection area and number of human figures includes: decoding the video stream data, extracting a single-frame image and adjusting it to a preset resolution; inputting the adjusted single-frame image into a preset deep learning human figure detection model to obtain human figure bounding box coordinates and confidence; generating detection area polygon data based on the human figure bounding box coordinates, the polygon data consisting of a sequence of bounding box vertex coordinates; counting the number of human figures in the adjusted single-frame image whose confidence is greater than a preset confidence threshold, and recording the current system time as the detection time; and structurally encapsulating the detection area polygon data, the number of human figures and the detection time to obtain image analysis data.

[0129] The preset resolution refers to adjusting the extracted single-frame image to a fixed resolution when processing video stream data to facilitate subsequent image analysis. In this embodiment, the preset resolution is 1920×1080, which ensures that the image has sufficient detail and clarity during analysis.

[0130] The preset deep learning humanoid detection model is a convolutional neural network (CNN)-based model. Specifically, it can adopt popular deep learning architectures such as YOLO (You Only Look Once) or SSD (Single Shot MultiBox Detector).

[0131] Taking YOLOv5 as an example, it mainly consists of three key modules: the first is the Backbone network, such as CSPDarknet53, which is used to extract the features of the input image. Through the convolutional layer and residual block structure, it can efficiently extract rich feature information; the second is the Neck network, which usually adopts the FPN structure to further process the features extracted by the backbone network. Through upsampling and feature fusion, it combines high-level semantic information with low-level detail information to generate multi-scale feature maps, thereby improving the model's detection ability for targets of different sizes; the last is the Head network, which is responsible for mapping the processed feature map to the detection result, outputting the coordinates and confidence of the human bounding box, and generating information such as the coordinates of the bounding box, category confidence, etc. through convolution operations.

[0132] The model training process first requires the preparation of a large amount of labeled data. This data includes images of people in various scenarios and their bounding box annotations, typically from public or custom datasets. The model is then trained using this labeled data, and weights are adjusted through a backpropagation algorithm to minimize the difference between the predicted results and the true annotations. The training goal is to optimize the loss function, which typically includes bounding box regression loss and category confidence loss. After training, the model is optimized through methods such as pruning and quantization to improve inference speed and deployment efficiency. Model performance is evaluated on a validation set, and hyperparameters are adjusted to achieve optimal performance. Finally, the trained model is deployed on an edge fusion gateway device for real-time human detection tasks.

[0133] The human figure bounding box coordinates refer to the position of the detected human figure in the image, usually represented by a rectangular box. In this embodiment, the human figure bounding box coordinates are composed of the coordinates of four vertices. These coordinates define the position and size of the rectangular box and are used to identify the specific position of the human figure in the image.

[0134] Confidence refers to the deep learning model's assessment of the reliability of the detection results. In this embodiment, the confidence level is a value between 0 and 1, indicating the model's confidence in the detected humanoid. A higher confidence level (e.g., greater than 0.8) indicates that the model is more confident in the detection result, while a lower confidence level indicates that the detection result may be unreliable.

[0135] The detection area polygon data refers to polygon data generated based on the coordinates of the human bounding box and is used to more accurately represent the detection area.

[0136] The bounding box vertex coordinate sequence refers to the coordinate sequence of the four vertices constituting the humanoid bounding box.

[0137] The preset confidence threshold is a threshold used to filter humanoid detection results with higher confidence levels when counting humanoids. In this embodiment, the preset confidence threshold is 0.8, indicating that only humanoid detection results with a confidence level greater than or equal to 0.8 will be counted as part of the humanoid count.

[0138] The current system time refers to the current time recorded by the system when a human figure is detected.

[0139] First, the video stream data is decoded and image data is extracted frame by frame. Each frame is resized to a preset resolution to ensure image quality and processing consistency. This step standardizes the input data. Second, the resized single-frame image is fed into a pre-set deep learning human figure detection model, which outputs the coordinates of the human figure's bounding box and a confidence score. This step identifies the human figure in the image and evaluates the detection accuracy. Then, polygon data for the detection area is generated based on the human figure's bounding box coordinates. The polygon is constructed by calculating the sequence of bounding box vertex coordinates. This step more accurately describes the shape of the detection area and improves detection accuracy. Next, the number of human figures in the resized single-frame image with a confidence score greater than a preset confidence threshold is counted, and the current system time is recorded as the detection time. This step selects high-confidence detection results and marks the specific moment of the detection, ensuring the timeliness and reliability of the detection results. Finally, the detection area polygon data, the number of human figures, and the detection time are structured and packaged to form complete image analysis data. This step integrates all key information for subsequent processing and decision-making.

[0140] Step S23: Associating the image analysis data with the device identifier of the camera device to generate image analysis resource information.

[0141] As you can understand, first, the unique device identifier associated with the camera is obtained from the system to ensure that subsequent data can be traced back to the specific camera. Next, various information in the image analysis data, such as detection time, detection area polygon data, and number of human figures, is bound to the device identifier to form a one-to-one correspondence. This step is to ensure the accuracy and traceability of the image analysis data. Finally, the bound data is integrated and packaged into image analysis resource information, allowing for rapid identification and use of this data in subsequent processing and analysis, while also facilitating interaction with other systems or modules.

[0142] This embodiment first obtains video stream data from the camera device through the Ethernet interface. This process ensures the efficient transmission and reception of video data and provides real-time image information for subsequent analysis. Next, the obtained video stream data is subjected to human detection analysis to obtain image analysis data containing the detection time, detection area, and number of human figures. This step uses deep learning algorithms to enhance the system's perception capabilities, and can identify human activities within the monitoring area in real time, thereby improving the intelligence level of the monitoring system. Finally, the image analysis data is associated with the device identifier of the camera device to generate image analysis resource information. This step enables the image analysis data to be traced back to the specific camera device, facilitating management and use, while providing accurate input for subsequent data fusion and decision-making. The entire process not only improves the real-time and accuracy of the monitoring system, but also enhances the intelligence and traceability of the system.

[0143] It should be noted that the above examples are only used to understand this application and do not constitute a limitation on the edge fusion intelligent monitoring method based on the rule engine of this application. More simple transformations based on this technical concept are all within the scope of protection of this application.

[0144] This application also provides an edge fusion intelligent monitoring device based on a rule engine, please refer to Figure 5 , the edge fusion intelligent monitoring device based on the rule engine includes:

[0145] The data access module 10 is used to access environmental sensor data through various hardware interfaces and generate sensor data resource information based on the environmental sensor data;

[0146] The image analysis module 20 is used to obtain video stream data from the camera device, perform image recognition analysis on the video stream data, and generate image analysis resource information;

[0147] A rule configuration module 30 is used to configure a logical association relationship between input resources and output resources in a rule engine, wherein the input resources include the sensor data resource information and the image analysis resource information, and the output resources include control instruction resource information;

[0148] The fusion control module 40 is used to perform fusion calculation on the sensor data resource information and the image analysis resource information according to the logical association relationship, generate the control instruction resource information, and send the control instruction resource information to the target device.

[0149] In one embodiment, the image analysis module 20 is further used to obtain video stream data collected by a camera device through an Ethernet interface; perform human detection analysis on the video stream data to obtain image analysis data including detection time, detection area and number of human figures; and associate the image analysis data with a device identifier of the camera device to generate image analysis resource information.

[0150] In one embodiment, the image analysis module 20 is further used to decode the video stream data, extract a single-frame image and adjust it to a preset resolution; input the adjusted single-frame image into a preset deep learning human figure detection model to obtain human figure bounding box coordinates and confidence; generate detection area polygon data based on the human figure bounding box coordinates, and the polygon data consists of a sequence of bounding box vertex coordinates; count the number of human figures in the adjusted single-frame image whose confidence is greater than a preset confidence threshold, and record the current system time as the detection time; and perform structured packaging on the detection area polygon data, the number of human figures and the detection time to obtain image analysis data.

[0151] In one embodiment, the rule configuration module 30 is further used to load the sensor data resource information and the image analysis resource information in the rule engine; associate the crane operation status in the sensor data resource information and the human detection result in the image analysis resource information as input resources; associate the alarm control instruction and the crane start-stop control instruction as output resources; and configure the logical association relationship between the input resource and the output resource to trigger the alarm control instruction and the crane start-stop control instruction when the crane operation status is started and the human detection result exists.

[0152] In one embodiment, the fusion control module 40 is further used to obtain the crane operation status in the sensor data resource information and the human detection result in the image analysis resource information; perform logical operations on the crane operation status and the human detection result according to the logical association relationship to obtain an operation result; when the operation result is true, generate an alarm instruction and a driving stop instruction; encapsulate the alarm instruction and the driving stop instruction into the control instruction resource information; and send the control instruction resource information to the target device.

[0153] In one embodiment, the fusion control module 40 is also used to call the protocol interface of the dynamic environment business module to generate a switch control signal that matches the alarm when the calculation result is true; call the industrial bus control interface to generate a shutdown protocol instruction for the crane operation; perform transaction association between the switch control signal and the shutdown protocol instruction to generate an associated instruction set; add a timestamp and a scene identifier to the associated instruction set to generate an alarm instruction and a crane stop instruction.

[0154] In one embodiment, the data access module 10 is also used to access various types of sensors through an industrial bus interface and an analog signal interface; parse the raw data transmitted by the sensor, and decode the raw data according to the protocol type of the industrial bus interface and the analog signal interface; convert the decoded data into a preset unified data format and store it in the dynamic environment business module; generate sensor data resource information including device type, data attributes and interface type based on the mapping relationship between sensor data, device identifier and resource identifier in the dynamic environment business module.

[0155] The rule-engine-based edge fusion intelligent monitoring device provided in this application adopts the rule-engine-based edge fusion intelligent monitoring method in the above-mentioned embodiment, which can solve the technical problem of how to coordinate multi-source heterogeneous data to achieve adaptive control of external devices. Compared with the existing technology, the beneficial effects of the rule-engine-based edge fusion intelligent monitoring device provided in this application are the same as the beneficial effects of the rule-engine-based edge fusion intelligent monitoring method provided in the above-mentioned embodiment, and the other technical features of the rule-engine-based edge fusion intelligent monitoring device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0156] The present application provides an edge fusion intelligent monitoring device based on a rule engine. The edge fusion intelligent monitoring device based on a rule engine includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the edge fusion intelligent monitoring method based on the rule engine in the above-mentioned embodiment one.

[0157] Reference below Figure 6 , which shows a schematic structural diagram of a rule-engine-based edge fusion intelligent monitoring device suitable for implementing the embodiments of the present application. The rule-engine-based edge fusion intelligent monitoring device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The edge fusion intelligent monitoring device based on the rule engine shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0158] like Figure 6As shown, the edge fusion intelligent monitoring device based on the rule engine may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the ROM (Read Only Memory) 1002 or the program loaded from the storage device 1003 to the RAM (Random Access Memory) 1004. Various programs and data required for the operation of the edge fusion intelligent monitoring device based on the rule engine are also stored in the RAM 1004. The processing device 1001, ROM 1002 and RAM 1004 are connected to each other via a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, an LCD (Liquid Crystal Display), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the edge fusion intelligent monitoring device based on the rule engine to communicate with other devices wirelessly or wired to exchange data. Although the figure shows an edge fusion intelligent monitoring device based on the rule engine with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0159] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0160] The rule-engine-based edge fusion intelligent monitoring device provided in this application adopts the rule-engine-based edge fusion intelligent monitoring method in the above-mentioned embodiment, which can solve the technical problem of how to coordinate multi-source heterogeneous data to achieve adaptive control of external devices. Compared with the existing technology, the beneficial effects of the rule-engine-based edge fusion intelligent monitoring device provided in this application are the same as the beneficial effects of the rule-engine-based edge fusion intelligent monitoring method provided in the above-mentioned embodiment, and the other technical features of the rule-engine-based edge fusion intelligent monitoring device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0161] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0162] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0163] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the edge fusion intelligent monitoring method based on the rule engine in the above embodiment.

[0164] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash memory), optical fiber, CD-ROM (CD-Read Only Memory, portable compact disk read-only memory), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0165] The above-mentioned computer-readable storage medium can be included in the edge fusion intelligent monitoring device based on the rule engine; or it can exist independently without being assembled into the edge fusion intelligent monitoring device based on the rule engine.

[0166] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the edge fusion intelligent monitoring device based on the rule engine, the edge fusion intelligent monitoring device based on the rule engine: accesses environmental sensor data through multiple hardware interfaces, and generates sensor data resource information based on the environmental sensor data; obtains video stream data of the camera device, and performs image recognition analysis on the video stream data to generate image analysis resource information; configures the logical association relationship between input resources and output resources in the rule engine, the input resources include the sensor data resource information and the image analysis resource information, and the output resources include control instruction resource information; performs fusion calculation on the sensor data resource information and the image analysis resource information according to the logical association relationship to generate the control instruction resource information, and sends the control instruction resource information to the target device.

[0167] The computer program code for performing the operations of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a LAN (Local Area Network) or a WAN (Wide Area Network), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0168] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0169] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0170] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., a computer program) for executing the above-mentioned rule-engine-based edge fusion intelligent monitoring method, and can solve the technical problem of how to coordinate multi-source heterogeneous data to achieve adaptive control of external devices. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the rule-engine-based edge fusion intelligent monitoring method provided in the above-mentioned embodiment, and will not be repeated here.

[0171] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned rule engine-based edge fusion intelligent monitoring method.

[0172] The computer program product provided in this application can solve the technical problem of how to coordinate multi-source heterogeneous data to achieve adaptive control of external devices. Compared with the existing technology, the beneficial effects of the computer program product provided in this application are the same as those of the rule-engine-based edge fusion intelligent monitoring method provided in the above embodiment, and will not be repeated here.

[0173] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An edge fusion intelligent monitoring method based on a rule engine, characterized in that: The method comprises: Accessing environmental sensor data through a variety of hardware interfaces and generating sensor data resource information based on the environmental sensor data; Acquire video stream data from a camera device, and perform image recognition analysis on the video stream data to generate image analysis resource information; Configuring a logical association relationship between input resources and output resources in a rule engine, wherein the input resources include the sensor data resource information and the image analysis resource information, and the output resources include the control instruction resource information; The sensor data resource information and the image analysis resource information are fused and calculated according to the logical association relationship to generate the control instruction resource information, and the control instruction resource information is sent to the target device.

2. The method according to claim 1, wherein The steps of acquiring video stream data of a camera device, performing image recognition analysis on the video stream data, and generating image analysis resource information include: Obtain video stream data collected by the camera device through the Ethernet interface; Performing human figure detection analysis on the video stream data to obtain image analysis data including detection time, detection area, and number of human figures; The image analysis data is associated with a device identifier of the camera device to generate image analysis resource information.

3. The method according to claim 2, wherein The step of performing human figure detection analysis on the video stream data to obtain image analysis data including detection time, detection area and number of human figures includes: Decoding the video stream data, extracting a single frame image and adjusting it to a preset resolution; Input the adjusted single-frame image into the preset deep learning human figure detection model to obtain the human figure bounding box coordinates and confidence level; generating detection area polygon data according to the humanoid bounding box coordinates, wherein the polygon data is composed of a bounding box vertex coordinate sequence; Counting the number of human figures in the adjusted single-frame image whose confidence level is greater than a preset confidence threshold, and recording the current system time as the detection time; The detection area polygon data, the number of human figures and the detection time are structurally packaged to obtain image analysis data.

4. The method according to claim 1, wherein The input resources include the sensor data resource information and the image analysis resource information, and the output resources include control instruction resource information; The step of configuring the logical association relationship between input resources and output resources in the rule engine includes: Loading the sensor data resource information and the image analysis resource information into a rule engine; associating the crane operation status in the sensor data resource information with the human figure detection result in the image analysis resource information as input resources; Associate the alarm control instructions and the crane start and stop control instructions as output resources; The logical association relationship between the input resource and the output resource is configured such that when the crane operation state is started and the human figure detection result exists, the alarm control instruction and the crane start-stop control instruction are triggered.

5. The method according to claim 1, wherein The step of performing fusion calculation on the sensor data resource information and the image analysis resource information according to the logical association relationship to generate the control instruction resource information, and sending the control instruction resource information to the target device includes: Obtaining the crane operation status in the sensor data resource information and the human figure detection result in the image analysis resource information; Performing a logical operation on the crane operation state and the human figure detection result according to the logical association relationship to obtain an operation result; When the operation result is true, an alarm instruction and a driving stop instruction are generated; Encapsulating the alarm instruction and the driving stop instruction into the control instruction resource information; The control instruction resource information is sent to the target device.

6. The method according to claim 5, wherein When the operation result is true, the step of generating an alarm instruction and a driving stop instruction includes: When the operation result is true, the protocol interface of the dynamic environment service module is called to generate a switch control signal matching the alarm; Call the industrial bus control interface to generate the crane's shutdown protocol instructions; Performing transaction association on the switch control signal and the shutdown protocol instruction to generate an associated instruction set; A timestamp and a scene identifier are added to the associated instruction set to generate an alarm instruction and a driving stop instruction.

7. The method according to any one of claims 1 to 6, characterized in that The steps of accessing environmental sensor data through multiple hardware interfaces and generating sensor data resource information based on the environmental sensor data include: Connect to various types of sensors through industrial bus interface and analog signal interface; Parsing the raw data transmitted by the sensor and decoding the raw data according to the protocol types of the industrial bus interface and the analog signal interface; Convert the decoded data into a preset unified data format and store it in the dynamic environment business module; According to the mapping relationship between the sensor data, device identifier and resource identifier in the dynamic environment business module, sensor data resource information including device type, data attribute and interface type is generated.

8. An edge fusion intelligent monitoring device based on a rule engine, characterized in that: The device comprises: A data access module, configured to access environmental sensor data through a variety of hardware interfaces and generate sensor data resource information based on the environmental sensor data; An image analysis module is used to obtain video stream data from a camera device, perform image recognition analysis on the video stream data, and generate image analysis resource information; A rule configuration module, configured to configure a logical association relationship between input resources and output resources in a rule engine, wherein the input resources include the sensor data resource information and the image analysis resource information, and the output resources include control instruction resource information; A fusion control module is used to perform fusion calculation on the sensor data resource information and the image analysis resource information according to the logical association relationship, generate the control instruction resource information, and send the control instruction resource information to the target device.

9. An edge fusion intelligent monitoring device based on a rule engine, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the rule engine-based edge fusion intelligent monitoring method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the edge fusion intelligent monitoring method based on a rule engine as described in any one of claims 1 to 7 are implemented.

Images