Method and device for collaborative work of multiple monitoring devices based on zhinet

Abstract

The present disclosure relates to the technical field of intelligent networking, and particularly relates to a multi-monitoring device cooperative operation method and device based on intelligent networking, comprising: in response to receiving a monitoring event reported by a monitoring device, acquiring real-time context data corresponding to the monitoring event, and obtaining a first cooperative operation rule based on the monitoring event and a cooperative rule library through a rule engine; generating a second cooperative operation strategy based on the real-time context data, the first cooperative operation strategy and a device topology map through a large language model; determining a target cooperative operation strategy based on the first cooperative operation strategy and the second cooperative operation strategy, and driving the monitoring device to perform cooperative operation based on the target cooperative operation strategy. Thus, an automatic multi-monitoring device cooperative operation process can be realized, and the rule engine can guarantee response speed and reliability; the large language model can improve the processing capability for uncertain and complex scenarios.

Description

Technical Field

[0001] This disclosure relates to the field of Internet of Things (IoT) technology, specifically to a method and apparatus for collaborative operation of multiple monitoring devices based on IoT. Background Technology

[0002] Collaborative operation of multiple monitoring devices allows them to work together to complete relevant monitoring tasks. In real-world scenarios, multiple monitoring devices with different functions and distributed in different locations can achieve more comprehensive and efficient monitoring results through mutual cooperation.

[0003] However, in related technologies, the collaborative operation between multiple monitoring devices requires manual intervention. For example, in a park security scenario involving cross-device target tracking, security personnel might spot a target on the first monitoring device. Once the target moves out of sight, the personnel need to rely on experience to guess the target's movement path, then manually switch to the next monitoring device that might cover it, and then re-search and identify the target in the new view. This process needs to be repeated, wasting a significant amount of manpower and time on repetitive searches and guesses, resulting in low efficiency. Furthermore, this process is highly dependent on human experience, making it extremely easy to lose the monitored target during switching and re-identification. Summary of the Invention

[0004] To overcome the problems existing in the related technologies, this disclosure provides a method and apparatus for collaborative operation of multiple monitoring devices based on the Internet of Things, in order to solve the defects in the related technologies.

[0005] According to a first aspect of the present disclosure, a method for collaborative operation of multiple monitoring devices based on the Internet of Things is provided, comprising:

[0006] In response to receiving a monitoring event reported by a monitoring device, the system obtains the real-time context data corresponding to the monitoring event, and obtains a first collaborative operation rule through a rule engine based on the monitoring event and a collaborative rule library. The collaborative rule library is used to store pre-configured collaborative operation rules for multiple monitoring devices, and each collaborative operation rule includes a preset trigger event, a preset execution condition, and a preset collaborative action sequence.

[0007] A second collaborative operation strategy is generated using a large language model based on the real-time context data, the first collaborative operation strategy, and the device topology map. The device topology map is obtained by processing the spatial attribute information of the multiple monitoring devices based on a unified spatiotemporal reference. The spatial attribute information includes spatial location, spatial orientation, and field of view parameters.

[0008] Based on the first collaborative operation strategy and the second collaborative operation strategy, a target collaborative operation strategy is determined, and based on the target collaborative operation strategy, the monitoring equipment is driven to perform collaborative operations.

[0009] According to a second aspect of the present disclosure, a multi-monitoring device collaborative operation apparatus based on the Internet of Things is provided, comprising:

[0010] The matching module is used to respond to a monitoring event reported by the first monitoring device, obtain the real-time context data corresponding to the monitoring event, and obtain a first collaborative operation rule based on the monitoring event and the collaborative rule library. The collaborative rule library is used to store pre-configured collaborative operation rules for multiple monitoring devices. Each collaborative operation rule includes a preset trigger event, a preset execution condition, and a preset collaborative action sequence.

[0011] The generation module is used to generate a second collaborative operation strategy based on the real-time context data, the first collaborative operation strategy, and the device topology map using a large language model. The device topology map is obtained based on the spatial attribute information of the multiple monitoring devices, and the spatial attribute information includes spatial location, spatial orientation, and field of view parameters.

[0012] The control module is used to determine a target collaborative operation strategy based on the first collaborative operation strategy and the second collaborative operation strategy, and to drive the monitoring equipment to perform collaborative operation based on the target collaborative operation strategy.

[0013] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0014] The multi-monitoring device collaborative operation method based on the Internet of Things provided in this disclosure can obtain a first collaborative operation rule based on monitoring events and a collaborative rule base. Then, a second collaborative operation strategy is generated based on real-time context data of monitoring events, the first collaborative operation strategy, and a device topology map using a large language model. Finally, a target collaborative operation strategy is determined based on the first and second collaborative operation strategies to drive the monitoring devices to perform collaborative operations. This enables an automated multi-monitoring device collaborative operation process, reduces manual intervention, and improves the efficiency of multi-monitoring device collaborative operations. Furthermore, the rule engine can handle high-frequency, deterministic monitoring scenarios, ensuring response speed and reliability; the large language model can improve its ability to handle uncertain and complex scenarios, understand task requirements described in natural language, and perform reasoning and resource scheduling based on a global device topology map and real-time context data, thereby enhancing the intelligence level of multi-monitoring device collaborative operations. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0016] Figure 1This is a flowchart illustrating an exemplary embodiment of the present disclosure of a method for collaborative operation of multiple monitoring devices based on the Internet of Things;

[0017] Figure 2 This is a schematic diagram of the structure of a multi-monitoring device collaborative operation device based on the Internet of Things, as shown in an exemplary embodiment of this disclosure. Detailed Implementation

[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0019] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0020] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information.

[0021] Firstly, at least one embodiment of this disclosure provides a method for collaborative operation of multiple monitoring devices based on the Internet of Things. Please refer to the appendix. Figure 1 It illustrates the process of the method, including steps S101 to S103.

[0022] In step S101, in response to receiving a monitoring event reported by a monitoring device, real-time context data corresponding to the monitoring event is obtained, and a first collaborative operation rule is obtained through the rule engine based on the monitoring event and the collaborative rule library. The collaborative rule library stores pre-configured collaborative operation rules for multiple monitoring devices, and each collaborative operation rule includes a preset trigger event, preset execution conditions, and a preset collaborative action sequence.

[0023] In step S102, a second collaborative operation strategy is generated based on real-time context data, the first collaborative operation strategy, and the device topology map using a large language model.

[0024] In step S103, a target collaborative operation strategy is determined based on the first collaborative operation strategy and the second collaborative operation strategy, and the monitoring equipment is driven to perform collaborative operation based on the target collaborative operation strategy.

[0025] This enables automated collaborative operation processes for multiple monitoring devices, reducing manual intervention and improving the efficiency of collaborative operation. Furthermore, the rule engine can handle high-frequency, deterministic monitoring scenarios, ensuring response speed and reliability; the large language model enhances its ability to handle uncertain and complex scenarios, understands task requirements described in natural language, and performs reasoning and resource scheduling based on a global device topology map and real-time contextual data, thereby improving the intelligence level of collaborative operation among multiple monitoring devices.

[0026] It should be understood that the Internet of Things (IoT) refers to a network architecture built upon the Internet, integrating advanced technologies such as artificial intelligence, big data, and cloud computing to intelligently connect and interact with various devices, systems, and personnel. Devices and systems not only achieve interconnectivity but also possess the capabilities of autonomous perception, intelligent decision-making, and collaborative execution. In this embodiment, a rule engine with spatiotemporal awareness and a large language model are used to drive multiple monitoring devices to dynamically, orderly, and predictively collaborate for a common monitoring task, forming a multi-monitoring device operating system based on the IoT, enabling continuous monitoring and trajectory reconstruction of moving targets.

[0027] To facilitate understanding of the collaborative operation of multiple monitoring devices provided in this disclosure, the following examples further illustrate the steps described above.

[0028] For example, a monitoring event is used to trigger the coordinated operation of multiple monitoring devices. For instance, it could be the movement of an intrusion target in a security scenario, or the abnormality of device monitoring in an anomaly detection scenario. This disclosure does not limit this, and it can be set according to actual needs.

[0029] For example, real-time context data may include at least one of the following: device real-time status data, environmental scene data, business rule data, and external system data. Specifically, device real-time status data may include the operating status, resource load, and current control parameters of the monitoring device reporting the monitoring event. Environmental scene data may include system time, meteorological data, and geographical information related to the monitoring event. Business rule data may include information on currently executing collaborative tasks and the security levels of different monitoring areas. External system data may include social network information and calendar system information related to the monitoring event.

[0030] In some embodiments, a first collaborative operation rule is obtained by a rule engine based on monitoring events and a collaborative rule base, including: matching collaborative rules based on monitoring events using the rule engine to obtain matching collaborative rules; predicting the movement path of the monitoring target based on the location information of the monitoring device and the movement direction of the monitoring target in the monitoring event to obtain a predicted movement path, and determining collaborative monitoring devices whose field of view covers the predicted movement path based on the device topology map; and generating a first collaborative operation strategy based on the collaborative monitoring devices and a preset collaborative action sequence in the matching collaborative rules.

[0031] For example, the collaborative rule base stores pre-configured collaborative operation rules. Each collaborative operation rule adopts an "event-condition-action" structure, including a preset trigger event, preset execution conditions, and a preset collaborative action sequence. The trigger event can be matched with monitoring events reported by monitoring devices. If a match is successful, the collaborative operation rule associated with that trigger event is taken as the matched collaborative operation rule. The preset execution conditions are used for rule verification. When the preset execution conditions are met, the matched collaborative operation rule is activated; otherwise, it is not activated. For example, the preset execution condition could be {Event.Confidence > 0.8 AND SourceDevice.Status == "Online" AND Time isDaytime}, meaning that the corresponding collaborative operation rule is activated only when the confidence level is > 0.8, the source device is online, and it is daytime. The preset collaborative action sequence includes abstract execution actions, and the subject of the action execution needs to be dynamically determined based on the actual scenario. For example, the preset collaborative action sequence might be: notify the next monitoring device to prepare; let the high-definition capture camera take a picture. In practical applications, the "next camera" and the "high-definition capture camera" need to be instantiated as specific monitoring devices.

[0032] For example, when monitoring device A detects target movement, the rule engine is triggered to perform matching in the collaborative rule base and obtain matching collaborative rules. Furthermore, based on the location of monitoring device A and the target's direction of movement, the rule engine predicts the target's movement path and then calculates the next monitoring device most likely to capture the target in the device topology map, identifying monitoring device B as the next such device. Finally, based on monitoring device B, the preset collaborative action sequence in the matching collaborative rules is updated to obtain the first collaborative operation strategy.

[0033] It should be understood that in related technologies, the data generated by different monitoring devices are fragmented in time and space, and the data cannot be correlated and fused under a unified spatiotemporal reference, resulting in collaborative operations remaining at the stage of manually switching video feeds. In the embodiments disclosed in this disclosure, the spatial attribute information of multiple monitoring devices is processed based on a unified spatiotemporal reference to obtain a device topology map, enabling the data of multiple monitoring devices to be automatically fused in a unified spatiotemporal context. This endows the rule engine with spatial reasoning and prediction capabilities, allowing it to perceive the positional relationships and field of view of all monitoring devices in the physical world, and then intelligently select the most suitable next camera. Compared with simple alarm linkage, this can achieve proactive dynamic collaboration based on spatial perception and prediction.

[0034] In some embodiments, the device topology map is obtained as follows: in response to the monitoring device accessing the network, the corresponding virtual device driver is loaded through a smart gateway, and a virtual device instance uniquely corresponding to the monitoring device is created in the video fusion platform; the functions of the physical monitoring device are abstracted into a standardized capability set, and the standardized capability set is reported to the video fusion platform through the smart gateway; spatial attribute information is added to the virtual device instance based on user input; the spatial relationship of each virtual device instance is determined based on the spatial attribute information, and a device topology map is generated based on the spatial relationship.

[0035] For example, virtual device drivers are used to abstract and encapsulate monitoring devices, thereby abstracting physical monitoring devices into a service-oriented interface with defined capabilities (such as PTZ control and face capture) for unified invocation. When a new monitoring device is connected, the smart gateway automatically matches and loads the corresponding virtual device driver from the repository based on information such as device model and manufacturer. If the driver is not found in the repository, it initiates a search and download from the cloud driver repository, achieving plug-and-play functionality.

[0036] For example, a standardized capability set is used to explicitly define all the capabilities that a monitoring device can provide, and can be an XML or JSON format file.

[0037] For example, spatial attribute information can be added manually. For instance, on the graphical device topology map interface provided by the video fusion platform, the installation location of the virtual device instance can be determined by dragging and dropping, and then the azimuth, pitch, and field of view of the virtual device instance can be manually configured through parameter input boxes.

[0038] In some embodiments, spatial attribute information can be added in a semi-automatic manner. For example, displaying video footage captured by a monitoring device to the user; in response to the user's selection of at least two reference points in the video footage, determining the pixel coordinates of the at least two reference points in the video footage; in response to the user's input of the actual physical distance between the at least two reference points, obtaining the spatial coordinates, spatial orientation, and field of view parameters of the monitoring device corresponding to the virtual device instance based on the pixel coordinates of the at least two reference points in the video footage and the actual physical distance corresponding to the input operation, and adding the spatial coordinates, spatial orientation, and field of view parameters to the virtual device instance.

[0039] For example, a user selects two prominent points on the ground in a video feed, such as the center of a manhole cover and the bottom of a wall corner, and inputs the actual distance between them, say 5 meters. Then, using computer vision algorithms, the focal length and orientation of the monitoring equipment can be calculated based on the pixel coordinates of the two reference points in the image and their actual distance. For instance, by solving the camera model, the height and pitch angle of the monitoring equipment can be obtained, and then the latitude, longitude, and yaw angle of the monitoring equipment can be calculated using the geometric relationship between the reference points. This allows the parameters of different monitoring equipment to be automatically converted to a unified spatiotemporal reference, reducing manual operation, improving the efficiency of adding spatial attribute information, and thus increasing the efficiency of generating equipment topology maps.

[0040] For example, the spatial relationships between virtual device instances include at least one of the following: adjacent field of view, overlapping field of view, and physical location relay relationship. Among these, the physical location relay relationship indicates that there exists a anticipated, continuous target movement path between the coverage areas of two or more monitoring devices. For instance, when a target leaves the coverage area of ​​monitoring device C, according to its movement pattern, it is most likely to enter the coverage area of ​​monitoring device D next; therefore, a physical location relay relationship is established between monitoring device C and monitoring device D.

[0041] Using the above methods, a device topology map can be obtained based on a unified spatiotemporal reference and the spatial attribute information of multiple monitoring devices. As a result, the rule engine can perceive the positional relationship and field of view of all monitoring devices in the physical world, and realize proactive dynamic collaboration based on spatial perception and prediction.

[0042] In some embodiments, a second collaborative operation strategy is generated using a large language model based on real-time context data, a first collaborative operation strategy, and a device topology map. This includes: analyzing resource dependencies among multiple monitoring devices in the device topology map using a large language model; predicting cascading resource occupancy effects that the first collaborative operation strategy may cause based on resource dependencies; generating a resource reservation scheme based on the cascading resource occupancy effects and the device topology map; predicting resource deadlock and competition relationships that the first collaborative operation strategy may cause based on real-time context data and a preset collaborative action sequence in the first collaborative operation strategy; generating an alternative resource allocation scheme based on the resource deadlock, competition, and the device topology map; and adjusting the first collaborative operation strategy according to the resource reservation scheme and the alternative resource allocation scheme to obtain the second collaborative operation strategy.

[0043] For example, the cascading resource occupancy effect describes a chain reaction or the diffusion of resource crowding out. For instance, in a target tracking scenario, the first collaborative operation strategy is to continuously schedule monitoring devices E1, E2, and E3. The large language model, through analysis of the device topology map, finds that monitoring device E3 is also the only video verification device for the access control system at entrance F. If monitoring device E3 is occupied for a long period for tracking, it will prevent all personnel needing to pass through entrance F from using facial recognition to open the door, thus causing congestion at entrance F. The tracking task does not directly compete with the access control task for monitoring device E3, but the prolonged occupation of the tracking task indirectly leads to a decrease in the performance of the access control task and business disruption. Therefore, it is determined that the first collaborative operation strategy may trigger a cascading resource occupancy effect.

[0044] Accordingly, the large language model can generate a resource reservation scheme based on this cascading resource occupancy effect and the device topology map. For example, if the large language model predicts that monitoring device E3 will be occupied by a tracking task for 2 minutes, and that monitoring device E3 needs to perform a 10-second access control verification task every 5 minutes, the generated resource reservation scheme could be: during the tracking task, reserve a 10-second uninterrupted time window for the access control verification task in the scheduling queue of monitoring device E3 every 5 minutes. Finally, the second collaborative operation strategy could be: adding a rule to the first collaborative operation strategy: when monitoring device E3 is performing a tracking task, periodically pause and relinquish control to the access control task for 10 seconds.

[0045] For example, a resource deadlock relationship represents the existence of a circular waiting chain, where task 1 holds resource R1 and requests resource R2, while task 2 holds resource R2 and requests resource R1, and neither can continue execution. Therefore, the large language model can perform topology analysis based on real-time context data and the preset cooperative action sequence in the first cooperative operation strategy to identify whether a circular waiting loop exists, thereby predicting the resource deadlock relationship that the first cooperative operation strategy may cause.

[0046] For example, competition indicates that the execution result depends on the precise timing of tasks or instructions; improper timing will lead to unexpected results. For instance, the first collaborative operation strategy includes two instructions for monitoring device E4: rotate to a preset position; and perform intelligent analysis of face capture. If the instruction for intelligent analysis of face capture takes effect before the device has fully rotated to the preset position, the face capture function will start from the wrong perspective, causing capture failure. The execution order and timing of these two instructions are crucial and therefore competitive. Thus, the large language model can perform timing and mutual exclusion analysis of the instruction sequence based on real-time context data and the preset collaborative action sequence in the first collaborative operation strategy, predicting the potential competition relationships arising from the first collaborative operation strategy.

[0047] In some embodiments, based on real-time context data and a preset sequence of collaborative actions in the first collaborative operation strategy, predicting the resource deadlock and competition relationships that the first collaborative operation strategy may cause includes: generating a natural language description of the current collaborative operation scenario based on real-time context data and the preset sequence of collaborative actions in the first collaborative operation strategy, and analyzing the natural language description using a large language model to obtain the resource deadlock relationships that the first collaborative operation strategy may cause, wherein the natural language description includes the identification information and resource occupancy information of the monitoring devices required by each monitoring task in the first collaborative operation strategy; and analyzing the execution timing sensitivity of the preset sequence of collaborative actions in the first collaborative operation strategy using a large language model to obtain the competition relationships that the first collaborative operation strategy may cause, wherein the execution sensitivity is used to characterize whether different action execution sequences correspond to different monitoring results.

[0048] For example, based on real-time context data and the preset collaborative action sequence in the first collaborative operation strategy, a natural language description of the current collaborative operation scenario is generated: The VIP escort task requires monitoring devices G1 and G2, currently holds G1, and is waiting for G2; the perimeter intrusion tracking task requires monitoring devices G1 and G2, currently holds G2, and is waiting for G1. The large language model performs analysis and reasoning based on this natural language description: First, it identifies key entities: VIP escort task, perimeter intrusion task, monitoring device G1, and monitoring device G2. Then, it performs relationship parsing, establishes an occupancy-waiting relationship matrix, and then performs loop detection to identify the resources held by the VIP escort task and other perimeter intrusion tracking tasks. The resources held by the perimeter intrusion tracking task and other VIP escort tasks exhibit a resource deadlock relationship.

[0049] It should be understood that if changing the execution order of the preset collaborative action sequence in the first collaborative operation strategy results in different monitoring results, it indicates that the first collaborative operation strategy has a competitive relationship; otherwise, no competitive relationship exists. Therefore, the large language model analyzes the execution timing sensitivity of the preset collaborative action sequence in the first collaborative operation strategy to obtain the competitive relationship that the first collaborative operation strategy may cause.

[0050] For example, generating alternative resource allocation schemes based on resource deadlock relationships, competition relationships, and device topology maps can be achieved by using a large language model to find candidate alternative devices in the device topology map that have functional equivalence and spatial coverage similarity with conflicting resources, and then selecting from all candidate alternative devices based on the principles of cost minimization and effect optimization.

[0051] For example, continuing with the above example, the VIP escort mission and the perimeter intrusion mission are deadlocked. Large language model analysis reveals that the field of view of monitoring device G3 overlaps 80% with that of monitoring device G1, and it is currently idle. The large language model then assesses that replacing monitoring device G1 with monitoring device G3 will have minimal impact on the tracking effect of the perimeter intrusion mission and will completely break the deadlock. Therefore, the generated alternative resource allocation scheme is: redirect the perimeter intrusion mission's request to monitoring device G1 to monitoring device G3, which has similar functionality and space. The second collaborative operation strategy could be: modify the instruction sequence of the perimeter intrusion mission in the first collaborative operation strategy, replacing monitoring device G1 with monitoring device G3.

[0052] Therefore, by using a large language model for deep semantic reasoning of resource-dependent networks, it is possible to predict in advance systemic risks such as cascading resource depletion, deadlock, and competition conditions. This allows for the generation of resource reservation or alternative allocation schemes before resource conflicts occur, enabling globally optimal resource utilization decisions. This reduces monitoring interruptions, target loss, or system deadlock caused by resource competition, and improves the robustness and reliability of multi-device collaborative operations in complex monitoring scenarios.

[0053] In some embodiments, a second collaborative operation strategy is generated using a large language model based on real-time context data, a first collaborative operation strategy, and a device topology map. This includes: understanding the task requirements of the collaborative monitoring task corresponding to the monitoring event using the large language model based on real-time context data and the first collaborative operation strategy; searching for potential monitoring devices that meet the task requirements in the device topology map; determining the matching degree between the potential monitoring devices and the task requirements based on the spatial location and capability characteristics of the potential monitoring devices; and replacing the monitoring devices in the first collaborative operation strategy with the potential monitoring devices that have the highest matching degree to obtain the second collaborative operation strategy.

[0054] For example, a fixed camera in a warehouse area detects a "motion detection" event. The rule engine generates a first collaborative operation strategy: dispatch two corridor monitoring devices H1 and H2 around the warehouse to conduct area lockdown and tracking. In this scenario, the large language model analyzes the task requirements of the current collaborative operation task through natural language understanding: cover all possible escape paths; acquire clear facial feature images; continuously track the target's movement trajectory; and prevent the target from discarding evidence. Then, the large language model can not be limited to the monitoring devices in the collaborative rule base, but can comprehensively search the device topology map to obtain potential monitoring devices that meet the task requirements. Based on the matching degree between the potential monitoring devices and the task requirements, the monitoring devices in the first collaborative operation strategy are replaced to obtain a second collaborative operation strategy.

[0055] Therefore, by leveraging the deep semantic understanding capabilities of large language models, discrete device resources can be transformed into on-demand service agents, realizing a shift from rule matching to demand matching. This enables the automatic discovery and recommendation of unlabeled but optimal monitoring resources in complex scenarios, significantly improving the globality and adaptability of resource utilization. This allows the monitoring system to flexibly mobilize resources across the entire domain to achieve monitoring goals, thereby improving the collaborative efficiency of multiple monitoring devices.

[0056] In some embodiments, a second collaborative operation strategy is generated using a large language model based on real-time context data, a first collaborative operation strategy, and a device topology map. This includes: analyzing the similarity between the collaborative monitoring task corresponding to the monitoring event and historical collaborative monitoring tasks using the large language model based on real-time context data and the first collaborative operation strategy; determining similar monitoring tasks similar to the collaborative monitoring task corresponding to the monitoring event based on the similarity; retrieving device control parameters for similar monitoring tasks from an association database, wherein the association database stores association data between device control parameters and the monitoring task completion effect, and the association data at least characterizes the relationship between gimbal zoom parameters, image quality parameters, and target tracking success rate in historical monitoring tasks; and adjusting a preset collaborative action sequence in the first collaborative operation strategy based on the device control parameters to obtain the second collaborative operation strategy.

[0057] Therefore, by establishing a database linking historical parameters and task results, and utilizing the pattern recognition and similarity matching capabilities of large language models, precise optimization of the control parameters of the current task equipment can be achieved. This improves the success rate of monitoring tasks, reduces monitoring risks such as target loss due to parameter mismatch, and enables collaborative operations among multiple monitoring devices to have human-like learning capabilities that allow them to autonomously accumulate operational experience and continuously improve performance, thereby enhancing the monitoring effectiveness of multiple monitoring devices.

[0058] In some embodiments, determining a target collaborative operation strategy based on a first collaborative operation strategy and a second collaborative operation strategy includes: calculating the confidence levels of the first collaborative operation strategy and the second collaborative operation strategy respectively; if the confidence level of the first collaborative operation strategy is greater than a first threshold, using the first collaborative operation strategy as the target collaborative operation strategy; if the confidence level of the first collaborative operation strategy is less than the first threshold and the confidence level of the second collaborative operation strategy is greater than a second threshold, using the second collaborative operation strategy as the target collaborative operation strategy; if the confidence level of the first collaborative operation strategy is less than the first threshold and the confidence level of the second collaborative operation strategy is less than the second threshold, outputting a human decision-making prompt, and in response to receiving the human decision-making information, obtaining the target collaborative operation process based on the human decision-making information.

[0059] For example, the confidence level of the first collaborative action strategy can be calculated by the rule engine using at least one of the confidence levels of the reported monitoring events, the historical accuracy of the first collaborative action strategy, and the real-time context data of the monitoring events. For instance, if the confidence level of the monitoring event itself is 0.65, the historical accuracy is 90%, and the real-time context data indicates that the current time is late at night, the false alarm rate will increase, therefore the attenuation coefficient is 0.8, and the final confidence level is: 0.65 × 0.9 × 0.8 = 0.468.

[0060] For example, the confidence level of the second collaborative operation strategy can be obtained by analyzing the second collaborative operation strategy based on the device topology map and real-time context data using a large language model. For instance, the large model performs logical consistency analysis, context fit analysis, and historical suggestion accuracy analysis, and then multiplies the three analysis results by quantifying them into values ​​between 0 and 1 to obtain the confidence level of the second collaborative operation strategy.

[0061] For example, the first threshold can be set to 0.75, and the second threshold can be set to 0.70.

[0062] When the confidence level of the rule engine is higher than the first threshold, the strategy generated by the rule engine is adopted first. When the confidence level of the rule engine is lower than the first threshold but the confidence level of the large language model is higher than the second threshold, the strategy generated by the large language model is adopted. When both confidence levels are lower than the threshold, a manual decision-making process is triggered. For example, the first collaboration strategy or the second collaborative operation strategy is determined manually, or a new collaborative operation strategy is input manually.

[0063] Therefore, when the rules are clear and the scenario is simple, a stable and reliable rule engine should be prioritized to ensure efficiency. When the rules are unclear or the scenario is complex, the intelligence of a large model should be introduced to enhance its ability to handle complex situations. When neither of these methods can handle the situation well, human decision-making should be performed to ensure that the system behavior is absolutely controllable, thereby improving the accuracy of collaborative operation of multiple monitoring devices.

[0064] It should be understood that the multi-monitoring device collaborative operation method provided in this disclosure can be integrated into a video fusion platform. Through device topology maps and unified spatiotemporal registration, the video fusion platform can acquire spatial awareness capabilities. It can not only perceive which monitoring devices are present, but also their positional relationships and field of view in the physical world. Therefore, through a large language model based on a global device topology map and real-time contextual data, it performs inference and resource scheduling, enabling multiple monitoring devices to achieve proactive and dynamic collaboration based on spatial awareness, thus improving the intelligence level of multi-monitoring device collaborative operation. Furthermore, during the collaborative operation of multiple devices, the video fusion platform can automatically correlate and fuse the data of multiple monitoring devices under a unified spatiotemporal reference based on the device topology map, generating a complete and continuous target motion trajectory in real time, facilitating motion trajectory analysis.

[0065] According to a second aspect of the embodiments of this disclosure, a multi-monitoring device collaborative operation apparatus based on the Internet of Things is provided. Please refer to the appendix. Figure 2 The multi-monitoring device collaborative operation device 200 based on the Internet of Things includes:

[0066] The matching module 201 is used to respond to receiving a monitoring event reported by the first monitoring device, obtain the real-time context data corresponding to the monitoring event, and obtain a first collaborative operation rule based on the monitoring event and the collaborative rule library. The collaborative rule library is used to store pre-configured collaborative operation rules for multiple monitoring devices. Each collaborative operation rule includes a preset trigger event, a preset execution condition, and a preset collaborative action sequence.

[0067] The generation module 202 is used to generate a second collaborative operation strategy based on the real-time context data, the first collaborative operation strategy, and the device topology map using a large language model. The device topology map is obtained based on the spatial attribute information of the multiple monitoring devices, and the spatial attribute information includes spatial location, spatial orientation, and field of view parameters.

[0068] The control module 203 is used to determine a target collaborative operation strategy based on the first collaborative operation strategy and the second collaborative operation strategy, and drive the monitoring equipment to perform collaborative operation based on the target collaborative operation strategy.

[0069] In some embodiments of this disclosure, the generation module 202 is used to:

[0070] The resource dependencies among multiple monitoring devices in the device topology map are analyzed using a large language model. Based on these dependencies, the cascading resource occupancy effect that the first collaborative operation strategy may trigger is predicted. Based on the cascading resource occupancy effect and the device topology map, a resource reservation scheme is generated. Based on the real-time context data and the preset collaborative action sequence in the first collaborative operation strategy, the resource deadlock and competition relationships that the first collaborative operation strategy may trigger are predicted. Based on the resource deadlock, competition, and the device topology map, an alternative resource allocation scheme is generated. The first collaborative operation strategy is adjusted according to the resource reservation scheme and the alternative resource allocation scheme to obtain a second collaborative operation strategy.

[0071] In some embodiments of this disclosure, the generation module 202 is used to:

[0072] Based on the real-time context data and the preset collaborative action sequence in the first collaborative operation strategy, a natural language description of the current collaborative operation scenario is generated. The natural language description is then analyzed by the large language model to obtain the resource deadlock relationship that the first collaborative operation strategy may cause. The natural language description includes the identification information and resource occupancy information of the monitoring devices required by each monitoring task in the first collaborative operation strategy.

[0073] By analyzing the execution timing sensitivity of the preset collaborative action sequence in the first collaborative operation strategy using the large language model, the competitive relationships that the first collaborative operation strategy may cause can be obtained. The execution sensitivity is used to characterize whether different action execution sequences correspond to different monitoring results.

[0074] In some embodiments of this disclosure, the generation module 202 is used to:

[0075] Based on the real-time context data and the first collaborative operation strategy, the large language model understands the task requirements of the collaborative monitoring task corresponding to the monitoring event. It then retrieves potential monitoring devices that meet the task requirements from the device topology map. Based on the spatial location and capability characteristics of the potential monitoring devices, it determines the matching degree between the potential monitoring devices and the task requirements. Finally, it replaces the monitoring devices in the first collaborative operation strategy with the potential monitoring devices that have the highest matching degree, thus obtaining the second collaborative operation strategy.

[0076] In some embodiments of this disclosure, the generation module 202 is used to:

[0077] Based on real-time context data and the first collaborative operation strategy, the large language model analyzes the similarity between the collaborative monitoring task corresponding to the monitoring event and the historical collaborative monitoring task, and based on the similarity, determines similar monitoring tasks that are similar to the collaborative monitoring task corresponding to the monitoring event.

[0078] The device control parameters of the similar monitoring tasks are retrieved from the association database, wherein the association database is used to store the association data between the device control parameters and the monitoring task completion effect, and the association data at least represents the relationship between the pan-tilt zoom parameters, image quality parameters and target tracking success rate in the historical monitoring tasks.

[0079] Based on the equipment control parameters, the preset collaborative action sequence in the first collaborative operation strategy is adjusted to obtain the second collaborative operation strategy.

[0080] In some embodiments of this disclosure, the matching module 201 is used for:

[0081] The matching collaborative rules are obtained by matching the monitoring events in the collaborative rules through the rule engine;

[0082] Based on the location information of the monitoring device and the movement direction of the monitoring target in the monitoring event, the movement path of the monitoring target is predicted to obtain the predicted movement path, and based on the device topology map, the collaborative monitoring devices whose field of view covers the predicted movement path are determined.

[0083] A first collaborative operation strategy is generated based on the collaborative monitoring device and the preset collaborative action sequence in the matching collaborative rules.

[0084] In some embodiments of this disclosure, the control module 203 is used for:

[0085] Calculate the confidence levels of the first collaborative operation strategy and the second collaborative operation strategy respectively;

[0086] If the first collaborative operation strategy is greater than the first threshold, the first collaborative operation strategy will be used as the target collaborative operation strategy.

[0087] If the confidence level of the first collaborative operation strategy is less than the first threshold and the confidence level of the second collaborative operation strategy is greater than the second threshold, the second collaborative operation strategy shall be used as the target collaborative operation strategy.

[0088] If the confidence level of the first collaborative operation strategy is less than the first threshold and the confidence level of the second collaborative operation strategy is less than the second threshold, a manual decision prompt is output, and in response to receiving the manual decision information, the target collaborative operation process is obtained based on the manual decision information.

[0089] In some embodiments of this disclosure, the device topology map is obtained through the following modules:

[0090] The loading module is used to load the corresponding virtual device driver through the smart gateway in response to the monitoring device accessing the network, and to create a virtual device instance in the video fusion platform that uniquely corresponds to the monitoring device.

[0091] The reporting module is used to abstract the functions of the physical monitoring equipment into a standardized capability set, and report the standardized capability set to the video fusion platform through the smart gateway;

[0092] An add module is used to add spatial attribute information to the virtual device instance based on user input;

[0093] The determination module is used to determine the spatial relationship of each virtual device instance based on the spatial attribute information, and to generate a device topology map based on the spatial relationship.

[0094] In some embodiments of this disclosure, the adding module is used for:

[0095] Display the video footage captured by the monitoring equipment to the user;

[0096] In response to the user's selection operation of at least two reference points in the video frame, the pixel coordinates of the at least two reference points in the video frame are determined;

[0097] In response to the user's input operation on the actual physical distance between the at least two reference points, based on the pixel coordinates of the at least two reference points in the video frame and the actual physical distance corresponding to the input operation, the spatial coordinates, spatial orientation, and field of view parameters of the monitoring device corresponding to the virtual device instance are obtained, and the spatial coordinates, spatial orientation, and field of view parameters are added to the virtual device instance.

[0098] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the method in the first aspect, and will not be elaborated upon here.

[0099] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0100] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0101] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A method for collaborative operation of multiple monitoring devices based on the Internet of Things, characterized in that, include: In response to receiving a monitoring event reported by a monitoring device, the system obtains the real-time context data corresponding to the monitoring event, and obtains a first collaborative operation rule through a rule engine based on the monitoring event and a collaborative rule library. The collaborative rule library is used to store pre-configured collaborative operation rules for multiple monitoring devices, and each collaborative operation rule includes a preset trigger event, a preset execution condition, and a preset collaborative action sequence. A second collaborative operation strategy is generated using a large language model based on the real-time context data, the first collaborative operation strategy, and the device topology map. The device topology map is obtained by processing the spatial attribute information of the multiple monitoring devices based on a unified spatiotemporal reference. The spatial attribute information includes spatial location, spatial orientation, and field of view parameters. Specifically... The resource dependencies between multiple monitoring devices in the device topology map are analyzed using a large language model. Based on these dependencies, the cascading resource occupancy effect that the first collaborative operation strategy may cause is predicted. Based on the cascading resource occupancy effect and the device topology map, a resource reservation scheme is generated. Based on the real-time context data and the preset collaborative action sequence in the first collaborative operation strategy, the resource deadlock and competition relationships that the first collaborative operation strategy may cause are predicted. Based on the resource deadlock, competition, and the device topology map, an alternative resource allocation scheme is generated. The first collaborative operation strategy is adjusted according to the resource reservation scheme and the alternative resource allocation scheme to obtain a second collaborative operation strategy. Based on the first collaborative operation strategy and the second collaborative operation strategy, a target collaborative operation strategy is determined, and based on the target collaborative operation strategy, the monitoring equipment is driven to perform collaborative operations.

2. The method for collaborative operation of multiple monitoring devices in the Internet of Things according to claim 1, characterized in that, The prediction of resource deadlock and competition relationships that may be caused by the first collaborative operation strategy, based on the real-time context data and the preset collaborative action sequence in the first collaborative operation strategy, includes: Based on the real-time context data and the preset collaborative action sequence in the first collaborative operation strategy, a natural language description of the current collaborative operation scenario is generated. The natural language description is then analyzed by the large language model to obtain the resource deadlock relationship that the first collaborative operation strategy may cause. The natural language description includes the identification information and resource occupancy information of the monitoring devices required by each monitoring task in the first collaborative operation strategy. By analyzing the execution timing sensitivity of the preset collaborative action sequence in the first collaborative operation strategy using the large language model, the competitive relationship that the first collaborative operation strategy may cause can be obtained. The execution timing sensitivity is used to characterize whether different action execution sequences correspond to different monitoring results.

3. The method for collaborative operation of multiple monitoring devices in the Internet of Things according to claim 1, characterized in that, The step of generating a second collaborative operation strategy based on the real-time context data, the first collaborative operation strategy, and the device topology map using a large language model includes: Based on the real-time context data and the first collaborative operation strategy, the large language model understands the task requirements of the collaborative monitoring task corresponding to the monitoring event. It then retrieves potential monitoring devices that meet the task requirements from the device topology map. Based on the spatial location and capability characteristics of the potential monitoring devices, it determines the matching degree between the potential monitoring devices and the task requirements. Finally, it replaces the monitoring devices in the first collaborative operation strategy with the potential monitoring devices that have the highest matching degree, thus obtaining the second collaborative operation strategy.

4. The method for collaborative operation of multiple monitoring devices in the Internet of Things according to claim 1, characterized in that, The step of generating a second collaborative operation strategy based on the real-time context data, the first collaborative operation strategy, and the device topology map using a large language model includes: Based on real-time context data and the first collaborative operation strategy, the large language model analyzes the similarity between the collaborative monitoring task corresponding to the monitoring event and the historical collaborative monitoring task, and based on the similarity, determines similar monitoring tasks that are similar to the collaborative monitoring task corresponding to the monitoring event. The device control parameters of the similar monitoring tasks are retrieved from the association database, wherein the association database is used to store the association data between the device control parameters and the monitoring task completion effect, and the association data at least represents the relationship between the pan-tilt zoom parameters, image quality parameters and target tracking success rate in the historical collaborative monitoring tasks. Based on the equipment control parameters, the preset collaborative action sequence in the first collaborative operation strategy is adjusted to obtain the second collaborative operation strategy.

5. The method for collaborative operation of multiple monitoring devices in an intelligent network according to any one of claims 1-4, characterized in that, The first collaborative operation rule is obtained by the rule engine based on the monitored events and the collaborative rule base, including: The matching collaborative rules are obtained by matching the monitoring events in the collaborative rules through the rule engine; Based on the location information of the monitoring device and the movement direction of the monitoring target in the monitoring event, the movement path of the monitoring target is predicted to obtain the predicted movement path, and based on the device topology map, the collaborative monitoring devices whose field of view covers the predicted movement path are determined. A first collaborative operation strategy is generated based on the collaborative monitoring device and the preset collaborative action sequence in the matching collaborative rules.

6. The method for collaborative operation of multiple monitoring devices in an intelligent network according to any one of claims 1-4, characterized in that, The step of determining the target collaborative operation strategy based on the first collaborative operation strategy and the second collaborative operation strategy includes: Calculate the confidence levels of the first collaborative operation strategy and the second collaborative operation strategy respectively; If the first collaborative operation strategy is greater than the first threshold, the first collaborative operation strategy will be used as the target collaborative operation strategy. If the confidence level of the first collaborative operation strategy is less than the first threshold and the confidence level of the second collaborative operation strategy is greater than the second threshold, the second collaborative operation strategy shall be used as the target collaborative operation strategy. If the confidence level of the first collaborative operation strategy is less than the first threshold and the confidence level of the second collaborative operation strategy is less than the second threshold, a manual decision prompt is output, and in response to receiving the manual decision information, the target collaborative operation process is obtained based on the manual decision information.

7. The method for collaborative operation of multiple monitoring devices in an intelligent network according to any one of claims 1-4, characterized in that, The device topology map was obtained in the following way: In response to the monitoring device accessing the network, the corresponding virtual device driver is loaded through the smart gateway, and a virtual device instance uniquely corresponding to the monitoring device is created in the video fusion platform; The functions of the monitoring equipment are abstracted into a standardized capability set, and the standardized capability set is reported to the video fusion platform through the smart gateway; Based on user input, spatial attribute information is added to the virtual device instance; Based on the spatial attribute information, the spatial relationships of each virtual device instance are determined, and a device topology map is generated based on the spatial relationships.

8. The method for collaborative operation of multiple monitoring devices based on the Internet of Things according to claim 7, characterized in that, The step of adding spatial attribute information to the virtual device instance based on user input includes: Display the video footage captured by the monitoring equipment to the user; In response to the user's selection operation of at least two reference points in the video frame, the pixel coordinates of the at least two reference points in the video frame are determined; In response to the user's input operation on the actual physical distance between the at least two reference points, based on the pixel coordinates of the at least two reference points in the video frame and the actual physical distance corresponding to the input operation, the spatial coordinates, spatial orientation, and field of view parameters of the monitoring device corresponding to the virtual device instance are obtained, and the spatial coordinates, spatial orientation, and field of view parameters are added to the virtual device instance.

9. A collaborative operation device for multiple monitoring devices based on the Internet of Things, characterized in that, include: The matching module is used to respond to a monitoring event reported by the first monitoring device, obtain the real-time context data corresponding to the monitoring event, and obtain a first collaborative operation rule based on the monitoring event and the collaborative rule library. The collaborative rule library is used to store pre-configured collaborative operation rules for multiple monitoring devices. Each collaborative operation rule includes a preset trigger event, a preset execution condition, and a preset collaborative action sequence. The generation module is used to generate a second collaborative operation strategy based on the real-time context data, the first collaborative operation strategy, and the device topology map using a large language model. The device topology map is obtained based on the spatial attribute information of the multiple monitoring devices, including spatial location, spatial orientation, and field of view parameters. Specifically... The resource dependencies between multiple monitoring devices in the device topology map are analyzed using a large language model. Based on these dependencies, the cascading resource occupancy effect that the first collaborative operation strategy may cause is predicted. Based on the cascading resource occupancy effect and the device topology map, a resource reservation scheme is generated. Based on the real-time context data and the preset collaborative action sequence in the first collaborative operation strategy, the resource deadlock and competition relationships that the first collaborative operation strategy may cause are predicted. Based on the resource deadlock, competition, and the device topology map, an alternative resource allocation scheme is generated. The first collaborative operation strategy is adjusted according to the resource reservation scheme and the alternative resource allocation scheme to obtain a second collaborative operation strategy. The control module is used to determine a target collaborative operation strategy based on the first collaborative operation strategy and the second collaborative operation strategy, and to drive the monitoring equipment to perform collaborative operation based on the target collaborative operation strategy.

Images