Cooperative system, method of inspection, cloud server, user terminal and storage medium

Abstract

The present application relates to the technical field of Internet of Things, and particularly relates to a cooperative system, a patrol method, a cloud server, a user terminal and a storage medium, the cooperative system comprising a cloud server, a user terminal, an edge gateway and a target device, the user terminal is used for generating task issuing information in response to a task creation instruction, and sending the task issuing information to the cloud server based on a specified protocol, the cloud server converts the task issuing information into task execution instructions of a communication protocol adapted to the target device, and sends the task execution instructions after protocol conversion to the edge gateway, the edge gateway forwards the task execution instructions to the target device, and the target device executes a target task according to the task execution instructions. The embodiment can not only solve the problem of multi-device heterogeneity, thereby improving system scalability, but also enable the user terminal to quickly interact with different devices during task scheduling, thereby improving interaction real-time performance and reducing patrol cooperation delay.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, specifically to a collaborative system, inspection method, cloud server, user terminal, and storage medium. Background Technology

[0002] Traditional technologies typically improve inspection efficiency by controlling smart devices to perform various inspection tasks through manual control or by building an Internet of Things (IoT) architecture. However, in traditional IoT architectures, when cloud servers interface with multiple devices and schedule tasks for these devices, there is often a heterogeneity issue with the protocols and data formats of these devices. Traditional technologies often struggle to shield the differences in the underlying protocols of these devices to manage them based on a unified protocol. This can easily lead to insufficient real-time interaction between the cloud server and the devices, thereby reducing the efficiency of multi-device collaboration. Summary of the Invention

[0003] The present invention aims to provide a collaborative system, inspection method, cloud server, user terminal and storage medium to improve the technical problem of insufficient real-time interaction between cloud server and device in the prior art.

[0004] In a first aspect, embodiments of the present invention provide a collaborative system, including a cloud server, a user terminal, an edge gateway, and a target device, wherein the cloud server is communicatively connected to the user terminal and the edge gateway, and the edge gateway is also communicatively connected to the target device; The user terminal responds to the task creation instruction to generate task distribution information, and sends the task distribution information to the cloud server based on a specified protocol, wherein the specified protocol is a unified protocol exposed to the outside world; The cloud server converts the task distribution information into a task execution instruction adapted to the communication protocol of the target device, and sends the protocol-converted task execution instruction to the edge gateway; The edge gateway forwards the task execution command to the target device; The target device executes the target task according to the task execution instruction.

[0005] Optionally, the task distribution information includes target task information and the identity information of the target device. The cloud server determines the target object model based on the identity information and converts the target task information into task execution instructions adapted to the communication protocol of the target device based on the target object model.

[0006] Optionally, the target device uploads task reporting information to the edge gateway based on a communication protocol adapted to the target device; The edge gateway forwards the task reporting information to the cloud server; The cloud server converts the task reporting information into task reporting information with a specified protocol based on the target object model, and sends the converted task reporting information to the user terminal. The user terminal monitors the task execution process of the target device based on the task reporting information.

[0007] Optionally, the task reporting information includes the live stream address of the target device; The target device will push the real-time video stream collected by capturing and monitoring the target to the real-time video server; The user terminal parses the live stream address from the task reporting information, retrieves the real-time video stream from the real-time video server based on the live stream address, and displays the real-time video stream on the user terminal.

[0008] Optionally, the user terminal sends a streaming request to the cloud server, and the streaming request includes the live stream address of the real-time video stream; When the cloud server receives the streaming request, it retrieves the real-time video stream from the real-time video server according to the live stream address, inputs the real-time video stream into a preset large model, obtains video analysis results, and sends the video analysis results to the user terminal. The user terminal determines whether there is an anomaly in the monitored target based on the video analysis results. If an anomaly is found, an alarm message is generated.

[0009] Optionally, the task reporting information includes the task completion identifier of the target task; The user terminal responds to the task end identifier to obtain the alarm information and target task information of the target task, and generates a task report based on the alarm information and target task information.

[0010] Optionally, the task reporting information includes the real-time status of the target device, task execution information, and fault information; The user terminal monitors the target device based on the real-time status, the task execution information, and the fault information.

[0011] Optionally, the task reporting information may also include the device attitude, device position, and target position of the monitored target; The user terminal generates target trajectory information based on the device attitude, the device position, and the target position, and sends the target trajectory information to the cloud server; The cloud server converts the target trajectory information into a trajectory movement command adapted to the communication protocol of the target device, and sends the protocol-converted trajectory movement command to the edge gateway; The edge gateway forwards the trajectory movement command to the target device; The target device moves to the designated position according to the trajectory movement command.

[0012] In a second aspect, embodiments of the present invention provide an inspection method applied to a cloud server as described in the first aspect above, the inspection method comprising: Obtain task distribution information sent by the user terminal based on a specified protocol, wherein the specified protocol is a unified protocol exposed to the outside world; The task distribution information is converted into a task execution instruction adapted to the communication protocol of the target device, and the protocol-converted task execution instruction is sent to the edge gateway so that the edge gateway forwards the task execution instruction to the target device. The task execution instruction is used to instruct the target device to execute the target task.

[0013] In a third aspect, embodiments of the present invention provide an inspection method applied to a user terminal as described in the first aspect above, the inspection method comprising: Respond to task creation instructions to generate task distribution information; The task distribution information is sent to the cloud server based on a specified protocol, so that the cloud server converts the task distribution information into a task execution instruction adapted to the communication protocol of the target device, and forwards the protocol-converted task execution instruction to the target device via the edge gateway. The task execution instruction is used to instruct the target device to execute the target task. The specified protocol is a unified protocol exposed to the outside world.

[0014] In a fourth aspect, embodiments of the present invention provide a cloud server, including a memory and a processor, wherein the memory is connected to the processor, and the processor is configured to execute one or more computer programs stored in the memory, wherein when the processor executes the one or more computer programs, the cloud server enables the inspection method described in the second aspect above.

[0015] In a fifth aspect, embodiments of the present invention provide a user terminal, including a memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, and the processor, when executing the one or more computer programs, causing the user terminal to implement the inspection method described in the third aspect above.

[0016] In a sixth aspect, embodiments of the present invention provide a storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the inspection method as described in the second or third aspect above.

[0017] Compared with the prior art, the embodiments of the present invention provide a collaborative system, an inspection method, a cloud server, a user terminal, and a storage medium. The collaborative system includes a cloud server, a user terminal, an edge gateway, and a target device. The cloud server is communicatively connected to both the user terminal and the edge gateway. The edge gateway is also communicatively connected to the target device. The user terminal is used to generate task distribution information in response to task creation instructions and send the task distribution information to the cloud server based on a specified protocol. The specified protocol is a unified protocol exposed to the outside world. The cloud server converts the task distribution information into task execution instructions adapted to the communication protocol of the target device and sends the protocol-converted task execution instructions to the edge gateway. The edge gateway forwards the task execution instructions to the target device, and the target device executes the target task according to the task execution instructions. Therefore, this embodiment constructs an IoT architecture of intelligent agent-edge gateway-cloud server-user terminal, and performs protocol conversion on downlink data on the cloud server side so that the user terminal can issue tasks based on the unified protocol exposed to the outside world. This is equivalent to seamless interaction between the user terminal and each intelligent agent. Thus, this embodiment can solve the problem of heterogeneity of multiple devices, thereby improving system scalability. On the other hand, it enables the user terminal to quickly interact with different devices during task scheduling, thereby improving the real-time performance of interaction, reducing inspection and collaboration delays, and achieving more real-time task scheduling. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram of the structure of a collaborative system provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of an inspection process for a collaborative system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating an interactive method for executing an action, provided by an embodiment of the present invention. Figure 4 A schematic diagram of a reporting link provided in an embodiment of the present invention; Figures 5 to 8A schematic diagram of a module creation process provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of a task reporting information processing flow provided in an embodiment of the present invention; Figure 10 A flowchart illustrating an inspection method provided in an embodiment of the present invention; Figure 11 A flowchart illustrating an inspection method according to another embodiment of the present invention; Figure 12 This is a schematic diagram of the structure of an inspection device provided in an embodiment of the present invention; Figure 13 This is a schematic diagram of the structure of an inspection device according to another embodiment of the present invention; Figure 14 This is a hardware schematic diagram of a device provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0021] It should be noted that, unless otherwise specified, the various features in the embodiments of this invention can be combined with each other, all of which are within the protection scope of this invention. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this invention do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0022] Please see Figure 1 This invention provides a collaborative system, such as... Figure 1 As shown, the collaborative system 100 includes a target device 101, an edge gateway 102, a cloud server 103, and a user terminal 104.

[0023] Target device 101 is an intelligent agent used to perform designated inspection tasks within a specified inspection area. The inspection area is a physical range pre-defined according to business needs and used for systematically carrying out inspection work. Designated inspection tasks may include pipeline oil leak inspection tasks, vehicle illegal parking inspection tasks, etc. Target device 101 can be an intelligent agent capable of independent thought and possessing mobility, able to move to any location within the designated inspection area. Target device 101 can be any type of intelligent agent, including but not limited to robots, robot dogs, or drones. There can be one or more target devices 101. When there are multiple target devices 101, they can include homogeneous intelligent agents or heterogeneous intelligent agents. Homogeneous intelligent agents are those with the same mechanical structure and operating environment, while heterogeneous intelligent agents are those with different mechanical structures and operating environments. For example, robot dogs and drones are heterogeneous intelligent agents.

[0024] In some embodiments, the target device 101 includes a camera, a controller, and a communication module. The camera is electrically connected to the controller and is used to collect environmental data when performing a specified inspection task. The controller is electrically connected to the communication module and is used to upload the environmental data to an upper-layer application through the communication module. The communication module can upload data based on any communication type, including but not limited to MQTT, HTTP, CoAP, Websocket, etc.

[0025] Edge gateway 102 is communicatively connected to target device 101 and cloud server 103, and is used to upload data sent by target device 101 to cloud server 103 via wired / wireless networks such as Wi-Fi, 5G, and LoRa. In some scenarios, a rule engine is required to perform protocol conversion. Edge gateway 102 can be a smart hardware device deployed at the network edge (close to the data source, such as in factory workshops, building sites, or equipment racks), and acts as a bridge connecting the physical and digital worlds, responsible for data communication, protocol conversion, edge computing, and intelligent processing between target device 101 and cloud server 103.

[0026] The cloud server 103 is communicatively connected to both the edge gateway 102 and the user terminal 104. It transmits downlink data from the user terminal 104 to the target device 101 via the edge gateway 102, such as sending task data to the target device 101. Alternatively, it receives uplink data reported by the target device 101 via the edge gateway 102, such as the status of the target device 101 and collected environmental data. The cloud server 103 can monitor the task execution process of the target device 101 based on its status and collected environmental data. For example, it can identify abnormal situations such as oil leaks in pipelines or illegal parking, allowing it to promptly notify the user terminal 104 and take appropriate measures, including alarms, in the event of an anomaly.

[0027] In some embodiments, the cloud server 103 can be a standalone physical server, a server cluster consisting of multiple physical servers, or a distributed system. The cloud server 103 serves as the core of the task monitoring system and is used to execute the inspection methods described below.

[0028] In some embodiments, regarding the security of uplink and downlink data transmission between the target device 101 and the user terminal 104, the cloud server 103 can perform device legitimacy verification. When the target device 101 establishes a long-connection channel, the cloud server 103 can provide HTTP or database-based device legitimacy verification through a message broker service. The cloud server 103 can provide certificates from both ends, and perform strong verification on the certificates of both ends when the target device 101 initiates a connection to ensure the security of transmission.

[0029] In some embodiments, in a weak network environment, the cloud server 103 can also provide a device shadow function to ensure message reachability. The device shadow function compares the current value reported by the target device 101 with the expected value to determine whether there is a situation where a command has not arrived. If the expected value and the reported value do not match, it is determined that a command needs to be sent synchronously.

[0030] The cloud server 103 can internally adapt to multiple protocols based on the object model. The object model is a digital abstract description of the target device on the cloud server, used to define the functions and data interaction methods of the target device. In some embodiments, the cloud server 103 may include an access layer, a platform layer, and an application layer. The access layer is used for uplink and downlink data exchange with the target device 101 via the edge gateway 102, integrating common device access protocols such as MQTT and WebSocket. The access layer is compatible with commonly used protocols and then converts them into a unified protocol based on the object model for use by the user terminal 104. The platform layer exposes unified atomic capabilities (such as device control) and real-time device status information, providing functions such as terminal discovery and registration, time-series data storage, security authentication, protocol conversion, and unified methods for uplink and downlink interaction with the application layer. The application layer, also known as the business layer, can be a specific business capability provider. Combining the channel capabilities and large model of the access layer with a visual interface, it empowers scenarios such as smart homes, industrial IoT, and smart cities. The application layer and platform layer can use the HTTP protocol to operate the device, receiving real-time device information and task process data sent by the platform layer via Kafka using a unified protocol.

[0031] User terminal 104 can be any type of electronic device. In some embodiments, the type of user terminal 104 includes, but is not limited to, smartphones, desktop computers, laptops, digital broadcast receivers, personal digital assistants (PDAs), portable Android devices (PADs) and other user equipment (UEs), handheld devices, in-vehicle devices, wearable devices, computing devices or other processing devices connected to a wireless modem, mobile stations (MS), mobile terminals, etc.

[0032] In some embodiments, please refer to Figure 2 The collaborative system provided in this embodiment of the invention can implement the following inspection method: S201. The user terminal responds to the task creation instruction, generates task distribution information, and sends the task distribution information to the cloud server based on the specified protocol, which is the unified protocol exposed to the outside world.

[0033] In this step, the task creation instruction is used to instruct the user terminal to create an inspection task. The user can input the task creation instruction by operating the user terminal. The specified protocol is the communication protocol uniformly adopted by the user terminal. This communication protocol can be configured according to business needs, for example, the communication protocol is HTTP. The task distribution information is used to issue instructions to the target device to execute the inspection task. In some embodiments, the task distribution information includes target task information and target device identity information. The target task information is the information of the target task that the target device needs to perform. The target task can be an inspection task, a monitoring task, etc. An inspection task can be a pipeline oil leak inspection, a vehicle illegal parking inspection, etc. A monitoring task can be a continuous image acquisition or video recording of monitoring targets such as pipelines and vehicles. The identity information is the information used to identify the target device and uniquely determine the specific target device.

[0034] S202, The cloud server converts the task distribution information into task execution instructions that are adapted to the communication protocol of the target device, and sends the converted task execution instructions to the edge gateway.

[0035] In this step, the communication protocol adapted to the target device can be a protocol that the target device can directly use to send and receive data. Typically, different target devices may require different communication protocols. For example, the target devices may include device A, device B, device C, and device D. Device A might use the MQTT protocol, device B the HTTP protocol, device C the CoAP protocol, and device D the WebSocket protocol. The task execution instruction is the command used to instruct the target device to execute the target task.

[0036] This embodiment converts task distribution information into task execution instructions that adapt to the communication protocol of the target device. Although the communication protocol adapted to each target device is different, the data structure of the task execution instructions can be adapted to the communication protocol adapted to each target device. Therefore, for each target device, the task execution instructions are a kind of undifferentiated intelligent agent instructions.

[0037] In some embodiments, the cloud server determines the target object model based on the identity information, and converts the target task information into task execution instructions that are adapted to the communication protocol of the target device based on the target object model.

[0038] In this embodiment, the target object model is an object model corresponding to the target device. By defining the target device's attributes (such as location and battery level), services (such as remote control), and events (such as fault alarms) using standardized data templates, a digital description of the target device can be achieved. The target object model is configured with a mapping relationship between the target device and the target protocol; different target devices may correspond to different target protocols. The object model is a digital mapping of an entity in physical space (such as a drone, robot dog, or robot) in the cloud. It describes what the entity is, what it can do, and what information it can provide from three dimensions: attributes, services, and events. Defining these three dimensions of the object model completes the definition of the product functionality.

[0039] Attributes are used to describe specific information and status of the device during operation, such as the OSD (On Screen Display) reported by the drone at a fixed frequency and the status information reported on demand. Specifically, it can include the drone's flight mode, altitude limit attribute, various payload capabilities of the drone (such as video recording capability), real-time latitude and longitude information, battery and backup battery status, various RTK (Real-time kinematic) capability information and flight route information, etc.

[0040] An action refers to an instruction or method that the target device can invoke externally. Input and output parameters can be set in a service call. Input parameters are the parameters used during service execution, and output parameters are the results after service execution. Compared to attributes, services can implement more complex business logic through a single instruction, such as executing a specific task. Services can be called asynchronously or synchronously. The capabilities of a drone may include device management, live streaming, flight path functionality, early warning capabilities, OTA (Over-the-Air Technology) capabilities, and command flight capabilities.

[0041] An event refers to information that a target device actively reports to the cloud during operation. It generally includes information that needs to be perceived and processed externally, such as progress, alarms, and faults. An event may contain multiple output parameters, such as AI (Artificial Intelligence) recognition and shooting result reporting, evidence collection status reporting, illegal parking detection information reporting, and flyto execution result event notifications.

[0042] In some embodiments, the access layer supports defining multiple sets of functions for a product, each set of functions can include attributes, services, and events. A collection of function definitions constitutes a device model, and multiple device models do not affect each other. When a device has many functions, it can be broken down into different modules. For example, a drone has a flight control module and a gimbal operation module. Breaking it down by modules makes things clearer. Multiple modules are relatively independent and can be developed in parallel, thus breaking down complex problems. Therefore, the device model solves the problem of complex device modeling in industrial scenarios, making it easier to develop devices with different functions under the same product, such as a drone's flight path management module and command flight module.

[0043] The product encompasses all metadata of a collection of smart devices deployed in a real-world scenario, including both the most core and fundamental functional attributes developed during product development, as well as all custom-added support actions. Each device can only be associated with one product, while a single product can have multiple devices. In some embodiments, the product may include device attributes, device actions, and device events.

[0044] For device attributes, attribute function points are often used to describe the state of a product, such as the latitude and longitude of a drone, or an RTK switch. Because the range of definitions is wide, a rich set of data types is provided. Device attributes can define function names, identifiers, and data types. Among them, the function name is the name of the function that the product has, the identifier can be used to define the function and is unique under the product, and the data type can be selected according to the characteristics of different attributes. Data types include, but are not limited to, numeric, character, time, boolean, enumeration, transparent, structure, array, and fault types.

[0045] For device actions, action function points are primarily used to describe the complex functions a product possesses, which have input and output definitions. A typical characteristic is that message sending exhibits a one-to-one response feature, often used in scenarios where a function requires a response. For example, if defined as "opening a door," after the door-opening action is completed, a confirmation message will be received, thus achieving complete function tracking. Functions of devices such as drones, drone nests, remote controllers, and robotic dogs can all be completed through action interaction methods. The interaction methods for action execution are as follows... Figure 3 As shown.

[0046] For device events, event function points are commonly used for device fault and alarm reporting. Information can be transmitted to the cloud by defining one or more output parameters for the function reporting, such as water temperature alarms and fire alarms. Events are actively reported by the device, and the reporting chain is as follows: Figure 4 As shown.

[0047] When modeling, firstly, to ensure compatibility with data reported by multiple device protocols, a unified object model can be converted to a standard protocol. Next, new products can be created for each device. Then, different modules can be broken down according to the product's functional categories. Finally, attributes, events, and services can be created for each module. The process of creating attributes, events, and services for each module is as follows: Figures 5 to 8 As shown.

[0048] To enable access to devices using various protocols, this invention proposes a generic gateway concept. It investigates commonly used device interfacing protocols such as EMQX and WebSocket, and provides a configuration function to support these common protocols. As a result, no changes to the access layer code are required during future device interfacing, and the access layer can seamlessly connect to third-party platforms.

[0049] After the access layer connects to the device, the device will report real-time data such as errors, alarms and attitude information on a timely / as-needed basis, thereby maintaining the health status of the device and reducing the cost of manual intervention.

[0050] S203, the edge gateway forwards the task execution command to the target device.

[0051] In this step, the edge gateway acts as a data forwarding center between the cloud server and the target device. It can forward downlink data (such as task execution instructions) from the cloud server to the target device, and it can also forward uplink data from the target device to the cloud server.

[0052] S204. The target device executes the target task according to the task execution instruction.

[0053] This embodiment constructs an IoT architecture consisting of a device-edge gateway-cloud server-user terminal, and performs protocol conversion on downlink data on the cloud server side so that the user terminal can issue tasks based on a unified protocol exposed to the outside world. This is equivalent to seamless interaction between the user terminal and various devices. Therefore, this embodiment can solve the problem of heterogeneity among multiple devices, thereby improving system scalability. On the other hand, it enables the user terminal to quickly interact with different devices during task scheduling, thereby improving the real-time performance of interaction, reducing inspection and collaboration delays, and achieving more real-time task scheduling.

[0054] In some embodiments, the target device uploads task reporting information to the edge gateway based on a communication protocol adapted to the target device. The edge gateway forwards the task reporting information to the cloud server. The cloud server converts the task reporting information into task reporting information of a specified protocol according to the target object model and sends the protocol-converted task reporting information to the user terminal. The user terminal monitors the task execution process of the target device based on the task reporting information.

[0055] In this embodiment, the task reporting information is the information that the target device needs to report to the user terminal during the execution of the target task, which is used to characterize the execution status of the target device in executing the target task.

[0056] Therefore, in this embodiment, the uplink data is converted on the cloud server side of the IoT architecture described above so that the user terminal can obtain the reported tasks based on the unified protocol exposed to the outside world. This is equivalent to seamless interaction between the user terminal and various devices. Therefore, this embodiment can solve the problem of heterogeneity of multiple devices on the one hand, thereby improving the system scalability. On the other hand, it enables the user terminal to quickly interact with different devices during the task scheduling process, thereby improving the real-time interaction, reducing the inspection and coordination delay, and realizing more real-time task scheduling.

[0057] In some embodiments, the task reporting information includes the real-time status of the target device, task execution information, and / or fault information, and the user terminal can monitor the target device based on the real-time status, task execution information, and / or fault information.

[0058] In this embodiment, real-time status may include real-time remaining battery power, real-time location, etc.; task execution information may include task start information, task interruption information, task end information, etc.; and fault information may include location signal loss, battery abnormality, communication interruption, power system failure, etc. By monitoring the target device in various dimensions, it can be ensured that the target device completes the target task more reliably.

[0059] Therefore, this embodiment constructs an IoT architecture of device-edge gateway-cloud server, and performs protocol conversion on uplink and downlink data through the access layer on the cloud server side, so that the business system can process the uplink information of the unified protocol and schedule tasks for different devices through the downlink information of the unified protocol. On the one hand, it can solve the problem of heterogeneity of multiple devices, thereby improving the system scalability. On the other hand, it can enable the business system to quickly interact with different devices during the task scheduling process, thereby improving the real-time interaction, reducing edge collaboration latency, and realizing more real-time task scheduling and monitoring.

[0060] In some embodiments, the target device uploads task reporting information to the edge gateway based on a communication protocol adapted to the target device. The edge gateway forwards the task reporting information to the cloud server. The cloud server converts the task reporting information into task reporting information of a specified protocol according to the target object model and sends the protocol-converted task reporting information to the user terminal. The user terminal monitors the task execution process of the target device based on the task reporting information.

[0061] In this embodiment, because the target object model is configured with a mapping relationship between the target device and its adapted communication protocol, the cloud server can report uplink messages from the target device according to the structure of the unified object model. This allows user terminals and target devices to interface through the target object model, thus solving the problem of device heterogeneity. If new types of devices are subsequently added, only a new object model needs to be created to support their access, thereby improving system scalability.

[0062] In some embodiments, the task creation instruction includes inspection task information, and the task execution instruction includes a motion trajectory file. The user terminal can generate a motion trajectory file based on the inspection task information and use the motion trajectory file and the identity information of the target device as task distribution information.

[0063] In this embodiment, the motion trajectory file is used to control the movement of the target device. The motion trajectory file can be a path file, a flight path file, etc. The flight path file can include flight route information, point information, and operation information at each point. The inspection task information can be configured according to business needs. The business operator can add one-time inspection tasks or periodic inspection tasks through the user terminal, thereby inputting the task creation command to the user terminal so that the user terminal can respond to the task creation command and issue the task.

[0064] In some embodiments, the task reporting information includes the live stream address of the target device. The target device pushes the real-time video stream collected by shooting and monitoring the target to the real-time video server. The user terminal parses the live stream address from the task reporting information, pulls the real-time video stream from the real-time video server according to the live stream address, and presents the real-time video stream on the user terminal.

[0065] In this embodiment, the live stream address can be the network address used by the target device for live streaming push or pull, usually starting with a specific protocol, such as RTMP, HTTP, RTSP, etc. When the motion trajectory file is sent to the target device, the live stream of the target device is started. The target device pushes the real-time video stream collected by shooting and monitoring the target to the Simple Realtime Server (SRS). The SRS is responsible for protocol conversion and multi-channel forwarding of video streams. The user terminal can obtain the real-time video stream from the SRS according to the live stream address and display the real-time video stream on the user terminal, thereby monitoring the entire process of the target device performing the target task in real time.

[0066] In some embodiments, the user terminal can send a streaming request to the cloud server. The streaming request includes the live stream address of the real-time video stream. When the cloud server receives the streaming request, it retrieves the real-time video stream from the real-time video server according to the live stream address, inputs the real-time video stream into a preset large model, obtains video analysis results, and sends the video analysis results to the user terminal. The user terminal determines whether there is an anomaly in the monitored target based on the video analysis results. If there is an anomaly, an alarm message is generated.

[0067] In this embodiment, the pull request is used to request the cloud server to pull the real-time video stream from the real-time video server. The preset large model can be a large model used for video analysis and anomaly detection of the real-time video stream, such as determining whether there are anomalies like oil leaks or illegal parking. If an anomaly is determined, the analysis result is sent to the video analysis service. The preset large model extracts structured information, recognizes patterns, and understands semantics from the real-time video stream, thereby achieving functions such as object detection, action recognition, and scene understanding. The preset large model can be based on a deep learning architecture, such as a Convolutional Neural Network (CNN) or Transformer model architecture, and trained on a large-scale video dataset to achieve high-precision real-time processing. Anomaly information is used to indicate the anomaly type, which can be an oil leak anomaly, illegal parking anomaly, etc. Alarm information includes, but is not limited to, anomaly severity, alarm type, and alarm time.

[0068] Therefore, this embodiment enables the target device to achieve embodied intelligence. Embodied intelligence enables the target device to perceive the environment, understand intentions, and control the collaborative work of multiple devices. By combining the target device with a preset large model and utilizing the reasoning and decision-making capabilities of the preset large model, the target device has autonomous learning capabilities, which is conducive to completing more complex tasks, such as traffic violation inspections and oil field detection. Furthermore, it can provide real-time and accurate anomaly alarms during task execution, which can greatly improve work efficiency and safety.

[0069] In some embodiments, the task reporting information includes a task completion identifier for the target task. The user terminal responds to the task completion identifier to obtain alarm information and target task information for the target task, and generates a task report based on the alarm information and target task information.

[0070] In this embodiment, the task completion identifier is used to indicate that the target device has completed the target task. By generating a task report containing alarm information when the target device completes the target task, business personnel can promptly understand the cause of the anomaly based on the task report and take corresponding measures accordingly.

[0071] In some embodiments, the task reporting information also includes the device attitude, device position, and target position of the monitored target. The user terminal generates target trajectory information based on the device attitude, device position, and target position, and sends the target trajectory information to the cloud server. The cloud server converts the target trajectory information into trajectory movement instructions adapted to the communication protocol of the target device, and sends the protocol-converted trajectory movement instructions to the edge gateway. The edge gateway forwards the trajectory movement instructions to the target device, and the target device moves to the designated position according to the trajectory movement instructions.

[0072] In this embodiment, device attitude is used to represent the orientation or direction of the target device in space, such as the rotation state of the target device relative to a reference coordinate system. Device position is used to represent the current position of the target device in space. Target position is used to represent the current position of the monitored target in space. Target trajectory information is information used to control the target device to move according to the target trajectory. The preset large model can also continuously perform motion control on the target device based on the attitude and position of the target device and the position of the monitored target, thereby giving the target device the ability to continuously monitor the monitored target.

[0073] Therefore, on the one hand, this embodiment can ensure that the target device always monitors the target in real time with an optimal attitude and position. On the other hand, the target trajectory information is a downlink message, and the access layer can provide a unified attribute and service downlink interface to parse it into the corresponding structure according to the target object model. This can ensure the target device's response speed to the downlink message, thereby improving the real-time performance of monitoring.

[0074] To illustrate in detail the task reporting information processing flow provided by the embodiments of the present invention, the following is combined with... Figure 9 The process is explained below: S901, the target device uploads task reporting information to the edge gateway based on the communication protocol adapted to the target device, and pushes the real-time video stream collected by shooting and monitoring the target to the real-time video server.

[0075] S902, the edge gateway forwards the task reporting information to the access layer.

[0076] S903. The access layer converts the task reporting information into task reporting information of a specified protocol according to the target object model, and sends the converted task reporting information to the task management module. The task reporting information includes the live stream address of the target device and the task end identifier of the target task.

[0077] S904 The task management module parses the live stream address from the task reporting information, pulls the real-time video stream from the real-time video server according to the live stream address, and triggers the alarm management module to perform alarm management.

[0078] S905, the alarm management module initiates video analysis to the video analysis server.

[0079] S906, the video analysis server calls the model processing module to perform video analysis.

[0080] S907 The model processing module retrieves the real-time video stream from the real-time video server based on the live stream address and returns the video analysis results to the video analysis server.

[0081] S908: When the video analysis server determines that there is an anomaly in the monitored target based on the video analysis results, it generates alarm information and sends the alarm information to the alarm management module.

[0082] S909. The target device shall report when it finishes performing the target task.

[0083] S910, The task management module responds to the task completion flag and sends the target task information to the reporting module.

[0084] S911 The reporting module obtains alarm information from the alarm management module and sends the target task information and alarm information to the agent server.

[0085] S912, the intelligent agent server generates a task report based on the target task information and alarm information.

[0086] In some embodiments, this invention provides an inspection method, wherein the execution entity of the inspection method is a cloud server as described above. Please refer to [link to relevant documentation]. Figure 10 The inspection method includes: S1001. Obtain task distribution information sent by the user terminal based on a specified protocol, wherein the specified protocol is a unified protocol exposed to the outside world.

[0087] S1002. Convert the task distribution information into task execution instructions that are adapted to the communication protocol of the target device.

[0088] S1003. Send the protocol-converted task execution instruction to the edge gateway so that the edge gateway forwards the task execution instruction to the target device. The task execution instruction is used to instruct the target device to execute the target task.

[0089] This embodiment constructs an IoT architecture consisting of a device-edge gateway-cloud server-user terminal, and performs protocol conversion on downlink data on the cloud server side so that the user terminal can issue tasks based on a unified protocol exposed to the outside world. This is equivalent to seamless interaction between the user terminal and various devices. Therefore, this embodiment can solve the problem of heterogeneity among multiple devices, thereby improving system scalability. On the other hand, it enables the user terminal to quickly interact with different devices during task scheduling, thereby improving the real-time performance of interaction, reducing inspection and collaboration delays, and achieving more real-time task scheduling.

[0090] It is understandable that the cloud server can also perform other inspection methods, which can be referred to in the above embodiments and will not be repeated here.

[0091] In some embodiments, this invention provides an inspection method, wherein the subject executing the inspection method is the user terminal as described above. Please refer to [link to relevant documentation]. Figure 11 The inspection method includes: S1101, Respond to the task creation command to generate task distribution information.

[0092] S1102. Based on the specified protocol, the task distribution information is sent to the cloud server, so that the cloud server converts the task distribution information into a task execution instruction adapted to the communication protocol of the target device, and forwards the protocol-converted task execution instruction to the target device through the edge gateway. The task execution instruction is used to instruct the target device to execute the target task. The specified protocol is a unified protocol exposed to the outside.

[0093] This embodiment constructs an IoT architecture consisting of a device-edge gateway-cloud server-user terminal, and performs protocol conversion on downlink data on the cloud server side so that the user terminal can issue tasks based on a unified protocol exposed to the outside world. This is equivalent to seamless interaction between the user terminal and various devices. Therefore, this embodiment can solve the problem of heterogeneity among multiple devices, thereby improving system scalability. On the other hand, it enables the user terminal to quickly interact with different devices during task scheduling, thereby improving the real-time performance of interaction, reducing inspection and collaboration delays, and achieving more real-time task scheduling.

[0094] It is understood that the user terminal can also perform other inspection methods, which can be referred to the above embodiments and will not be repeated here.

[0095] It should be noted that in the above embodiments, there is no necessarily a certain order between the steps. Those skilled in the art can understand from the description of the embodiments of the present invention that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0096] As another aspect of this invention, an inspection device is provided. This inspection device can be a software module, which includes several instructions stored in a memory. A processor can access the memory, call the instructions, and execute them to complete the inspection methods described in the various embodiments above.

[0097] In some embodiments, the inspection device can be constructed from hardware components. For example, the inspection device can be constructed from one or more chips, which can work in coordination to complete the inspection methods described in the various embodiments above. As another example, the inspection device can also be constructed from components such as general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, ARM (Acorn RISC Machines), programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components.

[0098] In some embodiments, please refer to Figure 12 An inspection device 1200 provided in this embodiment of the invention includes an acquisition module 1201, a conversion module 1202 and a first sending module 1203.

[0099] The acquisition module 1201 is used to acquire task distribution information sent by the user terminal based on a specified protocol. The specified protocol is a unified protocol exposed to the outside world. The conversion module 1202 is used to convert the task distribution information into a task execution instruction that adapts to the communication protocol of the target device. The first sending module 1203 is used to send the protocol-converted task execution instruction to the edge gateway so that the edge gateway forwards the task execution instruction to the target device. The task execution instruction is used to instruct the target device to execute the target task.

[0100] In some embodiments, please refer to Figure 13 An inspection device 1200 provided in this embodiment of the invention includes a generation module 1204 and a second sending module 1205.

[0101] The generation module 1204 is used to generate task distribution information in response to the task creation instruction. The second sending module 1205 is used to send the task distribution information to the cloud server based on the specified protocol, so that the cloud server can convert the task distribution information into a task execution instruction that adapts to the communication protocol of the target device, and forward the protocol-converted task execution instruction to the target device through the edge gateway. The task execution instruction is used to instruct the target device to execute the target task. The specified protocol is a unified protocol exposed to the outside.

[0102] It should be noted that the above-described inspection device can execute the inspection method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in the embodiments of the inspection device can be found in the inspection method provided in the embodiments of the present invention.

[0103] Please see Figure 14 , Figure 14 This is a schematic diagram of the hardware structure of a device provided in an embodiment of the present invention. The device can be a cloud server or a user terminal as described above, such as... Figure 14 As shown, device 1400 includes one or more processors 1401 and memory 1402. Figure 14 Take a processor 1401 as an example.

[0104] Processor 1401 is configured to support the computer device in performing the corresponding functions in the methods described in the above method embodiments. Processor 1401 may be a Central Processing Unit (CPU), a Network Processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip may be an Application Specific Integrated Circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The aforementioned PLD may be a Complex Programmable Logic Device (CPLD), a Field-Programmable Gate Array (FPGA), a Generic Array Logic (GAL), or any combination thereof.

[0105] Memory 1402 is used to store program code. Memory 1402 may include volatile memory (VM), such as random access memory (RAM); memory may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 1402 may also include combinations of the above types of memory.

[0106] The memory 1402 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the inspection method in the embodiments of the present invention. The processor 1401 executes various functional applications and data processing of the inspection method and inspection device by running the non-volatile software programs, instructions, and modules stored in the memory 1402, that is, it realizes the functions of each module or unit of the inspection method and inspection device provided in the above method embodiments.

[0107] The memory 1402 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function. The data storage area may store data created based on the use of the inspection device. In some embodiments, the memory 1402 may optionally include memory remotely configured relative to the processor, which can be connected to the inspection device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0108] The one or more modules are stored in the memory 1402. When executed by the one or more processors 1401, they perform the inspection method in any of the above method embodiments. For example, they perform the method steps described in the above method embodiments to realize the functions of the modules described in the above device embodiments.

[0109] This invention also provides a storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the inspection method as described in the foregoing embodiments.

[0110] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0111] Finally, it should be noted that the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. These embodiments are not intended to impose additional limitations on the content of the present invention; their purpose is to provide a more thorough and comprehensive understanding of the disclosure of the present invention. Furthermore, within the framework of the present invention, the above-described technical features can be combined with each other, and many other variations of different aspects of the present invention as described above exist, all of which are considered to be within the scope of the present invention specification. Moreover, those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A collaborative system, characterized in that, It includes a cloud server, a user terminal, an edge gateway, and a target device. The cloud server is communicatively connected to both the user terminal and the edge gateway, and the edge gateway is also communicatively connected to the target device. The user terminal responds to the task creation instruction to generate task distribution information, and sends the task distribution information to the cloud server based on a specified protocol, wherein the specified protocol is a unified protocol exposed to the outside world; The cloud server converts the task distribution information into a task execution instruction adapted to the communication protocol of the target device, and sends the protocol-converted task execution instruction to the edge gateway; The edge gateway forwards the task execution command to the target device; The target device executes the target task according to the task execution instruction.

2. The collaborative system according to claim 1, characterized in that, The task delivery information includes target task information and the identity information of the target device. The cloud server determines the target object model based on the identity information and converts the target task information into task execution instructions adapted to the communication protocol of the target device based on the target object model.

3. The collaborative system according to claim 2, characterized in that, The target device uploads task reporting information to the edge gateway based on a communication protocol adapted to the target device. The edge gateway forwards the task reporting information to the cloud server; The cloud server converts the task reporting information into task reporting information with a specified protocol based on the target object model, and sends the converted task reporting information to the user terminal. The user terminal monitors the task execution process of the target device based on the task reporting information.

4. The collaborative system according to claim 3, characterized in that, The task reporting information includes the live stream address of the target device; The target device will push the real-time video stream collected by capturing and monitoring the target to the real-time video server; The user terminal parses the live stream address from the task reporting information, retrieves the real-time video stream from the real-time video server based on the live stream address, and displays the real-time video stream on the user terminal.

5. The collaborative system according to claim 4, characterized in that, The user terminal sends a streaming request to the cloud server, and the streaming request includes the live stream address of the real-time video stream; When the cloud server receives the streaming request, it retrieves the real-time video stream from the real-time video server according to the live stream address, inputs the real-time video stream into a preset large model, obtains video analysis results, and sends the video analysis results to the user terminal. The user terminal determines whether there is an anomaly in the monitored target based on the video analysis results. If an anomaly is found, an alarm message is generated.

6. The collaborative system according to claim 5, characterized in that, The task reporting information includes the task completion identifier of the target task; The user terminal responds to the task end identifier to obtain the alarm information and target task information of the target task, and generates a task report based on the alarm information and target task information.

7. The collaborative system according to claim 3, characterized in that, The task reporting information includes the real-time status of the target device, task execution information, and / or fault information. The user terminal monitors the target device based on the real-time status, the task execution information, and / or the fault information.

8. The collaborative system according to claim 4, characterized in that, The task reporting information also includes the device attitude, device location, and target location of the monitored target; The user terminal generates target trajectory information based on the device attitude, the device position, and the target position, and sends the target trajectory information to the cloud server; The cloud server converts the target trajectory information into a trajectory movement command adapted to the communication protocol of the target device, and sends the protocol-converted trajectory movement command to the edge gateway; The edge gateway forwards the trajectory movement command to the target device; The target device moves to the designated position according to the trajectory movement command.

9. An inspection method, characterized in that, Applied to a cloud server as described in any one of claims 1 to 8, the inspection method includes: Obtain task distribution information sent by the user terminal based on a specified protocol, wherein the specified protocol is a unified protocol exposed to the outside world; The task distribution information is converted into task execution instructions that are adapted to the communication protocol of the target device; The protocol-converted task execution instruction is sent to the edge gateway, so that the edge gateway forwards the task execution instruction to the target device, and the task execution instruction is used to instruct the target device to execute the target task.

10. An inspection method, characterized in that, Applied to a user terminal as described in any one of claims 1 to 8, the inspection method includes: Respond to task creation instructions to generate task distribution information; The task distribution information is sent to the cloud server based on a specified protocol, so that the cloud server converts the task distribution information into a task execution instruction adapted to the communication protocol of the target device, and forwards the protocol-converted task execution instruction to the target device via the edge gateway. The task execution instruction is used to instruct the target device to execute the target task. The specified protocol is a unified protocol exposed to the outside world.

11. A cloud server, characterized in that, The system includes a memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, and the processor, when executing the one or more computer programs, causing the cloud server to implement the inspection method as described in claim 9.

12. A user terminal, characterized in that, The device includes a memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, and the processor, when executing the one or more computer programs, causing the user terminal to implement the inspection method as described in claim 10.

13. A storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, cause the processor to perform the inspection method as described in claim 9 or 10.

Images