Inspection end cloud cooperation system with inspection task guidance and task execution method

Abstract

The invention discloses an inspection end cloud cooperation system with inspection task guidance and a task execution method. The system comprises end side equipment and cloud equipment, the end side equipment is used for acquiring video data of a target inspection scene, and determining a first layered change detection result and a second layered change detection result according to layered change detection performed on the video data by the inspection task; performing key frame judgment based on the first hierarchical change detection result and / or the second hierarchical change detection result, generating structured key visual data corresponding to the inspection task, and then transmitting the structured key visual data to cloud equipment; and the cloud equipment performs inspection task analysis according to the structured key visual data, generates structured inspection judgment data and / or inspection adjustment guiding information, and feeds back the structured inspection judgment data and / or the inspection adjustment guiding information to the end side equipment. Therefore, high-precision and low-redundancy key frame extraction is realized, the bandwidth consumption of transmission data is reduced, and the inspection efficiency of the system is continuously and autonomously improved.

Description

Inspection task-oriented cloud-based collaborative system and task execution methods Technical Field

[0001] This application relates to the field of intelligent monitoring technology for industrial safety and the environment, specifically to a cloud-based collaborative system for inspection tasks. This application also relates to a method and apparatus for executing inspection tasks with an inspection task orientation. Furthermore, this application relates to a method for generating key data in inspection tasks, a method for judging inspection tasks, as well as computer storage media and electronic equipment. Background Technology

[0002] With the widespread adoption of video surveillance systems in urban management, industrial production, and community security, the number of cameras deployed in various scenarios has increased dramatically, generating massive amounts of video data. Faced with such a massive data volume, relying on manual real-time monitoring is not only inefficient but also highly susceptible to missed or misjudged critical events due to visual fatigue or distraction, failing to meet the timeliness and accuracy requirements of modern security management. Against this backdrop, utilizing artificial intelligence technology to automatically analyze surveillance videos and achieve intelligent identification and early warning of risk events (such as falls, fires and smoke, unauthorized intrusions, and abnormal equipment operation) has become an important development direction for improving inspection efficiency and ensuring public safety. This technological approach not only reduces the burden on manual labor but also significantly improves the response speed and handling efficiency of abnormal events, providing strong support for building an intelligent and proactive security and prevention system. Summary of the Invention

[0003] This application provides an inspection terminal-cloud collaborative system with inspection task orientation to solve the problems of wasted computing resources and bandwidth consumption in the prior art.

[0004] This application provides a cloud-edge collaborative inspection system with inspection task orientation, comprising: an edge device and a cloud device; the edge device is used to acquire video data of a target inspection scene, and determine a first layer change detection result and a second layer change detection result based on the layer change detection performed on the video data according to the inspection task; based on the first layer change detection result and / or the second layer change detection result, keyframe judgment is performed to generate structured key visual data corresponding to the inspection task, and then the structured key visual data is transmitted to the cloud device; the cloud device analyzes the inspection task according to the structured key visual data, generates structured inspection judgment data and / or inspection adjustment guidance information, and feeds it back to the edge device.

[0005] In some embodiments, the step of performing layered change detection on the inspection video data according to the inspection task requirements, and determining the first layered change detection result and the second layered change detection result, includes: filtering video frames and / or image sequences in the video data according to pre-configured invalid regions or pre-configured target regions to determine valid regions in the video frames and / or image sequences; performing the layered change detection on the valid regions to determine the first layered change detection result and the second layered change detection result in the video frames and / or image sequences.

[0006] In some embodiments, performing the layered change detection on the effective region to determine the first layered change detection result and the second layered change detection result in the video frame and / or image sequence includes: performing dynamic change detection on the effective region to obtain static attribute information and dynamic attribute information of the dynamically changing region in the video frame and / or image sequence; determining the static attribute information and dynamic attribute information of the dynamically changing region as the first layered change detection result; performing target inspection object change detection on the effective region to obtain static attribute information and dynamic attribute information of the target inspection object within a preset time range in the video frame and / or image sequence; and determining the static attribute information and dynamic attribute information of the target inspection object as the second layered change detection result.

[0007] In some embodiments, the step of detecting changes in the target inspection object within the effective area and obtaining the static and dynamic attribute information of the target inspection object within a preset time range in the video frame and / or image sequence includes: detecting the effective area within the preset time range and identifying the target inspection object; and creating a temporal state machine of the target inspection object based on the tracking of the target inspection object to record the static and dynamic attribute information of the target inspection object.

[0008] In some embodiments, the step of determining keyframes based on the first layer change detection result and / or the second layer change detection result to generate key visual data corresponding to the inspection task requirements includes: combining the event determination requirements corresponding to the inspection task with the first layer change detection result, and / or combining the event determination requirements corresponding to the inspection task with the second layer change detection result to determine video key regions corresponding to the inspection task from the video data; and generating structured key visual data corresponding to the inspection task based on the video key regions, static attribute information and / or dynamic attribute information in the first layer change detection result, and static attribute information and / or dynamic attribute information in the second layer change detection result.

[0009] In some embodiments, the step of performing inspection task analysis on the structured key visual data to generate structured inspection judgment data and / or inspection adjustment guidance information includes: obtaining inspection task configuration information according to the inspection task; and performing inspection task analysis using the inspection task configuration information, the structured key visual data, and the prompt words of the inspection task as input data for a large model to generate structured inspection judgment data and / or inspection adjustment guidance information.

[0010] In some embodiments, the step of performing inspection task analysis on the structured key visual data to generate structured inspection judgment data and / or inspection adjustment guidance information includes: generating structured early warning information when the analysis result of the inspection task analysis on the structured key visual data indicates a risk, or generating structured early warning information and guidance information for adjusting inspection requirements; and generating guidance information for adjusting inspection execution when the analysis result of the inspection task analysis on the resultified key visual data indicates a normal result.

[0011] In some embodiments, the cloud device performs inspection task analysis based on the structured key visual data, generates structured inspection judgment data and / or inspection adjustment guidance information, and feeds it back to the edge device, including: the cloud device preprocesses the structured key visual data to determine target structured visual data; performs inspection task analysis based on the target structured visual data, generates structured inspection judgment data and / or inspection adjustment guidance information, and feeds it back to the edge device.

[0012] In some embodiments, the cloud device preprocesses the structured key visual data to determine target structured visual data, including: acquiring key regions related to the current inspection task; determining changing target regions related to the current inspection task from the structured key visual data based on the key regions; generating target markers in the structured key visual data based on the changing target regions to obtain preprocessed target structured visual data; or, acquiring key regions related to the current inspection task; determining changing target regions related to the current inspection task in the structured key visual data based on the key regions; cropping images or videos in the structured key visual data based on the changing target regions to obtain local structured visual data; and determining the local structured visual data as the preprocessed target structured visual data.

[0013] This application also provides a patrol task execution method with patrol task orientation, comprising: acquiring video data of a target patrol scene; determining a first layer change detection result and a second layer change detection result based on layer change detection performed on the video data according to the patrol task; performing keyframe judgment based on the first layer change detection result and / or the second layer change detection result to generate structured key visual data corresponding to the patrol task; and performing patrol task analysis on the structured key visual data to generate structured patrol judgment data and / or patrol adjustment guidance information.

[0014] This application also provides an inspection task execution device with inspection task orientation, comprising: an acquisition unit for acquiring video data of a target inspection scene; a determination unit for determining a first layer change detection result and a second layer change detection result based on layer change detection performed on the video data according to the inspection task; a first generation unit for performing keyframe judgment based on the first layer change detection result and / or the second layer change detection result to generate structured key visual data corresponding to the inspection task; and a second generation unit for performing inspection task analysis on the structured key visual data to generate structured inspection judgment data and / or inspection adjustment guidance information.

[0015] This application also provides a method for generating key data in an inspection task, comprising: acquiring video data of a target inspection scene; determining a first layer change detection result and a second layer change detection result based on layer change detection performed on the video data by the inspection task; and performing key frame judgment based on the first layer change detection result and / or the second layer change detection result to generate structured key visual data corresponding to the inspection task.

[0016] This application also provides a method for determining inspection tasks, comprising: receiving structured key visual data generated in the method for generating key data in the inspection task as described above; performing inspection task analysis on the structured key visual data to generate structured inspection judgment data and / or guidance information for inspection adjustments.

[0017] This application also provides a computer storage medium, comprising: a computer program, which, when the computer program is executed, performs the steps as described above in the inspection task-oriented cloud collaborative system, or performs the steps as described above in the inspection task execution method, or performs the steps as described above in the method for generating key data in the inspection task, or performs the steps as described above in the inspection task discrimination method.

[0018] This application also provides an electronic device, including: a processor; and a memory for storing a program for processing data generated by the electronic device. When the program is read and executed by the processor, it performs the steps as described in the above-described inspection task-oriented cloud collaborative system, or performs the steps as described in the above-described inspection task-oriented inspection task execution method, or performs the steps as described in the above-described method for generating key data in an inspection task, or performs the steps as described in the above-described inspection task discrimination method.

[0019] Compared with existing technologies, this application has the following advantages: By performing layered change detection on video data through edge devices, redundant information is filtered to obtain key information relevant to the inspection task. This key information is then transmitted to cloud devices for inspection task analysis. The cloud devices, based on their deployed computing resources, provide corresponding feedback information based on the inspection task analysis results and transmit it back to the edge devices, which then make corresponding adjustments based on the feedback. This allows the cloud devices to reverse-optimize the preprocessing parameters, change detection parameters, and key frame screening requirements of the edge devices based on the effectiveness of risk identification, constructing a collaborative evolutionary closed loop of "perception → judgment → feedback → optimization," and achieving continuous autonomous improvement in the system's inspection efficiency. Attached Figure Description

[0020] Figure 1 is a schematic diagram of the framework of a cloud-based collaborative system for inspection tasks provided in this application.

[0021] Figure 2 is a flowchart of an embodiment of the generation of structured key video data by end-side devices in an inspection task-oriented end-cloud collaborative system provided in this application.

[0022] Figure 3 is a flowchart of an embodiment of inspection task analysis of cloud devices in an inspection task-oriented cloud collaborative system provided in this application.

[0023] Figure 4 is a flowchart of an inspection task execution method with inspection task orientation provided in this application.

[0024] Figure 5 is a structural schematic diagram of an inspection task execution device with inspection task orientation provided in this application.

[0025] Figure 6 is a flowchart of a method for generating key data in an inspection task provided in this application.

[0026] Figure 7 is a schematic diagram of the structure of a device for generating key data in an inspection task provided in this application.

[0027] Figure 8 is a flowchart of an inspection task discrimination method provided in this application.

[0028] Figure 9 is a structural schematic diagram of an inspection task discrimination device provided in this application.

[0029] Figure 10 is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0030] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0031] It should be noted that the terms "first," "second," "third," etc., in the claims, specification, and drawings of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. Such data are interchangeable where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown or described in this application. Furthermore, the terms "comprising," "having," and their variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0032] It should be understood that in the embodiments of this application, "at least one" means one or more, and "more than one" means two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the related objects before and after it are in an "or" relationship. "Contains A, B and / or C" means containing any one, two, or three of A, B, and C.

[0033] It should be understood that in the embodiments of this application, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0034] Based on the aforementioned background technology, it is known that the process of using artificial intelligence technology for intelligent inspection of industrial scenarios or environments involves automatically analyzing surveillance videos to achieve early warning of risk events. Existing video intelligent inspection analysis technologies can be categorized into two types: one is cloud-based analysis, where all or part of the video stream captured by the front-end camera is encoded and continuously transmitted to a cloud server. Cloud servers possess powerful computing capabilities, enabling the deployment of complex and high-precision artificial intelligence models for analysis. However, this method has drawbacks: extremely high network bandwidth requirements and huge transmission costs; real-time video data uploads are delayed, affecting the timeliness of event response; and a large number of redundant video clips without any events occurring are also uploaded, resulting in a serious waste of cloud computing resources.

[0035] Another approach is edge-side analysis, which involves deploying AI models directly on edge devices such as cameras or edge computing boxes for analysis. This solution can achieve low-latency real-time response without consuming network bandwidth. However, its drawbacks are: edge devices have limited computing power and storage space, making it difficult to run large, high-precision, and complex models; model updates and algorithm iterations are inconvenient, requiring operation on a large number of front-end devices one by one; and for inspection tasks that require global information or complex logical reasoning, the capabilities of edge-side models are limited.

[0036] In addition, current risk assessment methods mostly rely on fixed rules or single-modal image / video models, lacking the ability to understand complex semantic scenarios in a general way; they also lack a flexible risk definition mechanism based on task orientation, making it difficult to adapt to diverse inspection task requirements.

[0037] In summary, this application proposes an edge-cloud collaborative system that can solve the problem of wasted computing resources, reduce bandwidth consumption, and has an inspection-oriented approach, so as to achieve low-bandwidth, high-precision, and highly generalized risk event inspection.

[0038] As shown in Figure 1, the collaborative system includes an edge device 101 and a cloud device 102. The edge device can be one or more of the following: a camera, an edge computing box, or an electronic device with camera functionality, capable of acquiring one or more monitoring video streams in real time. The cloud device can be a server or a server cluster, etc.

[0039] The edge device 101 can be used to collect video data of the target inspection scene, and determine the first layer change detection result and the second layer change detection result based on the layer change detection performed on the video data according to the inspection task; perform key frame judgment based on the first layer change detection result and / or the second layer change detection result, generate structured key visual data corresponding to the inspection task, and then transmit the structured key visual data to the cloud device.

[0040] The cloud device 102 analyzes the inspection task based on the structured key visual data, generates structured inspection judgment data and / or guidance information for inspection adjustment, and feeds it back to the end device.

[0041] In this embodiment, when the end-side device 101 acquires video data of the target inspection scene, it can dynamically adjust the frame rate and / or image resolution of the acquired video according to the inspection task requirements issued by the cloud device 102 and in combination with its own computing resource load, thereby ensuring effective monitoring while dynamically and in real time saving the computing power of the end-side device.

[0042] The edge device can be one or more of the following: a camera, an edge computing box, or an electronic device with camera functionality. The edge computing box, also known as an edge module, is the core hardware carrier for edge computing, integrating or connecting to a camera. It is a device deployed near the data source and possessing a certain level of computing power.

[0043] To reduce the computational load on edge devices and avoid wasting computing resources, layered change detection is performed on video data. This can be achieved by embedding a lightweight algorithm within the edge device. Layered change detection can be determined based on the inspection task. As shown in Figure 2, in this embodiment, layered change detection is performed using changes in dynamically changing regions and specific target inspection objects as examples. Changes can refer to the appearance of new target objects, the disappearance of old target objects, or changes in the position or posture of existing target objects. Layered change detection can detect information about dynamically changing regions and specific target objects in the video data, enabling subsequent keyframe determination based on the layered change detection results. This improves the accuracy of keyframe determination and reduces bandwidth consumption for subsequent transmission to the cloud.

[0044] To avoid wasting computational resources and further improve the accuracy of keyframe judgment, redundant data is removed from the video data. Layered change detection is performed on the video data to determine the first and second layered change detection results. This can include removing some invalid regions from the video data, thereby eliminating some redundant data, narrowing the scope of dynamic change region detection and target inspection object change detection, and reducing computational load. Specifically, at least two methods can be used.

[0045] Method 1: Filter video frames and / or image sequences in the video data according to pre-configured invalid regions to determine valid regions in the video frames and / or image sequences; perform the layered change detection on the valid regions to determine the first layered change detection result and the second layered change detection result in the video frames and / or image sequences.

[0046] Method 2 is as follows: Based on a pre-configured target region, the video frames and / or image sequences in the video data are filtered to determine the effective region in the video frames and / or image sequences; the layer change detection is performed on the effective region to determine the first layer change detection result and the second layer change detection result in the video frames and / or image sequences.

[0047] In this embodiment, the invalid region or the target region can be segmented using background modeling (such as Gaussian Mixture Model (GMM)) or a lightweight semantic segmentation network, based on predefined invalid or target regions. The invalid region can be, for example, the sky, roof, walls, or other areas containing elements unrelated to the inspection task. In this embodiment, some meaningless content can be considered invalid content, which may be video clips that do not include inspection service requirements, do not trigger analysis needs, or are repetitive and redundant. The target region can be the area containing elements related to the inspection task. The invalid regions of video frames and / or image sequences are filtered according to the pre-configured invalid or target regions to determine or extract valid regions. After filtering invalid regions, layered change detection is performed within the valid regions. The specific implementation process may include: performing dynamic change detection on the valid regions to obtain static and dynamic attribute information of dynamically changing regions in the video frames and / or image sequences; wherein the static attribute information of the dynamically changing regions includes: the location and size of the dynamically changing regions. The dynamic attribute information of dynamically changing regions can include: the type of dynamic change, such as appearance, disappearance, movement, etc. Dynamic change detection of the effective region can be achieved through methods such as frame difference, background subtraction (e.g., Gaussian mixture model (GMM)), optical flow, or change detection models. Specifically, the process involves inputting video frames and / or image sequences → calculating changes → post-processing → outputting the changed region. Post-processing can include the original change mask → denoising filtering → hole filling → morphological operations → region filtering → boundary optimization → spatiotemporal consistency → outputting the result. The above method for detecting dynamically changing regions is merely an example and is based on existing detection techniques, which will not be described in detail here. However, considering computational resource constraints, factors such as the video data capture time and environment can be used as selection criteria for the above detection method. Under different shooting conditions or inspection requirements, using different detection methods can save computational power while still meeting the detection requirements.

[0048] The static and dynamic attribute information of the dynamically changing region are determined as the first layer change detection result.

[0049] The effective area is subjected to target object change detection to obtain the static and dynamic attribute information of the target object within a preset time range in the video frame and / or image sequence. The target object change detection can be implemented using a lightweight target detection / tracking model deployed on the edge device. The static attribute information of the target object may include: the target object's ID, category, location, size, etc.; the dynamic attribute information of the target object may include: appearance, disappearance, movement, entering / leaving the designated area, long-term stillness, etc.

[0050] The static and dynamic attribute information of the target inspection object is determined as the second layer change detection result.

[0051] The step of detecting changes in the target inspection object within the effective area and acquiring the static and dynamic attribute information of the target inspection object within a preset time range in the video frames and / or image sequences can be implemented by: detecting the effective area within the preset time range and identifying the target inspection object; wherein, the preset time range can be a periodic time range, such as once per second, and can be set according to actual inspection needs. The target inspection object can be personnel, vehicles, etc.

[0052] Based on the tracking of the target inspection object, a temporal state machine for the target inspection object is created to record its static and dynamic attribute information. The dynamic attribute information can be determined by comparing the detection results of the current video frame and / or the current image sequence with historical dynamic attributes to determine whether the state of the target inspection object in the current video frame and / or the current image sequence has changed, thereby determining the dynamic attribute information.

[0053] The edge device needs to perform keyframe judgment based on the first layer change detection result and / or the second layer change detection result to generate structured key visual data corresponding to the inspection task. The purpose is to transmit the structured key visual data to the cloud device so that the cloud device can perform inspection task analysis. The specific implementation process includes: combining the event judgment requirements corresponding to the inspection task with the first layer change detection result, and / or combining the event judgment requirements corresponding to the inspection task with the second layer change detection result, to determine the video key region corresponding to the inspection task from the video data; and generating structured key visual data corresponding to the inspection task based on the video key region, the static attribute information and / or dynamic attribute information in the first layer change detection result, and the static attribute information and / or dynamic attribute information in the second layer change detection result.

[0054] The event judgment requirements corresponding to the inspection task can include the following, although the following are just examples and are not limited to these: Event judgment requirement 1 (item left behind): The scene area undergoes significant dynamic changes, and the change type is "appearance"; Event judgment requirement 3 (item taken away): The scene area undergoes significant dynamic changes, and the change type is "disappearance"; Event judgment requirement 2 (new target intrusion): A new target in the "appearance" state appears in the target object state machine; Event judgment requirement 4 (abnormal stay): The "long-term static" state of a target object in the target object state machine exceeds a preset threshold.

[0055] For example, assuming the inspection task is fire lane obstruction, generating the structured key visual data corresponding to the inspection task requires extracting key frames and / or key image sequences with valid change areas from the video data, based on the event determination requirement 1 and the static and / or dynamic attribute information in the first layer change detection results, as well as structured data. Specifically, based on event determination requirement 1, the target dynamic change area with the dynamic attribute information "appeared" in the dynamic change detection results within the valid change area is extracted, and this target dynamic change area is used as the key frame and / or key image sequence. The static attribute information (e.g., dynamic change area location, size, camera ID, time, geographical location, etc.), dynamic attribute information (e.g., change type is "appeared and stationary"), and the bounding boxes, segmentation masks, and state change trajectories in the key frames and / or key image sequences from the dynamic change detection results are used as structured metadata. Structured key visual data is then generated based on the key frames and / or key image sequences and the structured metadata.

[0056] For example, assuming the inspection task involves an animal intrusion, generating the structured key visual data corresponding to the inspection requires extracting key frames and / or key image sequences containing effective change areas from the video data, based on the event determination requirement 2 and the static and / or dynamic attribute information in the second-layer change detection results, as well as structured data. Specifically, based on event determination requirement 2, the target change area where the dynamic attribute information of the target inspection object is "present" in the target inspection object change detection results within the effective change area is extracted, and this target change area is used as the key frame and / or key image sequence. The static attribute information (e.g., location, size, category, camera ID, time, geographical location, etc.), dynamic attribute information (e.g., change type: target intrusion), bounding boxes, segmentation masks, and state change trajectories in the key frames and / or key image sequences from the target inspection object change detection results are used as structured metadata. Structured key visual data is then generated based on the key frames and / or key image sequences and the structured metadata.

[0057] It is understood that the structured metadata may include: event information, change descriptions, target information, etc. The event information may include: timestamps, camera IDs, geographic locations, etc. The change descriptions may include: change types, such as: appearance (intrusion), disappearance, stay, movement, entering / leaving a designated area, long-term stillness, etc. Target information may include: if a list of inspected targets is involved, each inspected target includes the inspected target ID, type, bounding box in the keyframe, segmentation mask, and state change trajectory, etc.

[0058] Using the previous example of fire lane obstruction, assume that continuous surveillance video is obtained: from frame 1 to frame 20: based on the filtered invalid area or based on the configured target area, dynamic change area detection and target inspection object change detection are performed on the determined valid area. The dynamic change detection result of the valid area is no change, and the target inspection object change detection result is none.

[0059] Frames 21-30: The handcart enters the fire escape. Similarly, invalid areas are filtered out first, then dynamic change area detection and target inspection object change detection are performed on the valid fire escape area. Significant dynamic change was detected, classified as "occurring," and the area is located in the fire escape. The target inspection object change detection result is "none." Event judgment requirement 1 is triggered and met. Frames 21-30 are judged as "item left behind."

[0060] Therefore, the effective region of frames 21-30 is extracted as keyframes. It's understandable that if the trolley remains stationary after entering the fire lane within frames 21-30, then the video frames or image sequences after reaching that stationary state do not need to be used as keyframes. Therefore, frames 21-30 here are merely an example; within this range, some video frames or image sequences with insignificant state changes can be removed, further reducing computational load. Thus, keyframes can originate from effective regions of video frames or image sequences and can be determined based on the changing state.

[0061] In frame 31: the trolley is pushed away. Similarly, invalid areas are filtered first, then dynamic change area detection and target inspection object change detection are performed on the valid fire lane. Significant dynamic change was detected, typed as disappearance, and the area is located in the fire lane. Target inspection object change detection result is zero. Event judgment requirement 1 will not be triggered.

[0062] Using the previous example of an animal intrusion, let's assume we obtain continuous surveillance video from frame 1 to frame 20: First, filter out invalid areas, then perform dynamic change detection and target inspection object change detection on the valid areas. The result of dynamic change detection on the valid areas is no change, and the result of target inspection object change detection is no change.

[0063] Frames 21-30: An animal enters the target area. Similarly, invalid areas are filtered out first, then dynamic change detection and target object change detection are performed on the valid areas. No dynamic change detection is detected, and the target object change detection result is "present." Event judgment requirement 2 is triggered and met. Frames 21-30 are judged as "target intrusion."

[0064] Therefore, the effective region of frames 21-30 is extracted as keyframes.

[0065] The above examples are merely illustrative of the implementation process and are not intended to limit the principles of the technical solution. Furthermore, the examples demonstrate filtering invalid regions for each frame before performing layered change detection on the valid regions. Alternatively, invalid regions can be filtered for all video data beforehand, followed by valid region detection for each subsequent frame. For example, invalid regions can be filtered for video frames and / or image sequences in a surveillance video, and then dynamic change detection and target object change detection can be performed on the valid regions after excluding invalid regions. Alternatively, invalid regions can be filtered for each frame before valid region detection; the specific method is not limited.

[0066] The above examples illustrate fire lane obstruction and animal intrusion. It is understood that monitoring video can include various types of information, and this information can be used to extract various key details. For example, the video may contain structured key visual data corresponding to fire lane obstruction, animal intrusion, or the kitchen environment (such as the cleanliness of trash cans and the hygiene of food). Key information can be extracted from the effective area of ​​the video data according to the above event determination requirements, combined with dynamic change detection results and / or target inspection object change detection results, and structured key visual data can be generated.

[0067] In this embodiment, the keyframes and / or key image sequences included in the structured key visual data can be single or multiple images, such as before, during, and after a change; or they can be a short video edited based on extracted video information, such as content corresponding to a period of time before and after the inspection task (e.g., 5 seconds before and after the task). The keyframes and / or key image sequences can be frame images and / or image sequences determined based on the effective region, so that subsequent inspection task analysis only needs to target the effective region. Of course, the possibility of extracting complete frame images and / or image sequences from the effective region is not excluded. In this embodiment, the key information is mainly extracted from the effective region, that is, the extracted key information is the effective region portion.

[0068] The structured key visual data is transmitted to cloud devices for inspection task analysis. Because the structured key visual data undergoes invalid area filtering and effective area layer-by-layer detection, the generated structured key visual data avoids redundant information, thus reducing transmission bandwidth consumption. It also provides high-precision, low-redundancy data for subsequent inspection task analysis, saving computing resources.

[0069] As shown in Figure 3, in this embodiment, the inspection task analysis can be implemented using a large model deployed on a cloud device, such as a multimodal large model. The specific implementation process may include: obtaining inspection task configuration information based on the inspection task; the inspection task configuration information may include configuration information for key areas generated manually or automatically, such as task focus areas, shielded areas, etc.

[0070] The inspection task configuration information, the structured key visual data, and the prompts for the inspection task are used as input data for the large model to perform inspection task analysis, generating structured inspection judgment data and / or guidance information for inspection adjustments.

[0071] To enable the large model to have a more accurate understanding ability and accurately locate the inspection area, visual cues can be provided through preprocessing. Therefore, this embodiment may further include the following steps before the large model performs risk assessment: the cloud device preprocesses the structured key visual data to determine the target structured visual data; the inspection task is analyzed based on the target structured visual data to generate structured inspection judgment data and / or guidance information for inspection adjustments, and this data is fed back to the edge device. The preprocessing may include at least two of the following methods.

[0072] Method 1: Obtain key areas related to the current inspection task. This can be achieved by real-time detection of video data of the target inspection scene through a cloud detection port. For example, if the inspection object of the current task is a trash can, then the key area should include the trash can. Since the trash can may move, real-time detection is required to obtain the key area where the trash can is located. Based on the key area, determine the changing target area related to the current inspection task in the structured key visual data. Generate target markers in the structured key visual data based on the changing target areas to obtain preprocessed target structured visual data. The target markers can be in the form of semi-transparent masks, highlighted borders, or numerical markers. In this embodiment, the target marker range of the changing target area can be determined according to the inspection scene and / or the inspection object. For example, for an inspection object that may move, the target marker range can include the area outside the location of the inspection object; while for a fixed inspection object, the target marker can be the location area. That is, the range of the target markers can be flexibly expanded and contracted according to the specific inspection scene and / or inspection object. It can also be adjusted according to specific inspection needs.

[0073] Method 2: Obtain key areas related to the current inspection task; this can be achieved by real-time detection of video data of the target inspection scene through a cloud detection port. For example, if the inspection task is a trash can, the key area should include the trash can itself. Since the trash can may move, real-time detection is required to obtain the key area where the trash can is located. Based on the key area, determine the changing target area in the structured key visual data that is related to the current inspection task. Based on the changing target area, crop the images or videos in the structured key visual data to obtain local structured visual data. Determine the local structured visual data as the preprocessed target structured visual data.

[0074] The large model performs risk assessment on the target structured visual data, including: when the analysis result of the inspection task analysis of the target structured visual data indicates that there is a risk, generating structured early warning information, or generating structured early warning information and guidance information for adjusting inspection requirements; when the analysis result of the inspection task analysis of the target structured visual data indicates that it is normal, generating guidance information for adjusting inspection execution.

[0075] When the analysis results of the aforementioned large model indicate the existence of risk, the generated structured early warning information can be fed back. This structured early warning information may include: timestamp, location, risk type, confidence level, risk description, relevant images or video clips, etc., and can be transmitted and fed back via alarm interfaces such as SMS, email, platform messages, etc. Alternatively, based on the risk situation, guidance information corresponding to adjustments in inspection requirements can be generated, such as: risk level and corresponding risk handling methods, angle of the data acquisition equipment, acquisition time, trigger condition configuration, and guidance information to avoid omissions. That is, only the structured early warning information can be generated, or both structured early warning information and guidance information for adjusting inspection requirements can be generated.

[0076] When the large model analysis result is normal, guidance information corresponding to the adjustment of the inspection execution is generated, such as the data required for the next inspection task, thereby adjusting the corresponding acquisition equipment angle, clarity and other related acquisition requirements.

[0077] Regardless of whether the analysis result is risky or normal, cloud-based devices can generate relevant guiding information based on the specific inspection task results, including optimized edge-side inspection strategies, changes in detection parameters, and key information screening and judgment requirements. This information is then transmitted to the corresponding edge-side devices, thereby promoting the continuous progress of the overall inspection system. During the inspection task analysis process by cloud-based devices, it can be performed on multiple channels of structured key visual data; therefore, the guiding information fed back to the edge-side devices can also correspond to different edge-side devices.

[0078] In the aforementioned edge-cloud collaborative system, edge devices are responsible for filtering and compressing key information with low latency and low computing power, while cloud devices analyze inspection tasks using large models, forming an inspection collaboration paradigm of front-end filtering and back-end precision judgment. Through layered change detection of video data by edge devices, redundant information is filtered to obtain key information relevant to the inspection task. This key information is then transmitted to cloud devices for inspection task analysis. Cloud devices, based on their deployed computing resources, obtain the inspection task analysis results and provide corresponding feedback to edge devices, which then make adjustments based on the feedback. This allows cloud devices to reverse-optimize the preprocessing parameters, change detection parameters, and keyframe filtering requirements of edge devices based on the effectiveness of risk identification, constructing a collaborative evolutionary closed loop of "perception → judgment → feedback → optimization," achieving continuous and autonomous improvement in system inspection efficiency. Furthermore, cloud devices can dynamically and flexibly define and execute various complex inspection tasks with business logic during inspection task analysis, exhibiting high flexibility and scalability, enabling the collaborative system to quickly adapt to various monitoring scenarios.

[0079] The above is a detailed description of an embodiment of an inspection task-oriented cloud-based collaborative system provided in this application. Corresponding to the aforementioned embodiment of an inspection task-oriented cloud-based collaborative system, this application also discloses an embodiment of an inspection task execution method with inspection task orientation.

[0080] As shown in Figure 4, this method can be applied to cloud devices and / or edge devices. The specific implementation process includes: Step S401: Acquire video data of the target inspection scene; Step S402: Perform layered change detection on the video data according to the inspection task, and determine the first layered change detection result and the second layered change detection result; Step S403: Perform keyframe judgment based on the first layered change detection result and / or the second layered change detection result, and generate structured key visual data corresponding to the inspection task; Step S404: Perform inspection task analysis on the structured key visual data, and generate structured inspection judgment data and / or inspection adjustment guidance information.

[0081] For details regarding steps S401 to S404 above, please refer to the description of the above embodiment of the inspection terminal cloud collaborative system with inspection task orientation, which will not be elaborated here.

[0082] The above is a detailed description of an embodiment of an inspection task execution method with inspection task orientation provided in this application. Corresponding to the aforementioned embodiment of an inspection task execution method with inspection task orientation, this application also discloses an embodiment of an inspection task execution device with inspection task orientation.

[0083] As shown in Figure 5, the device includes: an acquisition unit 501 for acquiring video data of a target inspection scene; a determination unit 502 for determining a first layer change detection result and a second layer change detection result based on the layer change detection performed on the video data according to the inspection task; a first generation unit 503 for performing keyframe judgment based on the first layer change detection result and / or the second layer change detection result to generate structured key visual data corresponding to the inspection task; and a second generation unit 504 for performing inspection task analysis on the structured key visual data to generate structured inspection judgment data and / or guidance information for inspection adjustments.

[0084] Accordingly, the specific implementation process of the device can also be referred to the above description of the embodiment of the inspection terminal cloud collaborative system with inspection task orientation, which will not be detailed here.

[0085] Based on the above, this application also provides a method for generating key data in an inspection task, as shown in Figure 6. This method is applied to an end-side device and specifically includes: Step S601: acquiring video data of the target inspection scene; Step S602: determining the first layer change detection result and the second layer change detection result based on the layer change detection performed on the video data by the inspection task; Step S603: performing key frame judgment based on the first layer change detection result and / or the second layer change detection result to generate structured key visual data corresponding to the inspection task.

[0086] For the specific implementation process of steps S601 to S602, please refer to the relevant description in the above embodiment of the inspection end-cloud collaborative system with inspection task orientation, that is, the relevant description of the end-side device, which will not be detailed here.

[0087] Accordingly, this application also provides a device for generating key data in an inspection task, as shown in FIG7. The device includes: an acquisition unit 701 for acquiring video data of a target inspection scene; a determination unit 702 for determining a first layer change detection result and a second layer change detection result based on the layer change detection performed on the video data by the inspection task; and a generation unit 703 for performing key frame judgment based on the first layer change detection result and / or the second layer change detection result to generate structured key visual data corresponding to the inspection task.

[0088] Based on the above, this application also provides a method for judging inspection tasks, as shown in Figure 8. This method is applied to a cloud device and specifically includes: Step S801: receiving structured key visual data generated in the method for generating key data in the inspection task as described above; Step S802: performing inspection task analysis on the structured key visual data to generate structured inspection judgment data and / or guidance information for inspection adjustment.

[0089] For details regarding steps S801 and 802, please refer to the relevant content in the above system; they will not be elaborated here.

[0090] Accordingly, this application also provides an inspection task discrimination device, as shown in FIG9. The device specifically includes: a receiving unit 901: receiving structured key visual data generated in the above-mentioned method for generating key data in inspection tasks; and a generating unit 902: performing inspection task analysis on the structured key visual data to generate structured inspection judgment data and / or guidance information for inspection adjustment.

[0091] Based on the above, this application also provides a computer storage medium, characterized in that it includes: a computer program, which, when the computer program is running, executes the relevant content of the inspection task-oriented cloud collaborative system described above, or executes the relevant content of the inspection task execution method described above, or executes the relevant content of the key data generation method in the inspection task described above, or executes the relevant content of the inspection task discrimination method described above.

[0092] Based on the above, this application also provides an electronic device, as shown in FIG10. The electronic device includes: a processor 1001; a memory 1002 for storing a program for processing data generated by the electronic device. When the program is read and executed by the processor, it executes the relevant content of the inspection task-oriented cloud collaborative system described above, or executes the relevant content of the inspection task execution method described above, or executes the relevant content of the key data generation method in the inspection task described above, or executes the relevant content of the inspection task discrimination method described above.

[0093] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0094] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0095] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0096] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0097] 2. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0098] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A cloud-based collaborative system for inspection tasks, characterized in that, include: End-side devices and cloud devices; the end-side devices are used to acquire video data of the target inspection scene, and perform layered change detection on the video data according to the inspection task to determine the first layered change detection result and the second layered change detection result; Based on the first layer change detection result and / or the second layer change detection result, key frame judgment is performed to generate structured key visual data corresponding to the inspection task, and then the structured key visual data is transmitted to the cloud device. The cloud device analyzes the inspection task based on the structured key visual data, generates structured inspection judgment data and / or guidance information for inspection adjustments, and feeds it back to the edge device.

2. The system according to claim 1, characterized in that, The step of performing layered change detection on the inspection video data according to the inspection task requirements, and determining the first layered change detection result and the second layered change detection result, includes: filtering video frames and / or image sequences in the video data according to pre-configured invalid regions or pre-configured target regions to determine valid regions in the video frames and / or image sequences; performing the layered change detection on the valid regions to determine the first layered change detection result and the second layered change detection result in the video frames and / or image sequences.

3. The system according to claim 2, characterized in that, The step of performing layered change detection on the effective region to determine the first layered change detection result and the second layered change detection result in the video frame and / or image sequence includes: performing dynamic change detection on the effective region to obtain static attribute information and dynamic attribute information of the dynamically changing region in the video frame and / or image sequence; determining the static attribute information and dynamic attribute information of the dynamically changing region as the first layered change detection result; performing target inspection object change detection on the effective region to obtain static attribute information and dynamic attribute information of the target inspection object within a preset time range in the video frame and / or image sequence; and determining the static attribute information and dynamic attribute information of the target inspection object as the second layered change detection result.

4. The system according to claim 2, characterized in that, The step of detecting changes in the target inspection object within the effective area and obtaining the static and dynamic attribute information of the target inspection object within a preset time range in the video frame and / or image sequence includes: detecting the effective area within the preset time range and identifying the target inspection object; and creating a temporal state machine of the target inspection object based on the tracking of the target inspection object to record the static and dynamic attribute information of the target inspection object.

5. The system according to claim 1, characterized in that, The step of determining keyframes based on the first layer change detection result and / or the second layer change detection result to generate key visual data corresponding to the inspection task requirements includes: combining the event determination requirements corresponding to the inspection task with the first layer change detection result, and / or combining the event determination requirements corresponding to the inspection task with the second layer change detection result to determine the video key region corresponding to the inspection task from the video data; and generating structured key visual data corresponding to the inspection task based on the video key region, the static attribute information and / or dynamic attribute information in the first layer change detection result, and the static attribute information and / or dynamic attribute information in the second layer change detection result.

6. The system according to claim 1, characterized in that, The step of performing inspection task analysis on the structured key visual data to generate structured inspection judgment data and / or inspection adjustment guidance information includes: obtaining inspection task configuration information according to the inspection task; using the inspection task configuration information, the structured key visual data, and the prompt words of the inspection task as input data for a large model to perform inspection task analysis and generate structured inspection judgment data and / or inspection adjustment guidance information.

7. The system according to claim 1, characterized in that, The step of performing inspection task analysis on the structured key visual data to generate structured inspection judgment data and / or inspection adjustment guidance information includes: generating structured early warning information when the analysis result of the inspection task analysis on the structured key visual data indicates that there is a risk, or generating structured early warning information and guidance information for adjusting inspection requirements; and generating guidance information for adjusting inspection execution when the analysis result of the inspection task analysis on the resultified key visual data indicates that the inspection execution is normal.

8. The system according to claim 1, characterized in that, The cloud device analyzes the inspection task based on the structured key visual data, generates structured inspection judgment data and / or inspection adjustment guidance information, and feeds it back to the edge device. This includes: the cloud device preprocesses the structured key visual data to determine the target structured visual data; analyzes the inspection task based on the target structured visual data, generates structured inspection judgment data and / or inspection adjustment guidance information, and feeds it back to the edge device.

9. The system according to claim 8, characterized in that, The cloud device preprocesses the structured key visual data to determine the target structured visual data, including: acquiring key regions related to the current inspection task; determining changing target regions related to the current inspection task from the structured key visual data based on the key regions; generating target markers in the structured key visual data based on the changing target regions to obtain preprocessed target structured visual data; or, acquiring key regions related to the current inspection task; determining changing target regions related to the current inspection task in the structured key visual data based on the key regions; cropping images or videos in the structured key visual data based on the changing target regions to obtain local structured visual data; and determining the local structured visual data as the preprocessed target structured visual data.

10. A method for executing inspection tasks with an inspection task orientation, characterized in that, include: Acquire video data of the target inspection scene; Based on the layered change detection performed on the video data according to the inspection task, the first layered change detection result and the second layered change detection result are determined; Based on the first layer change detection result and / or the second layer change detection result, key frame judgment is performed to generate structured key visual data corresponding to the inspection task; The structured key visual data is analyzed for inspection tasks to generate structured inspection judgment data and / or guidance information for inspection adjustments.

11. An inspection task execution device with inspection task orientation, characterized in that, include: The acquisition unit is used to acquire video data of the target inspection scene. The determining unit is used to determine the first layer change detection result and the second layer change detection result based on the layer change detection performed on the video data according to the inspection task; The first generation unit is used to determine key frames based on the first layer change detection result and / or the second layer change detection result, and generate structured key visual data corresponding to the inspection task. The second generation unit is used to perform inspection task analysis on the structured key visual data and generate structured inspection judgment data and / or guidance information for inspection adjustment.

12. A method for generating key data in an inspection task, characterized in that, include: Acquire video data of the target inspection scene; Based on the layered change detection performed on the video data according to the inspection task, the first layered change detection result and the second layered change detection result are determined; Based on the first layer change detection result and / or the second layer change detection result, key frame determination is performed to generate structured key visual data corresponding to the inspection task.

13. A method for identifying inspection tasks, characterized in that, include: Receive structured key visual data generated in the method for generating key data in the inspection task as described in claim 12; The structured key visual data is analyzed for inspection tasks to generate structured inspection judgment data and / or guidance information for inspection adjustments.

14. A computer storage medium, characterized in that, include: A computer program, when the computer program is executed, performs the steps in the system as described in any one of claims 1 to 9, or performs the method as described in claim 10, 12, or 13.

15. An electronic device, characterized in that, include: processor; A memory for storing a program for processing data generated by an electronic device, the program, when read and executed by the processor, performing the steps in the system as described in any one of claims 1 to 9, or performing the method as described in claim 10, 12, or 13.

Images