Inspection terminal cloud cooperation system with inspection task orientation and task execution method

Abstract

The application discloses a kind of with the inspection task orientation of inspection terminal cloud cooperation system, task execution method, system includes: end side equipment and cloud end equipment;End side equipment is used to obtain the video data of target inspection scene, and according to the layered change detection of video data executed by inspection task, determine first layered change detection result and second layered change detection result;Key frame judgment is carried out based on first layered change detection result and / or second layered change detection result, generate the structured key visual data corresponding to the inspection task, then structured key visual data is transmitted to cloud end equipment;Cloud end equipment carries out inspection task analysis according to structured key visual data, generates structured inspection judgment data and / or the guidance information of inspection adjustment, and feedback to end side equipment.Thereby realize the key frame extraction of high accuracy, low redundancy, reduce the bandwidth consumption of transmission data, and the continuous autonomous promotion of system inspection efficiency.

Description

Technical Field

[0001] This application relates to the field of intelligent monitoring technology for industrial safety and the environment, specifically to a cloud-based collaborative system for inspection tasks. This application also relates to a method and apparatus for executing inspection tasks with an inspection task orientation. Furthermore, this application relates to a method for generating key data in inspection tasks, a method for judging inspection tasks, as well as computer storage media and electronic equipment. Background Technology

[0002] With the widespread adoption of video surveillance systems in urban management, industrial production, and community security, the number of cameras deployed in various scenarios has increased dramatically, generating massive amounts of video data. Faced with such a massive data volume, relying on manual real-time monitoring is not only inefficient but also highly susceptible to missed or misjudged critical events due to visual fatigue or distraction, failing to meet the timeliness and accuracy requirements of modern security management. Against this backdrop, utilizing artificial intelligence technology to automatically analyze surveillance videos and achieve intelligent identification and early warning of risk events (such as falls, fires and smoke, unauthorized intrusions, and abnormal equipment operation) has become an important development direction for improving inspection efficiency and ensuring public safety. This technological approach not only reduces the burden on manual labor but also significantly improves the response speed and handling efficiency of abnormal events, providing strong support for building an intelligent and proactive security and prevention system. Summary of the Invention

[0003] This application provides an inspection terminal-cloud collaborative system with inspection task orientation to solve the problems of wasted computing resources and bandwidth consumption in the prior art.

[0004] This application provides an inspection task-oriented end-cloud collaborative system, including: end-side devices and cloud devices;

[0005] The edge device is used to acquire video data of the target inspection scene, and perform layered change detection on the video data according to the inspection task to determine the first layered change detection result and the second layered change detection result; based on the first layered change detection result and / or the second layered change detection result, key frame judgment is performed to generate structured key visual data corresponding to the inspection task, and then the structured key visual data is transmitted to the cloud device.

[0006] The cloud device analyzes the inspection task based on the structured key visual data, generates structured inspection judgment data and / or guidance information for inspection adjustments, and feeds it back to the edge device.

[0007] In some embodiments, the step of performing layered change detection on the inspection video data according to the inspection task requirements, and determining the first layered change detection result and the second layered change detection result, includes:

[0008] Based on a pre-configured invalid region or a pre-configured target region, the video frames and / or image sequences in the video data are filtered to determine the valid regions in the video frames and / or image sequences;

[0009] Perform the layered change detection on the effective region to determine the first layered change detection result and the second layered change detection result in the video frame and / or image sequence.

[0010] In some embodiments, performing the layered change detection on the effective region to determine the first layered change detection result and the second layered change detection result in the video frame and / or image sequence includes:

[0011] Dynamic change detection is performed on the effective region to obtain static and dynamic attribute information of the dynamically changing region in the video frame and / or image sequence.

[0012] The static and dynamic attribute information of the dynamically changing region are determined as the first layer change detection result;

[0013] Detect changes in the target inspection object within the effective area, and obtain the static and dynamic attribute information of the target inspection object within a preset time range in the video frame and / or image sequence;

[0014] The static and dynamic attribute information of the target inspection object is determined as the second layer change detection result.

[0015] In some embodiments, the step of detecting changes in the target inspection object within the effective area and obtaining static and dynamic attribute information of the target inspection object within a preset time range in the video frame and / or image sequence includes:

[0016] The effective area is detected within a preset time range to identify the target inspection object;

[0017] Based on the tracking of the target inspection object, a time-series state machine of the target inspection object is created to record the static and dynamic attribute information of the target inspection object.

[0018] In some embodiments, the step of determining keyframes based on the first layer change detection result and / or the second layer change detection result, and generating key visual data corresponding to the inspection task requirements, includes:

[0019] Based on the event determination requirements corresponding to the inspection task and the first layer change detection result, and / or based on the event determination requirements corresponding to the inspection task and the second layer change detection result, determine the video key area corresponding to the inspection task from the video data;

[0020] Based on the key areas of the video, the static and / or dynamic attribute information in the first layered change detection result, and the static and / or dynamic attribute information in the second layered change detection result, structured key visual data corresponding to the inspection task is generated.

[0021] In some embodiments, the step of performing inspection task analysis on the structured key visual data to generate structured inspection judgment data and / or inspection adjustment guidance information includes:

[0022] Based on the inspection task, obtain the inspection task configuration information;

[0023] The inspection task configuration information, the structured key visual data, and the prompts for the inspection task are used as input data for the large model to perform inspection task analysis, generating structured inspection judgment data and / or guidance information for inspection adjustments.

[0024] In some embodiments, the step of performing inspection task analysis on the structured key visual data to generate structured inspection judgment data and / or inspection adjustment guidance information includes:

[0025] When the analysis result of the inspection task analysis of the structured key visual data is that there is a risk, a structured early warning message is generated, or a structured early warning message and guidance information for adjusting the inspection requirements are generated.

[0026] When the analysis result of the inspection task analysis on the resulting key visual data is normal, guidance information for adjusting the inspection execution is generated.

[0027] In some embodiments, the cloud device analyzes the inspection task based on the structured key visual data, generates structured inspection judgment data and / or guidance information for inspection adjustments, and feeds it back to the edge device, including:

[0028] The cloud device preprocesses the structured key visual data to determine the target structured visual data;

[0029] The inspection task is analyzed based on the target structured visual data to generate structured inspection judgment data and / or inspection adjustment guidance information, which is then fed back to the end-side device.

[0030] In some embodiments, the cloud device preprocesses the structured key visual data to determine the target structured visual data, including:

[0031] Identify key areas relevant to the current inspection task;

[0032] Based on the key regions, determine the changing target regions related to the current inspection task from the structured key visual data;

[0033] Based on the changed target region, target markers are generated in the structured key visual data to obtain preprocessed target structured visual data;

[0034] or,

[0035] Identify key areas relevant to the current inspection task;

[0036] Based on the key regions, determine the changing target regions in the structured key visual data that are related to the current inspection task;

[0037] Based on the changed target region, the images or videos in the structured key visual data are cropped to obtain local structured visual data;

[0038] The local structured visual data is identified as the preprocessed target structured visual data.

[0039] This application also provides a patrol task execution method with patrol task orientation, including:

[0040] Acquire video data of the target inspection scene;

[0041] Based on the layered change detection performed on the video data according to the inspection task, the first layered change detection result and the second layered change detection result are determined;

[0042] Based on the first layer change detection result and / or the second layer change detection result, key frame judgment is performed to generate structured key visual data corresponding to the inspection task;

[0043] The structured key visual data is analyzed for inspection tasks to generate structured inspection judgment data and / or guidance information for inspection adjustments.

[0044] This application also provides an inspection task execution device with inspection task orientation, comprising:

[0045] The acquisition unit is used to acquire video data of the target inspection scene.

[0046] The determining unit is used to determine the first layer change detection result and the second layer change detection result based on the layer change detection performed on the video data according to the inspection task;

[0047] The first generation unit is used to determine key frames based on the first layer change detection result and / or the second layer change detection result, and generate structured key visual data corresponding to the inspection task.

[0048] The second generation unit is used to perform inspection task analysis on the structured key visual data and generate structured inspection judgment data and / or guidance information for inspection adjustment.

[0049] This application also provides a method for generating key data in inspection tasks, including:

[0050] Acquire video data of the target inspection scene;

[0051] Based on the layered change detection performed on the video data according to the inspection task, the first layered change detection result and the second layered change detection result are determined;

[0052] Based on the first layer change detection result and / or the second layer change detection result, key frame determination is performed to generate structured key visual data corresponding to the inspection task.

[0053] This application also provides a method for determining inspection tasks, including:

[0054] Receive structured key visual data generated by the method for generating key data in the inspection task described above.

[0055] The structured key visual data is analyzed for inspection tasks to generate structured inspection judgment data and / or guidance information for inspection adjustments.

[0056] This application also provides a computer storage medium, comprising: a computer program, which, when the computer program is executed, performs the steps as described above in the inspection task-oriented cloud collaborative system, or performs the steps as described above in the inspection task execution method, or performs the steps as described above in the method for generating key data in the inspection task, or performs the steps as described above in the inspection task discrimination method.

[0057] This application also provides an electronic device, including:

[0058] processor;

[0059] The memory is used to store programs that process data generated by electronic devices. When the program is read and executed by the processor, it performs the steps as described in the inspection task-oriented cloud collaborative system, or the steps as described in the inspection task execution method, or the steps as described in the key data generation method in the inspection task, or the steps as described in the inspection task discrimination method.

[0060] Compared with the prior art, this application has the following advantages:

[0061] By performing layered change detection on video data using edge devices, redundant information is filtered out to obtain key information relevant to the inspection task. This key information is then transmitted to cloud devices for inspection task analysis. The cloud devices, based on their deployed computing resources, provide feedback based on the analysis results and transmit it back to the edge devices, which then make corresponding adjustments. This allows the cloud devices to reverse-engineer the preprocessing parameters, change detection parameters, and keyframe selection requirements of the edge devices based on the effectiveness of risk identification, constructing a collaborative evolutionary closed loop of "perception → judgment → feedback → optimization," and achieving continuous autonomous improvement in the system's inspection efficiency. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the framework of a cloud-based collaborative system for inspection tasks provided in this application.

[0063] Figure 2 This is a flowchart illustrating an embodiment of the generation of structured key video data by end-side devices in an inspection task-oriented end-cloud collaborative system provided in this application.

[0064] Figure 3 This is a flowchart illustrating an embodiment of cloud-based device inspection task analysis in an inspection task-oriented cloud collaborative system provided in this application.

[0065] Figure 4 This is a flowchart of an inspection task execution method with an inspection task orientation provided in this application.

[0066] Figure 5 This is a structural schematic diagram of an inspection task execution device with inspection task orientation provided in this application.

[0067] Figure 6 This is a flowchart of a method for generating key data in an inspection task, as provided in this application.

[0068] Figure 7 This is a schematic diagram of the structure of a device for generating key data in an inspection task, as provided in this application.

[0069] Figure 8 This is a flowchart of an inspection task discrimination method provided in this application.

[0070] Figure 9 This is a schematic diagram of the structure of an inspection task discrimination device provided in this application.

[0071] Figure 10 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0072] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0073] It should be noted that the terms "first," "second," "third," etc., in the claims, specification, and drawings of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. Such data are interchangeable where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown or described in this application. Furthermore, the terms "comprising," "having," and their variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0074] It should be understood that in the embodiments of this application, "at least one" means one or more, and "more than one" means two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the related objects before and after it are in an "or" relationship. "Contains A, B and / or C" means containing any one, two, or three of A, B, and C.

[0075] It should be understood that in the embodiments of this application, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0076] Based on the aforementioned background technology, it is clear that the process of using artificial intelligence technology for intelligent inspection of industrial scenarios or environments involves automatically analyzing surveillance videos to provide early warnings of risk events. Existing technologies for intelligent video inspection can include two categories of analysis:

[0077] One approach is cloud-based analytics, which involves encoding all or part of the video stream captured by the front-end camera and continuously transmitting it to a cloud server. Cloud servers possess powerful computing capabilities, enabling the deployment of complex and high-precision artificial intelligence models for analysis. However, this method has drawbacks: extremely high network bandwidth requirements and enormous transmission costs; real-time video data uploads are delayed, affecting the timeliness of event response; and a large number of redundant video clips without any events occurring are also uploaded, resulting in a significant waste of cloud computing resources.

[0078] Another approach is edge-side analysis, which involves deploying AI models directly on edge devices such as cameras or edge computing boxes for analysis. This solution can achieve low-latency real-time response without consuming network bandwidth. However, its drawbacks are: edge devices have limited computing power and storage space, making it difficult to run large, high-precision, and complex models; model updates and algorithm iterations are inconvenient, requiring operation on a large number of front-end devices one by one; and for inspection tasks that require global information or complex logical reasoning, the capabilities of edge-side models are limited.

[0079] In addition, current risk assessment methods mostly rely on fixed rules or single-modal image / video models, lacking the ability to understand complex semantic scenarios in a general way; they also lack a flexible risk definition mechanism based on task orientation, making it difficult to adapt to diverse inspection task requirements.

[0080] In summary, this application proposes an edge-cloud collaborative system that can solve the problem of wasted computing resources, reduce bandwidth consumption, and has an inspection-oriented approach, so as to achieve low-bandwidth, high-precision, and highly generalized risk event inspection.

[0081] like Figure 1 As shown, the collaborative system includes: an edge device 101 and a cloud device 102. The edge device can be one or more of the following: a camera, an edge computing box, or an electronic device with camera functionality, capable of acquiring one or more monitoring video streams in real time. The cloud device can be a server or a server cluster, etc.

[0082] The edge device 101 can be used to collect video data of the target inspection scene, and determine the first layer change detection result and the second layer change detection result based on the layer change detection performed on the video data according to the inspection task; perform key frame judgment based on the first layer change detection result and / or the second layer change detection result, generate structured key visual data corresponding to the inspection task, and then transmit the structured key visual data to the cloud device.

[0083] The cloud device 102 analyzes the inspection task based on the structured key visual data, generates structured inspection judgment data and / or guidance information for inspection adjustment, and feeds it back to the end device.

[0084] In this embodiment, when the end-side device 101 acquires video data of the target inspection scene, it can dynamically adjust the frame rate and / or image resolution of the acquired video according to the inspection task requirements issued by the cloud device 102 and in combination with its own computing resource load, thereby ensuring effective monitoring while dynamically and in real time saving the computing power of the end-side device.

[0085] The edge device can be one or more of the following: a camera, an edge computing box, or an electronic device with camera functionality. The edge computing box, also known as an edge module, is the core hardware carrier for edge computing, integrating or connecting to a camera. It is a device deployed near the data source and possessing a certain level of computing power.

[0086] To reduce the computational load on edge devices and avoid wasting computing resources, layered change detection is performed on video data. This can be achieved by embedding a lightweight algorithm within the edge device. The specific layered change detection method can be determined based on the inspection task. Figure 2 As shown, in this embodiment, layered change detection is performed using changes in dynamically changing regions and specific target inspection objects as examples. Changes can refer to the appearance of new target objects, the disappearance of old target objects, or changes in the position or posture of existing target objects. Layered change detection can detect information about dynamically changing regions and specific target objects in video data, enabling subsequent keyframe determination based on the layered change detection results. This improves the accuracy of keyframe determination and reduces bandwidth consumption for subsequent transmission to the cloud.

[0087] To avoid wasting computational resources and further improve the accuracy of keyframe judgment, redundant data is removed from the video data. Layered change detection is performed on the video data to determine the first and second layered change detection results. This can include removing some invalid regions from the video data, thereby eliminating some redundant data, narrowing the scope of dynamic change region detection and target inspection object change detection, and reducing computational load. Specifically, at least two methods can be used.

[0088] Method 1:

[0089] The video frames and / or image sequences in the video data are filtered according to the pre-configured invalid regions to determine the valid regions in the video frames and / or image sequences;

[0090] Perform the layered change detection on the effective region to determine the first layered change detection result and the second layered change detection result in the video frame and / or image sequence.

[0091] Method two is:

[0092] Based on a pre-configured target region, the video frames and / or image sequences in the video data are filtered to determine the effective region in the video frames and / or image sequences;

[0093] Perform the layered change detection on the effective region to determine the first layered change detection result and the second layered change detection result in the video frame and / or image sequence.

[0094] In this embodiment, the invalid region or the target region can be segmented using background modeling (such as Gaussian Mixture Model (GMM)) or a lightweight semantic segmentation network, based on predefined invalid or target regions. The invalid region can be, for example, the sky, roof, walls, or other areas containing elements unrelated to the inspection task. In this embodiment, some meaningless content can be considered invalid content, which may be video clips that do not include inspection service requirements, do not trigger analysis needs, or are repetitive and redundant. The target region can be the area containing elements related to the inspection task. The invalid regions of video frames and / or image sequences are filtered according to the pre-configured invalid or target regions to determine or extract valid regions from the video frames and / or image sequences. After filtering the invalid regions, layered change detection is performed within the valid regions. The specific implementation process may include:

[0095] Dynamic change detection is performed on the effective region to obtain static and dynamic attribute information of the dynamically changing region in the video frame and / or image sequence. The static attribute information of the dynamically changing region includes its location and size. The dynamic attribute information can include the type of dynamic change, such as appearance, disappearance, or movement. Dynamic change detection of the effective region can be performed using methods such as inter-frame difference, background subtraction (e.g., Gaussian mixture model (GMM)), optical flow, or change detection models. The process involves inputting a video frame and / or image sequence → calculating changes → post-processing → outputting the changed region. Post-processing can include the original change mask → denoising filtering → hole filling → morphological operations → region filtering → boundary optimization → spatiotemporal consistency → outputting the result. The above method for detecting dynamically changing regions is only a brief example, and the detection method used is an existing method, which will not be described in detail here. However, considering computational resource limitations, factors such as the video data shooting time and environment can be used as selection factors for the above detection method. Under different shooting conditions or inspection requirements, different detection methods can be used to save computational power while still meeting the detection requirements.

[0096] The static and dynamic attribute information of the dynamically changing region are determined as the first layer change detection result.

[0097] The effective area is subjected to target object change detection to obtain the static and dynamic attribute information of the target object within a preset time range in the video frame and / or image sequence. The target object change detection can be implemented using a lightweight target detection / tracking model deployed on the edge device. The static attribute information of the target object may include: the target object's ID, category, location, size, etc.; the dynamic attribute information of the target object may include: appearance, disappearance, movement, entering / leaving the designated area, long-term stillness, etc.

[0098] The static and dynamic attribute information of the target inspection object is determined as the second layer change detection result.

[0099] The specific implementation process of detecting changes in the target inspection object within the effective area and obtaining the static and dynamic attribute information of the target inspection object within a preset time range in the video frame and / or image sequence may include:

[0100] The effective area is detected within a preset time range to identify the target inspection object; wherein, the preset time range can be a periodic time range, such as once per second, and can be set according to actual inspection needs. The target inspection object can be personnel, vehicles, etc.

[0101] Based on the tracking of the target inspection object, a temporal state machine for the target inspection object is created to record its static and dynamic attribute information. The dynamic attribute information can be determined by comparing the detection results of the current video frame and / or the current image sequence with historical dynamic attributes to determine whether the state of the target inspection object in the current video frame and / or the current image sequence has changed, thereby determining the dynamic attribute information.

[0102] The edge device needs to perform keyframe judgment based on the first layer change detection result and / or the second layer change detection result to generate structured key visual data corresponding to the inspection task. The purpose is to transmit the structured key visual data to the cloud device so that the cloud device can perform inspection task analysis. The specific implementation process includes:

[0103] Based on the event determination requirements corresponding to the inspection task and the first layer change detection result, and / or based on the event determination requirements corresponding to the inspection task and the second layer change detection result, determine the video key area corresponding to the inspection task from the video data;

[0104] Based on the key areas of the video, the static and / or dynamic attribute information in the first layered change detection result, and the static and / or dynamic attribute information in the second layered change detection result, structured key visual data corresponding to the inspection task is generated.

[0105] The event determination requirements corresponding to the inspection task may include the following, although the following are just examples and are not limited to these:

[0106] Event Judgment Requirement 1 (Item Residual): The scene area undergoes significant dynamic changes, and the change type is "appearance";

[0107] Event Judgment Requirement 3 (Item Taken): The scene area undergoes significant dynamic changes, and the change type is disappearance;

[0108] Event Judgment Requirement 2 (New Target Intrusion): A new "Appearance" state appears in the target object's state machine;

[0109] Event Judgment Requirement 4 (Abnormal Stay): The "long-term static" state of a target object in the target object state machine exceeds the preset threshold.

[0110] For example, assuming the inspection task is fire lane obstruction, generating the structured key visual data corresponding to the inspection task requires extracting key frames and / or key image sequences with valid change areas from the video data, based on the event determination requirement 1 and the static and / or dynamic attribute information in the first layer change detection results, as well as structured data. Specifically, based on event determination requirement 1, the target dynamic change area with the dynamic attribute information "appeared" in the dynamic change detection results within the valid change area is extracted, and this target dynamic change area is used as the key frame and / or key image sequence. The static attribute information (e.g., dynamic change area location, size, camera ID, time, geographical location, etc.), dynamic attribute information (e.g., change type is "appeared and stationary"), and the bounding boxes, segmentation masks, and state change trajectories in the key frames and / or key image sequences from the dynamic change detection results are used as structured metadata. Structured key visual data is then generated based on the key frames and / or key image sequences and the structured metadata.

[0111] For example, assuming the inspection task involves an animal intrusion, generating the structured key visual data corresponding to the inspection requires extracting key frames and / or key image sequences containing effective change areas from the video data, based on the event determination requirement 2 and the static and / or dynamic attribute information in the second-layer change detection results, as well as structured data. Specifically, based on event determination requirement 2, the target change area where the dynamic attribute information of the target inspection object is "present" in the target inspection object change detection results within the effective change area is extracted, and this target change area is used as the key frame and / or key image sequence. The static attribute information (e.g., location, size, category, camera ID, time, geographical location, etc.), dynamic attribute information (e.g., change type: target intrusion), bounding boxes, segmentation masks, and state change trajectories in the key frames and / or key image sequences from the target inspection object change detection results are used as structured metadata. Structured key visual data is then generated based on the key frames and / or key image sequences and the structured metadata.

[0112] It is understood that the structured metadata may include: event information, change descriptions, target information, etc. The event information may include: timestamps, camera IDs, geographic locations, etc. The change descriptions may include: change types, such as: appearance (intrusion), disappearance, stay, movement, entering / leaving a designated area, long-term stillness, etc. Target information may include: if a list of inspected targets is involved, each inspected target includes the inspected target ID, type, bounding box in the keyframe, segmentation mask, and state change trajectory, etc.

[0113] Using the fire lane obstruction example from before, let's assume we obtain continuous surveillance video:

[0114] From frame 1 to frame 20: Based on the filtered invalid area or the configured target area, perform dynamic change area detection and target inspection object change detection on the determined valid area. The dynamic change detection result of the valid area is no change, and the target inspection object change detection result is none.

[0115] Frames 21-30: The handcart enters the fire escape. Similarly, invalid areas are filtered out first, then dynamic change area detection and target inspection object change detection are performed on the valid fire escape area. Significant dynamic change was detected, classified as "occurring," and the area is located in the fire escape. The target inspection object change detection result is "none." Event judgment requirement 1 is triggered and met. Frames 21-30 are judged as "item left behind."

[0116] Therefore, the effective region of frames 21-30 is extracted as keyframes. It's understandable that if the trolley remains stationary after entering the fire lane within frames 21-30, then the video frames or image sequences after reaching that stationary state do not need to be used as keyframes. Therefore, frames 21-30 here are merely an example; within this range, some video frames or image sequences with insignificant state changes can be removed, further reducing computational load. Thus, keyframes can originate from effective regions of video frames or image sequences and can be determined based on the changing state.

[0117] In frame 31: the trolley is pushed away. Similarly, invalid areas are filtered first, then dynamic change area detection and target inspection object change detection are performed on the valid fire lane. Significant dynamic change was detected, typed as disappearance, and the area is located in the fire lane. Target inspection object change detection result is zero. Event judgment requirement 1 will not be triggered.

[0118] Continuing with the previous example of an animal breaking in, let's assume we obtain continuous surveillance video:

[0119] From frame 1 to frame 20: First, invalid areas are filtered out, and then dynamic change area detection and target inspection object change detection are performed on the valid areas. The dynamic change detection result of the valid areas is no change, and the target inspection object change detection result is no change.

[0120] Frames 21-30: An animal enters the target area. Similarly, invalid areas are filtered out first, then dynamic change detection and target object change detection are performed on the valid areas. No dynamic change detection is detected, and the target object change detection result is "present." Event judgment requirement 2 is triggered and met. Frames 21-30 are judged as "target intrusion."

[0121] Therefore, the effective region of frames 21-30 is extracted as keyframes.

[0122] The above examples are merely illustrative of the implementation process and are not intended to limit the principles of the technical solution. Furthermore, the examples demonstrate filtering invalid regions for each frame before performing layered change detection on the valid regions. Alternatively, invalid regions can be filtered for all video data beforehand, followed by valid region detection for each subsequent frame. For example, invalid regions can be filtered for video frames and / or image sequences in a surveillance video, and then dynamic change detection and target object change detection can be performed on the valid regions after excluding invalid regions. Alternatively, invalid regions can be filtered for each frame before valid region detection; the specific method is not limited.

[0123] The above examples illustrate fire lane obstruction and animal intrusion. It is understood that monitoring video can include various types of information, and this information can be used to extract various key details. For example, the video may contain structured key visual data corresponding to fire lane obstruction, animal intrusion, or the kitchen environment (such as the cleanliness of trash cans and the hygiene of food). Key information can be extracted from the effective area of ​​the video data according to the above event determination requirements, combined with dynamic change detection results and / or target inspection object change detection results, and structured key visual data can be generated.

[0124] In this embodiment, the keyframes and / or key image sequences included in the structured key visual data can be single or multiple images, such as before, during, and after a change; or they can be a short video edited based on extracted video information, such as content corresponding to a period of time before and after the inspection task (e.g., 5 seconds before and after the task). The keyframes and / or key image sequences can be frame images and / or image sequences determined based on the effective region, so that subsequent inspection task analysis only needs to target the effective region. Of course, the possibility of extracting complete frame images and / or image sequences from the effective region is not excluded. In this embodiment, the key information is mainly extracted from the effective region, that is, the extracted key information is the effective region portion.

[0125] The structured key visual data is transmitted to cloud devices for inspection task analysis. Because the structured key visual data undergoes invalid area filtering and effective area layer-by-layer detection, the generated structured key visual data avoids redundant information, thus reducing transmission bandwidth consumption. It also provides high-precision, low-redundancy data for subsequent inspection task analysis, saving computing resources.

[0126] like Figure 3 As shown, in this embodiment, the inspection task analysis can be implemented using a large model deployed on a cloud device, such as a multimodal large model. The specific implementation process may include:

[0127] Based on the inspection task, obtain the inspection task configuration information; the inspection task configuration information may include configuration information of key areas generated manually or automatically, such as: task focus area, shielded area, etc.

[0128] The inspection task configuration information, the structured key visual data, and the prompts for the inspection task are used as input data for the large model to perform inspection task analysis, generating structured inspection judgment data and / or guidance information for inspection adjustments.

[0129] To enable the large model to have a more accurate understanding ability and accurately locate the inspection area, visual cues can be provided through preprocessing. Therefore, this embodiment may further include the following steps before the large model performs risk assessment: the cloud device preprocesses the structured key visual data to determine the target structured visual data; the inspection task is analyzed based on the target structured visual data to generate structured inspection judgment data and / or guidance information for inspection adjustments, and this data is fed back to the edge device. The preprocessing may include at least two of the following methods.

[0130] Method 1:

[0131] Obtain the key area related to the current inspection task; this can be obtained by real-time detection of video data of the target inspection scene through a cloud detection port. For example, if the inspection object of the current inspection task is a trash can, then the key area should include the trash can. Since the trash can may move, real-time detection is required to obtain the key area where the trash can is located.

[0132] Based on the key regions, determine the changing target regions in the structured key visual data that are related to the current inspection task;

[0133] Target markers are generated in the structured key visual data based on the changed target region, resulting in preprocessed target structured visual data. The target markers can be in the form of semi-transparent masks, highlighted borders, or numerical markers. In this embodiment, the range of the target markers for the changed target region can be determined based on the inspection scenario and / or the inspection object. For example, for inspection objects that may move, the target marker range can include the area outside the location of the inspection object; while for fixed inspection objects, the target marker can be the area of ​​the location itself. That is, the range of the target markers can be flexibly expanded or contracted based on the specific inspection scenario and / or inspection object. It can also be adjusted according to specific inspection requirements.

[0134] Method 2:

[0135] Obtain the key area related to the current inspection task; this can be obtained by real-time detection of video data of the target inspection scene through a cloud detection port. For example, if the inspection task is a trash can, then the key area should include the trash can. Since the trash can may move, real-time detection is required to obtain the key area where the trash can is located.

[0136] Based on the key regions, determine the changing target regions in the structured key visual data that are related to the current inspection task;

[0137] Based on the changed target region, the images or videos in the structured key visual data are cropped to obtain local structured visual data;

[0138] The local structured visual data is identified as the preprocessed target structured visual data.

[0139] The large model performs risk assessment on the target structured visual data, including:

[0140] When the analysis result of the inspection task analysis of the target structured visual data is that there is a risk, a structured early warning message is generated, or a structured early warning message and guidance information for adjusting the inspection requirements are generated.

[0141] When the analysis result of the inspection task analysis of the target structured visual data is normal, guidance information for adjusting the inspection execution is generated.

[0142] When the analysis results of the aforementioned large model indicate the existence of risk, the generated structured early warning information can be fed back. This structured early warning information may include: timestamp, location, risk type, confidence level, risk description, relevant images or video clips, etc., and can be transmitted and fed back via alarm interfaces such as SMS, email, platform messages, etc. Alternatively, based on the risk situation, guidance information corresponding to adjustments in inspection requirements can be generated, such as: risk level and corresponding risk handling methods, angle of the data acquisition equipment, acquisition time, trigger condition configuration, and guidance information to avoid omissions. That is, only the structured early warning information can be generated, or both structured early warning information and guidance information for adjusting inspection requirements can be generated.

[0143] When the large model analysis result is normal, guidance information corresponding to the adjustment of the inspection execution is generated, such as the data required for the next inspection task, thereby adjusting the corresponding acquisition equipment angle, clarity and other related acquisition requirements.

[0144] Regardless of whether the analysis result is risky or normal, cloud-based devices can generate relevant guiding information based on the specific inspection task results, including optimized edge-side inspection strategies, changes in detection parameters, and key information screening and judgment requirements. This information is then transmitted to the corresponding edge-side devices, thereby promoting the continuous progress of the overall inspection system. During the inspection task analysis process by cloud-based devices, it can be performed on multiple channels of structured key visual data; therefore, the guiding information fed back to the edge-side devices can also correspond to different edge-side devices.

[0145] In the aforementioned edge-cloud collaborative system, edge devices are responsible for filtering and compressing key information with low latency and low computing power, while cloud devices analyze inspection tasks using large models, forming an inspection collaboration paradigm of front-end filtering and back-end precision judgment. Through layered change detection of video data by edge devices, redundant information is filtered to obtain key information relevant to the inspection task. This key information is then transmitted to cloud devices for inspection task analysis. Cloud devices, based on their deployed computing resources, obtain the inspection task analysis results and provide corresponding feedback to edge devices, which then make adjustments based on the feedback. This allows cloud devices to reverse-optimize the preprocessing parameters, change detection parameters, and keyframe filtering requirements of edge devices based on the effectiveness of risk identification, constructing a collaborative evolutionary closed loop of "perception → judgment → feedback → optimization," achieving continuous and autonomous improvement in system inspection efficiency. Furthermore, cloud devices can dynamically and flexibly define and execute various complex inspection tasks with business logic during inspection task analysis, exhibiting high flexibility and scalability, enabling the collaborative system to quickly adapt to various monitoring scenarios.

[0146] The above is a detailed description of an embodiment of an inspection task-oriented cloud-based collaborative system provided in this application. Corresponding to the aforementioned embodiment of an inspection task-oriented cloud-based collaborative system, this application also discloses an embodiment of an inspection task execution method with inspection task orientation.

[0147] like Figure 4 As shown, this method can be applied to cloud devices and / or edge devices, and the specific implementation process includes:

[0148] Step S401: Acquire video data of the target inspection scene;

[0149] Step S402: Based on the layered change detection performed on the video data according to the inspection task, determine the first layered change detection result and the second layered change detection result;

[0150] Step S403: Based on the first layer change detection result and / or the second layer change detection result, key frame judgment is performed to generate structured key visual data corresponding to the inspection task;

[0151] Step S404: Perform inspection task analysis on the structured key visual data to generate structured inspection judgment data and / or inspection adjustment guidance information.

[0152] For details regarding steps S401 to S404 above, please refer to the description of the above embodiment of the inspection terminal cloud collaborative system with inspection task orientation, which will not be elaborated here.

[0153] The above is a detailed description of an embodiment of an inspection task execution method with inspection task orientation provided in this application. Corresponding to the aforementioned embodiment of an inspection task execution method with inspection task orientation, this application also discloses an embodiment of an inspection task execution device with inspection task orientation.

[0154] like Figure 5 As shown, the device includes:

[0155] Acquisition unit 501 is used to acquire video data of the target inspection scene;

[0156] The determining unit 502 is used to determine the first layer change detection result and the second layer change detection result based on the layer change detection performed on the video data according to the inspection task;

[0157] The first generation unit 503 is used to determine key frames based on the first layer change detection result and / or the second layer change detection result, and generate structured key visual data corresponding to the inspection task.

[0158] The second generation unit 504 is used to perform inspection task analysis on the structured key visual data and generate structured inspection judgment data and / or guidance information for inspection adjustment.

[0159] Accordingly, the specific implementation process of the device can also be referred to the above description of the embodiment of the inspection terminal cloud collaborative system with inspection task orientation, which will not be detailed here.

[0160] Based on the above, this application also provides a method for generating key data in inspection tasks, such as... Figure 6 As shown, this method is applied to end-side devices and specifically includes:

[0161] Step S601: Acquire video data of the target inspection scene;

[0162] Step S602: Based on the layered change detection performed on the video data according to the inspection task, determine the first layered change detection result and the second layered change detection result;

[0163] Step S603: Based on the first layer change detection result and / or the second layer change detection result, key frame judgment is performed to generate structured key visual data corresponding to the inspection task.

[0164] For the specific implementation process of steps S601 to S602, please refer to the relevant description in the above embodiment of the inspection end-cloud collaborative system with inspection task orientation, that is, the relevant description of the end-side device, which will not be detailed here.

[0165] Accordingly, this application also provides a device for generating key data in inspection tasks, such as... Figure 7 As shown, the device includes:

[0166] Acquisition Unit 701: Acquires video data of the target inspection scene;

[0167] Determination unit 702: Determines the first layer change detection result and the second layer change detection result based on the layer change detection performed on the video data according to the inspection task;

[0168] Generation unit 703: Based on the first layer change detection result and / or the second layer change detection result, key frame judgment is performed to generate structured key visual data corresponding to the inspection task.

[0169] Based on the above, this application also provides a method for determining inspection tasks, such as... Figure 8 As shown, this method is applied to cloud devices and specifically includes:

[0170] Step S801: Receive the structured key visual data generated in the method for generating key data in the inspection task described above;

[0171] Step S802: Perform inspection task analysis on the structured key visual data to generate structured inspection judgment data and / or inspection adjustment guidance information.

[0172] For details regarding steps S801 and 802, please refer to the relevant content in the above system; they will not be elaborated here.

[0173] Accordingly, this application also provides an inspection task discrimination device, such as... Figure 9 As shown, the device specifically includes:

[0174] Receiving unit 901: Receives structured key visual data generated in the method for generating key data in the above-mentioned inspection task;

[0175] Generation unit 902: Performs inspection task analysis on the structured key visual data to generate structured inspection judgment data and / or guidance information for inspection adjustment.

[0176] Based on the above, this application also provides a computer storage medium, characterized in that it includes: a computer program, which, when the computer program is running, executes the relevant content of the inspection task-oriented cloud collaborative system described above, or executes the relevant content of the inspection task execution method described above, or executes the relevant content of the key data generation method in the inspection task described above, or executes the relevant content of the inspection task discrimination method described above.

[0177] Based on the above, this application also provides an electronic device, such as... Figure 10 As shown, the electronic device includes:

[0178] Processor 1001;

[0179] The memory 1002 is used to store a program for processing data generated by the electronic device. When the program is read and executed by the processor, it executes the relevant content of the inspection task-oriented cloud collaborative system described above, or executes the relevant content of the inspection task execution method described above, or executes the relevant content of the key data generation method in the inspection task described above, or executes the relevant content of the inspection task discrimination method described above.

[0180] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0181] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0182] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0183] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0184] 2. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A cloud-based collaborative system for inspection tasks, characterized in that, include: End-side devices and cloud devices; The end-side device is used to acquire video data of the target inspection scene, and perform layered change detection on the video data according to the inspection task to determine the first layered change detection result and the second layered change detection result. Based on the first layered change detection result and / or the second layered change detection result, keyframe judgment is performed to generate structured key visual data corresponding to the inspection task, and then the structured key visual data is transmitted to the cloud device; wherein, the layered change detection includes: filtering video frames and / or image sequences in the video data according to pre-configured invalid regions or pre-configured target regions to determine valid regions in the video frames and / or image sequences; performing layered change detection on the valid regions to obtain static attribute information and dynamic attribute information of dynamically changing regions in the video frames and / or image sequences; determining the static attribute information and dynamic attribute information of the dynamically changing regions as the first layered change detection result; performing target inspection object change detection on the valid regions to obtain static attribute information and dynamic attribute information of the target inspection object in the video frames and / or image sequences within a preset time range; determining the static attribute information and dynamic attribute information of the target inspection object as the second layered change detection result; The cloud device analyzes the inspection task based on the structured key visual data, generates structured inspection judgment data and / or guidance information for inspection adjustments, and feeds it back to the edge device.

2. The system according to claim 1, characterized in that, The step of detecting changes in the target inspection object within the effective area and obtaining the static and dynamic attribute information of the target inspection object within a preset time range in the video frame and / or image sequence includes: Within a preset time range, the effective area is detected to identify the target inspection object; Based on the tracking of the target inspection object, a time-series state machine of the target inspection object is created to record the static and dynamic attribute information of the target inspection object.

3. The system according to claim 1, characterized in that, The step of determining keyframes based on the first layer change detection result and / or the second layer change detection result, and generating key visual data corresponding to the inspection task requirements, includes: Based on the event determination requirements corresponding to the inspection task and the first layer change detection result, and / or based on the event determination requirements corresponding to the inspection task and the second layer change detection result, determine the video key area corresponding to the inspection task from the video data; Based on the key areas of the video, the static and / or dynamic attribute information in the first layered change detection result, and the static and / or dynamic attribute information in the second layered change detection result, structured key visual data corresponding to the inspection task is generated.

4. The system according to claim 1, characterized in that, The step of performing inspection task analysis on the structured key visual data to generate structured inspection judgment data and / or inspection adjustment guidance information includes: Based on the inspection task, obtain the inspection task configuration information; The inspection task configuration information, the structured key visual data, and the prompts for the inspection task are used as input data for the large model to perform inspection task analysis, generating structured inspection judgment data and / or guidance information for inspection adjustments.

5. The system according to claim 1, characterized in that, The step of performing inspection task analysis on the structured key visual data to generate structured inspection judgment data and / or inspection adjustment guidance information includes: When the analysis result of the inspection task analysis of the structured key visual data is that there is a risk, a structured early warning message is generated, or a structured early warning message and guidance information for adjusting the inspection requirements are generated. When the analysis result of the inspection task analysis of the structured key visual data is normal, guidance information for adjusting the inspection execution is generated.

6. The system according to claim 1, characterized in that, The cloud device analyzes the inspection task based on the structured key visual data, generates structured inspection judgment data and / or guidance information for inspection adjustments, and feeds it back to the edge device, including: The cloud device preprocesses the structured key visual data to determine the target structured visual data; The inspection task is analyzed based on the target structured visual data to generate structured inspection judgment data and / or inspection adjustment guidance information, which is then fed back to the end-side device.

7. The system according to claim 6, characterized in that, The cloud device preprocesses the structured key visual data to determine the target structured visual data, including: Identify key areas relevant to the current inspection task; Based on the key regions, determine the changing target regions related to the current inspection task from the structured key visual data; Based on the changed target region, target markers are generated in the structured key visual data to obtain preprocessed target structured visual data; or, Identify key areas relevant to the current inspection task; Based on the key regions, determine the changing target regions in the structured key visual data that are related to the current inspection task; Based on the changed target region, the images or videos in the structured key visual data are cropped to obtain local structured visual data; The local structured visual data is identified as the preprocessed target structured visual data.

8. A method for executing inspection tasks with an inspection task orientation, characterized in that, include: Acquire video data of the target inspection scene; Based on the layered change detection performed on the video data according to the inspection task, a first layered change detection result and a second layered change detection result are determined; this includes: filtering video frames and / or image sequences in the video data according to pre-configured invalid regions or pre-configured target regions to determine valid regions in the video frames and / or image sequences; performing layered change detection on the valid regions to obtain static and dynamic attribute information of dynamically changing regions in the video frames and / or image sequences; determining the static and dynamic attribute information of the dynamically changing regions as the first layered change detection result; performing target inspection object change detection on the valid regions to obtain static and dynamic attribute information of the target inspection object in the video frames and / or image sequences within a preset time range; and determining the static and dynamic attribute information of the target inspection object as the second layered change detection result. Based on the first layer change detection result and / or the second layer change detection result, key frame judgment is performed to generate structured key visual data corresponding to the inspection task; The structured key visual data is analyzed for inspection tasks to generate structured inspection judgment data and / or guidance information for inspection adjustments.

9. An inspection task execution device with inspection task orientation, characterized in that, include: The acquisition unit is used to acquire video data of the target inspection scene. A determining unit is configured to determine a first layered change detection result and a second layered change detection result based on layered change detection performed on the video data according to the inspection task; including: filtering video frames and / or image sequences in the video data according to pre-configured invalid regions or pre-configured target regions to determine valid regions in the video frames and / or image sequences; performing layered change detection on the valid regions to obtain static and dynamic attribute information of dynamically changing regions in the video frames and / or image sequences; determining the static and dynamic attribute information of the dynamically changing regions as the first layered change detection result; performing target inspection object change detection on the valid regions to obtain static and dynamic attribute information of the target inspection object in the video frames and / or image sequences within a preset time range; and determining the static and dynamic attribute information of the target inspection object as the second layered change detection result. The first generation unit is used to determine key frames based on the first layer change detection result and / or the second layer change detection result, and generate structured key visual data corresponding to the inspection task. The second generation unit is used to perform inspection task analysis on the structured key visual data and generate structured inspection judgment data and / or guidance information for inspection adjustment.

10. A method for generating key data in an inspection task, characterized in that, include: Acquire video data of the target inspection scene; Based on the layered change detection performed on the video data according to the inspection task, a first layered change detection result and a second layered change detection result are determined; this includes: filtering video frames and / or image sequences in the video data according to pre-configured invalid regions or pre-configured target regions to determine valid regions in the video frames and / or image sequences; performing layered change detection on the valid regions to obtain static and dynamic attribute information of dynamically changing regions in the video frames and / or image sequences; determining the static and dynamic attribute information of the dynamically changing regions as the first layered change detection result; performing target inspection object change detection on the valid regions to obtain static and dynamic attribute information of the target inspection object in the video frames and / or image sequences within a preset time range; and determining the static and dynamic attribute information of the target inspection object as the second layered change detection result. Based on the first layer change detection result and / or the second layer change detection result, key frame determination is performed to generate structured key visual data corresponding to the inspection task.

11. A computer storage medium, characterized in that, include: A computer program, when the computer program is executed, performs the steps in the system as described in any one of claims 1 to 7, or performs the method as described in claim 8 or 10.

12. An electronic device, characterized in that, include: processor; A memory for storing a program for processing data generated by an electronic device, the program, when read and executed by the processor, performing the steps in the system as described in any one of claims 1 to 7, or performing the method as described in claim 8 or 10.

Images