Job violation identification method and system based on adaptive model arrangement

Abstract

The invention discloses an operation violation identification method and system based on adaptive model arrangement, relates to the technical field of operation safety supervision, and solves the problems that an operation safety supervision system is insufficient in flexibility and is difficult to adapt to different demands of different scenes. According to the embodiment of the invention, through the business scene label and the model arrangement rule base, the same set of hardware and model pool can support differentiated monitoring strategies, and a new scene can be quickly adapted through different configurations, so that the expandability and flexibility of the system are improved, and the problem of configuration rigidity is solved; through the available computing resource information of the computing system, a processing assembly line can be dynamically degraded according to real-time computing resources, and a lighter model combination is adopted during high load, so that the overall response speed of the system is guaranteed, congestion is avoided, and resource-sensitive self-adaptive scheduling is realized. By adopting the method provided by the embodiment of the invention, the flexibility of the operation safety monitoring system can be improved, and the differentiated monitoring requirements of different operation scenes can be better met.

Description

Technical Field

[0001] This invention relates to the field of work safety monitoring technology, and in particular to a method and system for identifying work violations based on adaptive model orchestration. Background Technology

[0002] In industrial production, engineering construction, and other fields, operational safety supervision is a crucial link in ensuring personnel safety and preventing production safety accidents. With the rapid development of artificial intelligence technology, AI vision-based operational safety supervision solutions have been widely used in various operational scenarios due to their advantages such as real-time performance, automation, and non-contact operation. They have effectively replaced the traditional supervision model that relies on manual inspections, significantly improving supervision efficiency and coverage.

[0003] Currently, the core architecture of existing Artificial Intelligence (AI) visual safety monitoring systems mainly falls into two categories, both employing static and fixed model deployment methods to identify violations. The first type of architecture uses a single, multimodal large model that integrates multiple perception modalities such as vision, infrared, and sound, achieving high recognition accuracy. Its design philosophy is to use a single model to uniformly analyze all monitoring video streams, achieving comprehensive recognition of various violations. The second type of architecture, however, targets specific violations such as not wearing a safety helmet or unauthorized crossing, deploying multiple lightweight convolutional neural network (CNN) small models. This multi-model parallel processing improves recognition speed and meets the basic requirements of real-time monitoring.

[0004] However, the existing system's model deployment process and recognition strategy are fixed and cannot be dynamically adjusted according to the differences in actual business scenarios and the real-time computing resource load, making it difficult to adapt to the differentiated needs of different scenarios.

[0005] Therefore, a method and system for identifying job violations based on adaptive model orchestration is needed. Summary of the Invention

[0006] To address the problem that existing work safety monitoring systems lack flexibility and struggle to adapt to the differentiated needs of various scenarios, this invention provides a work violation identification method and system based on adaptive model orchestration. This method improves system flexibility and better adapts to the diverse needs of different scenarios. The specific technical solution is as follows: In a first aspect, embodiments of this application provide a method for identifying job violations based on adaptive model orchestration, including: The system receives a recognition task request, which includes at least monitoring video stream data, monitoring audio stream data, and a business scenario label. The monitoring video stream data and the monitoring audio stream data are associated. Based on the business scenario label, the system retrieves the corresponding initial model orchestration graph from a predefined model orchestration rule base. The initial model orchestration graph is a directed acyclic graph, which includes multiple nodes and directed edges connecting the nodes. Each node includes a lightweight unimodal model and / or a multimodal large model. The directed edges are used to indicate data flow and triggering logic. The system obtains the available computing resources information of the system and performs a simplification operation on the initial model orchestration graph based on the available computing resources information to obtain an executable model orchestration graph. The system loads and instantiates all nodes in the executable model orchestration graph and schedules the monitoring video stream data and audio stream data to flow sequentially through each node for processing based on the executable model orchestration graph. Based on the processing results of the output nodes of the executable model orchestration graph, a job violation recognition report is generated.

[0007] Preferably, the simplification operation includes at least one of the following: replacing the nodes corresponding to the multimodal large model with a subgraph composed of multiple lightweight single-modal models working together, or setting the nodes with higher accuracy and longer time consumption in the parallel branches of the initial model orchestration graph to an inactive state.

[0008] Preferably, the method of obtaining the corresponding initial model orchestration diagram from a predefined model orchestration rule library based on the business scenario label includes: retrieving the model orchestration diagram corresponding to the business scenario label in the model orchestration rule library; if the model orchestration diagram corresponding to the business scenario label does not exist, retrieving the model orchestration diagram corresponding to the parent scenario label of the business scenario label based on the predefined scenario label hierarchical inheritance structure and the business scenario label; if the model orchestration diagram corresponding to the parent scenario label does not exist, calculating the semantic vector similarity between the business scenario label and each scenario label in the model orchestration rule library; selecting the model orchestration diagram corresponding to the target scenario label with the highest semantic vector similarity among the target scenario labels whose semantic vector similarity is greater than the similarity threshold as the initial model orchestration diagram; and if the model orchestration diagram corresponding to the business scenario label exists, or the model orchestration diagram corresponding to the parent scenario label exists, obtaining the initial model orchestration diagram based on the retrieval results.

[0009] Preferably, there are multiple parent scene tags, and the search results include basic model arrangement diagrams that correspond one-to-one with the parent scene tags. Obtaining the initial model arrangement diagram based on the search results includes: obtaining the basic model arrangement diagram; identifying the common functional subgraphs and unique functional modules of the basic arrangement diagram; obtaining the corresponding fusion constraints and optimization objectives from the rule base based on the business scene tag; constructing an initial fusion diagram based on the fusion constraints, using the common functional subgraph as the basic framework and each unique functional module as a selective branch; and optimizing the structure of the initial fusion diagram based on the optimization objectives to generate the initial model arrangement diagram.

[0010] Preferably, the fusion constraints include at least functional integrity constraints, real-time constraints, and security rule constraints, and the optimization objectives include at least minimizing end-to-end latency and maximizing critical violation detection coverage.

[0011] Preferably, the simplification operation of the initial model orchestration diagram based on the available computing resource information to obtain an executable model orchestration diagram includes: calculating the resource stress score of the system; when the resource stress score is greater than the trigger threshold, selecting the simplification operation with the highest benefit-cost ratio from a predefined simplification strategy library based on the resource requirements and performance profiles of each node in the initial model orchestration diagram, where the benefit-cost ratio is the ratio of the estimated resource saving benefit to the estimated recognition accuracy loss; iteratively applying the selected simplification operation until the simplified model orchestration diagram meets the resource constraints or reaches the maximum number of iterations, thereby obtaining the executable model orchestration diagram.

[0012] Preferably, replacing the nodes corresponding to the multimodal large model with a subgraph composed of multiple collaborative lightweight single-modal models includes: identifying target multimodal large models that meet the replacement conditions in the initial model orchestration graph; selecting multiple target lightweight single-modal models from the model pool based on the scene-model capability mapping matrix and the currently available computing resources of the system; wherein, the elements in the scene-model capability mapping matrix represent the applicability score of the corresponding model in a specific scene, which is calculated based on the model's historical accuracy, recall rate, and latency performance in the specific scene; the combined capability range of the multiple target lightweight single-modal models covers the core capabilities of the target multimodal large model, and the total combined consumption is lower than that of the target multimodal large model; constructing the subgraph based on the multiple target lightweight single-modal models, wherein the internal nodes of the subgraph are connected in a serial or parallel manner to collaboratively complete the analysis function of the original large model node.

[0013] Secondly, embodiments of this application provide a job violation identification system based on adaptive model orchestration, applied to the method described in the first aspect, the system comprising: The receiving module receives a recognition task request, which includes at least monitoring video stream data, monitoring audio stream data, and business scenario tags, and the monitoring video stream data and the monitoring audio stream data are associated. The acquisition module is used to obtain the corresponding initial model orchestration graph from the predefined model orchestration rule library based on the business scenario label. The initial model orchestration graph is a directed acyclic graph, which includes multiple nodes and directed edges connecting the nodes. The nodes include lightweight unimodal models and / or multimodal large models. The directed edges are used to indicate data flow and triggering logic. The simplification module is used to obtain the available computing resources information of the system and simplify the initial model orchestration diagram based on the available computing resources information to obtain an executable model orchestration diagram. The processing module is used to load and instantiate all nodes in the executable model orchestration diagram, and based on the executable model orchestration diagram, schedule the monitoring video stream data and audio stream data to flow through each node for processing in sequence; The generation module is used to generate a job violation identification report based on the processing results of the output nodes of the executable model orchestration diagram.

[0014] Thirdly, embodiments of this application provide a computing device, including: a memory for storing a program; and a processor for loading the program to execute the method as described in the first aspect.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the method described in the first aspect.

[0016] Compared with existing technologies, the beneficial effects of this invention are as follows: By using business scenario tags and a predefined model orchestration rule library, the same set of hardware and model pool can support multiple differentiated monitoring strategies. New scenarios can be quickly adapted by configuring different directed acyclic graphs, greatly improving the system's scalability and flexibility and overcoming the problem of rigid configuration. By calculating the available computing resources of the computing system, the processing pipeline can be dynamically degraded based on real-time computing resources. Under high load, a lighter model combination can be automatically adopted to ensure the overall system response speed, avoid congestion, and achieve resource-sensitive adaptive scheduling. Using the method of the embodiments of this application can improve the flexibility of the job safety monitoring system and better adapt to the differentiated monitoring needs of different job scenarios. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0018] Figure 1 A flowchart illustrating a job violation identification method based on adaptive model orchestration provided in this application embodiment; Figure 2 A schematic diagram of the structure of a job violation identification system based on adaptive model orchestration provided in this application embodiment; Figure 3 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0021] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0023] To address the problem that existing work safety monitoring systems lack flexibility and struggle to adapt to the differentiated needs of different scenarios, this invention provides a work violation identification method and system based on adaptive model orchestration, which can improve system flexibility and better adapt to the differentiated needs of different scenarios.

[0024] Please see Figure 1 , Figure 1 This application provides a flowchart illustrating a job violation identification method based on adaptive model orchestration, which is applied to a computing device in a job violation identification system based on adaptive model orchestration. Figure 1 As shown, the method includes: Step 101: The computing device receives the recognition task request.

[0025] The computing device can be a server or terminal in the violation identification system, which can communicate with external systems to obtain relevant data from the external systems and perform analysis and identification for the corresponding external systems.

[0026] Specifically, the external system includes a video management platform, a security management system, or a work order system, etc.

[0027] For example, a video management platform can initiate analysis and identification tasks for specific video streams to computing devices based on scheduling plans or event triggers; a safety management system can collect audio and video data at the work site in real time based on audio and video acquisition devices deployed at the work site, and request computing devices to monitor this audio and video data in real time to determine whether there are any violations, and to correct or stop them in a timely manner.

[0028] The request includes at least monitoring video stream data, monitoring audio stream data, and business scenario tags, with the monitoring video stream data and the monitoring audio stream data being associated.

[0029] Specifically, the computing device can receive surveillance video stream data and surveillance audio stream data in parallel through a network socket listening service.

[0030] The external system can maintain a camera-scene table, which can be queried to obtain the business scene tags corresponding to the camera; it can also classify the monitoring video stream data through a classification model and output business scene tags; and it can also receive business scene tags manually entered or selected by the user.

[0031] For example, business scenario tags include standardized tags such as high-altitude operations, hot work, confined space operations, electrical work, and general inspections.

[0032] Upon receiving a recognition task request, the computing device can align the surveillance video stream data and the surveillance audio stream data using hardware timestamps or content feature alignment, and then generate associated metadata for both. This associated metadata includes an alignment quality score, a modal complementarity index, and an association identifier. The alignment quality score is calculated based on the standard deviation of the alignment error; the modal complementarity index is the mutual information between the audio and video sequences, used to indicate the redundancy of information provided by the two modalities; and the association identifier is used to identify the aligned audio and video data.

[0033] The computing device can then calculate the priority of the recognition task, using the following formula: ; in, The risk coefficient corresponds to the business scenario label. High-risk scenarios have a high coefficient, such as hot work. For alignment quality scoring; The level of urgency is determined by whether it is entered through an external system or marked by a user. These are the weighting coefficients, and the sum of the three is 1.

[0034] Then, the computing device can insert task requests into a priority queue according to their priority. The queue is implemented using a binary heap data structure to ensure that high-priority requests are dequeued first. At the same time, request metadata is written to a log database for auditing and performance monitoring.

[0035] Step 102: The computing device obtains the corresponding initial model orchestration diagram from the predefined model orchestration rule library based on the business scenario label.

[0036] The model orchestration rule base employs a hybrid storage architecture combining relational and graph databases. The relational database stores the rule data, while the graph database stores the complex graph structures.

[0037] Specifically, the computing device predefines a rule base rather than a graph library. The rule data in the rule base can serve as a decision support system containing multi-level knowledge. It includes: strategy rules for defining how to select or construct an orchestration graph based on scenario characteristics; construction rules for describing the constraints and optimization objectives of the graph structure; component rules for specifying the applicability, configuration parameters, and compatibility requirements of a specific model under specific conditions; and an instance library of model orchestration graphs.

[0038] Specifically, the graph database stores an orchestration graph master table, a node definition table, and an edge definition table. The orchestration graph master table stores basic information about the model orchestration graph, the node definition table stores information about the nodes in the graph, and the edge definition table stores the connection relationships between the nodes.

[0039] In this way, computing devices can form an intelligent decision-making system based on a rule base, capable of understanding scenario requirements, adhering to security constraints, and dynamically generating adaptive solutions, rather than a static database that can only perform keyword matching. The rule base is the brain of the computing device, and the orchestration diagram is the action instructions generated after the brain's thinking.

[0040] The initial model orchestration graph is a directed acyclic graph, which includes multiple nodes and directed edges connecting the nodes. The nodes include lightweight unimodal models and / or multimodal large models, and the directed edges are used to indicate data flow and triggering logic.

[0041] Preferably, the computing device also maintains a scenario-model capability mapping matrix M in the rule base. This is a sparse matrix with dimensions S×M, where S is the size of the business scenario label set, M is the size of the available model combination, and the matrix element M[s][m] represents the applicability score of model m under scenario s. The score is calculated using the following formula: ; in, Let m be the average accuracy of model m on the test set of scene s. Let m be the average recall of model m on the test set of scene s. To handle the single-frame delay of model m on standard hardware, normalize it to the [0,1] interval; These are the weighting coefficients, and the sum of the three is 1.

[0042] For example, The values ​​are 0.4, 0.4, and 0.2 respectively.

[0043] The matrix is ​​continuously updated through statistical learning of historical task execution results, providing data support for subsequent graph retrieval and optimization.

[0044] Preferably, the computing device can retrieve the model orchestration diagram corresponding to the business scenario label from the model orchestration rule base; if the model orchestration diagram corresponding to the business scenario label does not exist, it can retrieve the model orchestration diagram corresponding to the parent scenario label of the business scenario label based on the predefined scenario label hierarchical inheritance structure and the business scenario label; if the model orchestration diagram corresponding to the parent scenario label does not exist, it can calculate the semantic vector similarity between the business scenario label and each scenario label in the model orchestration rule base; select the model orchestration diagram corresponding to the target scenario label with the highest semantic vector similarity among the target scenario labels with a semantic vector similarity greater than the similarity threshold as the initial model orchestration diagram; if the model orchestration diagram corresponding to the business scenario label exists, or if the model orchestration diagram corresponding to the parent scenario label exists, it can obtain the initial model orchestration diagram based on the retrieval results.

[0045] The computing device can retrieve records corresponding to business scenario tags from the orchestration diagram main table through SQL queries. If the applicable business scenario field of a record in the orchestration diagram main table matches the business scenario tag precisely, the computing device can obtain the corresponding model orchestration diagram as the initial model orchestration diagram.

[0046] When the exact match result is empty, the computing device can enable tag inheritance system retrieval. The system maintains an inheritance tree structure of scene tags, which records the hierarchical relationship between business scene tags. For example, operation safety supervision is the root node of level 0, the child nodes of the root node in level 1 include high-altitude operation, hot work operation and confined space operation, and the child nodes of high-altitude operation in level 2 include scaffolding operation and suspended platform operation.

[0047] The computing device can trace back up the inheritance tree to the parent scenario label of the business scenario label until a usable model orchestration diagram is found, or the maximum backtracking depth is reached.

[0048] When the hierarchical retrieval results are also empty, the computing device can use semantic similarity calculation to convert scene labels into vectors through a pre-trained word embedding model. Then calculate Compared with all active scene label vectors in the rule base The cosine similarity is used. After determining the target scene label, the computing device can record the similarity of the fuzzy match as a confidence score for subsequent process decisions.

[0049] In cases of complex business scenario tags, the inheritance tree structure may not have corresponding tag records. In such cases, the computing device can split the complex business scenario tag into a set of basic tags. For example, high-altitude hot work cross-operation can be split into high-altitude operation and hot work operation. These basic tags can serve as the parent scenario tags of the complex business scenario tag.

[0050] Preferably, there are multiple parent scene tags, and the search results include basic model arrangement diagrams that correspond one-to-one with the parent scene tags; the computing device can obtain the basic model arrangement diagram; identify the common functional subgraphs and unique functional modules of the basic arrangement diagram; based on the business scene tag, obtain the corresponding fusion constraints and optimization objectives from the rule base; based on the fusion constraints, construct an initial fusion diagram with the common functional subgraph as the basic framework and each unique functional module as a selective branch; based on the optimization objective, perform structural optimization on the initial fusion diagram to generate the initial model arrangement diagram.

[0051] Among them, the closer the level in the inheritance number is to the root node, the wider the scope of its recognition, and the closer it is to basic general detection and recognition. Therefore, the model arrangement diagram corresponding to the parent scene label can be called the basic model arrangement diagram.

[0052] For multiple basic orchestration graphs, the computing device can calculate the functional signature of each node in these graphs. The functional signature is a hash value of the node model type, model input / output format, and functional description. Then, a bipartite graph matching problem is constructed between the source graph node set and the target graph node set. The Hungarian algorithm is then used to solve for the maximum weighted matching, where the weight is the similarity of the functional signatures. The matching node pairs in the solution result constitute a candidate set of common functional subgraphs. Finally, a variant of the McSplit algorithm is used to determine the common functional subgraph between the source graph and the target graph.

[0053] Specifically, the bipartite graph matching problem can be transformed from finding the most similar pair of nodes into finding a set of edges in a bipartite graph that matches each node exactly once and maximizes the total weight. This can be efficiently solved using the mature Hungarian algorithm. While the matched node pairs obtained from the bipartite graph matching problem only satisfy node similarity, the common functional subgraph requires both node similarity and similar connections between nodes. A variant of the McSplit algorithm can be used to verify the topological consistency of candidate node pairs and filter out the true common subgraph.

[0054] Specifically, a variant of the McSplit algorithm uses a recursive algorithm and a greedy strategy to prioritize matching the node with the highest degree, and gradually find the largest subgraph in two graphs with the same node labels and consistent topology.

[0055] The computing device can employ the Louvain algorithm to iteratively optimize the modularity Q, dividing each graph into multiple functional modules with tightly connected internal connections and sparse external connections. Then, natural language processing techniques are used to analyze node descriptions, extract functional keywords, establish a function-module mapping, and identify the unique functional modules in each graph.

[0056] Then, the computing device can obtain the fusion constraints corresponding to the business scenario tags from the rule base; identify the dependencies between each subgraph; and then fuse these subgraphs based on the fusion constraints and dependencies. The fusion strategies include serial fusion, which connects the subgraphs sequentially, with the output of the previous module serving as the input of the next module; parallel fusion, where multiple subgraphs are executed independently and in parallel, and their results are converged at subsequent nodes; and conditional fusion, which dynamically selects the branch to be executed next based on an intermediate result or context information.

[0057] Computing devices can combine the above strategies based on actual conditions to form more complex topologies.

[0058] Preferably, the fusion constraints include at least functional integrity constraints, real-time constraints, and security rule constraints.

[0059] Among them, the real-time constraint is used to constrain the time required for the initial fusion graph to go from input to output, and the safety rule constraint is used to indicate the detection priority, such as life safety detection being higher than equipment safety detection, and equipment safety detection being higher than behavioral norm detection.

[0060] For example, hot work requirements must include fire source detection and safe distance monitoring, and the two are logically dependent, requiring the fire source to be detected first and the distance to be assessed.

[0061] Preferably, the optimization objective includes at least minimizing end-to-end latency and maximizing critical violation detection coverage.

[0062] The computing device can eliminate redundancy by identifying and merging nodes with overlapping functions; and reduce end-to-end latency by rearranging the node order.

[0063] Specifically, computing devices can identify critical paths using the critical path method, and then parallelize the nodes of non-critical paths to reduce overall latency.

[0064] At the same time, computing devices need to maximize the coverage of key violation detection while ensuring the performance of the model orchestration graph, and retain the model orchestration graph's ability to detect key violations or phenomena.

[0065] Optionally, after obtaining the initial model orchestration graph of the fusion, the computing device can also calculate the complexity index of the initial model orchestration graph based on its number of nodes, number of edges, and number of conditional branches.

[0066] Then, the computing device can evaluate the end-to-end latency and peak memory usage of the initial model orchestration graph; if one or more of these two metrics exceed the corresponding threshold, the initial model orchestration graph will be preferentially simplified in step 103.

[0067] Step 103: The computing device obtains the available computing resource information of the system, and simplifies the initial model orchestration diagram based on the available computing resource information to obtain an executable model orchestration diagram.

[0068] The computing device can periodically collect multi-dimensional resource metrics through the operating system-level monitoring interface, including CPU, memory, GPU, storage I / O, network, and process-level metrics. Based on these metrics, the system resource stress level is determined, and when system resources are strained, the initial model orchestration diagram is simplified to ensure the smooth execution of the recognition task.

[0069] Preferably, the computing device can calculate the resource stress score of the system; if the resource stress score is greater than the trigger threshold, based on the resource requirements and performance profiles of each node in the initial model orchestration diagram, the device selects the simplification operation with the highest benefit-cost ratio from a predefined simplification strategy library, where the benefit-cost ratio is the ratio of the estimated resource saving benefit to the estimated recognition accuracy loss; the selected simplification operation is applied iteratively until the simplified model orchestration diagram meets the resource constraints or reaches the maximum number of iterations, thus obtaining the executable model orchestration diagram.

[0070] The resource stress score can be calculated based on at least one of the normalized CPU stress metrics, GPU stress metrics, memory stress metrics, storage I / O stress metrics, and network stress metrics.

[0071] For example, the formula for calculating the resource stress score may include: ; In the above formula, U represents the normalized index, f represents the nonlinear transformation function, w represents the weighting coefficient, mem represents memory, and net represents the network.

[0072] Among these features, the computing device can model the remaining available computing resources of the system, which can be represented as a resource availability vector. The elements in the denominator represent the absolute or relative amounts of available resources for CPU, GPU, memory, storage I / O, and network, respectively.

[0073] For example, ; .

[0074] It is understandable that iterative application refers to applying simplification operations sequentially and checking whether the resource constraint is met or the maximum number of iterations is reached after each application. If the constraint is not met and the maximum number of iterations is not reached, the simplification operation with the highest benefit-cost ratio is determined again and applied until the simplified model orchestration diagram meets the resource constraint or the maximum number of iterations is reached.

[0075] Preferably, the computing device can also use an ARIMA model or simple exponential smoothing to calculate and predict resource stress for each dimension of the index. If the predicted resource stress continues to rise and may exceed the corresponding threshold, simplification is triggered in advance.

[0076] The resource requirements and performance profiles of the nodes are tested in advance and stored in the computing device. These profiles include the computation latency, peak memory usage, number of CPU cycles required for computation, and device preference (CPU or GPU) for each node. Then, the computing device can further calculate the sum of the computation latencies of all nodes on the critical path of the initial model orchestration graph as the estimated total latency. By simulating the execution timeline, the memory requirements of the active node set at each moment are calculated, and the maximum value is taken as the estimated peak memory requirement.

[0077] Among them, high-priority recognition tasks can have their trigger thresholds appropriately increased, meaning they are more tolerant of resource constraints and try to preserve the original image as much as possible; low-priority tasks are more likely to be triggered and simplified.

[0078] The computing device also maintains a simplification strategy library, which contains a variety of predefined simplification operations. Each operation includes the scope of application, execution function, resource impact estimate, and accuracy impact estimate. The resource impact estimate refers to the expected reduction in computing latency, memory, etc.; the accuracy impact estimate refers to the expected impact on the final recognition accuracy, which can be estimated based on historical data or model performance curves.

[0079] Preferably, the simplification operation includes at least one of the following: for computationally intensive large model nodes, the node corresponding to the multimodal large model is replaced with a subgraph composed of multiple lightweight single-modal models working together; for parallel branch structures, nodes with higher accuracy and longer processing time in the parallel branches of the initial model orchestration graph are set to an inactive state; for visual model nodes, the model input resolution or processing frame rate can be reduced.

[0080] When simplification conditions are triggered, the computing device can calculate the benefit-cost ratio of all feasible simplification operations, including the following formulas: ; ; ; Here, 'op' represents a feasible simplified operation. To simplify operations and improve efficiency, To simplify the cost of operation, To prevent extremely small positive numbers from being divided by zero, For resource weight, To estimate the reduction in delay, To estimate the total delay, To estimate the memory reduction, For the estimated peak memory usage, To estimate the energy reduction, To estimate energy consumption; This is the accuracy importance coefficient, because the model accuracy ΔAcc is usually negative, and taking the negative yields the positive.

[0081] The process of simplifying operations through iterative application is as follows: Evaluate the resource requirements of the initial graph or the current graph after the previous simplification; if the resource requirements meet the resource constraints, stop simplification; otherwise, select the currently applicable set of operations from the strategy library; for each operation, estimate the new resource requirements and performance impact after application; select the best operation and apply it to the graph; repeat the iterative operation until the constraints are met or the maximum number of iterations is reached.

[0082] Understandably, resource constraints during iteration are dynamic because resource status is continuously monitored. If resource status improves during simplification, the computing device can stop or even roll back certain simplification operations earlier.

[0083] After simplification, the computing device can reanalyze the parallel potential of the graph. It identifies the set of nodes that can be executed in parallel and adjusts the degree of parallelism based on currently available resources; for parallelizable branches, the computing device can set a concurrent execution limit to avoid excessive parallelism leading to resource contention.

[0084] If a simplified solution that meets the resource constraints cannot be found within the specified number of iterations, the computing device can reduce the quality of the input data to further reduce the amount of computation; or it can directly alert the monitoring system, prompting that manual intervention or resource expansion is required.

[0085] Preferably, replacing the nodes corresponding to the multimodal large model with a subgraph composed of multiple collaborative lightweight single-modal models includes: identifying target multimodal large models that meet the replacement conditions in the initial model orchestration graph; selecting multiple target lightweight single-modal models from the model pool based on the scene-model capability mapping matrix and the currently available computing resources of the system; wherein, the elements in the scene-model capability mapping matrix represent the applicability score of the corresponding model in a specific scene, which is calculated based on the model's historical accuracy, recall rate, and latency performance in the specific scene; the combined capability range of the multiple target lightweight single-modal models covers the core capabilities of the target multimodal large model, and the total combined consumption is lower than that of the target multimodal large model; constructing the subgraph based on the multiple target lightweight single-modal models, wherein the internal nodes of the subgraph are connected in a serial or parallel manner to collaboratively complete the analysis function of the original large model node.

[0086] The replacement conditions include that the node type is a multimodal large model and the estimated resource consumption is greater than the product of the current available resources and the safety factor, which may affect the normal operation of the system.

[0087] The computing device can query the rule base to obtain the function list, performance baseline, and interface specifications of the large model node. For example, the function list of a large model for hot work risk analysis might be: [flame detection, smoke detection, personnel distance analysis, safety equipment identification].

[0088] Then, for each sub-function in the function list, the computing device can retrieve all models marked as lightweight and possessing the capability from the model repository, and the system's current available computing resources can run the model or a combination of models; then, it can use the scenario-model capability mapping matrix M[s][m] for initial screening, retaining only models whose applicability score is higher than the threshold in the current business scenario s.

[0089] The computing device can then select one candidate model from each sub-function to form a candidate model combination. If the number of combinations is too large, the computing device can use heuristic search to avoid combination explosion.

[0090] Then, the computing device evaluates the candidate model combinations based on the coverage of the target multimodal large model function list, interface compatibility, performance evaluation, and resource consumption of the combination capability range, and determines the optimal candidate model combination as the multiple target lightweight single-modal model.

[0091] Specifically, the computing device can use a weighted scoring method or find the Pareto optimal solution to select the one with the highest overall score from all feasible combinations.

[0092] Step 104: The computing device loads and instantiates all nodes in the executable model orchestration diagram, and based on the executable model orchestration diagram, schedules the monitoring video stream data and audio stream data to flow sequentially through each node for processing.

[0093] The computing device can generate a list of execution order of nodes based on the node dependencies in the executable model orchestration diagram, determine the execution level of each node, and then allocate hardware resources to each node.

[0094] The computing device can load model files from a centralized model repository into memory and instantiate them into executable objects. Specifically, for nodes in the first three layers of the execution path, the computing device can load them immediately. These nodes are at the front in the initial topology sort and will be executed first. For nodes in subsequent layers, the expected startup time is calculated, and the expected loading time is subtracted from the expected startup time to obtain the loading time, which is then used for loading. For branch nodes with low probability in the conditional branches, they are only loaded when the branch is activated.

[0095] Then, the computing device can schedule the monitoring video stream data and audio stream data to the corresponding input nodes in the executable model orchestration diagram, allowing these data to flow sequentially through each node of the corresponding pipeline for processing, and then obtain the output results of one or more output nodes in the executable model orchestration diagram.

[0096] Step 105: The computing device generates a job violation identification report based on the processing results of the output nodes of the executable model orchestration diagram.

[0097] Specifically, for detected operational violations in the output results, the computing device can first merge and deduplicate detection results that are similar in time and space; then, it can calculate the severity score for each deduplicated operational violation, using the following formula: ; in, For severity score, A preset baseline severity score is assigned to each violation of the work rules. As a context-enhancing factor, it considers the scenario risk level, time factors, and weather conditions to increase the severity of operational violations; As a persistence factor, the longer the operational violation lasts, the larger the value. As a diffusion factor, the larger the number of people involved or the larger the geographical area affected by the violation, the higher the value.

[0098] The computing device can then convert these operational violations into standard reporting event objects and generate an operational violation identification report; and trigger the corresponding level of alarm behavior based on the severity level corresponding to the highest severity score.

[0099] For example, if the severity score of a work violation is greater than 0.8, corresponding to the highest severity level, a level 1 alarm is triggered. The computing device can issue an alarm through sound and flashing output devices to notify the on-site safety officer and management, triggering the emergency plan.

[0100] The work violation identification report may include a report summary, a detailed list of violations, data analysis and insights, evidence, conclusions and recommendations.

[0101] Specifically, data analysis and insights can include violation trend analysis, hotspot area identification, and cause speculation; conclusions and recommendations include urgent action items, short-term improvement suggestions, and long-term preventive measures.

[0102] In this embodiment, by using business scenario tags and a predefined model orchestration rule library, the same set of hardware and model pool can support multiple differentiated monitoring strategies. New scenarios can be quickly adapted by configuring different directed acyclic graphs, greatly improving the system's scalability and flexibility and overcoming the problem of rigid configuration. By calculating the available computing resources of the computing system, the pipeline can be dynamically degraded based on real-time computing resources. Under high load, a lighter model combination can be automatically adopted to ensure the overall system response speed, avoid congestion, and achieve resource-sensitive adaptive scheduling. Using the method of this embodiment can improve the flexibility of the job safety monitoring system and better adapt to the differentiated monitoring needs of different job scenarios.

[0103] The method provided in the embodiments of this application has been described above. The system provided in the embodiments of this application will be described below.

[0104] Please see Figure 2 , Figure 2 A schematic diagram of a work violation identification system based on adaptive model orchestration provided in this application embodiment is shown below. Figure 2 As shown, the system 20 includes: The receiving module 201 receives a recognition task request, the request including at least monitoring video stream data, monitoring audio stream data, and business scenario tags, wherein the monitoring video stream data and the monitoring audio stream data are associated. The acquisition module 202 is used to acquire the corresponding initial model orchestration graph from the predefined model orchestration rule library based on the business scenario label; the initial model orchestration graph is a directed acyclic graph, the initial model orchestration graph includes multiple nodes and directed edges connecting the nodes, the nodes include lightweight single-modal models and / or multimodal large models, and the directed edges are used to indicate data flow and triggering logic; The simplification module 203 is used to obtain the available computing resource information of the system, and to simplify the initial model orchestration diagram based on the available computing resource information to obtain an executable model orchestration diagram; The processing module 204 is used to load and instantiate all nodes in the executable model orchestration diagram, and based on the executable model orchestration diagram, schedule the monitoring video stream data and the audio stream data to flow through each of the nodes for processing in sequence; The generation module 205 is used to generate a job violation identification report based on the processing results of the output nodes of the executable model orchestration diagram.

[0105] The aforementioned modules can be integrated into one or more computing devices of system 20.

[0106] The work violation identification system based on adaptive model orchestration provided in this application embodiment can be understood by referring to the relevant content in the foregoing method embodiment section, and will not be repeated here.

[0107] like Figure 3 As shown, Figure 3 This is a schematic diagram of a possible logical structure of a computing device provided in an embodiment of this application. The computing device 30 includes a processor 301, a communication interface 302, a memory 303, and a bus 304. The processor 301, the communication interface 302, and the memory 303 are interconnected via the bus 304. In an embodiment of this application, the processor 301 is used to control and manage the operation of the computing device 30. For example, the processor 301 is used to execute... Figure 1 The steps in the embodiments and / or other processes used in the techniques described herein. Communication interface 302 is used to support communication by computing device 30. Memory 303 is used to store program code and data of computing device 30.

[0108] The processor 301 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc. The bus 304 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0109] In another embodiment of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the above-described... Figure 1 The method described in the embodiments.

[0110] Those skilled in the art will recognize that the units of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the invention.

[0111] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0112] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0115] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A method for identifying job violations based on adaptive model orchestration, characterized in that, A computing device applied to a job violation detection system based on adaptive model orchestration; the method includes: Receive an identification task request, the request including at least monitoring video stream data, monitoring audio stream data, and business scenario tags, wherein the monitoring video stream data and the monitoring audio stream data are associated; Based on the business scenario tags, the corresponding initial model orchestration graph is obtained from the predefined model orchestration rule library; the initial model orchestration graph is a directed acyclic graph, which includes multiple nodes and directed edges connecting the nodes. The nodes include lightweight unimodal models and / or multimodal large models, and the directed edges are used to indicate data flow and triggering logic. Obtain the available computing resource information of the system, and simplify the initial model orchestration diagram based on the available computing resource information to obtain an executable model orchestration diagram; Load and instantiate all nodes in the executable model orchestration graph, and based on the executable model orchestration graph, schedule the monitoring video stream data and the audio stream data to flow sequentially through each of the nodes for processing; Based on the processing results of the output nodes of the executable model orchestration diagram, a job violation identification report is generated.

2. The method according to claim 1, characterized in that, The simplification operation includes at least one of the following: replacing the nodes corresponding to the multimodal large model with a subgraph composed of multiple lightweight single-modal models working together, or setting the nodes with higher accuracy and longer time consumption in the parallel branches of the initial model orchestration graph to an inactive state.

3. The method according to claim 1, characterized in that, The step of obtaining the corresponding initial model orchestration diagram from a predefined model orchestration rule library based on the business scenario tags includes: Retrieve the model orchestration diagram corresponding to the business scenario tag from the model orchestration rule base; If the model orchestration diagram corresponding to the business scenario label does not exist, the model orchestration diagram corresponding to the parent scenario label of the business scenario label is retrieved based on the predefined scenario label hierarchical inheritance structure and the business scenario label. If the model orchestration diagram corresponding to the parent scene label does not exist, calculate the semantic vector similarity between the business scene label and each scene label in the model orchestration rule base; select the model orchestration diagram corresponding to the target scene label with the highest semantic vector similarity among the target scene labels with semantic vector similarity greater than the similarity threshold as the initial model orchestration diagram; If a model orchestration diagram corresponding to the business scenario label exists, or a model orchestration diagram corresponding to the parent scenario label exists, the initial model orchestration diagram is obtained based on the search results.

4. The method according to claim 3, characterized in that, The number of parent scene tags is multiple, and the search results include basic model arrangement diagrams that correspond one-to-one with the parent scene tags; obtaining the initial model arrangement diagram based on the search results includes: Obtain the basic model arrangement diagram; Identify the common functional sub-graphs and unique functional modules of the basic arrangement diagram; Based on the business scenario tags, the corresponding fusion constraints and optimization objectives are obtained from the rule base; Based on the aforementioned fusion constraints, an initial fusion graph is constructed using the common functional subgraph as the basic framework and each of the aforementioned unique functional modules as selective branches. Based on the optimization objective, the initial fusion graph is structurally optimized to generate the initial model arrangement graph.

5. The method according to claim 4, characterized in that, The fusion constraints include at least functional integrity constraints, real-time constraints, and security rule constraints, and the optimization objectives include at least minimizing end-to-end latency and maximizing critical violation detection coverage.

6. The method according to any one of claims 1-5, characterized in that, The step of simplifying the initial model orchestration diagram based on the available computing resource information to obtain an executable model orchestration diagram includes: Calculate the resource stress score of the system; When the resource stress score is greater than the trigger threshold, based on the resource requirements and performance profiles of each node in the initial model orchestration diagram, the simplification operation with the highest cost-benefit ratio is selected from the predefined simplification strategy library. The cost-benefit ratio is the ratio of the estimated resource saving benefit to the estimated recognition accuracy loss. The selected simplification operation is applied iteratively until the simplified model orchestration graph meets the resource constraints or reaches the maximum number of iterations, thus obtaining the executable model orchestration graph.

7. The method according to claim 2, characterized in that, The step of replacing the nodes corresponding to the multimodal large model with a subgraph composed of multiple lightweight single-modal models working together includes: Identify the target multimodal large model that meets the replacement conditions in the initial model arrangement diagram; Based on the scenario-model capability mapping matrix and the currently available computing resources of the system, multiple target lightweight single-modal models are selected from the model pool; wherein, the elements in the scenario-model capability mapping matrix represent the applicability score of the corresponding model in a specific scenario, and the score is calculated based on the model's historical precision, recall rate, and latency performance in the specific scenario; the combined capability range of the multiple target lightweight single-modal models covers the core capabilities of the target multimodal large model, and the total combined cost is lower than that of the target multimodal large model; The subgraph is constructed based on the multiple target lightweight single-modal models. The internal nodes of the subgraph are connected in a serial or parallel manner to collaboratively complete the analysis function of the original large model nodes.

8. A job violation identification system based on adaptive model orchestration, characterized in that, The system includes: The receiving module receives a recognition task request, the request including at least monitoring video stream data, monitoring audio stream data, and business scenario tags, wherein the monitoring video stream data and the monitoring audio stream data are associated. The acquisition module is used to acquire the corresponding initial model orchestration graph from the predefined model orchestration rule library based on the business scenario label; the initial model orchestration graph is a directed acyclic graph, the initial model orchestration graph includes multiple nodes and directed edges connecting the nodes, the nodes include lightweight single-modal models and / or multimodal large models, and the directed edges are used to indicate data flow and triggering logic; A simplification module is used to obtain available computing resource information of the system and simplify the initial model orchestration diagram based on the available computing resource information to obtain an executable model orchestration diagram; The processing module is used to load and instantiate all nodes in the executable model orchestration diagram, and based on the executable model orchestration diagram, schedule the monitoring video stream data and the audio stream data to flow through each of the nodes for processing in sequence; The generation module is used to generate a job violation identification report based on the processing results of the output nodes of the executable model orchestration diagram.

9. A computing device, characterized in that, include: Memory, used to store programs; A processor for loading the program to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method of any one of claims 1-7.

Images