Work area detection method, device, apparatus, and storage medium

Abstract

The present application relates to the technical field of image processing, and more particularly to a work area detection method, device and equipment and storage medium, wherein a work image of a work area is acquired; the work image is detected by a work detection model to obtain a relationship triple, the relationship triple being used to indicate a spatial relationship between a worker and a load carrier in the work image; a graph structure corresponding to the worker is constructed according to the relationship triple and a prior height of the load carrier; the graph structure is classified by a graph classification model to obtain a classification result used to indicate whether the worker is in an aerial work state, compared with the prior art, the present application can improve the inference accuracy of the aerial work identification task, provide reliable technical support for the safety supervision of aerial work, and reduce the safety risks caused by false detection or missed detection.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for detecting a work area, a device for detecting a work area, a computer device, and a computer-readable storage medium. Background Technology

[0002] Falls from heights are a major safety hazard in industries such as construction, power maintenance, and wind power operation and maintenance. Current technologies often rely on manual inspections or image recognition methods based on fixed thresholds to determine whether workers are in a high-altitude environment. These methods struggle to accurately perceive spatial relationships in complex scenarios and are easily affected by perspective, lighting, and occlusion, resulting in high false positive and false negative rates. Summary of the Invention

[0003] This invention provides a method, device, computer equipment, and computer-readable storage medium for detecting work areas, which can improve the inference accuracy of high-altitude work identification tasks, provide reliable technical support for the safety supervision of high-altitude work, and reduce safety risks caused by false detection or missed detection.

[0004] On one hand, the work area detection method provided by the present invention includes:

[0005] Obtain the work image of the work area;

[0006] The operation detection model detects the operation image and obtains relation triples, which are used to indicate the spatial relationship between the workers and the load in the operation image.

[0007] Based on the relation triples and the prior height of the load, construct a graph structure corresponding to the workers;

[0008] The graph structure is classified using a graph classification model to obtain classification results that indicate whether workers are in a high-altitude working state.

[0009] Secondly, the work area detection device provided by the present invention includes:

[0010] The image acquisition module is used to acquire images of the work area.

[0011] The relationship acquisition module is used to detect the work image through the work detection model and obtain relationship triples. The relationship triples are used to indicate the spatial relationship between the workers and the load in the work image.

[0012] The graph construction module is used to construct a graph structure corresponding to the workers based on the relation triples and the prior height of the carrier;

[0013] The graph classification module is used to classify graph structures using a graph classification model to obtain classification results that indicate whether workers are in a high-altitude work state.

[0014] Optionally, in one embodiment, the graph construction module is used to use workers as personnel nodes, loads as object nodes, prior height as height nodes, and establish evidence edges between personnel nodes and object nodes, as well as constraint edges between object nodes and height nodes; the confidence level corresponding to the relation triple is used as the weight of the evidence edge, and the weight of the constraint edge is configured to a preset constant value to obtain the graph structure.

[0015] Optionally, in one embodiment, the graph classification module is used to perform embedding processing on each node in the graph structure to obtain the node embedding representation of each node; to perform multi-head attention enhancement on the node embedding representation of each node through the graph classification model to obtain the node enhanced representation of each node; to perform attention weighted aggregation on the node enhanced representation according to the weight of the edges in the graph structure through the graph classification model to obtain the aggregated representation of each node; to generate a graph embedding representation according to the aggregated representation of each node through the graph classification model; and to classify according to the graph embedding representation through the graph classification model to obtain a classification result used to indicate whether the worker is in a high-altitude work state.

[0016] Optionally, in one embodiment, the work area detection device provided by the present invention further includes a basis generation module, used to obtain the attention weight corresponding to the aggregate representation of each node; backtrack along the path corresponding to the highest attention weight to determine the key node combination that affects the classification result; and generate an interpretable classification basis corresponding to the classification result based on the key node combination.

[0017] Optionally, in one embodiment, the job area detection device provided by the present invention further includes a model training module, used to acquire sample job images and acquire labeled relation triples of the labeled sample job images; acquire prompt information for instructing the visual language big model to extract relation triples from the input images; input the prompt information and sample job images into the visual language big model to obtain the output sample relation triples; and fine-tune the visual language big model according to the difference between the sample relation triples and the labeled relation triples to obtain the job detection model.

[0018] Optionally, in one embodiment, the model training module is used to load a large visual language model using a preset bit-width quantization; based on the difference between the sample relation triplet and the label relation triplet, the quantized large visual language model is fine-tuned using low-rank adaptation to obtain the job detection model.

[0019] Optionally, in one embodiment, the model training module is used to perform low-rank adaptation fine-tuning on at least one of the visual encoder projection layer, QKV vector generation layer, language model attention layer, and feedforward network layer in the quantized visual language large model based on the difference between the sample relation triplet and the label relation triplet, so as to obtain the job detection model.

[0020] Thirdly, the computer device provided by the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the working area detection method provided by the present invention.

[0021] Fourthly, the computer-readable storage medium provided by the present invention stores a computer program, which, when executed by a processor, implements the working area detection method provided by the present invention.

[0022] This invention provides a work area detection scheme, which involves acquiring a work area image; detecting the work image using a work detection model to obtain relation triples, which indicate the spatial relationship between workers and load-bearing objects in the work image; constructing a graph structure corresponding to the workers based on the relation triples and the prior height of the load-bearing objects; and classifying the graph structure using a graph classification model to obtain a classification result indicating whether the workers are in a high-altitude work state. In this way, by deeply fusing real-time observation information and prior physical knowledge to construct a graph structure corresponding to the workers, the high-altitude work identification task is transformed into a semantic reasoning process of the graph structure. This not only enhances the model's ability to understand spatial relationships in complex scenarios but also significantly improves the reasoning accuracy of the classification results, providing reliable technical support for high-altitude work safety supervision and effectively reducing safety risks caused by false detections or missed detections. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the work area detection method provided in an embodiment of the present invention;

[0025] Figure 2 This is an example diagram of the graph structure provided in the embodiments of the present invention;

[0026] Figure 3 This is another example diagram of the graph structure provided in the embodiments of the present invention;

[0027] Figure 4 This is another flowchart illustrating the work area detection method provided in this embodiment of the invention;

[0028] Figure 5 This is a schematic diagram of the working area detection device provided in an embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0030] To make the technical problems solved, the technical solutions, and the beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0031] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0032] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0033] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0034] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0035] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0036] This invention provides a work area detection method, a work area detection device, a computer device, and a computer-readable storage medium. The work area detection method can be executed by the work area detection device or by a computer device integrating the work area detection device. The method involves: acquiring a work image of the work area; detecting the work image using a work detection model to obtain relation triples, which indicate the spatial relationship between the worker and the load in the work image; constructing a graph structure corresponding to the worker based on the relation triples and the prior height of the load; and classifying the graph structure using a graph classification model to obtain a classification result indicating whether the worker is in a high-altitude work state.

[0037] Please refer to Figure 1 This is a flowchart illustrating a work area detection method disclosed in an embodiment of the present invention, as shown below. Figure 1 As shown, the process of this work area detection method can be as follows:

[0038] In S110, the work image of the work area is acquired.

[0039] The work area includes, but is not limited to, construction sites, bridge construction areas, power facility maintenance areas, high-altitude cleaning areas, and other places where there is a risk of working at height.

[0040] The images of the operation are captured by visual acquisition devices deployed in the operation area. These visual acquisition devices can be fixed surveillance cameras, PTZ cameras, etc., or wearable cameras deployed on mobile devices, camera devices carried by drones, etc.

[0041] In practical implementation, the selection and deployment of visual acquisition equipment must cover key work locations to ensure that personnel and objects are fully represented in the acquired images. Furthermore, the resolution and frame rate of the acquired images should meet the basic requirements for work behavior recognition; for example, the resolution should be no less than 720p and the frame rate no less than 15fps to ensure the accuracy and real-time performance of subsequent image analysis. In addition, to adapt to complex lighting and occlusion scenarios, imaging equipment with infrared or depth sensing capabilities can be used to enhance environmental robustness.

[0042] The following embodiments illustrate the method for detecting the work area provided by the present invention using a computer device.

[0043] The computer equipment, following a configured image acquisition strategy, acquires images of the work area using visual acquisition devices located within the work area, and these images are recorded as work images. The image acquisition strategy includes settings such as acquisition frequency and resolution. For example, it automatically increases resolution and supplemental lighting intensity at night or in low-light conditions, while reducing resolution during the day or in well-lit conditions to save bandwidth and storage resources, and so on.

[0044] In S120, the work image is detected by the work detection model to obtain relation triples, which are used to indicate the spatial relationship between the workers and the load in the work image.

[0045] This invention provides a job detection model, which is trained based on a deep learning framework and configured to identify worker and carrier instances in an input image and output a spatial relationship triplet between the two.

[0046] It should be noted that the embodiments of the present invention do not limit the specific network structure of the job detection model. For example, it can be based on object detection architectures such as Faster R-CNN, YOLO, or Mask R-CNN, and introduce a relation reasoning module to output spatial relation triples of "worker - relation predicate - carrier". Among them, the relation predicate includes semantic tags such as "standing", "leaning", "suspended" and "close to", which are used to accurately describe the actual spatial relationship between the worker and the carrier, such as "standing on scaffolding", "leaning on guardrail" or "suspended on safety rope", etc.

[0047] Furthermore, this embodiment of the invention does not specifically limit the training method of the work detection model. Supervised learning can be adopted, using a dataset of sample work images labeled with triplet relationships to ensure that the model can accurately learn the spatial relationship patterns between workers and load-bearing objects. Moreover, data augmentation processing can be applied to the sample work images, such as motion blur, random lighting adjustment, viewpoint transformation, and occlusion simulation, to improve the model's generalization ability in complex construction site scenarios.

[0048] Accordingly, after acquiring the work area image, the computer equipment inputs the image into a pre-trained work detection model. The model extracts instances of workers and load-bearing objects from the image, analyzes their spatial relationships, and outputs corresponding relationship triples and confidence scores. These triples indicate the spatial relationship between workers and load-bearing objects in the image. Understandably, during actual operations, the amount of data in the relationship triples output by the work detection model will dynamically adjust with the number of workers and load-bearing objects in the work area, exhibiting complex many-to-many, many-to-one, or one-to-many spatial relationships. For example, a single worker may simultaneously be "standing" and "leaning" against multiple load-bearing objects, while multiple workers on the same scaffold may be associated with it in either a "standing" or "suspended" manner, and so on.

[0049] In addition, computer devices can perform post-processing based on relation triples and their confidence levels, including but not limited to:

[0050] (1) Remove relation triples with a confidence level lower than 0.05;

[0051] (2) Merge the triplet of the same person-the same carrier-the same space relationship by taking the maximum confidence level.

[0052] In S130, a graph structure corresponding to the operator is constructed based on the relation triples and the prior height of the load.

[0053] The prior height of a load-bearing object refers to the vertical height value of the load-bearing object's spatial location, which is pre-set in the work scenario or obtained through measurement, such as a scaffolding platform 2.5 meters above the ground or the bottom of a suspended platform 4 meters above the ground.

[0054] A graph structure is a data structure consisting of nodes and edges connecting the nodes, used to represent relationships between objects.

[0055] In this embodiment of the invention, after acquiring the relation triples of the work image, the computer device further acquires the prior height corresponding to the load-bearing object within it. Based on the relation triples and the prior height, a graph structure corresponding to the worker corresponding to the relation triples is constructed. In this way, by fusing real-time observed spatial relationships and prior height information, a joint representation including spatial topology and geometric attributes is formed, achieving refined modeling of the three-dimensional spatial relationship between workers and load-bearing objects in the work scene, providing reliable structured input for subsequent classification.

[0056] Optionally, in one embodiment, a graph structure corresponding to the operator is constructed based on the relation triples and the prior height of the load, including:

[0057] The operator is designated as the operator node, the load-bearing object as the object node, and the prior height as the height node. Evidence edges are established between the operator node and the object node, and constraint edges are established between the object node and the height node.

[0058] The confidence level corresponding to the relation triple is used as the weight of the evidence edge, and the weight of the constraint edge is configured to a preset constant value to obtain the graph structure.

[0059] In constructing the graph structure corresponding to the workers, each worker node is connected to its associated object nodes via evidence edges. The weight of these evidence edges is determined by the confidence level of the corresponding spatial relationship in the relation triplet, reflecting the reliability of the model's judgment on that relationship. Object nodes are connected to their corresponding height nodes via constraint edges, with the weights of these constraint edges set to preset constant values, representing the certainty of prior height information. The resulting graph structure accurately models the spatial relationships of the workers, providing a structured basis for subsequent risk assessment of high-altitude operations. It is understandable that the number of graph structures constructed depends on the number of workers; that is, each worker corresponds to an independent graph structure used to characterize their spatial relationship with the associated load-bearing structures.

[0060] It should be noted that the setting of the preset constant value is not limited in the embodiments of the present invention. It can be configured to 1.0 or other fixed values ​​according to the actual application scenario. For example, when the preset constant value is 1, it represents an inviolable rigid constraint, which means that the height information of the load is absolutely accurate prior knowledge and does not participate in the propagation of uncertainty.

[0061] For example, please refer to Figure 2 , Figure 2 In the graph structure shown, the personnel node P1 is connected to the object node S1 (scaffolding) via an evidence edge with a weight of 0.93, representing a high confidence level of the "standing" relationship. S1 is connected to the height node H1 via a constraint edge with a weight of 1.0, indicating that the platform height is a definite prior value. Meanwhile, the personnel node P1 is connected to another object node S2 (high formwork) via an evidence edge with a weight of 0.76, representing a medium confidence level of the "leaning" relationship. S2 is connected to the height node H2 via a constraint edge with a weight of 1.0, reflecting the certainty of its prior height.

[0062] For example, please refer to Figure 3 , Figure 3 In the graph structure shown, the person node P2 is connected to the object node S3 (ground) through an evidence edge with a weight of 0.98, representing a high confidence judgment of "standing" on the ground; S3 is connected to the height node H3 through a constraint edge with a weight of 1.0, reflecting the prior certainty of the ground height as an absolute reference benchmark.

[0063] In S140, the graph structure is classified using a graph classification model to obtain classification results that indicate whether the workers are in a high-altitude working state.

[0064] In addition to the above-mentioned work detection model, this embodiment of the invention also provides a graph classification model. This graph classification model is built on a graph neural network architecture and is configured to perform end-to-end classification on the input graph structure, outputting a classification result indicating whether the worker is in a high-altitude work state. In specific implementations, there is no limitation on the type of graph neural network architecture used; graph convolutional networks (GCN), graph attention networks (GAT), or graph isomorphic networks (GIN), etc., can be selected.

[0065] Similarly, this embodiment of the invention does not limit the training method of the graph classification model. For example, the sample operation image dataset can be reused to generate corresponding graph structure samples, and the graph structure samples can be labeled based on the labeled label relationship triples. If the label relationship triples indicate that the person is at a height of 2 meters or above, it is labeled as a positive sample of "high-altitude operation"; otherwise, it is labeled as a negative sample of "non-high-altitude operation". Subsequently, the graph classification model is trained using a supervised learning method, with the loss function being the binary cross-entropy loss and L2 regularization added. The training is carried out for 3 to 10 epochs with a learning rate of 1e-3 to 1e-5. Training is stopped when the best F1 score is reached, and the optimal model parameters are saved.

[0066] Accordingly, in this embodiment of the invention, the computer device inputs the acquired graph structure into the trained graph classification model, calculates the classification probability through forward propagation, and if the probability value is greater than or equal to the upper probability threshold, the classification result is "high-altitude operation"; if the probability value is less than the lower probability threshold, it is determined to be "non-high-altitude operation". The specific values ​​of the upper and lower probability thresholds are not limited here and can be adjusted according to the accuracy requirements of the actual application scenario. For example, the upper probability threshold can be set to 0.7 and the lower probability threshold to 0.3 to retain an intermediate uncertainty range for manual verification.

[0067] In practical deployment, the work area detection solution provided by this invention can be integrated into the work area monitoring system to achieve automatic identification and early warning of workers' high-altitude work status. When it is finally determined that a worker is working at height, the monitoring system can be linked to trigger corresponding control measures, such as starting real-time recording, sending alarm information to the supervision platform, or prompting the wearing of protective equipment, etc. This improves the level of work safety management and reduces the risk of accidents. This solution can be widely used in high-risk scenarios such as building construction and power maintenance.

[0068] Optionally, in one embodiment, the graph structure is classified using a graph classification model to obtain a classification result indicating whether the worker is in a high-altitude work state, including:

[0069] The nodes in the graph structure are embedded to obtain the node embedding representation of each node;

[0070] Multi-head attention is applied to the node embedding representation of each node using a graph classification model to obtain the node augmentation representation of each node.

[0071] Based on the weights of the edges in the graph structure, attention-weighted aggregation of the node augmentation representations is performed using a graph classification model to obtain the aggregated representations of each node;

[0072] Based on the aggregated representation of each node, a graph embedding representation is generated using a graph classification model;

[0073] Based on graph embedding representation, classification is performed using a graph classification model to obtain classification results indicating whether workers are in a high-altitude work state.

[0074] In this embodiment of the invention, the graph classification model includes a first graph attention convolutional layer, a second graph attention convolutional layer, a graph readout layer, and a classification head. The first graph attention convolutional layer is configured to perform multi-head attention enhancement on the input node embedding representation to generate an enhanced node representation. The second graph attention convolutional layer is configured to perform attention-weighted aggregation on the enhanced node representation according to the edge weights to obtain an aggregated representation of each node. The graph readout layer is configured to integrate the aggregated representations of each node into a graph embedding representation. The classification head is configured to perform classification based on the graph embedding representation and output the predicted probability of the high-altitude operation status.

[0075] Accordingly, when classifying graph structures using a graph classification model, for a given graph structure, the computer device first performs embedding processing on each node in the graph structure to obtain node embedding representations. For example, learnable embedding vectors can be pre-assigned to all nodes. For instance, 33 categories can be pre-defined, including person nodes, 26 object nodes, and 6 height node types. Each category is initialized with a 32- to 64-dimensional learnable embedding vector, initially set to a random normal distribution. This is then continuously optimized through backpropagation during the training of the graph classification model to obtain the node embedding representations for each type of node. During embedding processing, the node embedding representations for each node in the graph structure can be directly retrieved from a lookup table.

[0076] Subsequently, the computer device inputs the node embedding representations into the first graph attention convolutional layer, and enhances the node embedding representations through a multi-head attention mechanism to generate enhanced node representations for each node. For example, the first graph attention convolutional layer uses four attention heads, with a hidden feature dimension of 128. After ReLU activation, the outputs of the four attention heads are concatenated to obtain a 512-dimensional enhanced node representation for each node.

[0077] Subsequently, the computer device inputs the node augmented representations into the second graph attention convolutional layer. Combining the edge weights in the graph structure, it performs attention-weighted aggregation of the augmented representations of adjacent nodes, while simultaneously reducing the dimensionality of the features to obtain the aggregated representation of each node. For example, the second graph attention convolutional layer employs a single-head attention mechanism, with a hidden feature dimension of 64, and outputs a 64-dimensional aggregated representation of each node after ReLU activation.

[0078] Subsequently, the computer device inputs the aggregated representations of each node into the graph readout layer. Through global pooling, it integrates the aggregated representations of all nodes into a fixed-dimensional graph embedding representation. This graph embedding representation comprehensively depicts the semantic relationships and topological features between nodes in the graph structure, effectively fusing key contextual information from high-altitude operation scenarios. For example, an average pooling strategy is used to aggregate the 64-dimensional aggregated representations of each node into a 64-dimensional graph embedding representation.

[0079] Finally, the computer device embeds the graph representation into the classification head, performs classification mapping through the classification head, and outputs the predicted probability of the high-altitude work status. For example, the classification head consists of two fully connected network layers (with an intermediate dimension of 32), using the ReLU activation function followed by a Dropout layer to prevent overfitting, and finally a Sigmoid activation function to output a predicted probability value between 0 and 1, used to determine whether the worker corresponding to the input graph structure is in a high-altitude work status.

[0080] Optionally, in one embodiment, after classifying according to the graph embedding representation using a graph classification model to obtain a classification result indicating whether the worker is in a high-altitude work state, the method further includes:

[0081] Obtain the attention weights corresponding to the aggregated representation of each node;

[0082] Backtrack along the path corresponding to the highest attention weight to determine the key node combination that affects the classification result;

[0083] Based on the combination of key nodes, an interpretable classification basis corresponding to the classification result is generated.

[0084] In this embodiment of the invention, by introducing an attention weight backtracking mechanism, it is possible to locate the key node combination that contributes the most to the classification decision, thereby generating interpretable evidence corresponding to the classification result and revealing the logical path behind the model's decision.

[0085] In this process, after the computer device completes the classification task of the graph structure through the image classification model, it can further extract the distribution of attention weights among the nodes in the second graph attention convolutional layer, identify the edge corresponding to the highest attention weight and the node combination connected to it; by tracing the semantic role and topological position of the path in the original graph structure, the key node combination that affects the classification result is determined; and finally, an interpretable basis corresponding to the classification result is generated based on the key node combination.

[0086] For example, for a graph structure (with a confidence level of 0.94 for the correspondence triples), the graph classification model ultimately determines that the worker corresponding to this graph structure is in a high-altitude work state. Backtracking using attention weights reveals:

[0087] Classification results ← scaffolding object node (α=0.92) ← personnel node - scaffolding object node edge (α=0.87) ← personnel node (α=0.75) ← scaffolding object node - height node (5, 150) edge (α=1.0);

[0088] The resulting interpretable classification is based on the following: the worker was performing high-altitude work on scaffolding at a height of 5-150 meters with a confidence level of 0.94.

[0089] Alternatively, in one embodiment, the job detection model is obtained as follows:

[0090] Obtain the sample job images and the label relationship triples of the labeled sample job images;

[0091] Obtain prompting information to instruct the visual language model to extract relation triples from the input image;

[0092] Input the prompt information and sample task images into the visual language big model to obtain the output sample relationship triples;

[0093] Based on the differences between the sample relation triplet and the label relation triplet, the large visual language model is fine-tuned to obtain the task detection model.

[0094] This invention provides an optional training method for a work detection model. By using prompts, the visual language model is guided to accurately capture personnel entities and carrier entities in the work area and their spatial semantic relationships, thereby improving the accuracy and robustness of relation triple extraction.

[0095] The Visual-Language Big Model is a deep learning architecture that integrates visual and linguistic modalities, capable of semantically parsing image content and generating natural language descriptions. It obtains cross-modal joint representation capabilities through large-scale image and text pre-training.

[0096] In this embodiment of the invention, the computer device first acquires sample operation images, which may be screenshots from monitoring videos of actual operation scenarios or historical archived images, etc. In addition, it acquires the labeled relation triples of the sample operation images, denoted as label relation triples.

[0097] For example, spatial relation triples can be fully labeled in the following format:

[0098] {

[0099] "persons": [

[0100] {

[0101] "person_id": 0,

[0102] "person_bbox": [x1, y1, x2, y2],

[0103] "relations": [

[0104] {

[0105] "predicate": "standing_on",

[0106] "object_name": "scaffold",

[0107] "object_bbox": [x1, y1, x2, y2],

[0108] "confidence": 1.0

[0109] } ]

[0111] } ]

[0113] }

[0114] In this document, "person_id" corresponds to the unique identifier of the worker, "person_bbox" represents the bounding box coordinates of the worker in the format [x1, y1, x2, y2], i.e., the coordinates of the top left and bottom right corners; the "relations" list contains descriptions of the spatial relationship between the worker and the load-bearing object, "predicate" represents the relationship predicate, such as "standing_on" for standing, "leaning_on" for leaning, "hanging_from" for hanging, "near" for being close to, etc.; "object_name" represents the type of load-bearing object, such as "scaffold" for scaffolding, "gondola" for hanging basket, etc.; "object_bbox" is the bounding box coordinate of the load-bearing object, also in the format [x1, y1, x2, y2]; confidence is the label confidence level, used to reflect the reliability of the label. This label format can support the expression of complex scenarios with multiple workers and multiple relationships.

[0115] In addition, the computer equipment also obtains prompts to instruct the visual language big data model to extract relation triples from the input image. These prompts describe the task requirements in natural language, such as, "Please carefully observe this high-altitude operation monitoring image and identify all construction workers in the image. For each worker, analyze their spatial relationship with the surrounding load-bearing structures, using only four relation predicates: standing, near, hanging, and leaning. The load-bearing structure must be one of the following: scaffolding, suspended platform, ... work platform. The following structure must be strictly followed without adding any unnecessary explanations or code block identifiers: {The specific output structure is omitted here; for example, the same data structure as the label relation triples can be used}."

[0116] Subsequently, the computer device inputs the aforementioned prompt information and sample operation images into the visual language big model, driving it to perform reasoning based on the image content and task instructions, outputting sample relationship triples that conform to the specified format, and fine-tuning the visual language big model based on the difference between the sample relationship triples and the corresponding label relationship triples, to obtain an operation detection model that can accurately identify the spatial relationship between the operator and the load.

[0117] Optionally, in one embodiment, before inputting the prompt information and sample task image into the visual language large model to obtain the output sample relationship triplet, the method further includes:

[0118] Large visual language models are loaded using preset bit-width quantization.

[0119] Based on the differences between sample relation triples and label relation triples, the large visual language model is fine-tuned to obtain the task detection model, including:

[0120] Based on the differences between the sample relation triplet and the label relation triplet, the quantized visual language model is fine-tuned using low-rank adaptation to obtain the job detection model.

[0121] To improve job detection efficiency and reduce computational resource consumption, this embodiment of the invention employs quantization techniques to compress the model parameter size, while combining a low-rank adaptation (LoRA) strategy to maintain model performance while reducing the number of trainable parameters.

[0122] Specifically, the computer device uses a preset bit width to quantize and load the large visual language model, which means converting the model weights of the large visual language model from high-precision floating-point numbers to low-precision integer representations, thereby significantly reducing memory usage and computational overhead. For example, the computer device can use 8-bit integer quantization to compress the original 32-bit floating-point weights into 8-bit integers, achieving a four-fold storage compression ratio while ensuring controllable loss of model inference accuracy. Alternatively, it can use 4-bit integer quantization to compress the original 32-bit floating-point weights into 4-bit integers, further achieving an eight-fold storage compression ratio while balancing accuracy and efficiency, significantly improving the feasibility of deploying the model on edge devices.

[0123] Building upon this foundation, a low-rank adaptation fine-tuning strategy is further introduced. Low-rank decomposition matrices are introduced as trainable parameters alongside the weight parameters of the quantized visual language large model. During the fine-tuning process, these low-rank decomposition matrices are updated only based on the differences between the sample relation triplet and the label relation triplet, while the weights of the visual language large model itself are frozen. This significantly reduces training costs while preserving the original performance of the model, resulting in a job detection model that combines high accuracy and high efficiency.

[0124] In addition, after deploying the job detection model, for graph structures whose probability values ​​are between the upper and lower probability thresholds, the corresponding job image, relation triplet, and probability value are pushed to the manual review queue for review. Based on the review results, incremental samples corresponding to the job detection model are generated for subsequent periodic incremental fine-tuning of the job detection model.

[0125] Optionally, in one embodiment, based on the difference between the sample relation triplet and the label relation triplet, the quantized visual language large model is fine-tuned using low-rank adaptation to obtain a job detection model, including:

[0126] Based on the difference between the sample relation triplet and the label relation triplet, at least one layer in the quantized visual language model, including the visual encoder projection layer, the QKV vector generation layer, the attention layer of the language model, and the feedforward network layer, is fine-tuned using low-rank adaptation to obtain the job detection model.

[0127] In this embodiment of the invention, the computer device can perform low-rank adaptation fine-tuning on at least one layer of the visual encoder projection layer, QKV vector generation layer, attention layer of the language model, and feedforward network layer in the quantized visual language large model based on the difference between the sample relation triplet and the label relation triplet. This reduces the number of training parameters while accurately adjusting the model's ability to model multimodal semantic alignment.

[0128] Furthermore, after obtaining the job detection model, the computer equipment can also use Temperature Scaling or Platt Scaling on an independent validation set to calibrate the confidence level of the job detection model's output, so that the confidence level of its output has a statistically reliable probabilistic interpretation, thereby improving the decision-making credibility of the model in real-world application scenarios.

[0129] Please refer to Figure 4 , Figure 4 This is another flowchart illustrating the work area detection method provided in this embodiment of the invention, as shown below. Figure 4 As shown, the process of this work area detection method can be as follows:

[0130] Training phase: The computer equipment first acquires images of the real work area, and uses the acquired real work area images as sample work images.

[0131] The computer device receives the structured annotation data for the sample work images as input, forms the corresponding label relationship triples, and performs data augmentation processing on the sample work images (such as motion blur, random lighting adjustment, viewpoint transformation and occlusion simulation, etc.), and finally obtains a dataset consisting of sample work images and label relationship triples.

[0132] The computer equipment uses a preset bit-width quantization to load the visual large language model.

[0133] The computer equipment uses fixed prompts and combines them with the dataset to perform low-rank adaptation fine-tuning on the visual large language model to obtain the job detection model.

[0134] Reasoning stage:

[0135] The computer equipment acquires the work area images that need to be detected, inputs them into the work detection model along with fixed prompts, performs relational reasoning through the work detection model, and outputs relational triples and corresponding confidence scores.

[0136] The computer equipment combines the prior heights of different loads recorded in the prior knowledge base with the height of the loads to construct a corresponding graph structure for each worker's relation triplet.

[0137] The computer equipment inputs the graph structure of each worker into the graph classification model, and outputs the probability of each worker being in a high-altitude work state through the graph classification model. In addition, the computer equipment also generates an interpretable classification basis corresponding to the high-altitude work probability, and outputs the high-altitude work probability and the interpretable classification basis together.

[0138] The computer equipment also collects graph structures whose high-altitude operation probability is between the upper probability threshold and the lower probability threshold. The operation image, relation triplet, and probability value corresponding to the graph structure are pushed to the manual review queue for review. Based on the review results, incremental samples corresponding to the operation detection model are generated for subsequent periodic incremental fine-tuning of the operation detection model.

[0139] It should be noted that for any parts not described in detail in the embodiments of the present invention, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0140] As described above, the work area detection scheme provided by this invention acquires work images of the work area; detects the work images using a work detection model to obtain relation triples, which indicate the spatial relationship between workers and load-bearing objects in the work images; constructs a graph structure corresponding to the workers based on the relation triples and the prior height of the load-bearing objects; and classifies the graph structure using a graph classification model to obtain a classification result indicating whether the workers are in a high-altitude work state. Thus, by deeply fusing real-time observation information and prior physical knowledge to construct a graph structure corresponding to the workers, the high-altitude work identification task is transformed into a semantic reasoning process of the graph structure. This not only enhances the model's ability to understand spatial relationships in complex scenarios but also significantly improves the reasoning accuracy of the classification results, providing reliable technical support for high-altitude work safety supervision and effectively reducing safety risks caused by false detections or missed detections.

[0141] To facilitate better implementation of the above-described work area detection method, this embodiment of the invention also provides a corresponding work area detection device. The meanings of the terms used are the same as in the above-described work area detection method; for specific implementation details, please refer to the descriptions in the above method embodiments.

[0142] Please refer to Figure 5 The work area detection device may include a behavior image acquisition module 210, a relationship acquisition module 220, a graph construction module 230, and a graph classification module 240. Detailed descriptions of each functional module are as follows:

[0143] Image acquisition module 210 is used to acquire work images of the work area;

[0144] The relationship acquisition module 220 is used to detect the work image through the work detection model and obtain relationship triples. The relationship triples are used to indicate the spatial relationship between the workers and the load in the work image.

[0145] Graph construction module 230 is used to construct a graph structure corresponding to the operator based on the relation triples and the prior height of the carrier;

[0146] The graph classification module 240 is used to classify graph structures using a graph classification model to obtain classification results that indicate whether workers are in a high-altitude work state.

[0147] Optionally, in one embodiment, the graph construction module 230 is used to use the workers as personnel nodes, the load as object nodes, the prior height as height nodes, and to establish evidence edges between personnel nodes and object nodes, as well as constraint edges between object nodes and height nodes; the confidence level corresponding to the relation triple is used as the weight of the evidence edge, and the weight of the constraint edge is configured to a preset constant value to obtain the graph structure.

[0148] Optionally, in one embodiment, the graph classification module 240 is used to perform embedding processing on each node in the graph structure to obtain the node embedding representation of each node; to perform multi-head attention enhancement on the node embedding representation of each node through the graph classification model to obtain the node enhanced representation of each node; to perform attention weighted aggregation on the node enhanced representation according to the weight of the edges in the graph structure through the graph classification model to obtain the aggregated representation of each node; to generate a graph embedding representation according to the aggregated representation of each node through the graph classification model; and to classify according to the graph embedding representation through the graph classification model to obtain a classification result used to indicate whether the operator is in a high-altitude working state.

[0149] Optionally, in one embodiment, the work area detection device provided by the present invention further includes a basis generation module, used to obtain the attention weight corresponding to the aggregate representation of each node; backtrack along the path corresponding to the highest attention weight to determine the key node combination that affects the classification result; and generate an interpretable classification basis corresponding to the classification result based on the key node combination.

[0150] Optionally, in one embodiment, the job area detection device provided by the present invention further includes a model training module, used to acquire sample job images and acquire labeled relation triples of the labeled sample job images; acquire prompt information for instructing the visual language big model to extract relation triples from the input images; input the prompt information and sample job images into the visual language big model to obtain the output sample relation triples; and fine-tune the visual language big model according to the difference between the sample relation triples and the labeled relation triples to obtain the job detection model.

[0151] Optionally, in one embodiment, the model training module is used to load a large visual language model using a preset bit-width quantization; based on the difference between the sample relation triplet and the label relation triplet, the quantized large visual language model is fine-tuned using low-rank adaptation to obtain the job detection model.

[0152] Optionally, in one embodiment, the model training module is used to perform low-rank adaptation fine-tuning on at least one of the visual encoder projection layer, QKV vector generation layer, language model attention layer, and feedforward network layer in the quantized visual language large model based on the difference between the sample relation triplet and the label relation triplet, so as to obtain the job detection model.

[0153] Specific limitations regarding the work area detection device can be found in the limitations of the work area detection method described above, and will not be repeated here. Each module in the aforementioned work area detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0154] In one embodiment, a computer device is provided, the internal structure of which can be shown as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface connects to external wireless clients, providing wireless network access services to the connected clients. When executed by the processor, the computer program implements the work area detection method provided by this invention.

[0155] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the job area detection method described in the above embodiment, for example:

[0156] Obtain the work image of the work area;

[0157] The operation detection model detects the operation image and obtains relation triples, which are used to indicate the spatial relationship between the workers and the load in the operation image.

[0158] Based on the relation triples and the prior height of the load, construct a graph structure corresponding to the workers;

[0159] The graph structure is classified using a graph classification model to obtain classification results that indicate whether workers are in a high-altitude working state.

[0160] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the job area detection method described in the above embodiment, for example:

[0161] Obtain the work image of the work area;

[0162] The operation detection model detects the operation image and obtains relation triples, which are used to indicate the spatial relationship between the workers and the load in the operation image.

[0163] Based on the relation triples and the prior height of the load, construct a graph structure corresponding to the workers;

[0164] The graph structure is classified using a graph classification model to obtain classification results that indicate whether workers are in a high-altitude working state.

[0165] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0166] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0167] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0168] It should be noted that when the above embodiments of the present invention are applied to specific products or technologies, and user-related data is involved, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A method for detecting a work area, characterized in that, include: Obtain the work image of the work area; The operation image is detected by the operation detection model to obtain relation triples. The relation triples are used to indicate the spatial relationship between the operator and the load in the operation image. The spatial relationship is standing, leaning, hanging, or close. For duplicate relation triples, merge the triples by taking the maximum confidence score; The operator is designated as a personnel node, the load-bearing object as an object node, and the prior height of the load-bearing object as a height node. Evidence edges are established between the personnel node and the object node, and constraint edges are established between the object node and the height node. The confidence level of the spatial relationship in the relation triple is used as the weight of the evidence edge, and the weight of the constraint edge is configured to a preset constant value to obtain the graph structure corresponding to the operator. The preset constant value is used to represent the certainty of the prior height. The graph structure is classified using a graph classification model to obtain a classification result indicating whether the worker is in a high-altitude work state.

2. The work area detection method according to claim 1, characterized in that, The step of classifying the graph structure using a graph classification model to obtain a classification result indicating whether the worker is in a high-altitude work state includes: The nodes in the graph structure are embedded to obtain the node embedding representation of each node; The node embedding representation of each node is enhanced by multi-head attention through the graph classification model to obtain the node enhanced representation of each node. Based on the weights of the edges in the graph structure, the enhanced representations of the nodes are aggregated by attention weighting using the graph classification model to obtain the aggregated representations of each node; Based on the aggregated representation of each node, a graph embedding representation is generated using the graph classification model; Based on the graph embedding representation, the graph classification model is used to classify the data to obtain a classification result indicating whether the worker is in a high-altitude work state.

3. The work area detection method according to claim 2, characterized in that, After classifying the data according to the graph embedding representation using the graph classification model to obtain a classification result indicating whether the worker is in a high-altitude work state, the method further includes: Obtain the attention weights corresponding to the aggregated representation of each node; Backtrack along the path corresponding to the highest attention weight to determine the key node combination that affects the classification result; Based on the combination of key nodes, an interpretable classification basis corresponding to the classification result is generated.

4. The work area detection method according to claim 1, characterized in that, The job detection model is obtained as follows: Obtain sample job images and obtain the labeled triplet of the sample job images; Obtain prompting information to instruct the visual language model to extract relation triples from the input image; The prompt information and the sample task image are input into the visual language big model to obtain the output sample relationship triplet; Based on the difference between the sample relation triplet and the label relation triplet, the visual language big model is fine-tuned to obtain the task detection model.

5. The work area detection method according to claim 4, characterized in that, Before inputting the prompt information and the sample task image into the visual language big model to obtain the output sample relation triplet, the method further includes: The large visual language model is loaded using preset bit-width quantization; The step of fine-tuning the visual language model based on the difference between the sample relationship triplet and the label relationship triplet to obtain the task detection model includes: Based on the difference between the sample relation triplet and the label relation triplet, the quantized visual language large model is fine-tuned using low-rank adaptation to obtain the task detection model.

6. The work area detection method according to claim 5, characterized in that, The step of performing low-rank adaptation fine-tuning on the quantized visual language large model based on the difference between the sample relationship triplet and the label relationship triplet to obtain the task detection model includes: Based on the difference between the sample relation triplet and the label relation triplet, at least one layer of the visual encoder projection layer, QKV vector generation layer, language model attention layer, and feedforward network layer in the quantized visual language big model is fine-tuned using low-rank adaptation to obtain the job detection model.

7. A work area detection device, characterized in that, The work area detection device includes: The image acquisition module is used to acquire images of the work area. The relationship acquisition module is used to detect the work image through the work detection model to obtain relationship triples. The relationship triples are used to indicate the spatial relationship between the worker and the load in the work image. The spatial relationship is standing, leaning, hanging, or close. The module also merges duplicate relationship triples by taking the maximum confidence score. The graph construction module is used to treat the workers as personnel nodes, the load as object nodes, and the prior height of the load as height nodes. It establishes evidence edges between the personnel nodes and the object nodes, as well as constraint edges between the object nodes and the height nodes. The module uses the confidence level of the spatial relationship in the relation triples as the weight of the evidence edges and configures the weights of the constraint edges to preset constant values, thereby obtaining a graph structure corresponding to the workers. The preset constant values ​​represent the certainty of the prior height. The graph classification module is used to classify the graph structure using a graph classification model to obtain a classification result indicating whether the worker is in a high-altitude work state.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the working area detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the working area detection method according to any one of claims 1 to 6.

Images