Power transmission and transformation project construction site three-dimensional risk identification method fusing large model and YOLOv8

Abstract

The invention discloses a power transmission and transformation project construction site three-dimensional risk identification method fusing a large model and YOLOv8, and the method comprises the following steps: achieving the multi-target real-time detection of a construction site based on a YOLOv8 model, and obtaining a two-dimensional detection result set; realizing conversion from the two-dimensional detection result to a three-dimensional space through coordinate mapping; converting the three-dimensional dynamic scene data into structured text description, and generating a scene text; performing feature extraction, fusion and optimization on the scene text and the three-dimensional dynamic scene data to obtain multi-modal features of fused memory information; and based on the optimized large model, carrying out comprehensive risk calculation on the multi-modal features. According to the invention, intelligent risk identification from two-dimensional visual perception to three-dimensional semantic understanding is realized, and the intelligent level of construction site safety management and the accuracy and timeliness of risk prevention and control are improved.

Description

Technical Field

[0001] This invention relates to the field of risk identification technology at power construction sites, specifically to a three-dimensional risk identification method for power transmission and transformation engineering construction sites that integrates a large model with YOLOv8. Background Technology

[0002] Currently, power transmission and transformation construction sites present complex and dynamic environments, with numerous potential risks such as high-altitude operations, heavy machinery handling, and proximity to high-voltage equipment, posing a severe challenge to personnel and equipment safety. Traditional safety monitoring primarily relies on manual inspections and two-dimensional video analysis from fixed cameras. This method is not only inefficient and labor-intensive but also has significant limitations in perception. Manual monitoring is prone to missed reports due to fatigue or negligence, while conventional two-dimensional visual recognition technology struggles to accurately determine the actual spatial location, distance relationships, and complex interactive behaviors of targets, failing to effectively identify and warn of complex risk scenarios such as personnel entering live electrical zones or insufficient safety distances. With the advancement of smart construction site development, the industry urgently needs an intelligent risk identification method that can bridge the gap between two-dimensional perception and three-dimensional semantic understanding, enabling real-time, accurate, and automated assessment and control of risks at construction sites, thereby fundamentally improving safety management levels. Summary of the Invention

[0003] The purpose of this invention is to provide a three-dimensional risk identification method for power transmission and transformation engineering construction sites that integrates large models and YOLOv8. This method not only significantly improves the efficiency and coverage of risk identification, but also achieves deep semantic understanding and accurate three-dimensional positioning of complex risk scenarios. It effectively solves the problems of traditional methods that rely on manual labor, have slow response times, and cannot effectively identify complex violations.

[0004] The technical solution of the present invention to solve the above-mentioned technical problems is as follows:

[0005] A method for three-dimensional risk identification at power transmission and transformation engineering construction sites that integrates large-scale models and YOLOv8 includes the following steps:

[0006] S1: Real-time multi-target detection at the construction site is achieved based on the YOLOv8 model, resulting in a two-dimensional detection result set;

[0007] S2: The two-dimensional detection results are converted to three-dimensional space through coordinate mapping to obtain three-dimensional dynamic scene data containing three-dimensional position information;

[0008] S3: Convert the three-dimensional dynamic scene data into a structured text description to generate scene text;

[0009] S4: The scene text generated in step S3 and the 3D dynamic scene data obtained in step S2 are used to obtain a unified representation vector of multimodal features after feature extraction and fusion; the unified representation vector of multimodal features is introduced into a memory mechanism to obtain multimodal features with fused memory information; based on the optimized large model, a comprehensive risk calculation is performed on the multimodal features.

[0010] Furthermore, step S1 specifically involves extracting video frame images based on the YOLOv8 model, extracting multi-scale features through the backbone network, performing feature fusion through the neck network, and finally outputting a prediction tensor by the detection head. By decoding the prediction results and performing non-maximum suppression processing, a set of detection results containing target category, confidence level, and two-dimensional bounding box coordinates is obtained.

[0011] Furthermore, step S2 specifically involves first calculating the pixel coordinates of the midpoint of the bottom edge of the target bounding box, and then obtaining the normalized coordinates after distortion correction by the camera intrinsic parameter matrix; next, using a depth estimation network to obtain the target depth information, and combining the camera extrinsic parameters to transform the 3D points in the camera coordinate system to the world coordinate system.

[0012] Furthermore, in step S2, for each detected target d i Calculate the pixel coordinates p of the midpoint of the bottom edge of its bounding box. i =(u i ,v i ),

[0013] v i =y 2i

[0014] Among them, u i v is the x-coordinate of pixels. i The vertical coordinate is the pixel coordinate.

[0015] Using the camera's intrinsic parameter matrix K and distortion coefficients, the pixel coordinates p i After distortion correction, the coordinates p′ in the normalized camera coordinate system are obtained. i =(x′) i ,y′ i ),

[0016]

[0017] Where K is a 3x3 camera intrinsic parameter matrix, containing the focal length f x ,f y and principal point coordinates c x ,c y ;

[0018] A pre-trained depth estimation network is used to predict the depth value of each pixel in an image. Then target d i depth Take the average depth value of the bottom region of its bounding box.

[0019]

[0020] Among them, R bottom It is a region at the bottom of the bounding box, where N is the number of pixels within that region, and the 3D coordinates are... Calculated in the camera coordinate system using the following formula:

[0021]

[0022] Using the camera's extrinsic parameters, namely the rotation matrix R and the translation vector T, the three-dimensional points in the camera coordinate system are... Transform to world coordinate system

[0023]

[0024] Obtain 3D dynamic scene data D with 3D position information 3D ,

[0025]

[0026] Furthermore, step S3 specifically involves generating individual descriptions containing target categories and three-dimensional coordinates based on a preset template, calculating the Euclidean distance between targets and the spatial relationship between targets and restricted areas, and aggregating all descriptions to form a complete scene text containing object attributes and spatial relationships.

[0027] Furthermore, in step S3, a scene description template T is defined, which specifies the elements that need to be described.

[0028] For 3D dynamic scene data D 3D For each target in the list, generate its individual description desc i ,

[0029]

[0030] Calculate the Euclidean distance dist between target i and target j. ij :

[0031]

[0032] Aggregate all individual descriptions and relationship descriptions to form the scene text S that is ultimately input into the large model.

[0033] S = T(desc1,desc2,...,desc) 12 Personnel 1 is inside restricted area A1.

[0034] Furthermore, in step S4, the scene text S is encoded by a text encoder f. t (·) Obtain the semantic feature vector

[0035]

[0036] In the formula, f is a semantic feature vector; t (·) is a text encoder function used to extract language features;

[0037] 3D dynamic scene data D 3D via visual encoder f v (·) is transformed into spatial feature vectors

[0038]

[0039] In the formula, f is a spatial eigenvector; v (·) represents the visual encoder function used to extract spatial features;

[0040] Modal alignment is performed using the fusion module:

[0041]

[0042] In the formula, α is the unified representation vector of multimodal features after fusion; α is the modal weight coefficient.

[0043] Furthermore, in step S4, optimizing the large model specifically involves using the safety procedure knowledge base K. safe Embedded alignment enables large models to possess semantic priors for power safety rules.

[0044]

[0045] In the formula, To constrain the model to comply with safety procedures, the language generation loss is used; i For safe behavior labels; S i Let K be the i-th scene text; safe This is a knowledge base for power safety regulations; θ represents the model parameters.

[0046] Constructing an alignment loss between 3D features and text semantics:

[0047]

[0048] In the formula, The loss is the alignment loss between text semantics and 3D space.

[0049] We introduce a fine-tuning strategy based on reward modeling, where the reward function is defined as:

[0050] r = η1Acc rule +η2Cons logic +η3Coh lang

[0051] In the formula, r is the reward score for the sample output generated by the model during training; Acc rule To output the accuracy of whether safety procedures are met; Cons logic Score for semantic and logical consistency; Coh lang η1, η2, and η3 are the language fluency indicators; η1, η2, and η3 are the weights of each indicator.

[0052] Final composite optimization objective:

[0053]

[0054] In the formula, The overall optimization objective of the model.

[0055] Furthermore, in step S4, obtaining the multimodal features of fused memory information specifically involves:

[0056] Unified representation vector of fused multimodal features The Transformer decoding structure is calculated layer by layer:

[0057]

[0058] In the formula, h (l) is the hidden state vector of the l-th layer; MHAtn(·) is the multi-head attention mechanism module, used to capture the semantic dependencies between different spatial objects; FFN(·) is the feedforward network layer, which realizes non-linear feature transformation;

[0059] A semantic memory module is introduced to enhance the model's understanding of temporal scenarios:

[0060]

[0061] In the formula, m new (t) represents the newly generated memory vector at the current moment; γ is the forgetting factor, controlling the proportion of old memories retained; m old (t-1) is the memory vector from the previous time step;

[0062] Cumulative updates to the historical semantic memory:

[0063]

[0064] In the formula, It serves as a historical semantic memory bank, accumulating semantic vectors from various moments in time.

[0065] Retrieve the memory fragment most relevant to the current scene to obtain the context vector:

[0066]

[0067] In the formula, m ctx (t) is the context memory vector; a j Let m be the attention weight for the j-th memory; j Let j be the memory vector;

[0068] Finally, multimodal features that integrate memory information are generated:

[0069]

[0070] In the formula, The multimodal features are used to integrate memory information; μ represents the memory enhancement weight.

[0071] Furthermore, in step S4, the comprehensive risk calculation specifically involves:

[0072] Combining multimodal features Safety Regulations Knowledge K safe With historical memory bank Model generates risk logic chain:

[0073]

[0074] Output the risk logic chain R in natural language form logic This includes the type of violation, its location, and the level of risk.

[0075] For R logic Perform semantic analysis to extract the risk element set:

[0076]

[0077] In the formula, Γ represents the set of risk factors; r k This represents the k-th risk type. λ represents the three-dimensional coordinates corresponding to the risk. k For model semantic confidence;

[0078] Calculate the overall risk score:

[0079]

[0080] In the formula, R score For the overall risk score; β k For procedural weights; This is the spatial risk weighting function.

[0081] The present invention has the following beneficial effects:

[0082] This invention provides a 3D risk identification method for power transmission and transformation engineering construction sites that integrates a large-scale model with YOLOv8, effectively overcoming the limitations of traditional monitoring methods, such as reliance on manual labor, slow response, and difficulty in identifying complex risk scenarios. By combining 2D visual detection with 3D spatial positioning, real-time and accurate identification and semantic understanding of multiple targets in the construction environment are achieved. This method utilizes a large-scale model to integrate multimodal features, incorporates safety procedure knowledge, enhances risk logic reasoning capabilities, and can generate comprehensive risk scores. This significantly improves the intelligence level of construction site safety management, enhances the accuracy and timeliness of risk identification, and provides reliable technical support for power engineering safety prevention and control. Attached Figure Description

[0083] Figure 1 The flowchart shows the method for three-dimensional risk identification at the construction site of power transmission and transformation projects that integrates a large model with YOLOv8, as provided in this invention. Detailed Implementation

[0084] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0085] Example

[0086] like Figure 1 As shown, the present invention provides a three-dimensional risk identification method for power transmission and transformation engineering construction sites that integrates large-scale models and YOLOv8, comprising the following steps:

[0087] S1: Real-time multi-target detection at construction sites based on the YOLOv8 model. First, video frame images are extracted, multi-scale features are extracted via a backbone network, and feature fusion is performed by a neck network. Finally, the detection head outputs a prediction tensor. By decoding the prediction results and applying non-maximum suppression, a set of detection results containing target category, confidence level, and two-dimensional bounding box coordinates is obtained.

[0088] This step aims to quickly and accurately locate and identify all targets of interest from the raw video stream.

[0089] S1.1: Acquire real-time video streams from fixed or mobile cameras deployed at the construction site, and extract images frame by frame. t , where t is the frame number.

[0090] S1.2: Transfer image I t The input is fed into a pre-trained YOLOv8 model. YOLOv8 performs forward propagation through its backbone network, neck network, and detector head.

[0091] The backbone network is responsible for feature extraction and outputs multi-scale feature maps F. l , where l represents the feature layer level.

[0092] The neck network fuses multi-scale features, enhancing the representational power of the feature pyramid.

[0093] The detection head outputs the final prediction tensor P.

[0094] S1.3: Decode the prediction tensor P and perform non-maximum suppression processing to obtain the final detection result set D. 2D .

[0095] Each test result d i ∈D 2D Includes target category c i Confidence level s i 2D bounding box coordinates (bbox) i =(x 1i ,y 1i ,x 2i ,y 2i ), where (x 1i ,y 1i (x) represents the coordinates of the top-left corner. 2i ,y 2i () represents the coordinates of the lower right corner.

[0096] S2: The transformation from 2D detection results to 3D space is achieved through coordinate mapping. First, the pixel coordinates of the midpoint of the bottom edge of the target bounding box are calculated, and then normalized coordinates are obtained after distortion correction using the camera intrinsic parameter matrix. Next, a depth estimation network is used to obtain the target depth information, and combined with camera extrinsic parameters, the 3D points in the camera coordinate system are transformed to the world coordinate system. Finally, an enhanced detection result set containing 3D position information is output, i.e., 3D dynamic scene data.

[0097] This step aims to transform the two-dimensional detection result D obtained in step S1 into... 2D Mapped to a real-world three-dimensional coordinate system, it provides spatial information for risk assessment.

[0098] S2.1: For each detected target d i Calculate the pixel coordinates p of the midpoint of the bottom edge of its bounding box. i =(u i ,v i This point is typically considered to be the point of contact between the target and the ground, and its three-dimensional coordinates (X, Y, Z) are... i ,Y i Z i Z in ) i (Height) can be initialized to 0.

[0099] v i=y 2i

[0100] Among them, u i v is the x-coordinate of pixels. i The vertical coordinate is the pixel coordinate.

[0101] S2.2: Using the camera's intrinsic parameter matrix K and distortion coefficients, the pixel coordinates p i After distortion correction, the coordinates p′ in the normalized camera coordinate system are obtained. i =(x′) i ,y′ i ).

[0102]

[0103] Where K is a 3x3 camera intrinsic parameter matrix, containing the focal length f x ,f y and principal point coordinates c x ,c y .

[0104] S2.3: Use a pre-trained depth estimation network to predict the depth value of each pixel in the image. Then target d i depth The average depth value of the bottom region of its bounding box can be taken.

[0105]

[0106] Among them, R bottom This is a region at the bottom of the bounding box, where N is the number of pixels within that region. Then, the 3D coordinates... In the camera coordinate system, it can be calculated using the following formula:

[0107]

[0108] S2.4: Using the camera's extrinsic parameters (rotation matrix R and translation vector T), the 3D points in the camera coordinate system are... Transform to world coordinate system

[0109]

[0110] Ultimately, we obtain 3D dynamic scene data D with 3D position information. 3D ,

[0111]

[0112] S3: Transform 3D dynamic scene data into structured text descriptions. First, generate individual descriptions containing target categories and 3D coordinates based on a preset template. Then, calculate the Euclidean distance between targets and the spatial relationship between targets and restricted areas. Finally, aggregate all descriptions to form a complete scene text containing object attributes and spatial relationships.

[0113] This step aims to transform the 3D dynamic scene data D obtained in S2 into... 3D Transform it into a structured natural language description that large models can understand.

[0114] S3.1: Define a scene description template T, which specifies the elements to be described, such as objects, attributes, and spatial relationships.

[0115] S3.2: For D 3D For each target in the list, generate its individual description desc i .

[0116]

[0117] S3.3: Calculate the spatial relationships between key targets, especially the distances.

[0118] Calculate the Euclidean distance dist between target i and target j. ij :

[0119]

[0120] Calculate the relationship between target i and the predefined restricted area A k The relative positional relationships (defined by the set of corner points). Determine the points. Is it in polygon A? k Inside.

[0121] S3.4: Aggregate all individual descriptions and relationship descriptions to form the scene text S that is the final input to the large model.

[0122] S = T(desc1,desc2,...,desc) 12 Personnel 1 is inside restricted area A1.

[0123] S4: Based on the scene text generated in S3 and the enhanced detection result set obtained in S2, a unified representation vector of multimodal features after fusion is obtained through feature extraction and fusion; and safety procedure knowledge is introduced to fine-tune the large model, enabling it to acquire the semantic understanding capability of power safety; during inference, based on the unified representation vector of multimodal features after fusion and by introducing a memory mechanism, multimodal features with fused memory information are obtained; finally, the final risk logic chain and comprehensive risk score are generated by combining all the aforementioned information.

[0124] S4.1: Extract the semantic feature vector of the scene text S based on S3 and the 3D dynamic scene data D based on S2. 3D The spatial feature vectors are obtained, and modality alignment is achieved through weighted fusion to obtain a unified representation vector of multimodal features after fusion.

[0125] The scene text S based on S3 is encoded by the text encoder f. t (·) Obtain the semantic feature vector

[0126]

[0127] In the formula, f is a semantic feature vector; t (·) is a text encoder function used to extract language features.

[0128] Simultaneously, the 3D dynamic scene data D output by S2 will be... 3D via visual encoder f v (·) is transformed into spatial feature vectors

[0129]

[0130] In the formula, f is a spatial eigenvector; v (·) is the visual encoder function used to extract spatial features.

[0131] Then, modal alignment is performed using the fusion module:

[0132]

[0133] In the formula, α is the unified representation vector of multimodal features after fusion; α is the modal weight coefficient.

[0134] S4.2: To make the large model It possesses the ability to understand the semantics of power safety, enabling efficient fine-tuning of its parameters;

[0135] Through the safety procedure knowledge base K safe Embedded alignment enables the model to possess semantic priors for power safety rules.

[0136]

[0137] In the formula, To constrain the model to comply with safety procedures, the language generation loss is used; i For safe behavior labels; S i Let K be the i-th scene text; safeThis is a knowledge base for power safety regulations; θ represents the model parameters.

[0138] Constructing an alignment loss between 3D features and text semantics:

[0139]

[0140] In the formula, The loss is the alignment loss between text semantics and 3D space.

[0141] We introduce a fine-tuning strategy based on reward modeling, where the reward function is defined as:

[0142] r = η1Acc rule +η2Cons logic +η3Coh lang

[0143] In the formula, r is the reward score for the sample output generated by the model during training; Acc rule To output the accuracy of whether safety procedures are met; Cons logic Score for semantic and logical consistency; Coh lang η1, η2, and η3 are the language fluency indicators; η1, η2, and η3 are the weights of each indicator.

[0144] Final composite optimization objective:

[0145]

[0146] In the formula, The overall optimization objective of the model.

[0147] After training, the model obtains the optimized parameters θ. * .

[0148] S4.3: During the inference phase, multimodal features that integrate memory information are generated through processing by the Transformer decoder and semantic memory module;

[0149] During the inference phase, the fused multimodal feature unified representation vector obtained from S4.1 is input. The Transformer decoding structure is calculated layer by layer:

[0150]

[0151] In the formula, h (l) is the hidden state vector of the l-th layer; MHAtn(·) is the multi-head attention mechanism module, used to capture the semantic dependencies between different spatial objects; FFN(·) is the feedforward network layer, which realizes nonlinear feature transformation.

[0152] To enhance the model's understanding of temporal scenes, a semantic memory module is introduced:

[0153]

[0154] In the formula, m new (t) represents the newly generated memory vector at the current moment; γ is the forgetting factor, controlling the proportion of old memories retained; m old (t-1) is the memory vector from the previous time step;

[0155] And cumulatively update the historical semantic memory:

[0156]

[0157] In the formula, It serves as a historical semantic memory bank, accumulating semantic vectors from various moments in time.

[0158] Then, retrieve the memory fragment most relevant to the current scene from it to obtain the context vector:

[0159]

[0160] In the formula, m ctx (t) is the context memory vector; a j Let m be the attention weight for the j-th memory; j Let j be the j-th memory vector.

[0161] Finally, multimodal features that integrate memory information are generated:

[0162]

[0163] In the formula, The multimodal features are used to integrate memory information; μ represents the memory enhancement weight.

[0164] This mechanism enables knowledge transfer across time frames, giving the model contextual understanding when identifying persistent violations (such as people continuously approaching high-voltage areas).

[0165] S4.4: Combine multimodal features, safety procedures, and historical memory to generate a risk logic chain, analyze risk elements, and calculate a comprehensive risk score.

[0166] Combining multimodal features Safety Regulations Knowledge K safe With historical memory bank Model generates risk logic chain:

[0167]

[0168] Output the risk logic chain R in natural language form logic This includes the type of violation, location, and risk level.

[0169] For R logic Perform semantic analysis to extract the risk element set:

[0170]

[0171] In the formula, Γ represents the set of risk factors; r k This represents the k-th risk type. λ represents the three-dimensional coordinates corresponding to the risk. k This represents the semantic confidence of the model.

[0172] Calculate the overall risk score:

[0173]

[0174] In the formula, R score For the overall risk score; β k For procedural weights; This is the spatial risk weighting function.

[0175] This invention also provides a storage medium storing a computer program. When executed by a processor, the computer program implements some or all of the steps in various embodiments of the present invention's method for three-dimensional risk identification at power transmission and transformation engineering construction sites, which integrates a large model and YOLOv8. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0176] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0177] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for three-dimensional risk identification at power transmission and transformation engineering construction sites that integrates large-scale models and YOLOv8, characterized in that, Includes the following steps: S1: Real-time multi-target detection at the construction site is achieved based on the YOLOv8 model, resulting in a two-dimensional detection result set; S2: The two-dimensional detection results are converted to three-dimensional space through coordinate mapping to obtain three-dimensional dynamic scene data containing three-dimensional position information; S3: Convert the three-dimensional dynamic scene data into a structured text description to generate scene text; S4: The scene text generated in step S3 and the 3D dynamic scene data obtained in step S2 are used to obtain a unified representation vector of multimodal features after feature extraction and fusion; the unified representation vector of multimodal features is introduced into a memory mechanism to obtain multimodal features with fused memory information; Based on the optimized large model, a comprehensive risk calculation is performed on multimodal features.

2. The method for three-dimensional risk identification at power transmission and transformation engineering construction sites that integrates a large model and YOLOv8 as described in claim 1, characterized in that, Specifically, step S1 involves extracting video frame images based on the YOLOv8 model, extracting multi-scale features through the backbone network, fusing features through the neck network, and finally outputting a prediction tensor by the detection head. By decoding the prediction results and performing non-maximum suppression processing, a set of detection results containing target category, confidence level, and two-dimensional bounding box coordinates is obtained.

3. The method for three-dimensional risk identification at power transmission and transformation engineering construction sites that integrates a large model and YOLOv8 as described in claim 1, characterized in that, Specifically, step S2 involves first calculating the pixel coordinates of the midpoint of the bottom edge of the target bounding box, and then obtaining the normalized coordinates after distortion correction by the camera intrinsic parameter matrix; next, using a depth estimation network to obtain the target depth information, and combining the camera extrinsic parameters to transform the 3D points in the camera coordinate system to the world coordinate system.

4. The method for three-dimensional risk identification at power transmission and transformation engineering construction sites that integrates a large model and YOLOv8 as described in claim 3, is characterized in that... In step S2, for each detected target d i Calculate the pixel coordinates p of the midpoint of the bottom edge of its bounding box. i =(u i ,v i ), Among them, u i v is the x-coordinate of pixels. i The vertical coordinate is the pixel coordinate. Using the camera's intrinsic parameter matrix K and distortion coefficients, the pixel coordinates p i After distortion correction, the coordinates p′ in the normalized camera coordinate system are obtained. i =(x′) i ,y′ i ), Where K is a 3x3 camera intrinsic parameter matrix, containing the focal length f x ,f y and principal point coordinates c x ,c y ; A pre-trained depth estimation network is used to predict the depth value of each pixel in an image. Then target d i depth Take the average depth value of the bottom region of its bounding box. Among them, R bottom It is a region at the bottom of the bounding box, where N is the number of pixels within that region, and the 3D coordinates are... Calculated in the camera coordinate system using the following formula: Using the camera's extrinsic parameters, namely the rotation matrix R and the translation vector T, the three-dimensional points in the camera coordinate system are... Transform to world coordinate system Obtain 3D dynamic scene data D with 3D position information 3D , 5. The method for three-dimensional risk identification at power transmission and transformation engineering construction sites that integrates a large model and YOLOv8 as described in claim 1, characterized in that, In step S3, individual descriptions containing target categories and three-dimensional coordinates are generated according to a preset template, the Euclidean distance between targets and the spatial relationship between targets and restricted areas are calculated, and all descriptions are aggregated to form a complete scene text containing object attributes and spatial relationships.

6. The method for three-dimensional risk identification at power transmission and transformation engineering construction sites, which integrates a large model with YOLOv8, as described in claim 5, is characterized in that... In step S3, a scene description template T is defined, which specifies the elements that need to be described. For 3D dynamic scene data D 3D For each target in the list, generate its individual description desc i , desc i ="There exists a c" i ,lie in Calculate the Euclidean distance dist between target i and target j. ij : Aggregate all individual descriptions and relationship descriptions to form the scene text S that is ultimately input into the large model. S = T(desc1,desc2,...,desc) 12 Personnel 1 is inside restricted area A1.

7. The method for three-dimensional risk identification at power transmission and transformation engineering construction sites that integrates large-scale models and YOLOv8 as described in claim 1, characterized in that, In step S4, the scene text S is encoded by a text encoder f. t (·) Obtain the semantic feature vector In the formula, f is a semantic feature vector; t (·) is a text encoder function used to extract language features; 3D dynamic scene data D 3D via visual encoder f v (·) is transformed into spatial feature vectors In the formula, f is a spatial eigenvector; v (·) represents the visual encoder function used to extract spatial features; Modal alignment is performed using the fusion module: In the formula, α is the unified representation vector of multimodal features after fusion; α is the modal weight coefficient.

8. The method for three-dimensional risk identification at power transmission and transformation engineering construction sites that integrates large-scale models and YOLOv8 according to claim 1, characterized in that, In step S4, optimizing the large model specifically involves using the safety procedure knowledge base K. safe Embedded alignment enables large models to possess semantic priors for power safety rules. In the formula, To constrain the model to comply with safety procedures, the language generation loss is used; i Labeling for safe behaviors; S i Let K be the i-th scene text; safe This is a knowledge base for power safety regulations; θ represents the model parameters. Constructing an alignment loss between 3D features and text semantics: In the formula, The loss is the alignment loss between text semantics and 3D space. We introduce a fine-tuning strategy based on reward modeling, where the reward function is defined as: r=η1Acc rule +η2Cons logic +η3Coh lang In the formula, r is the reward score for the sample output generated by the model during training; Acc rule To output the accuracy of whether safety procedures are met; Cons logic Score for semantic and logical consistency; Coh lang η1, η2, and η3 are the language fluency indicators; η1, η2, and η3 are the weights of each indicator. Final composite optimization objective: In the formula, The overall optimization objective of the model.

9. The method for three-dimensional risk identification at power transmission and transformation engineering construction sites that integrates large-scale models and YOLOv8 according to claim 1, characterized in that, In step S4, the multimodal features obtained by fusing memory information are specifically as follows: Unified representation vector of fused multimodal features The Transformer decoding structure is calculated layer by layer: In the formula, h (l) is the hidden state vector of the l-th layer; MHAtn(·) is the multi-head attention mechanism module, used to capture the semantic dependencies between different spatial objects; FFN(·) is the feedforward network layer, which realizes non-linear feature transformation; A semantic memory module is introduced to enhance the model's understanding of temporal scenarios: In the formula, m new (t) represents the newly generated memory vector at the current moment; γ is the forgetting factor, controlling the proportion of old memories retained; m old (t-1) is the memory vector from the previous time step; Cumulative updates to the historical semantic memory: In the formula, It serves as a historical semantic memory bank, accumulating semantic vectors from various moments in time. Retrieve the memory fragment most relevant to the current scene to obtain the context vector: In the formula, m ctx (t) is the context memory vector; a j Let m be the attention weight for the j-th memory; j Let j be the j-th memory vector; Finally, multimodal features that integrate memory information are generated: In the formula, To integrate multimodal features of memory information; μ represents the memory enhancement weight.

10. The method for three-dimensional risk identification at power transmission and transformation engineering construction sites, which integrates a large model with YOLOv8, as described in claim 1, is characterized in that... In step S4, the comprehensive risk calculation specifically involves, Combining multimodal features Safety Regulations Knowledge K safe With historical memory bank Model generates risk logic chain: Output the risk logic chain R in natural language form logic This includes the type of violation, its location, and the level of risk. For R logic Perform semantic analysis to extract the risk element set: In the formula, Γ represents the set of risk factors; r k This represents the k-th risk type. λ represents the three-dimensional coordinates corresponding to the risk. k For model semantic confidence; Calculate the overall risk score: In the formula, R score For the overall risk score; β k For procedural weights; This is the spatial risk weighting function.

Images