Jib operation dangerous point identification method based on visual language large model

Abstract

The invention provides a jib operation dangerous point identification method based on a visual language large model, and relates to the technical field of electric power safety, and the method comprises the steps: obtaining first visual data in the operation process of a jib in an actual operation scene of the jib; inputting the first visual data and a preset cue word into a visual language large model Janus-Pro, and outputting at least one group of first image text pairs; screening the first image text pairs in combination with the condition of the actual operation site of the boom truck, and screening out second image text pairs which are inferred correctly; performing fine tuning on the visual language large model Janus-Pro by taking the second image text pair as a model fine tuning data set to obtain a fine-tuned visual language large model Janus-Pro; second visual data in the operation process of the boom vehicle are obtained in real time; and inputting the second visual data and a preset prompt word into the fine-tuned visual language large model Janus-Pro, and outputting an identification result of the boom truck operation dangerous point. According to the scheme, the accuracy of recognizing the operation dangerous point of the boom truck can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power safety, in particular to a boom truck operation dangerous point identification method based on a visual language large model. BACKGROUND

[0002] As a special engineering vehicle with telescopic and rotatable boom structure, the core function of the boom truck is to realize the lifting, handling and installation of heavy objects through the lifting, amplitude and rotation of the boom, combined with hooks, grabs and other accessories, and it is widely used in key fields such as power engineering, construction, municipal operation and maintenance, and traffic rescue, and is one of the core equipment supporting industrial production and infrastructure construction. However, there are many high-risk hidden dangers in the operation process of the boom truck, and the operation scene often involves high-voltage, high-altitude and densely populated areas. Once an accident occurs, it is easy to cause personnel casualties, equipment damage and even public safety incidents. For example, in the operation area, especially in the high-voltage live working area and the high-altitude operation area below, personnel are strictly prohibited from entering the warning area. Once irrelevant personnel mistakenly enter the warning area, they may face direct risks such as boom collision, heavy object falling and high-voltage electric shock, and such risks are sudden and the personnel reaction time is extremely short. For example, in the live working of power, safety helmets can prevent high-altitude falling objects from hitting the head, and insulation equipment can block current conduction. If the operator does not wear protective equipment according to the specification, once he or she comes into contact with a live device or encounters a falling object, it will directly lead to electric shock or brain injury and other fatal accidents. Therefore, accurate and real-time identification of dangerous points in operation is of great significance in protecting personnel safety, reducing property and economic losses, and standardizing the operation safety process.

[0003] However, the monitoring of dangerous area intrusion behavior in the operation scene of the boom truck currently mainly relies on traditional behavior recognition models, and there is still a lack of effective intelligent detection methods. Although the target detection model in the field of artificial intelligence provides a feasible idea for behavior recognition, the accuracy is low in actual application. SUMMARY

[0004] Therefore, in view of the above shortcomings, it is necessary to provide a boom truck operation dangerous point identification method based on a visual language large model to improve the accuracy of identifying the dangerous points of the boom truck operation.

[0005] In a first aspect, the present application provides a boom truck operation dangerous point identification method based on a visual language large model, comprising:

[0006] In the actual operation scene of the boom truck, first visual data in the operation process of the boom truck is obtained;

[0007] The first visual data and a preset prompt word are input into a visual language large model Janus-Pro, and at least one group of first image text pairs is output;

[0008] Screening each first image-text pair in combination with the actual working site of the crane truck, and screening a second image-text pair with correct reasoning;

[0009] Fine-tuning the visual language large model Janus-Pro by taking the second image-text pair as a model fine-tuning dataset, to obtain a fine-tuned visual language large model Janus-Pro;

[0010] Real-time acquisition of second visual data in the crane truck working process;

[0011] Inputting the second visual data and the preset prompt words into the fine-tuned visual language large model Janus-Pro, and outputting the identification result of the crane truck working dangerous point.

[0012] Preferably, the first visual data in the crane truck working process comprises:

[0013] Acquiring first visual data of at least three key positions in the crane truck working area; wherein the key positions include the front of the crane truck operating position, the entrance side of the working warning area, and the highest point of the farthest area covered by the crane arm radius; and the first visual data includes workers, protective equipment and working tools.

[0014] Preferably, the first visual data is further subjected to image preprocessing before being input into the visual language large model Janus-Pro; the image preprocessing process comprises:

[0015] Scaling the first visual data to a preset size and processing the image data by bilinear interpolation;

[0016] Performing histogram equalization processing on low-light images in the image data;

[0017] Saving the equalized image data in RGB format, and eliminating the alpha channel corresponding to the transparent information in the image.

[0018] Preferably, the prompt words include basic element prompt words, scene-specific prompt words and risk association prompt words;

[0019] The basic element prompt words include: describing the number of personnel in the image, the position, and whether the protective equipment is worn; describing whether the safety fence in the image is complete and the state of the crane arm legs;

[0020] The scene-specific prompt words include: whether there is a high-voltage line mark in the image; whether there is other construction machinery under the crane arm;

[0021] The risk association prompt words include: if the personnel are in the fence and do not wear safety helmets, the risk combination needs to be clearly marked.

[0022] Preferably, the filtering of each first image-text pair based on the actual working conditions of the crane truck includes:

[0023] The first image text pair was independently screened by at least two qualified personnel with electrical safety qualifications, based on the actual work site, and marked as pass, fail, and need correction using a labeling tool;

[0024] When the same first image text is inconsistently labeled by at least two qualified personnel with electrical safety qualifications, a senior engineer shall review it and determine the final labeling result.

[0025] For the first image-text pair that needs correction, it is manually corrected and supplemented before being marked as passed.

[0026] Each of the first image-text pairs marked as passed is identified as the selected second image-text pair.

[0027] Preferably, the step of using the second image-text pair as a model fine-tuning dataset to fine-tune the visual language large model Janus-Pro includes:

[0028] The second image-text pair, which serves as the model fine-tuning dataset, will be input into the model in batches.

[0029] Visual feature vectors are extracted from the second image-text pair using ViT;

[0030] The semantic feature vector of the second image-text pair is extracted through word embedding and language modeling;

[0031] Calculate the matching degree between the visual feature vector and the semantic feature vector;

[0032] The matching degree is compared with the preset matching labels, and the parameters of the trainable layer are adjusted through the backpropagation algorithm, so that the model gradually learns the correspondence between images and safety text in the crane operation scenario.

[0033] Preferably, when fine-tuning the visual language large model Janus-Pro, the parameters of the first 10 layers of ViT and the first 15 layers of DeepSeek-LLM in the visual language large model Janus-Pro are frozen, and only the top-level cross-modal attention layer and MLP layer are trained.

[0034] Preferably, after outputting the identification results of the hazard points in the crane operation, the process further includes:

[0035] Based on the output of the crane operation hazard identification results, the corresponding target hazard level is queried from the preset hazard level specification table, and a hazard alarm is triggered through the alarm method corresponding to the target hazard level; wherein, the hazard level specification table includes: hazard level, area intrusion, protective equipment, safety fence, judgment logic and alarm method.

[0036] Preferably, when the actual operating scenario of the crane changes or new types of hazards appear, the visual language large model Janus-Pro is fine-tuned again.

[0037] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform any of the methods described in the first aspect.

[0038] As can be seen from the above technical solution, the crane operation hazard identification method based on the visual language large model provided by this solution firstly acquires the first visual data during the crane operation in the actual operation scenario. Then, the first visual data and preset prompt words are input into the visual language large model Janus-Pro, and at least one first image-text pair is output. Further, each first image-text pair is filtered according to the actual operation site of the crane, and the correctly inferred second image-text pair is selected. The second image-text pair is used as the model fine-tuning dataset to fine-tune the visual language large model Janus-Pro, resulting in the fine-tuned visual language large model Janus-Pro. Further, the second visual data during the crane operation is acquired in real time, and the second visual data and preset prompt words are input into the fine-tuned visual language large model Janus-Pro, outputting the identification result of the crane operation hazard. As can be seen from the above technical solution, this solution, based on a general multimodal large-scale model for end-to-end image understanding, integrates image input, text generation, and model optimization into a unified process. This reduces the error accumulation caused by staged processing in traditional methods and improves the accuracy of the model in identifying and describing intrusion behavior in dangerous areas and safety equipment in crane operation scenarios. Simultaneously, this solution abandons the traditional separation of feature extraction and text generation, achieving deep fusion of image and text features in a deep network, enhancing cross-modal understanding capabilities. By inputting images and corresponding prompts into the model, it guides it to generate text descriptions of the target objects. High-quality image-text pairs are selected from the generated results to construct a dataset for fine-tuning the model. Finally, by adapting and fine-tuning the original large-scale model for the task, its inference accuracy and stability on the target task are significantly improved. Furthermore, this solution quickly focuses on the target task through a general visual language large-scale model and prompts, while combining the selection and construction of a fine-tuning dataset with targeted fine-tuning of the large-scale model. Compared to the traditional method of directly training the model using a large dataset, this effectively reduces dependence on data, improves scene adaptability, and reduces training costs. Attached Figure Description

[0039] Figure 1 The flowchart illustrates a method for identifying hazardous points in crane operations based on a large visual language model, as provided in this embodiment of the invention.

[0040] Figure 2 This is a schematic diagram of a framework for identifying hazard points in crane operations based on a large visual language model, provided as an embodiment of the present invention. Detailed Implementation

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Currently, monitoring of intrusion behavior in hazardous areas of crane operation scenarios mainly relies on traditional behavior recognition models, lacking effective intelligent detection methods. Object detection models in the field of artificial intelligence provide a feasible approach to behavior recognition. This involves first detecting key targets in video frames, then performing temporal analysis to determine the behavior; alternatively, key point detection and motion analysis can be used to identify behavior. While these methods can achieve the goal of recognition, they have significant limitations in practical applications. For example, at the data level, annotation costs are too high, and multiple angles and environments need to be covered. Furthermore, the limited number of samples for certain behaviors can lead to long-tailed distributions, limiting the accuracy of behavior recognition. Therefore, this solution considers using information filtering and large-scale visual language model fine-tuning techniques to effectively detect and identify hazardous points in crane operation scenarios. The aim is to use the Janus-pro large-scale visual language model to intelligently identify and describe the intrusion behavior and protective equipment wearing status of personnel in the crane operation environment, and to determine the level of hazardous behavior accordingly. Specifically, for example... Figure 1 and 2 As shown, this solution provides a method for identifying hazards in crane operations based on a large visual language model. This method may include the following steps:

[0043] Step 101: In the actual operating scenario of the crane truck, acquire the first visual data during the crane truck's operation;

[0044] Step 102: Input the first visual data and the preset prompt words into the visual language large model Janus-Pro, and output at least one first image-text pair;

[0045] Step 103: Based on the actual working conditions of the crane truck, filter each first image-text pair and select the second image-text pairs with correct reasoning;

[0046] Step 104: Use the second image-text pair as the model fine-tuning dataset to fine-tune the visual language large model Janus-Pro, and obtain the fine-tuned visual language large model Janus-Pro;

[0047] Step 105: Acquire second-vision data in real time during the crane operation process;

[0048] Step 106: Input the second visual data and the preset prompt words into the fine-tuned visual language large model Janus-Pro, and output the identification results of the hazard points of the crane operation.

[0049] In this embodiment, the Janus-Pro general visual language model is first guided to describe the image using pre-given prompts. The model outputs corresponding text information based on the given prior information. Then, the generated text is filtered for effective information, selecting the correct image-text pairs from the output to form a high-quality dataset for fine-tuning the model. This dataset is then used for model fine-tuning training, adapting the visual language model to the scenario of identifying hazards in live-line crane operations, resulting in a more accurate understanding of input images. Furthermore, this embodiment not only provides a dataset generation scheme, effectively improving the efficiency of dataset creation and overcoming the shortcomings of traditional manual image-text dataset creation, but also offers a multimodal fine-tuning approach for large-scale models applicable to specific tasks, providing a reference solution for hazard warning for workers in crane operation environments.

[0050] For step 101, in the actual working scenario of the crane truck, acquire the first visual data during the crane truck's operation.

[0051] In this step, when acquiring the first visual data, the first visual data of at least three key locations in the crane operation area can be acquired; wherein, the key locations include the front of the crane operation position, the entrance side of the operation warning zone, and the highest point of the farthest area covered by the crane's rotation radius; the first visual data includes the operators, protective equipment, and operating tools.

[0052] For example, portable cameras with a resolution of at least 1080P, such as industrial-grade high-definition cameras or drone-mounted cameras, can be selected to ensure clear capture of details. The camera needs to be fixed in three key locations within the work area: 5 meters directly in front of the crane operator's position, 3 meters to the side of the work warning zone entrance, and at the highest point of the furthest area covered by the crane's rotation radius, to achieve comprehensive coverage without blind spots. First-person view data should include important elements such as protective equipment and tools, including workers, the crane, safety barriers, and helmets. Specifically, core elements in the first-person view data can include: key components of the crane such as the boom, operator's cab, and outriggers; the complete outline of the safety barrier and safety facilities such as warning signs; and a full-body image of the worker, clearly showing the head, hands, and feet to determine whether they are wearing safety helmets, protective equipment, and insulated shoes.

[0053] The acquired first-person visual data is stored according to data storage specifications. For example, videos are named according to "work date-work location-equipment number", such as 20251019-XX substation-crane 001, and are divided into one file every 30 minutes; images are extracted from the video at a frequency of 1 frame per second, and frames without blur or obstruction are selected and saved in JPG format, with the size of a single image controlled between 2-5MB.

[0054] Step 102: Input the first visual data and the preset prompt words into the visual language large model Janus-Pro, and output at least one first image-text pair.

[0055] In this embodiment, Janus-Pro is based on the open-source multimodal AI model released by DeepSeek in January 2025. This model's text generation process for images is based on the Transformer model. First, the input image undergoes preprocessing operations such as resizing, cropping, and normalization. Then, text prompts guide the model to output descriptions in a specific format or content, allowing the model to focus on specific details, such as the helmet. Next, an image of a crane operation is input into the model, which performs inference operations on the preprocessed image, including feature extraction and fusion of visual and linguistic information. After obtaining visual features, the model combines them with prompts or other contextual information, achieving effective fusion of visual and linguistic information through a cross-modal attention mechanism, finally generating the desired text.

[0056] Specifically, the first step in describing the content of an image during crane operation is to extract image features so that the text generation model can understand the image. The visual feature extraction stage of the Janus-Pro model is based on ViT (VisionTransformer). This model consists of three modules: Linear Projection of Flattened Patches (Embedding layer), Transformer Encoder, and MLP Head (the final layer structure used for classification). First, image preprocessing is performed, cutting the complete image into several small "patches". The original image is divided equally, and each small image is a patch. This process is called patch partitioning. Each patch is flattened into a one-dimensional vector and transformed linearly (similar to a fully connected layer) to obtain a vector (token) of length 768. This token sequence is the input required by the standard Transformer model. The preprocessed patch features with positional encoding are input into the Transformer Encoder. Through multi-layer multi-head attention and a feedforward network (MLP), the model learns the global relationships between patches to extract more abstract image features. Finally, the features corresponding to the classification tokens are extracted from the output of the Transformer Encoder, and the final classification is performed using an MLP (Multilayer Perceptron).

[0057] When generating text, the text generation capability of the Janus-Pro model is supported by its infrastructure, which is based on a large language model (LLM) developed by DeepSeek. Janus-Pro is built on top of DeepSeek-LLM-1.5b and DeepSeek-LLM-7b. The overall structure of DeepSeek-LLM is based on the LLaMA (Large Language Model Meta AI) architecture. LLaMA is implemented based on the Transformer model architecture, consisting of one embedding layer, multiple Transformer layers, and one RMSNorm layer. The Transformer layer includes two RMSNorm layers, one multi-head attention layer, and one fully connected layer. The input text is first segmented into multiple tokens, then the tokens are converted into vector representations by the embedding layer, then processed sequentially through multiple Transformer blocks, and finally the output layer yields the prediction result.

[0058] In this step, during polar image preprocessing, the photo can first be scaled to 512×512 pixels using OpenCV to meet the optimal input size of the Janus-Pro model, and bilinear interpolation can be used to avoid distortion. Then, adaptive histogram equalization is performed on the low-light image, and the brightness of the overexposed image is reduced by 15%-20% to ensure clear features. Furthermore, it can be uniformly saved in RGB format, and the alpha channel of transparency information can be removed.

[0059] The prompt words can include basic element prompt words, scenario-specific prompt words, and risk-related prompt words;

[0060] The basic element prompts include: describing the number of people in the picture, their location (whether they are inside the safety fence), whether they are wearing safety helmets / insulating gloves / insulating clothing and other protective equipment; describing whether the safety fence in the picture is intact, the status of the boom outriggers, etc.

[0061] The scene-specific prompts include: whether there are high-voltage line markings in the image; whether there are other construction machinery under the crane boom;

[0062] The risk association warning words include: if a person is inside the fence and is not wearing a safety helmet, this risk combination must be clearly marked.

[0063] Specifically, in practical applications, each input should consist of one image and one type of prompt word to avoid confusion in type judgment due to multiple questions. For example, input: "20251019-XX Substation-Crane 001_001.jpg" + "Are there any people inside the safety fence in the picture? If so, how many are there?"

[0064] Step 103: Based on the actual working conditions of the crane truck, filter each first image-text pair and select the second image-text pairs with correct reasoning;

[0065] Because the native large model Janus-Pro is suitable for general domains and was not pre-trained for power scenarios, errors may occur in the text responses output by the model. Therefore, manual assistance is needed to filter correct text information and construct a reliable "image-text pair" dataset. Janus-Pro can then be fine-tuned based on these correct "image-text pairs" to suit the task requirements of specific scenarios. Specifically, this step considers establishing multi-level filtering rules to ensure a perfect match between text and image. For example, the first-level rule is basic matching, which determines whether the core elements mentioned in the text, such as people, safety fences, and safety helmets, actually exist in the image, avoiding errors such as "the image shows people but the text says no people"; the second-level rule is detail matching, which ensures that the details described in the text are consistent with the image, such as "1 person" matching the actual number of people in the image, and "the safety helmet is red" matching the color of the safety helmet in the image; the third-level rule is logical matching, which ensures that the text description has no logical contradictions, such as "the person is inside the safety fence" matching the scene in the image where "the person's position is inside the fence boundary," avoiding logical errors such as "the person is outside the fence but described as inside."

[0066] In the specific screening process, at least two qualified personnel with electrical safety certifications can first independently screen the first image-text pairs based on actual work conditions, marking them as pass, fail, or require correction using a labeling tool. Then, if at least two qualified personnel with electrical safety certifications have inconsistent labeling for the same first image-text pair, a senior engineer will review the data and determine the final labeling result. Furthermore, for first image-text pairs requiring correction, manual correction and supplementation are performed before marking them as pass. For example, if the text mentions missing insulating clothing, the description is manually supplemented and the data is re-included in the dataset. Finally, the first image-text pairs marked as pass are determined as the selected second image-text pairs.

[0067] Finally, the selected high-quality image-text pairs were divided into training, validation, and test sets in a 7:2:1 ratio to serve as the model fine-tuning dataset for subsequent model fine-tuning.

[0068] Step 104: Use the second image-text pair as the model fine-tuning dataset to fine-tune the visual language large model Janus-Pro, and obtain the fine-tuned visual language large model Janus-Pro.

[0069] This step, when fine-tuning the model, can be achieved in the following way:

[0070] The second image-text pair, which serves as the model fine-tuning dataset, will be input into the model in batches.

[0071] Visual feature vectors are extracted from the second image-text pair using ViT;

[0072] The semantic feature vector of the second image-text pair is extracted through word embedding and language modeling;

[0073] Calculate the matching degree between the visual feature vector and the semantic feature vector;

[0074] The matching degree is compared with the preset matching labels, and the parameters of the trainable layer are adjusted through the backpropagation algorithm, so that the model gradually learns the correspondence between images and safety text in the crane operation scenario.

[0075] In this embodiment, the model fine-tuning process based on "image-text pairs" revolves around data-driven weight optimization. The core is to enable the model to establish associations between paired visual and linguistic information, thereby adjusting internal parameters to adapt to specific task requirements. The model's learning process is a dynamic process of continuously correcting the target information weights through backpropagation. Initially, the model's parameter weights are pre-trained on a large-scale general dataset, limiting its adaptability to power scenarios. When a filtered "image-text pair" is input, the model first extracts features from the image and text separately: the image branch extracts visual feature vectors using ViT, while the text branch generates semantic feature vectors using word embeddings and a language model. Subsequently, the model calculates the matching degree of these two sets of features and compares it with a preset "matching label" (i.e., whether the image and text are a valid pair). The resulting error is backpropagated to each layer of the network. During error feedback, the model selectively adjusts the neuron connection weights related to the target information, thereby achieving model fine-tuning.

[0076] Specifically, the fine-tuning environment can be set up based on an NVIDIA RTX 3090 graphics card, using a Linux system environment, installing CUDA 11.7 (for GPU acceleration), PyTorch 2.0 framework, configuring the batch size to 16 (to balance training speed and memory usage), and the learning rate to 1e. -5 To avoid excessively high learning rates that could cause model parameter oscillations, during validation and tuning after model training, the training set is iterated once every epoch, and the model performance is tested using the validation set. If the validation set accuracy does not improve for three consecutive epochs, the learning rate is reduced to 1 / 10 of its original value until 20 epochs have been trained or the validation set accuracy has stabilized.

[0077] In one embodiment, when fine-tuning the visual language large model Janus-Pro, the parameters of the first 10 layers of ViT and the first 15 layers of DeepSeek-LLM in the visual language large model Janus-Pro are frozen, and only the top-level cross-modal attention layer and MLP layer are trained to improve the training efficiency of the model.

[0078] Furthermore, it's important to note that fine-tuning is not necessary before each prediction. The core purpose of fine-tuning is to enable the general visual language model, Janus-Pro, to learn the specific rules of crane operation scenarios, such as "electrical work requires the identification of insulated safety helmets" and "the boundary of the warning zone is 1 meter outside the fence," thus transforming it from a "general model" into a "crane operation-specific model." This only requires two conditions to be met: First, upon initial deployment, a fine-tuning should be performed using selected high-quality "image-text pairs," allowing the model to adapt to the target scenario, such as live-line electrical work. Only when the rules of the operation scenario undergo significant changes, such as the addition of a requirement for "workers to wear reflective vests," or the emergence of numerous new types of hazards, such as new safety fences or protective equipment, is a periodic fine-tuning using a supplementary dataset necessary to update the model's understanding. After fine-tuning, the model parameters are permanently saved, and subsequent predictions can directly utilize this "specific model" without repeated training.

[0079] Step 105: Acquire second-vision data in real time during the crane operation process;

[0080] In this step, following the equipment deployment in step 101, one image data is collected every 2 seconds, or when infrared human body sensors, vibration sensors, etc. detect someone approaching the warning area or the crane arm shaking violently, high-speed acquisition is automatically triggered to capture instantaneous risks.

[0081] Step 106: Input the second visual data and the preset prompt words into the fine-tuned visual language large model Janus-Pro, and output the identification results of the hazard points of the crane operation.

[0082] In this step, the image data collected in step 105 and the prompt words in step 102 are used as input. The finely tuned visual language model Janus-Pro is then used for prediction, and the result of hazard identification is output. For example, given an image collected in the trigger mode and the prompt words, the model outputs: "Number of people: 2, Personnel location: 1 meter outside the safety fence (approaching trend), Safety helmet: not wearing, Safety fence: fully closed."

[0083] In one embodiment, after outputting the identification results of the hazard points in the crane operation, a warning can be further issued based on the hazard identification results. Specifically, based on the output identification results of the crane operation hazards, the corresponding target hazard level can be queried from a preset hazard level specification table, and a hazard alarm can be issued through an alarm method corresponding to the target hazard level; wherein, the hazard level specification table includes: hazard level, area intrusion, protective equipment, safety fence, judgment logic, and alarm method.

[0084] In this embodiment, the hazard level demonstration table can be shown in Table 1 below:

[0085] Table 1 Hazard Level Specification Table for Work Sites

[0086] Furthermore, a tiered response is implemented based on the level of hazard. For example, the judgment logic for Level 0 hazard is "no safety hazards," and the corresponding alarm method is "no alarm"; the judgment logic for Level 1 hazard is "boundary breach or minor protective deficiencies," and the corresponding alarm method is "flashing yellow light + voice prompt (please pay attention to protection and area boundaries)"; the judgment logic for Level 2 hazard is "boundary breach and minor protective deficiencies, or severe protective deficiencies," and the corresponding alarm method is "flashing orange light + high-frequency voice prompt (immediately leave the danger zone and put on protective equipment)"; the judgment logic for Level 3 hazard is "boundary breach and severe protective deficiencies," and the corresponding alarm method is "flashing red light + high-frequency buzzer + remote push alarm information to the monitoring center." In this way, by transforming the text information generated by the model into clear hazard levels and actionable alarm actions, a closed loop of "identification-judgment-alarm" is achieved, enabling timely avoidance of safety risks.

[0087] The effectiveness of this scheme will be further explained below with specific experiments.

[0088] The dataset for this model primarily integrates 1000 frame-by-frame images of personnel working on crane operation sites, with each image category manually labeled. Experiments were conducted using an NVIDIA RTX 3090 graphics card, employing the Janus-Pro-tiny model to test the proposed method. Software-wise, a Linux system (including CUDA environment and PyTorch integrated framework) was deployed to run the Janus-Pro model. Core components included designing the prompt word library and calculating accuracy metrics, with manual verification added to ensure experimental rigor. By inputting different datasets and prompt words and recording the model's output, the performance and effectiveness of the Janus-Pro-tiny model were ultimately validated.

[0089] In the experiment, all images in the dataset were first input into the model, and the correct "image-text pairs" were selected from the output to form an optimized dataset. The optimized dataset was then used for model training and fine-tuning. The following are the experimental results of detecting intrusion behavior and the wearing status of security equipment in various dangerous areas under three different scenarios.

[0090] Table 2. Experimental results on regional intrusion behavior and security equipment wearing status under three scenarios.

[0091] Table 2 shows that the Janus-Pro model, without fine-tuning and without prior information such as cue words, has weak generalization ability for intrusion behavior of workers in dangerous areas and the wearing status of safety equipment in power scenarios, failing to meet the task requirements. Adding cue words as prior information significantly improves the model's accuracy; further improvement is achieved after fine-tuning the model based on the optimized dataset and adding cue words. This result demonstrates that the proposed method is highly effective for problems such as intrusion into dangerous areas.

[0092] In summary, the crane operation hazard identification method based on a large visual language model provided by this invention has at least the following beneficial effects:

[0093] (1) This solution is based on the end-to-end image understanding and model fine-tuning of the multimodal large model (Janus model under Deepseek). It integrates image input, text generation and model optimization into a unified process, which reduces the error accumulation caused by the staged processing in traditional methods and improves the accuracy of the model in identifying and describing dangerous area intrusion behavior and safety equipment in the crane operation scenario.

[0094] (2) This scheme abandons the traditional method of separating feature extraction and text generation. The proposed method achieves deep fusion of image and text features in deep networks, which enhances cross-modal understanding capabilities.

[0095] (3) This approach guides the model to generate text descriptions of the target objects by inputting images and corresponding prompts into the model, and automatically selects high-quality "image-text pairs" from the generated results to construct a dataset for fine-tuning. Finally, by adapting and fine-tuning the original large model for the task, its inference accuracy and stability on the target task are significantly improved.

[0096] (4) Traditional methods require the preparation of large-scale labeled datasets (usually thousands to tens of thousands of labeled images) in advance, and must cover all scene details. This solution reconstructs the data production logic through a "generation + filtering" model, which has significant advantages. For example, traditional methods require manual labeling of "danger point labels" in images one by one. Labeling 1,000 images requires 2-3 professionals to spend 1-2 weeks, which is extremely costly. This solution first allows the model to automatically generate "image-text pairs". The human only needs to filter "correct / incorrect", without labeling each point. The filtering of 1,000 images can be completed in 1-2 days, which significantly improves efficiency. Moreover, in crane operations, rare danger points such as "personnel not wearing insulated helmets" and "crane boom close to high-voltage lines" are difficult to collect in large quantities through traditional methods, resulting in low accuracy of the model in recognizing such high-risk events. This solution can guide the model to generate "image-text pairs" containing long-tail scenes by designing targeted prompts, and then form samples through filtering, quickly supplementing the long-tail dataset and solving the problem of "incomplete sample coverage" in traditional methods.

[0097] (5) Traditionally, after training a model using a general dataset, it is necessary to fine-tune it with secondary labeled data for the crane truck scenario. This process is cumbersome and the model is prone to "incompatibility". This solution can enable the model to quickly focus on the target task through "prompt words + scenario-based fine-tuning". For example, if the traditional method is to train a model adapted to the crane truck scenario from scratch, it is necessary to first train a basic target detection model (about 1-2 weeks) and then fine-tune it with scenario data (about 3-5 days), with a total cycle of nearly 2 weeks. This solution is based on the existing Janus-Pro visual language model and only requires fine-tuning for 20 epochs (about 1-2 days) with a selected small batch dataset (such as 800 pairs of "image-text pairs") to adapt to the scenario, which greatly shortens the model deployment cycle. In addition, traditional methods of training models require large-scale GPU clusters (such as 8 RTX3090s), and the computing power cost of training one model is about several thousand yuan. This solution only requires a single RTX3090 graphics card for fine-tuning, and the amount of training data is small (1 / 5 of the traditional method), which greatly reduces the amount of computation and the computing power cost, making it more suitable for deployment by small and medium-sized enterprises or field operation teams.

[0098] (6) Crane operation scenarios are complex and the operation rules may change with the scenario. For example, power operations require additional identification of insulating equipment. Traditional methods are difficult to respond quickly to changes. This solution can easily adjust and iterate the model according to the operation rules. Its technical ideas are more in line with actual application needs.

[0099] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods in any of the embodiments of the specification.

[0100] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method in any of the embodiments of the specification.

[0101] The device embodiments provided by the present invention are based on the same inventive concept as the method embodiments in this specification. For details, please refer to the description in the method embodiments of this specification, which will not be repeated here.

[0102] The modules or units in the device of this invention can be merged, divided, and deleted according to actual needs. The above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the invention. Those skilled in the art will understand that implementing all or part of the processes of the above embodiments and making equivalent changes according to the claims of this invention still fall within the scope of the invention.

Claims

1. A method for identifying hazard points in crane operation based on a large visual language model, characterized in that, include: In the actual operation scenario of the crane, acquire first-view data during the operation of the crane; Input the first visual data and the preset prompt words into the visual language large model Janus-Pro, and output at least one first image-text pair; Based on the actual working conditions of the crane truck, each first image-text pair was filtered to select the second image-text pairs with correct reasoning. The second image-text pair is used as the model fine-tuning dataset to fine-tune the visual language large model Janus-Pro, resulting in the fine-tuned visual language large model Janus-Pro. Real-time acquisition of second-vision data during crane operation; The second visual data and the preset prompt words are input into the fine-tuned visual language large model Janus-Pro, and the output is the identification result of the hazard points of the crane operation.

2. The method for identifying hazard points in crane operation based on a large visual language model according to claim 1, characterized in that, The acquisition of first visual data during the crane operation includes: Acquire first visual data of at least three key locations in the crane's operating area; wherein, the key locations include the area directly in front of the crane's operating position, the entrance side of the work warning zone, and the highest point of the farthest area covered by the crane's rotation radius; the first visual data includes the workers, protective equipment, and working tools.

3. The method for identifying hazard points in crane operation based on a large visual language model according to claim 1, characterized in that, Before being input into the visual language large model Janus-Pro, the first visual data undergoes further image preprocessing; this image preprocessing process includes: The first visual data is scaled to a preset size, and bilinear interpolation is used to process the image data; Perform histogram equalization on low-light images in image data; The equalized image data is saved in RGB format, while the alpha channel corresponding to the transparency information in the image is removed.

4. The method for identifying hazard points in crane operation based on a large visual language model according to claim 3, characterized in that, The prompts include basic element prompts, scenario-specific prompts, and risk-related prompts; The basic element prompts include: describing the number and location of people in the image, and whether they are wearing protective equipment; describing whether the safety fence in the image is intact and the status of the crane outriggers; The scene-specific prompts include: whether there are high-voltage line markings in the image; whether there are other construction machinery under the crane boom; The risk association warning words include: if a person is inside the fence and is not wearing a safety helmet, this risk combination must be clearly marked.

5. The method for identifying hazard points in crane operation based on a large visual language model according to claim 1, characterized in that, The filtering of each first image-text pair based on the actual working conditions of the crane truck includes: The first image text pair was independently screened by at least two qualified personnel with electrical safety qualifications, based on the actual work site, and marked as pass, fail, and need correction using a labeling tool; When the same first image text is inconsistently labeled by at least two qualified personnel with electrical safety qualifications, a senior engineer shall review it and determine the final labeling result. For the first image-text pair that needs correction, it is manually corrected and supplemented before being marked as passed. Each of the first image-text pairs marked as passed is identified as the selected second image-text pair.

6. The method for identifying hazard points in crane operation based on a large visual language model according to claim 1, characterized in that, The step of using the second image-text pair as a model fine-tuning dataset to fine-tune the visual language large model Janus-Pro includes: The second image-text pair, which serves as the model fine-tuning dataset, will be input into the model in batches. Visual feature vectors are extracted from the second image-text pair using ViT; The semantic feature vector of the second image-text pair is extracted through word embedding and language modeling; Calculate the matching degree between the visual feature vector and the semantic feature vector; The matching degree is compared with the preset matching labels, and the parameters of the trainable layer are adjusted through the backpropagation algorithm, so that the model gradually learns the correspondence between images and safety text in the crane operation scenario.

7. The method for identifying hazard points in crane operation based on a large visual language model according to claim 6, characterized in that, When fine-tuning the visual language large model Janus-Pro, the parameters of the first 10 layers of ViT and the first 15 layers of DeepSeek-LLM in Janus-Pro are frozen, and only the top cross-modal attention layer and MLP layer are trained.

8. The method for identifying hazard points in crane operation based on a large visual language model according to claim 1, characterized in that, After outputting the results of identifying hazardous points in crane operations, the following further steps are included: Based on the output of the crane operation hazard identification results, the corresponding target hazard level is queried from the preset hazard level specification table, and a hazard alarm is triggered through the alarm method corresponding to the target hazard level; wherein, the hazard level specification table includes: hazard level, area intrusion, protective equipment, safety fence, judgment logic and alarm method.

9. The method for identifying hazard points in crane operation based on a large visual language model according to any one of claims 1-8, characterized in that, When the actual operating scenario of the crane changes or new types of hazards appear, the visual language model Janus-Pro is fine-tuned again.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method as described in any one of claims 1-9.

Images