Model training method, risk behavior identification method, system, device and medium

Abstract

The application provides a model training method, a risk behavior identification method, a system, a device and a medium. The training method comprises: acquiring a training data set; the training data set comprises image data and text data; a risk behavior identification model is trained based on the training data set by using a multi-stage distillation training strategy, and a trained risk behavior identification model is acquired; the multi-stage distillation training strategy comprises distillation training on a semantic representation, an attention distribution and / or a risk label stage of the risk behavior identification model. The application trains the risk behavior identification model by using the multi-stage distillation training strategy, so that the small model has a multi-task performance close to the level of a large model while significantly reducing memory and computing power consumption, thereby solving the problems of a large number of parameters of a conventional visual language model for identifying risk behaviors and difficulty in edge deployment in the prior art.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and relates to a model training method, a risk behavior recognition method, a system, a device and a medium. BACKGROUND

[0002] Industrial safety has always been a core issue in modern production systems. With the development of automation and intensive operations, high-risk sites such as construction sites, factory sites, warehouses, and the like are increasingly complex, and accidents caused by violations occur frequently, posing a major threat to personnel safety and property protection. Dangerous factors such as high temperature, high altitude, high pressure, chemicals, and high-speed rotating machinery exist widely in various industrial operating environments. Once the operating personnel violate the operating specifications, do not use personal protective equipment correctly, or enter restricted areas by mistake, it is easy to cause accidents and even systemic safety hazards. Therefore, how to realize the timely perception of operating behavior and the automatic identification of violation risks has become a key link in the intelligent management of industry.

[0003] Traditional manual supervision methods such as manual and video monitoring have obvious bottlenecks. On the one hand, relying on human identification of risk behavior cannot meet the needs of large-scale and high-density operations in terms of personnel allocation and response speed, and it is difficult to achieve continuity and comprehensive coverage. On the other hand, traditional visual algorithms based on simple rules or threshold triggering can only identify predefined single events and cannot flexibly cope with complex and variable actual working conditions. In the face of dynamic changes in the field environment and the demand for multi-source information fusion, intelligent recognition technology based on computer vision is increasingly introduced into the field of industrial safety to assist in realizing real-time, high-precision violation behavior detection and early warning response.

[0004] The current widely used image recognition method is mostly based on target detection and action recognition models such as YOLO, FasterR-CNN, I3D, etc. Although these models have certain static or dynamic target analysis capabilities, they have obvious limitations in multi-factor condition combination judgment and complex semantic relationship understanding. In particular, in the industrial scene, violation behaviors often have high semantic dependency and context sensitivity. For example, the behavior of "working at a high place without wearing a safety belt" requires not only spatial positioning and target recognition, but also understanding of the relationship between the operating state and the personnel equipment. Such tasks are beyond the expression ability of traditional classification or detection models.

[0005] In recent years, the development of visual language models has brought new solutions. By jointly modeling images and text, models can understand more complex semantic instructions and perform tasks such as image question answering and image-text matching, demonstrating strong multi-modal perception capabilities in general scenarios. However, such models typically have a large number of parameters, high training and inference costs, and rely on large-scale computing resources, making them unsuitable for edge computing environments, especially for deployment on site equipment such as construction sites and factory floors. At the same time, their training data is mostly sourced from general corpora and internet images, which has significant transfer differences for professional risk behavior recognition in industrial scenarios. Models often struggle to accurately determine the nature of specific risk operations and lack the ability to model causal relationships between entities, behaviors, and environments, resulting in biased or ambiguous recognition results.

[0006] In addition, industrial safety scenarios place higher demands on the explainability, response speed, and deployment flexibility of models. Models that only focus on classification accuracy or detection performance cannot meet the actual needs of precise tracing and real-time warning of abnormal behavior. SUMMARY

[0007] The present application provides a model training method, a risk behavior recognition method, a system, a device, and a medium, which are used to solve the problem of large number of parameters of traditional visual language models for identifying risk behaviors and difficulty in edge deployment in the prior art.

[0008] In a first aspect, the present application provides a training method of a risk behavior recognition model, comprising: obtaining a training data set; the training data set comprises image data and text data; training the risk behavior recognition model based on the training data set using a multi-stage distillation training strategy to obtain a trained risk behavior recognition model; the multi-stage distillation training strategy comprises distillation training on the semantic representation, attention distribution, and / or risk label stage of the risk behavior recognition model.

[0009] In an implementation form of the first aspect, the training of the risk behavior recognition model based on the training data set using the multi-stage distillation training strategy to obtain the trained risk behavior recognition model comprises: training the risk behavior recognition model and a reference risk behavior recognition model based on the training data set using the multi-stage distillation training strategy; the number of training times of the risk behavior recognition model is less than or equal to a preset training time threshold; if the similarity between the current training result of the risk behavior recognition model and the current training result of the reference risk behavior recognition model meets a preset threshold within the preset training time threshold, the risk behavior recognition model at this time is taken as the trained risk behavior recognition model; if the similarity between the training result of the risk behavior recognition model and the training result of the reference risk behavior recognition model does not meet the preset threshold within the preset training time threshold, the weight parameters of the risk behavior recognition model are adjusted and retrained.

[0010] In an implementation form of the first aspect, the preset threshold comprises a first preset threshold, a second preset threshold and a third preset threshold; the similarity between the current training result of the risk behavior recognition model and the current training result of the reference risk behavior recognition model satisfies the preset threshold, and the risk behavior recognition model at this time is taken as the trained risk behavior recognition model comprises: obtaining a first training result and a second training result; the first training result is the training result of the risk behavior recognition model; the second training result is the training result of the reference risk behavior recognition model; the training result comprises joint representation, attention distribution and task prediction result; similarity between the joint representation, the attention distribution and the task prediction result in the first training result and the joint representation, the attention distribution and the task prediction result in the second training result is judged respectively; if the similarity between the joint representation in the first training result and the joint representation in the second training result is greater than or equal to the first preset threshold, the difference between the attention distribution in the first training result and the attention distribution in the second training result is lower than the second preset threshold, and the similarity between the task prediction result in the first training result and the task prediction result in the second training result is greater than or equal to the third preset threshold, the risk behavior recognition model at this time is taken as the trained risk behavior recognition model.

[0011] In an implementation form of the first aspect, the similarity between the training result of the risk behavior recognition model and the training result of the reference risk behavior recognition model does not satisfy the preset threshold, and the adjustment of the weight parameter of the risk behavior recognition model and the retraining comprise: if the similarity between the joint representation in the first training result and the joint representation in the second training result is less than the first preset threshold, the weight parameter of the semantic alignment distillation loss function of the risk behavior recognition model is adjusted and retrained; if the difference between the attention distribution in the first training result and the attention distribution in the second training result is greater than or equal to the second preset threshold, the weight parameter of the attention distillation loss function of the risk behavior recognition model is adjusted and retrained; if the similarity between the task prediction result in the first training result and the task prediction result in the second training result is less than the third preset threshold, the weight parameter of the task distillation loss function of the risk behavior recognition model is adjusted and retrained.

[0012] In a second aspect, the present application provides a risk behavior identification method, the method comprising: obtaining image data and text data; inputting the image data and the text data into a trained risk behavior identification model obtained by the training method described above to obtain a risk behavior identification result of the image data; the risk behavior identification result comprising image description text of the image data and a risk label list; the risk label list comprising labels of each risk behavior existing in the image data.

[0013] In an implementation form of the second aspect, inputting the image data and the text data into the trained risk behavior identification model obtained by the training method described above to obtain the risk behavior identification result of the image data comprises: performing feature extraction on the image data and the text data respectively to obtain corresponding image features and text features; performing semantic compression processing based on the image features and the text features; obtaining the image features and the text features after the semantic compression processing; performing image-text fusion based on the image features and the text features after the semantic compression processing to obtain a joint representation of the image data and the text data; the joint representation comprising spatial structure information of the image data and risk element semantics of the text data; obtaining the image description text of the image data and the risk label list based on the joint representation; the risk label list comprising risk labels of each risk behavior existing in the image data.

[0014] In a third aspect, the present application provides a risk behavior identification system, the system comprising: an obtaining module configured to obtain image data and text data; a risk behavior identification module configured to input the image data and the text data into a trained risk behavior identification model obtained by the training method described above to obtain a risk behavior identification result of the image data; the risk behavior identification result comprising image description text of the image data and a risk label list; the risk label list comprising labels of each risk behavior existing in the image data.

[0015] In an implementation form of the third aspect, the risk behavior identification module comprises: a feature extraction unit configured to perform feature extraction based on the image data and the text data respectively to obtain corresponding image features and text features; a semantic compression unit configured to perform semantic compression processing based on the image features and the text features; obtain image features and text features after semantic compression processing; a graph-text fusion unit configured to perform graph-text fusion based on the image features and the text features after semantic compression processing to obtain joint representation of the image data and the text data; the joint representation comprises spatial structure information of the image data and risk element semantics of the text data; a multi-task output unit configured to obtain image description text of the image data and a risk label list based on the joint representation; the risk label list comprises risk labels of each risk behavior existing in the image data.

[0016] In a fourth aspect, the present application provides an electronic device, comprising: a memory storing a computer program; a processor connected in communication with the memory, and when the computer program is invoked, the processor executes the training method of the risk behavior identification model or the risk behavior identification method as described above.

[0017] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the training method of the risk behavior identification model or the risk behavior identification method as described above is implemented.

[0018] As described above, the model training method, the risk behavior identification method, the system, the device and the medium provided by the present application have the following beneficial effects:

[0019] The present application obtains a training data set; the training data set comprises image data and text data; a risk behavior identification model is trained based on the training data set using a multi-stage distillation training strategy to obtain a trained risk behavior identification model; the multi-stage distillation training strategy comprises distillation training of semantic representation, attention distribution and / or risk label stage of the risk behavior identification model. The present application trains the risk behavior identification model using the multi-stage distillation training strategy, so that the small model has a multi-task performance close to the level of large model while significantly reducing the memory and computing power consumption, solving the problem of large parameter quantity of the traditional visual language model for identifying risk behaviors in the prior art and difficult to deploy on the edge, and also solving the problem that small models are difficult to balance semantic understanding and risk behavior identification accuracy under limited parameter scale.

[0020] The application trains a risk behavior recognition model through a multi-stage distillation training strategy. First, a large-scale pre-trained visual language model is used as a teacher network to extract knowledge from three levels of image-text alignment, intermediate representation, and behavior attention distribution. Second, a behavior semantic guidance mechanism is introduced in a lightweight student model, using the teacher model attention map as a supervision signal to guide the small model to accurately focus on the key risk areas in the image. Finally, the training stability and model convergence speed are improved through freezing part of the parameters and training in stages, so that the small model significantly reduces memory and computing power consumption while maintaining multi-task performance close to that of large models. Through multi-stage knowledge distillation training, the small model effectively inherits the multi-modal knowledge of the large model, maintaining high recognition accuracy and robustness while significantly reducing model size.

[0021] The application introduces a lightweight visual backbone network in the model architecture design of the risk behavior recognition model, effectively compresses the parameter quantity and computational complexity of the visual coding module by using an efficient coding structure. At the same time, a Prompt-aware Semantic Bottleneck module is integrated inside the model, which can dynamically focus on semantic information related to task prompts during image-text interaction, automatically compress redundant attention paths, and improve image-text alignment efficiency and expression ability of downstream tasks.

[0022] The application constructs a multi-task output structure to support multi-label violation behavior recognition and image question answering tasks while ensuring backbone sharing. By guiding the model to understand complex semantic commands such as "whether there is high-altitude operation without safety rope", the model can simultaneously perform visual entity recognition, semantic judgment, and behavior classification tasks, significantly improving the model's generalization ability and practicality.

[0023] The application proposes a risk behavior modeling method based on causal graph. This method constructs a scene graph structure based on the semantic relationship between entities and behaviors in the image, and generates "behavior-consequence" causal paths combined with task instructions, achieving semantic modeling and reasoning judgment of implicit violation operations. This mechanism significantly improves the recognition effect of combined high-risk behaviors and has certain interpretability. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 An application scenario diagram of the training method of the risk behavior recognition model and the risk behavior recognition method described in the embodiments of the application is shown.

[0025] Figure 2 A structural diagram of the risk behavior recognition model described in the embodiments of the application is shown.

[0026] Figure 3A flowchart of a training method of a risk behavior identification model is shown.

[0027] Figure 4 A flowchart of a risk behavior identification method is shown.

[0028] Figure 5 A structural diagram of a trained risk behavior identification model is shown.

[0029] Figure 6 A structural diagram of a risk behavior identification system is shown.

[0030] Figure 7 A structural diagram of a risk behavior identification system is shown.

[0031] Figure 8 A structural diagram of an electronic device is shown.

[0032] Element number explanation DETAILED DESCRIPTION

[0033] The embodiments of the present application will be described in detail hereinafter with specific reference on the attached drawings. It is to be noted that the following description is only exemplary and explanatory in nature and it is intended to include all technical and structural equivalents of the elements included in the present application. The following embodiments and characteristics in the embodiments can be combined with each other without conflict under the premise that they do not conflict.

[0034] It should be noted that the drawings provided in the following embodiments only schematically illustrate the basic concept of the present application, and only the components related to the present application are shown in the drawings, not the number, shape and size of the components when actually implemented. The actual implementation of each component may be randomly changed in type, number and proportion, and the layout type of the components may also be more complex.

[0035] The following embodiments of the present application provide a model training method, a risk behavior identification method, a system, a device and a medium, which solve the problem of large parameter quantity and difficulty in edge deployment of the conventional visual language model for identifying risk behavior in the prior art.

[0036] Figure 1This diagram illustrates an application scenario of the training method and risk behavior recognition method for the risk behavior recognition model described in this application. The risk behavior recognition device 100 can be used to implement the training method and risk behavior recognition method for the risk behavior recognition model provided in the embodiments of this application; however, the application scenarios of the training method and risk behavior recognition method for the risk behavior recognition model provided in the embodiments of this application are not limited to these specific scenarios. Figure 1 The risk behavior identification device 100 shown is an example. Figure 1 As shown, the risk behavior recognition device 100 includes a data acquisition device 101, a processing unit 102, and a display terminal 103. The training method and risk behavior recognition method of the risk behavior recognition model provided in this application embodiment can be applied to the processing unit 102.

[0037] in, Figure 1 The processing unit 102 can be a single processor, a processor cluster consisting of multiple processors, or a cloud computing center, etc., and is not specifically limited here. Although Figure 1 Only one acquisition device 101, one processing unit 102, and one display terminal 103 are shown in the diagram, but it should be understood that... Figure 1 The examples in this paper are only for understanding this solution. The specific number of processing units 102 and display terminals 103 should be flexibly determined based on the actual situation.

[0038] In some other implementations, the risk behavior recognition device 100 may not include a display terminal 103, but only a processing unit 102 with display function and a data acquisition device 101. The training method and risk behavior recognition method of the risk behavior recognition model provided in this application embodiment can be applied to the processing unit 102. The processing unit 102 with display function may include tablet computers, laptops, handheld computers, mobile phones, personal computers, and voice interaction devices, or it may be a monitoring device, etc., which is not limited here.

[0039] In some embodiments, to address the issue of lightweight structure in traditional visual language models, this application proposes a compact modeling method that combines prompting semantic guidance with cross-modal alignment to construct a low-parameter visual language model for identifying risky behaviors in industrial scenarios, namely, a risky behavior identification model.

[0040] Figure 2 The diagram shown is a structural schematic of the risk behavior recognition model described in an embodiment of this application. Figure 2As shown, the risk behavior recognition model includes an input module, an extracted feature module, a feature alignment module, and an output module. The input module is used to input image data and text data, the extracted feature module is used to extract features from the input image data and text data to extract image features and text features, the feature alignment module is used to perform semantic alignment and compression on the extracted image features and text features to obtain a joint representation of the image data and the text data, the joint representation including spatial structure information of the image data and risk element semantics of the text data, and the output module is used to output image description text of the image data and a risk label list according to the joint representation of the image data and the text data, the risk label list including risk labels of each risk behavior existing in the image data.

[0041] In some embodiments, the construction method of the risk behavior recognition model includes: first, in the visual coding part, an efficient and lightweight backbone network such as MobileViT or GhostNet is used to extract features from the input image, to reduce the spatial resolution while maintaining the representation of key information, thereby significantly compressing the parameter size of the visual encoder. Second, a Prompt-aware Semantic Bottleneck (PSB) is introduced, which dynamically filters image feature channels strongly related to risk prompt sentences (such as “whether there is a person without wearing a safety helmet”) using a Cross-Attention mechanism during the image-text interaction stage, filters redundant attention information, and realizes semantic alignment and compression under low computational load. Finally, a unified multi-task output structure is constructed at the top of the model, which supports image description generation and multi-label risk behavior recognition tasks in parallel on the shared joint semantic representation, ensuring the expression ability and adaptability of the model in multiple tasks.

[0042] In some embodiments, for the problem of semantic modeling of complex industrial behaviors, the application introduces a graph-aware behavior relationship modeling method. First, entity and behavior action information such as people, objects, and locations are extracted from the input image to construct a scene structure graph; second, a “risk causal path” is generated in combination with the task prompt, modeling the violation event as a graph structure path between entities and actions (such as “person + high place + no safety rope”); finally, a lightweight graph neural network is introduced to propagate and fuse the causal graph, improving the understanding and explanation ability of the model for combined violation behaviors.

[0043] In some embodiments, for the constructed risk behavior identification model, the present application proposes a multi-level distillation training strategy: synchronous distillation at three levels of semantic representation, attention distribution, and risk label. Considering that existing distillation mostly focuses on the final output layer or a single feature layer, which cannot fully inherit the cross-modal capabilities of large models, the present application proposes a multi-level distillation training strategy, which specifically includes: (a) inheriting cross-modal semantic alignment capability through joint representation distillation; (b) guiding the student model to learn the “attention area” of the large model through attention distillation; (c) combining teacher prediction and real label double supervision in multi-label identification through task distillation. Combined with the phased optimization strategy of freezing-unfreezing, this method realizes the balance of performance retention and parameter compression, and realizes near real-time inference on edge devices.

[0044] The low-parameter visual language model proposed in the present application adopts a multi-stage distillation training strategy to solve the problem that small models are difficult to balance semantic understanding and risk behavior identification accuracy under limited parameter scale. The training process is as follows:

[0045] First, prepare the input data. The industrial scene image is preprocessed, including uniform scaling to 224x224 resolution, normalization, and channel standardization; the prompt sentence is input in Chinese form and embedded in a predefined template, for example, “Is there a person without a safety helmet in the picture?”.

[0046] Second, input the image and prompt sentence into the teacher model (such as Qwen-VL or BLIP-2) and the student model at the same time. The teacher model, as a large-scale pre-trained visual language model, can provide multi-level knowledge supervision signals, including: (1) joint representation features of image and text; (2) attention weight distribution of Cross-Attention layer; (3) risk behavior label prediction results. The student model is composed of a lightweight visual encoder (MobileViT or GhostNet), a self-developed text encoder, a Prompt-aware semantic compression module, an image-text fusion module, and a multi-task output head, which respectively output corresponding joint representation, attention distribution, and task prediction results.

[0047] Then, design multiple distillation loss functions to guide the student model to approximate the performance of the teacher model:

[0048] Semantic alignment distillation loss: use mean square error (MSE) to constrain the difference between student and teacher in joint representation of image and text, to ensure that the student model inherits the cross-modal semantic understanding capability;

[0049] Attention distillation loss: use Kullback-Leibler divergence (KL divergence) to minimize the difference between the Cross-Attention attention matrices of the two, to guide the student model to focus on the image regions and text segments consistent with the teacher during inference;

[0050] Task distillation loss: the cross-entropy loss is used to constrain the multi-label risk behavior prediction results of the student model to be consistent with the teacher model;

[0051] Supervised loss: for training samples with artificial labels, the joint form of cross-entropy and Dice Loss is used to optimize the output of the student model and the true label.

[0052] In the training process, the AdamW optimizer is used, the initial learning rate is set to 1x10^(-4), the batch size is 32, and the total training round is 50. The learning rate is gradually decayed to 1x10^(-6) according to the cosine decay strategy. To improve the training stability, the bottom encoder of the student model is frozen at the beginning of training, and only the top module parameters are optimized; as the training iteration deepens, the bottom network is gradually unfrozen and the whole model is fine-tuned.

[0053] After training by the above distillation strategy, the parameter amount of the student model is controlled between 0.3B and 0.5B, but the risk behavior recognition accuracy is close to or even better than that of the small model without distillation. The deployment test results show that the average inference delay of the student model in this embodiment on edge devices such as Jetson Nano and Ascend 310 is less than 100 milliseconds, which can meet the strict real-time requirements of industrial safety scenarios.

[0054] Through the collaborative design of structure optimization, distillation training and causal relationship modeling, the application effectively improves the accuracy, flexibility and deployment adaptability of visual language models in industrial risk identification tasks while greatly compressing the model parameter amount and computational overhead. This method has good scalability and practicality, and can be widely applied to safety-sensitive scenarios such as construction sites, intelligent factories and warehouse logistics, helping to build a real-time, intelligent and highly reliable industrial safety supervision system.

[0055] It should be noted that the model training method, risk behavior identification method, system, device and medium described in the following embodiments of the application are designed to accurately identify high-risk violations in industrial scenarios, improve the intelligent level of work safety management, and meet the comprehensive needs of real-time, high performance and deployability in the fields of smart construction sites, intelligent factories, warehouse logistics, etc.

[0056] The technical solutions in the embodiments of the application will be described in detail below with reference to the drawings in the embodiments of the application.

[0057] As shown in Figure 3 The training method of the risk behavior identification model provided by the embodiment includes the following steps S301-S302.

[0058] Step S301, obtaining a training data set; the training data set includes image data and text data.

[0059] In some embodiments, the training data set of the present application includes image data and text data, the image data is live image from a camera or an upload system, and the text data is a task instruction prompt text set manually or built-in in the system, in the form of "Is there a person without a safety helmet in the picture?" "Is there a high-altitude operation without protective measures?" and the like.

[0060] In some embodiments, the image data from the industrial scene image is subjected to a preprocessing operation, including uniform scaling to a resolution of 224x224, normalization and channel standardization; the text data and the like prompt statements are input in Chinese form and embedded in a predefined template, such as "Is there a person without a safety helmet in the picture?".

[0061] Step S302, training a risk behavior recognition model based on the training data set using a multi-stage distillation training strategy, and obtaining a trained risk behavior recognition model; the multi-stage distillation training strategy includes distillation training of semantic representation, attention distribution and / or risk label stage of the risk behavior recognition model.

[0062] The multi-stage distillation training strategy is adopted for the risk behavior recognition model constructed in the present application, and distillation is simultaneously performed at three levels of semantic representation, attention distribution and risk label, so as to solve the problem that a small model is difficult to balance semantic understanding and risk behavior recognition accuracy under a limited parameter scale.

[0063] In an embodiment of the present application, training a risk behavior recognition model based on the training data set using a multi-stage distillation training strategy, and obtaining a trained risk behavior recognition model includes the following steps S3021 to S3023.

[0064] Step S3021, training a risk behavior recognition model and a reference risk behavior recognition model based on the training data set using a multi-stage distillation training strategy; the number of training times of the risk behavior recognition model is less than or equal to a preset training time threshold.

[0065] Among them, the risk behavior recognition model is the student model in the above-mentioned embodiments, and the reference risk behavior recognition model is the teacher model in the above-mentioned embodiments. The teacher model adopts a large-scale pre-trained visual language model, for example, Qwen-VL or BLIP-2.

[0066] Step S3022, if the similarity between the current training result of the risk behavior recognition model and the current training result of the reference risk behavior recognition model meets a preset threshold within a preset training number threshold, the risk behavior recognition model at this time is taken as the trained risk behavior recognition model.

[0067] The preset training number threshold is the total training round number of model training. Taking 50 as an example, the preset training number threshold is 50.

[0068] Step S3023, if the similarity between the training result of the risk behavior recognition model and the training result of the reference risk behavior recognition model does not meet the preset threshold within the preset training number threshold, the weight parameter of the risk behavior recognition model is adjusted and retrained.

[0069] In some embodiments, within the preset training number threshold, if the similarity between the training result of the risk behavior recognition model and the training result of the reference risk behavior recognition model meets the preset threshold, the risk behavior recognition model at this time is taken as the trained risk behavior recognition model, otherwise the weight parameter of the risk behavior recognition model is adjusted and retrained.

[0070] In an embodiment of the present application, the preset threshold includes a first preset threshold, a second preset threshold and a third preset threshold; the similarity between the current training result of the risk behavior recognition model and the current training result of the reference risk behavior recognition model meets the preset threshold, and the risk behavior recognition model at this time is taken as the trained risk behavior recognition model, including steps S30221 to S30223.

[0071] Step S30221, a first training result and a second training result are obtained; the first training result is the training result of the risk behavior recognition model; the second training result is the training result of the reference risk behavior recognition model; the training result includes joint representation, attention distribution and task prediction result.

[0072] Step S30222, similarity judgment is performed on the joint representation, attention distribution and task prediction result in the first training result and the joint representation, attention distribution and task prediction result in the second training result respectively.

[0073] Step S30223, if the similarity between the joint representation in the first training result and the joint representation in the second training result is greater than or equal to a first preset threshold, the difference between the attention distribution in the first training result and the attention distribution in the second training result is lower than a second preset threshold, and the similarity between the task prediction result in the first training result and the task prediction result in the second training result is greater than or equal to a third preset threshold, the risk behavior recognition model at this time is taken as the trained risk behavior recognition model.

[0074] In this embodiment, the similarity between the joint representation in the first training result and the joint representation in the second training result is greater than or equal to a first preset threshold, the difference between the attention distribution in the first training result and the attention distribution in the second training result is lower than a second preset threshold, and the similarity between the task prediction result in the first training result and the task prediction result in the second training result is greater than or equal to a third preset threshold, all of which must be met at the same time, so that the risk behavior recognition model at this time is taken as the trained risk behavior recognition model. If any one of them is not met, the weight parameter of the corresponding loss function needs to be adjusted, and the model needs to be retrained.

[0075] In an embodiment of the present application, if the similarity between the training result of the risk behavior recognition model and the training result of the reference risk behavior recognition model does not meet the preset threshold, adjusting the weight parameter of the risk behavior recognition model and retraining include the following steps S30231 and S30233.

[0076] Step S30231, if the similarity between the joint representation in the first training result and the joint representation in the second training result is less than the first preset threshold, the weight parameter of the semantic alignment distillation loss function of the risk behavior recognition model is adjusted and retrained.

[0077] Step S30232, if the difference between the attention distribution in the first training result and the attention distribution in the second training result is greater than or equal to the second preset threshold, the weight parameter of the attention distillation loss function of the risk behavior recognition model is adjusted and retrained.

[0078] Step S30233, if the similarity between the task prediction result in the first training result and the task prediction result in the second training result is less than the third preset threshold, the weight parameter of the task distillation loss function of the risk behavior recognition model is adjusted and retrained.

[0079] The application proposes a low-parameter visual language model risk behavior recognition method, aiming to accurately identify high-risk violations in industrial scenes in a computing resource-limited environment, improve the intelligent level of work safety management, and meet the comprehensive needs of real-time performance, high performance, and deployability in fields such as smart construction sites, intelligent factories, and warehouse logistics.

[0080] Firstly, the application introduces a lightweight visual backbone network in the model architecture design, effectively compressing the parameter quantity and computational complexity of the visual coding module by using efficient coding structures such as MobileViT and GhostNet. At the same time, a Prompt-aware Semantic Bottleneck module is integrated into the model, which can dynamically focus on semantic information related to task prompts during image-text interaction, automatically compress redundant attention paths, and improve image-text alignment efficiency and downstream task expression ability.

[0081] Secondly, the application constructs a multi-task output structure, which supports multi-label violation behavior recognition and image question answering tasks while ensuring backbone sharing. By guiding the model to understand the combination of "image + question", such as "whether there is high-altitude work without safety rope", the model can simultaneously perform visual entity recognition, semantic judgment, and behavior classification tasks, significantly improving the generalization ability and practicality of the model.

[0082] In terms of training strategy, the application designs a multi-stage distillation training method. First, use a large-scale pre-trained visual language model (such as Qwen-VL or BLIP-2) as a teacher network to extract knowledge from image-text alignment, intermediate representation, and behavior attention distribution; second, introduce a behavior semantic guidance mechanism in the lightweight student model, using the teacher model attention map as a supervision signal to guide the small model to accurately focus on key risk areas in the image; finally, freeze part of the parameters and train in stages to improve training stability and model convergence speed, so that the small model can perform close to the large model level while significantly reducing memory and computing power consumption.

[0083] In addition, to enhance the model's understanding of behavior semantic logic, the application proposes a risk behavior modeling method based on causal graph. This method constructs a scene graph structure through the semantic relationship between entities (such as personnel, equipment, and location) and behaviors (such as standing, climbing, and using tools) in the image, and generates "behavior-consequence" causal paths combined with task instructions, achieving semantic modeling and reasoning judgment of implicit violation operations. This mechanism significantly improves the recognition effect of combined high-risk behaviors and has certain interpretability.

[0084] Finally, the model parameters of the application are controlled between 0.3B and 0.5B, with high adaptability, low delay and high deployment characteristics, adapting to mainstream edge devices, and can be directly applied to industrial safety scenes such as construction sites, factory inspections, and warehouse operations, to realize real-time analysis of the state of workers, automatic identification of illegal behavior and risk prompts and other functions.

[0085] In summary, the low-parameter visual language model risk behavior identification method proposed in the application has the following significant advantages: it significantly reduces the model parameter quantity and reasoning resource consumption while ensuring multi-modal understanding and identification accuracy, balances task generalization and deployment flexibility, has good scalability and engineering landing potential, and is of great significance for promoting the popularization of AI-assisted safety supervision systems in industrial scenarios.

[0086] Figure 4 The flowchart shows the risk behavior identification method described in the embodiments of the application. As shown in Figure 4 The risk behavior identification method includes the following steps S401-S402.

[0087] Step S401, obtaining image data and text data.

[0088] In some embodiments, in actual application, the application provides two types of input: one is the live image from the camera or the uploading system, and the other is the task instruction prompt text set by human or built-in in the system, such as "whether there is a worker without safety helmet in the picture", "whether there is a high-altitude operation without protective measures" and the like. This double-input structure enables the model to adjust its identification focus according to the specific task target, realizing image understanding based on human instructions, which is more flexible and practical compared with the traditional classification or detection method relying only on the image itself.

[0089] Step S402, inputting the image data and the text data into the trained risk behavior identification model obtained by the training method described above to obtain the risk behavior identification result of the image data; the risk behavior identification result includes the image description text of the image data and the risk label list; the risk label list includes the labels of each risk behavior existing in the image data.

[0090] The application provides a low-parameter visual language model risk behavior identification method, which aims to solve the problems of high computing resource occupation, high deployment difficulty and insufficient scene generalization ability of existing models, and is especially suitable for industrial safety scenarios that need to run in real time on edge devices. Through steps such as structure compression, image-text semantic fusion, prompt-guided identification and graph modeling, accurate identification and semantic analysis of risk behaviors in images are realized under limited resource conditions.

[0091] In an embodiment of the present application, the image data and the text data are input into the trained risk behavior recognition model obtained by the training method described above to obtain a risk behavior recognition result of the image data, including the following steps S4021-S4024.

[0092] In step S4021, image features and text features are extracted based on the image data and the text data, respectively.

[0093] In some embodiments, the present application separately encodes the image and text inputs: the image part uses a lightweight convolutional neural network (such as GhostNet, MobileViT, etc.) to extract multi-level image features, i.e., image features; the text part uses a small Transformer language encoder to extract semantic vectors of task instructions, i.e., text features.

[0094] In step S4022, semantic compression processing is performed based on the image features and the text features; and image features and text features after the semantic compression processing are obtained.

[0095] In some embodiments, the present application inputs the extracted image features and text features into a Prompt-aware Semantic Bottleneck (PSB) module, which dynamically compresses the image features by calculating the correlation between the image features and the text prompts, and only retains the part of the features that are strongly related to the semantic content of the prompts, thereby greatly reducing the computational overhead of the image-text interaction module without sacrificing the recognition performance.

[0096] In step S4023, image-text fusion is performed based on the image features and the text features after the semantic compression processing to obtain a joint representation of the image data and the text data; the joint representation includes spatial structure information of the image data and risk element semantics of the text data.

[0097] In some embodiments, the present application performs image-text fusion on the image features and the text features after the semantic compression processing, and fuses the compressed and aligned image features and prompt vector under a shallow cross-attention mechanism to generate a joint representation with strong semantic guidance capability. This representation not only reflects the spatial structure information in the image, but also embeds the risk elements concerned by the task instructions.

[0098] In step S4024, image description text and a risk label list of the image data are obtained based on the joint representation; the risk label list includes risk labels of each risk behavior existing in the image data.

[0099] In some embodiments, the present application will obtain the image description text and the risk label list of the image data according to the joint representation, that is, input the joint representation into two output branches: the image description module is responsible for generating natural language text to summarize the key behavior scene in the image; the risk behavior recognition module judges whether several predefined industrial risk behaviors appear in the image based on a multi-label classification head, such as not wearing a safety helmet, working at a high place, not using a safety belt, etc. The multi-label output form allows the model to identify multiple behaviors simultaneously in a single inference, with higher identification efficiency and practical value.

[0100] In some embodiments, in order to improve the ability of the model in combined behavior understanding and causal chain modeling, the present application introduces a graph perception mechanism to construct a scene causal graph. First, the present application uses a trained auxiliary module to identify entity nodes (such as personnel, safety equipment, and work equipment) and behavior nodes (such as standing, operating, and climbing) from the image. Then, through a triple relationship (entity-behavior-space / equipment), a risk causal graph is constructed. The graph is input into a lightweight graph neural network for propagation and inference, thereby judging the existence of certain “combined” violation behaviors, such as “personnel + high place + not wearing a safety belt”. This mechanism enhances the model’s understanding ability of complex behavior scenes and provides structural support for subsequent explanatory output.

[0101] In some embodiments, the present application integrates the results of image-text fusion and causal graph reasoning to output two types of information:

[0102] The first type is the image semantic description, that is, a brief natural language directly describing the main behaviors and scene elements in the image, such as: “In the image, a construction worker is working at a high altitude without wearing a safety helmet and is approaching a hoisting device.” This text can be used for voice broadcasting, log recording, or auxiliary manual review;

[0103] The second type is the structured risk behavior recognition result table, which is a two-dimensional table containing “risk behavior name” and “identification result (True / False)”, used to clearly list the high-risk behaviors identified by the model and mark their judgment results, facilitating subsequent alarm, statistical analysis, or compliance assessment.

[0104] Figure 5 The structural schematic diagram of the trained risk behavior recognition model described in the embodiments of the present application is shown. As shown in Figure 5 The trained risk behavior recognition model includes an input module, a feature extraction module, a feature alignment module, and an output module. The feature extraction module includes a lightweight visual encoder and a text encoder for encoding image and text features. The feature alignment module includes a Prompt-aware semantic compression module, an image-text fusion module, and a multi-task output module. The functions of each module are as follows:

[0105] Input module: receives images and Chinese prompt sentences as input. Among them, the image is first pre-processed, including uniformly adjusting the image resolution to 224x224 pixels, performing pixel value normalization processing, and standardizing the image channel to meet the model input requirements. The prompt sentence is provided in Chinese form and is embedded into a pre-defined template structure to form a standardized task instruction text. For example, a template format similar to "Is there a person without a safety helmet in the picture?" can be used to describe the risk behavior of interest. This dual input structure enables the model to analyze image content in a targeted manner in combination with specific instructions.

[0106] Feature extraction module, image and text feature encoding: the pre-processed image is input into a lightweight visual encoder for feature extraction. The application preferably uses high-efficiency visual backbone networks such as MobileViT or GhostNet to extract multi-scale semantic features of the image layer by layer, thereby greatly reducing the model parameter quantity and computational quantity while preserving key information. At the same time, the Chinese prompt sentence is input into a Transformer language encoder to extract the semantic vector representation of the text. The language encoder is optimized for Chinese instructions and can generate high-quality text semantic features with fewer parameters. After the above processing, the image feature tensor and the text feature vector are prepared for subsequent cross-modal fusion.

[0107] Prompt-aware semantic bottleneck module: the application inputs the image features and text features into the Prompt-aware Semantic Bottleneck (PSB) module. This module calculates the correlation between the features of each region of the image and the semantic of the text prompt based on the cross-attention mechanism, and uses an attention compression strategy to filter and compress the image features. Specifically, the PSB module only retains the image semantic information channels that are highly related to the text prompt content, and filters out features that are irrelevant to the task or have low attention weights. By dynamically focusing on key semantics during image-text interaction, this module compresses redundant information without significantly sacrificing recognition performance, reduces the computational overhead of subsequent fusion, and improves the efficiency and accuracy of image-text alignment.

[0108] Image-text fusion module: The image features and text features after semantic compression are sent to the image-text fusion module to perform cross-modal feature interaction and fusion at a deeper level. The module uses further cross-attention mechanisms and combines gate mechanisms to control the fusion process and gradually build a semantically consistent joint representation. On the one hand, cross-attention is used to guide the model to embed the semantic attention of the text prompt into the image feature representation; on the other hand, the introduced gate unit adjusts the information flow of the image and text features, suppresses noise or inconsistent feature components, and ensures that the fusion result is consistent with the prompt requirements in semantics. Through the above fusion operation, the output joint representation contains both the spatial structure information of the image and the risk element semantics emphasized by the text prompt, laying a foundation for subsequent multi-task decision-making.

[0109] Multi-task output module: The joint representation is input into the multi-task output module to simultaneously complete the image description generation and risk behavior recognition tasks.

[0110] First, the image description generation sub-module uses a self-attention Transformer decoder to decode the joint representation and generate a fluent Chinese natural language description text, which is used to depict the key behavior scenes and risk situations in the image. For example, the model can generate a sentence describing, for example, "two workers are working at a high place without wearing safety belts."

[0111] Second, the risk behavior recognition sub-module uses a multi-label classification head to judge and output a plurality of pre-defined industrial risk violation behaviors. The multi-label classification head outputs a Boolean value (True / False, i.e., Yes / No) for each risk behavior (such as not wearing a safety helmet, illegal high-altitude operation, not using a safety belt, etc.), indicating whether the risk behavior exists in the image. The multi-label setting allows the model to simultaneously determine the existence of multiple risk behaviors in one inference, greatly improving the coverage and efficiency of the identification. When the output is True, it indicates that the corresponding violation behavior is detected, and when it is False, it indicates that the behavior is not found. Finally, the text description and the risk behavior label judgment results generated by the model will be jointly used as the identification output of the input image.

[0112] Model training strategy: The application proposes an efficient training method for the above model structure.

[0113] The AdamW optimizer is used in the training process to iteratively update the model parameters with a batch size of 32; the initial learning rate is set to 1x10^(-4), and the cosine decay learning rate scheduling strategy is used to gradually reduce the learning rate during the training process, thereby smoothing the convergence process and avoiding falling into local optima. To compensate for the limited capacity of small models that may result in insufficient performance, a multi-modal distillation strategy is introduced during the training phase: a large-scale pre-trained visual language model (e.g., Qwen-VL or BLIP-2) is selected as the teacher model, and the representation ability of the student model is improved through multi-stage knowledge distillation.

[0114] Specifically, the distillation method combines attention weight alignment, image-text representation alignment, and risk label supervision: first, the cross-attention distribution of the student model is made to simulate the teacher model as much as possible to align the image-text attention pattern; second, the difference between the student model and the teacher model in the image-text joint representation is minimized to enable the student model to better inherit the multi-modal understanding ability of the teacher model; and third, the risk behavior labels provided by the teacher model or artificial labeling are used to supervise the student model at the output end to ensure the discriminative ability of the small model on the target task. Through the above distillation training strategies and reasonable hyperparameter configuration, the student model gradually converges and masters similar image-text semantic alignment and behavior recognition ability to the teacher model after about 50 epochs of training iteration. During the entire training process, the application can also use the method of freezing part of the layers first and then gradually unfreezing and fine-tuning to stabilize the training process, thereby improving the generalization performance and convergence speed of the small model.

[0115] Output result form: After the model is trained and finalized, it can output information containing two parts for any input image and prompt sentence in the inference stage. On the one hand, the model outputs a semantic description text (Chinese) about the image content, which clearly describes the recognized personnel behavior and its potential risks; on the other hand, the model outputs a two-dimensional table containing multiple rows and columns, where each row corresponds to a pre-defined risk behavior category, and each column gives the judgment result (True / False) of whether the behavior of that category is detected. For example, for an input image, the model can give a textual description of "there is a person on site who does not wear a safety helmet and enters a restricted area", and generate corresponding table entries such as [does not wear a safety helmet]: True [high-altitude operation: False] [does not wear a safety belt: True], etc., to visually present the identification of various risk behaviors. This output form combines qualitative description and quantitative labels, facilitating system integration and rapid understanding of model judgment results by human supervisors.

[0116] To sum up, the application realizes a low-parameter but complete visual language model risk behavior recognition method through the above technical solutions. With the help of light-weight visual coding and prompt-guided feature compression, the method greatly reduces the model parameter size and computing cost, enabling the model to run smoothly on edge devices such as Jetson Nano, Huawei Ascend, etc., meeting the real-time requirements of industrial sites. At the same time, cross-modal semantic fusion and multi-task output design enable the model to flexibly handle various violation behavior recognition tasks in complex scenarios, outputting both human-readable text descriptions and detailed risk label judgment results, improving the interpretability and practical value of the recognition results. Through multi-stage knowledge distillation training, the small model effectively inherits the multi-modal knowledge of the large model, maintaining high recognition accuracy and robustness while greatly compressing the model size. In summary, the method of the application has high efficiency, accuracy and deployment convenience, and can be widely applied to safety monitoring fields such as construction sites, factory inspections, warehouse operations, etc., and has important significance for improving the intelligent level of industrial safety management.

[0117] The protection scope of the risk behavior recognition method described in the embodiments of the application is not limited to the order of steps listed in the embodiments, and any scheme realized by adding, replacing or changing steps of the prior art according to the principles of the application is included in the protection scope of the application.

[0118] The embodiments of the application also provide a risk behavior recognition system, which can implement the risk behavior recognition method described in the application. However, the implementation device of the risk behavior recognition method described in the application includes but is not limited to the structure of the risk behavior recognition system listed in the embodiments, and any structure deformation and replacement of the prior art according to the principles of the application is included in the protection scope of the application.

[0119] Figure 6 The structure of the risk behavior recognition system described in the embodiments of the application is shown in the structure diagram. As shown in Figure 6 The risk behavior recognition system 600 includes an acquisition module 601 and a risk behavior recognition module 602.

[0120] The acquisition module 601 is configured to acquire image data and text data.

[0121] The risk behavior recognition module 602 is configured to input the image data and the text data into the trained risk behavior recognition model obtained by the training method described above to obtain a risk behavior recognition result of the image data; the risk behavior recognition result includes an image description text of the image data and a risk label list; the risk label list includes labels of each risk behavior existing in the image data.

[0122] Figure 7 A structural schematic diagram of a risk behavior identification system is shown in the embodiments of the present application. As shown, the risk behavior identification module 602 includes a feature extraction unit 6021, a semantic compression unit 6022, a graph-text fusion unit 6023, and a multi-task output unit 6024. Figure 7

[0123] The feature extraction unit 6021 is configured to perform feature extraction based on the image data and the text data respectively, and obtain corresponding image features and text features.

[0124] The semantic compression unit 6022 is configured to perform semantic compression processing based on the image features and the text features, and obtain image features and text features after semantic compression processing.

[0125] The graph-text fusion unit 6023 is configured to perform graph-text fusion based on the image features and the text features after semantic compression processing, and obtain a joint representation of the image data and the text data; the joint representation includes spatial structure information of the image data and risk element semantics of the text data.

[0126] The multi-task output unit 6024 is configured to obtain image description text of the image data and a risk label list based on the joint representation; the risk label list includes risk labels of each risk behavior existing in the image data.

[0127] It should be noted that the functions or operations of the obtaining module 601 and the risk behavior identification module 602, the feature extraction unit 6021, the semantic compression unit 6022, the graph-text fusion unit 6023, and the multi-task output unit 6024 described in the embodiments of the present disclosure correspond one by one to the steps in the risk behavior identification method described above, and therefore will not be repeated here.

[0128] In several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules / units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, device or module or unit indirect coupling or communication connection, which can be electrical, mechanical or other forms.

[0129] ​The modules / units described as separated components may or may not be physically separated, and the components shown as modules / units may or may not be physical modules, i.e., may be located in one place, or may be distributed on multiple network units. Part or all of the modules / units can be selected according to actual needs to achieve the purposes of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application can be integrated in one processing module, or each module / unit can be physically present alone, or two or more modules / units can be integrated in one module / unit.

[0130] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0131] The embodiment provides an electronic device, which comprises a memory and a processor.

[0132] The memory stores a computer program.

[0133] The processor is connected in communication with the memory, and when the computer program is invoked, executes the training method of the risk behavior identification model or the risk behavior identification method as described above.

[0134] Figure 8 The structure schematic diagram of the electronic device 800 described in the embodiments of the present application is shown. As shown in the figure, Figure 8 The electronic device 800 in the embodiment comprises a memory 801 and a processor 802.

[0135] The memory 801 is used for storing a computer program; preferably, the memory 801 comprises a ROM, a RAM, a magnetic disc, a U disk, a memory card, or an optical disc, and various media capable of storing program codes.

[0136] In particular, the memory 801 can include a computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The electronic device 800 can further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory 801 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application. It should be appreciated that the memory 801 can be in the form of volatile memory or non-volatile memory, or both. The non-volatile memory can be read only memory (ROM), programmable read only memory (PROM), and used as external cache memory. By way of example, and not limitation, a number of forms of RAM can be used, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in embodiments of the application is intended to include, but not be limited to, these and any other suitable types of memory.

[0137] The processor 802 is connected with the memory 801, and is configured to execute the computer program stored in the memory 801, so that the electronic device 800 executes the training method of the risk behavior recognition model or the risk behavior recognition method described above.

[0138] Optionally, the processor 802 can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0139] Optionally, the electronic device 800 in the embodiment can further include a display 803. The display 803 is connected with the memory 801 and the processor 802, and is configured to display a related graphical user interface (GUI) of the training method of the risk behavior recognition model or the risk behavior recognition method.

[0140] The embodiment of the present application further provides a computer readable storage medium. Those skilled in the art can understand that all or part of the steps of the method described in the above embodiment can be completed by a program instructing a processor, and the program can be stored in a computer readable storage medium. The storage medium is a non-transitory medium, for example, a random access memory, a read only memory, a flash memory, a hard disk, a solid state disk, a magnetic tape, a floppy disk, an optical disc and any combination thereof. The storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center and the like integrated with one or more available medium sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital video disc (DVD)) or a semiconductor medium (for example, a solid state disk (SSD)) and the like.

[0141] The embodiment of the present application can further provide a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in the embodiments of the present application are generated. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer or data center to another website, computer or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave and the like) mode.

[0142] When the computer program product is executed by a computer, the computer executes the method described in the foregoing method embodiment. The computer program product can be a software installation package, and when the foregoing method needs to be used, the computer program product can be downloaded and executed on the computer.

[0143] As described above, the model training method, the risk behavior recognition method, the system, the device and the medium provided by the present application have the following beneficial effects:

[0144] The application obtains a training data set, the training data set includes image data and text data, trains a risk behavior recognition model based on the training data set by using a multi-stage distillation training strategy, and obtains a trained risk behavior recognition model. The multi-stage distillation training strategy includes distillation training of semantic representation, attention distribution and / or risk label stage of the risk behavior recognition model. The application trains the risk behavior recognition model by using the multi-stage distillation training strategy, so that the small model has a multi-task performance close to the level of the large model while significantly reducing memory and computing power consumption, solving the problem of large parameter quantity of the conventional visual language model for identifying risk behavior in the prior art and difficult edge deployment, and also solving the problem of difficulty of small model in balancing semantic understanding and risk behavior recognition accuracy under limited parameter scale.

[0145] The application trains the risk behavior recognition model by using the multi-stage distillation training strategy, first uses a large-scale pre-trained visual language model as a teacher network, extracts knowledge from three levels of image-text alignment, intermediate representation and behavior attention distribution; secondly, a behavior semantic guidance mechanism is introduced in the lightweight student model, taking the teacher model attention map as a supervision signal to guide the small model to accurately focus on the key risk area in the image; finally, the training stability and model convergence speed are improved by freezing part of the parameters and training in stages, so that the small model has a multi-task performance close to the level of the large model while significantly reducing memory and computing power consumption. Through multi-stage knowledge distillation training, the small model effectively inherits the multi-modal knowledge of the large model, and still maintains high recognition accuracy and robustness in the case of greatly compressing the model volume.

[0146] The application introduces a lightweight visual backbone network in the model architecture design of the risk behavior recognition model, effectively compresses the parameter quantity and computational complexity of the visual coding module by using an efficient coding structure. At the same time, a Prompt-aware Semantic Bottleneck module is integrated in the model, which can dynamically focus on semantic information related to the task prompt during image-text interaction, automatically compress redundant attention paths, and improve the efficiency of image-text alignment and the expression ability of downstream tasks.

[0147] The application constructs a multi-task output structure, which supports multi-label violation behavior recognition and image question answering tasks while ensuring backbone sharing. By guiding the model to understand the combination of “image + question”, such as “whether there is high-altitude operation without safety rope”, the model can simultaneously perform visual entity recognition, semantic judgment and behavior classification tasks, and significantly improves the generalization ability and practicality of the model.

[0148] The application proposes a risk behavior modeling method based on a causal graph. The method constructs a scene graph structure through the semantic relationship between entities and behaviors in an image, and generates a "behavior-consequence" causal path in combination with task instructions, to realize semantic modeling and reasoning judgment of implicit violation operations. The mechanism significantly improves the recognition effect of the model on combined high-risk behaviors, and has certain interpretability.

[0149] The description of the flow or structure corresponding to each of the above figures has its own emphasis. The parts not described in detail in a certain flow or structure can be referred to the related description of other flows or structures.

[0150] The above embodiments only exemplarily illustrate the principles and effects of the application, and are not used to limit the application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the application. Therefore, all equivalent modifications or changes completed by those skilled in the art without departing from the spirit and technical thought disclosed by the application should be covered by the claims of the application.

Claims

1. A training method for a risk behavior recognition model, characterized in that, The method comprises: obtaining a training data set; the training data set comprises image data and text data; training a risk behavior recognition model based on the training data set using a multi-stage distillation training strategy, obtaining a trained risk behavior recognition model; the multi-stage distillation training strategy comprises distillation training on the semantic representation, attention distribution and / or risk label stage of the risk behavior recognition model; training the risk behavior recognition model and the reference risk behavior recognition model based on the training data set using a multi-stage distillation training strategy; the number of training times of the risk behavior recognition model is less than or equal to a preset training number threshold; if the similarity between the current training result of the risk behavior recognition model and the current training result of the reference risk behavior recognition model meets the preset threshold within the preset training number threshold, the risk behavior recognition model at this time is taken as the trained risk behavior recognition model; if the similarity between the training result of the risk behavior recognition model and the training result of the reference risk behavior recognition model does not meet the preset threshold within the preset training number threshold, the weight parameters of the risk behavior recognition model are adjusted and retrained; the preset threshold comprises a first preset threshold, a second preset threshold and a third preset threshold; the similarity between the current training result of the risk behavior recognition model and the current training result of the reference risk behavior recognition model meets the preset threshold, and the risk behavior recognition model at this time is taken as the trained risk behavior recognition model, which comprises: obtaining a first training result and a second training result; the first training result is the training result of the risk behavior recognition model; the second training result is the training result of the reference risk behavior recognition model; the training result comprises joint representation, attention distribution and task prediction result; based on the similarity between the joint representation, attention distribution and task prediction result in the first training result and the joint representation, attention distribution and task prediction result in the second training result; if the similarity between the joint representation in the first training result and the joint representation in the second training result is greater than or equal to the first preset threshold, the difference between the attention distribution in the first training result and the attention distribution in the second training result is lower than the second preset threshold, and the similarity between the task prediction result in the first training result and the task prediction result in the second training result is greater than or equal to the third preset threshold, the risk behavior recognition model at this time is taken as the trained risk behavior recognition model. 2.The method of claim 1, wherein, if the similarity between the joint representation in the first training result and the joint representation in the second training result is less than the first preset threshold, the weight parameters of the semantic alignment distillation loss function of the risk behavior recognition model are adjusted and retrained; ​ If a difference between the attention distribution in the first training result and the attention distribution in the second training result is greater than or equal to a second preset threshold, a weight parameter of an attention distillation loss function of the risk behavior recognition model is adjusted and retrained. If a similarity between a task prediction result in the first training result and a task prediction result in the second training result is less than a third preset threshold, a weight parameter of a task distillation loss function of the risk behavior recognition model is adjusted and retrained.

3. A risk behavior recognition method, characterized by, The method comprises: obtaining image data and text data; inputting the image data and the text data into the trained risk behavior recognition model obtained by the training method in any one of claims 1 to 2, to obtain a risk behavior recognition result of the image data; the risk behavior recognition result comprises image description text of the image data and a risk label list; and the risk label list comprises labels of each risk behavior existing in the image data.

4. The risk behavior recognition method of claim 3, wherein, The method comprises: performing feature extraction on the image data and the text data respectively, to obtain corresponding image features and text features; performing semantic compression processing based on the image features and the text features; and obtaining image features and text features after the semantic compression processing; performing image-text fusion based on the image features and the text features after the semantic compression processing, to obtain a joint representation of the image data and the text data; the joint representation comprises spatial structure information of the image data and risk element semantics of the text data; obtaining image description text of the image data and a risk label list based on the joint representation; and the risk label list comprises risk labels of each risk behavior existing in the image data.

5. A risk behavior identification system characterized by, The method comprises: an obtaining module configured to obtain image data and text data; a risk behavior recognition module configured to input the image data and the text data into the trained risk behavior recognition model obtained by the training method in any one of claims 1 to 2, to obtain a risk behavior recognition result of the image data; the risk behavior recognition result comprises image description text of the image data and a risk label list; and the risk label list comprises labels of each risk behavior existing in the image data.

6. The risk behavior identification system of claim 5, wherein, The risk behavior recognition module comprises: a feature extraction unit configured to perform feature extraction on the image data and the text data respectively, to obtain corresponding image features and text features; a semantic compression unit configured to perform semantic compression processing based on the image features and the text features; and obtain image features and text features after the semantic compression processing; an image-text fusion unit configured to perform image-text fusion based on the image features and the text features after the semantic compression processing, to obtain a joint representation of the image data and the text data; the joint representation comprises spatial structure information of the image data and risk element semantics of the text data; a multi-task output unit configured to obtain image description text of the image data and a risk label list based on the joint representation; and the risk label list comprises risk labels of each risk behavior existing in the image data.

7. An electronic device, comprising: The method comprises: a memory storing a computer program; a processor, communicatively connected with the memory, invoking the computer program to execute the training method of the risk behavior identification model according to any one of claims 1 to 2 or the risk behavior identification method according to any one of claims 3 to 4.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the training method of the risk behavior identification model according to any one of claims 1 to 2 or the risk behavior identification method according to any one of claims 3 to 4.

Images