Operation site safety management and control method and system based on multi-modal large model

Abstract

The invention discloses an operation site safety management and control method and system based on a multi-modal large model, and the method comprises the steps: collecting site violation data, and carrying out the preprocessing, and obtaining multi-modal data; constructing a multi-modal large model; carrying out training and optimization on the multi-modal large model by utilizing the preprocessed multi-modal data; and using the optimized multi-modal large model to realize operation site safety management and control. According to the invention, a visible light image, a safety violation knowledge base and other modal data are fused, and a comprehensive safety monitoring and management platform is constructed by using a large model and a big data processing technology, so that real-time monitoring, early warning and intelligent response of potential safety hazards of a complex operation site are realized, and the safety management level of the operation site is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of engineering operation safety management, and specifically relates to a method and system for operation site safety management and control based on a multi-modal large model. Background Art

[0002] With the rapid development of global industrialization, the safety of work sites has gradually become the focus of social attention. In high-risk industries such as construction sites, chemical parks, and mines, it is particularly important to ensure the safety of workers and effectively prevent accidents. Traditional safety management and control methods, such as manual inspections and video surveillance, are often difficult to deal with various safety hazards and violations that may occur in complex and changing environments due to many limitations such as personnel quality, equipment performance, and environmental factors.

[0003] In recent years, with the rapid development of artificial intelligence technology, especially breakthroughs in computer vision and deep learning, more and more artificial intelligence algorithms based on target detection have been applied to the field of safety management and control. These algorithms can automatically identify key elements such as personnel, equipment, and environment at the work site to a certain extent, and monitor their status changes in real time, thereby assisting or replacing manual work in efficient safety inspections and risk warnings.

[0004] However, although these algorithms based on single-modal data (such as video images) have achieved certain results, there are still many challenges in practical application. First, the processing capacity of single-modal data is limited, and it is impossible to fully utilize the multiple information sources at the work site (such as audio, text, sensor data, etc.), which limits the system's understanding of complex scenes and its ability to identify potential risks. For example, in a noisy construction environment, it may be difficult to accurately determine whether a worker is wearing a helmet based solely on visual information; and in conditions of insufficient light or severe occlusion, a single visual data may make it impossible for the system to effectively identify violations.

[0005] Secondly, the existing artificial intelligence algorithms still need to improve their ability to understand and identify complex violations. For example, for some violations that require a combination of multiple actions and environmental factors to judge (such as not wearing a safety rope when working at height, not setting up warning signs in dangerous areas, etc.), single-mode data processing methods often find it difficult to accurately grasp the full picture of the event, resulting in a high rate of missed detection and false detection. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a job site safety management and control method and system based on a multimodal large model in response to the deficiencies in the above-mentioned prior art, so as to solve the technical problem that safety hazards in existing complex job sites cannot be monitored, warned and responded to intelligently in real time.

[0007] The present invention adopts the following technical solutions: A method for safety management and control of a work site based on a multimodal large model comprises the following steps: Collect on-site violation data and pre-process them to obtain multimodal data; Build a large multimodal model; Use preprocessed multimodal data to train and optimize large multimodal models; The optimized multimodal large model is used to output the judgment results of illegal behaviors and realize safety management and control of the work site.

[0008] Preferably, data preprocessing is specifically as follows: The on-site violation images are collected and annotated, a safety violation knowledge base is constructed, the on-site violation images are fused with the safety violation knowledge base to form a multimodal dataset, the multimodal dataset is enhanced and cleaned, and then the multimodal dataset is divided into training set, validation set and test set.

[0009] Preferably, the annotation content includes the type of violation, location, time, and persons or vehicles involved.

[0010] Preferably, the safety violation knowledge base includes violation classification, specific description, relevant law and regulation references, and case analysis.

[0011] Preferably, each data sample in the multimodal data set includes one or more violation images, and corresponding text information such as description, type, hazard, and preventive measures of the violation behavior.

[0012] Preferably, the enhancement processing includes rotation, scaling, cropping and adding noise.

[0013] Preferably, a multimodal large model is constructed to map multimodal data to the same feature space, so that images and texts with similar semantics are closest to each other in the feature space.

[0014] Preferably, the loss function of the multimodal large model is L as follows:

[0015] Where D represents the training data set, (x i ,y i ) represents a pair of images and text in the dataset; z i Represents image x i The eigenvector of i Represents text y i The feature vector of ; s() represents the similarity function; τ is the temperature parameter used to control the smoothness of the similarity; N and M are the total number of texts and images in the dataset, respectively.

[0016] Preferably, the preprocessed data is used to train the constructed multimodal large model in the following manner: During the training process, the multimodal model continuously learns from the training dataset and updates its internal parameters, fits the data, and predicts the output of unknown samples; In each cycle, the multimodal model traverses the entire training data set once, performs forward propagation on each sample to calculate the predicted value, then calculates the loss between the predicted value and the true value, and back-propagates the loss value back to each layer of the network through the back-propagation algorithm to update the weight and bias parameters; Through continuous repetition, the multimodal model gradually learns the inherent laws of the data and optimizes its predictive ability.

[0017] Preferably, the optimization process is as follows: Through tuning methods such as grid search, we can find the hyperparameter configuration that can minimize the loss on the validation set and optimize the generalization ability of large multimodal models.

[0018] In a second aspect, an embodiment of the present invention provides a work site safety management and control system based on a multimodal large model, including: The data module collects on-site violation data and pre-processes it to obtain multi-modal data; Building modules and constructing large multimodal models; The training module uses the preprocessed multimodal data to train and optimize the multimodal large model; The output module uses the optimized multimodal large model to output the judgment results of illegal behaviors and realize safety management and control of the work site.

[0019] Preferably, data preprocessing is specifically as follows: The on-site violation images are collected and annotated, a safety violation knowledge base is constructed, the on-site violation images are fused with the safety violation knowledge base to form a multimodal dataset, the multimodal dataset is enhanced and cleaned, and then the multimodal dataset is divided into training set, validation set and test set.

[0020] Preferably, the annotation content includes the type of violation, the location and time of occurrence, and the persons or vehicles involved; The safety violation knowledge base includes violation classification, specific description, relevant law and regulation references, and case analysis; Each data sample in the multimodal dataset includes one or more violation images, as well as the corresponding text information such as the violation description, type, hazard, preventive measures, etc.; enhancement processing includes rotation, scaling, cropping and adding noise.

[0021] Preferably, a multimodal large model is constructed to map the multimodal data to the same feature space, so that images and texts with similar semantics are closest in the feature space. The loss function of the multimodal large model is L as follows:

[0022] Where D represents the training data set, (x i ,y i ) represents a pair of images and text in the dataset; z i Represents image x i The eigenvector of i Represents text y i The feature vector of ; s() represents the similarity function; τ is the temperature parameter used to control the smoothness of the similarity; N and M are the total number of texts and images in the dataset, respectively.

[0023] In a third aspect, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for job site safety management and control based on a multimodal large model when executing the computer program.

[0024] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for job site safety management and control based on a multimodal large model.

[0025] In a fifth aspect, a chip comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for job site safety management and control based on a multimodal large model when executing the computer program.

[0026] In a sixth aspect, an embodiment of the present invention provides an electronic device, comprising a computer program, which, when executed by the electronic device, implements the steps of the above-mentioned method for job site safety management and control based on a multimodal large model.

[0027] Compared with the prior art, the present invention has at least the following beneficial effects: A job site safety management and control method based on a multimodal large model integrates multimodal data, comprehensively analyzes violations at the job site, realizes real-time monitoring, early warning and intelligent response to safety hazards at complex job sites, and improves the safety management level of the job site.

[0028] Furthermore, by collecting and annotating on-site violation images, the accuracy and relevance of the data can be ensured. High-quality data is the basis for training efficient and accurate models. Furthermore, a safety violation knowledge base is constructed, and on-site violation images are integrated with the safety violation knowledge base to form a multimodal dataset. This process itself relies on data preprocessing to ensure the consistency and compatibility of data from different sources. Preprocessing helps integrate image information and text descriptions (such as violation type, location, etc.), making the knowledge base more comprehensive and easy to retrieve, providing rich resources for subsequent analysis and model training.

[0029] Furthermore, data availability is improved. This annotation method makes the data more available in subsequent analysis, mining, and model training, which helps the model build the logical connection between the equipment and safety violations in the diagram, supporting more complex analysis tasks and more accurate model predictions.

[0030] Furthermore, to enrich data diversity, image enhancement processing generates a series of new images that are similar to but not identical to the original images by performing various transformations on the original images, such as rotation, scaling, translation, flipping, contrast adjustment, color transformation, etc. These new images, as training samples, greatly enrich the diversity of the data set and help the model learn more comprehensive and robust feature representations.

[0031] Furthermore, the data processing process is simplified. By mapping multimodal data to the same feature space, complex conversion and matching between different modalities can be avoided, thereby simplifying the data processing process and better supporting data mining and analysis; data relevance is enhanced. In the feature space, semantically similar images and texts are represented as similar points, which helps the algorithm automatically identify and understand the relevance between data and better assist model training.

[0032] Furthermore, this loss function aims to maximize the similarity between positive sample pairs (i.e., semantically similar images and texts) while minimizing the similarity between negative sample pairs (i.e., semantically dissimilar images and texts). By optimizing two types of logarithmic losses simultaneously, the model's ability to distinguish between similar and dissimilar sample pairs is enhanced.

[0033] Furthermore, through training, the multimodal large model can more accurately understand and process data from different modes, learn the logical relationship between data, and thus better identify violations. Furthermore, tuning methods such as grid search can significantly improve the performance of the model by systematically traversing the parameter space to find the optimal or near-optimal parameter combination. This includes improving the model's accuracy, recall, F1 score and other evaluation indicators, making the model perform better on the training set and test set.

[0034] It can be understood that the beneficial effects of the second aspect mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0035] In summary, the present invention integrates multiple modal data such as visible light images and safety violation knowledge bases, and uses big models and big data processing technologies to build a comprehensive safety monitoring and management platform, aiming to achieve real-time monitoring, early warning and intelligent response to safety hazards in complex work sites, thereby improving the safety management level of the work site.

[0036] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a structural diagram of a multi-modal large model of the present invention; Figure 2 It is a schematic diagram of the interactive interface of the present invention; Figure 3 A schematic diagram of a computer device provided by an embodiment of the present invention; Figure 4 A block diagram of a chip provided according to an embodiment of the present invention; Figure 5 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0040] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0041] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0042] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0043] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0044] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0045] The present invention provides a method for safety control of work sites based on a multimodal large model, which integrates data from visible light images and safety violation knowledge bases to construct a multi-source heterogeneous data set. A multimodal large model is constructed using a deep learning algorithm based on a self-attention mechanism and big data processing technology to achieve comprehensive perception, accurate analysis, and intelligent decision-making of the safety status of the work site.

[0046] The multimodal large model can process and analyze data from different sources at the same time. By learning the violation images, the multimodal large model can extract the visual features of the scene and behavior of the violation; by learning the safety knowledge base, the multimodal large model can understand the context of the violation and the semantic features of the behavior. Combined with knowledge transfer, the image and semantic features are transferred to each other's modality respectively, and the data of different modalities are mutually verified and supplemented, so that the model can more comprehensively understand the safety violation judgment criteria and reduce the false detection rate and missed detection rate of recognition.

[0047] See also Figure 5 The present invention provides a method for safety management and control of a work site based on a multi-modal large model, comprising the following steps: S1, data preprocessing; In the data preprocessing stage, the on-site violation images and the safety violation knowledge base are integrated to construct a multimodal dataset, which mainly includes the following four aspects: a) Collection and annotation of on-site violation images Image collection: Collect a large number of on-site violation images through various means such as surveillance cameras, drone photography, and on-site law enforcement personnel. These images should cover different types of violations, such as traffic violations, workplace safety violations, etc.

[0048] Image annotation: Detailed annotation of the collected images. The annotation content may include the type of violation, location, time, people or vehicles involved, etc. The annotation work can be done manually by professionals or assisted by semi-automatic or automatic image recognition technology.

[0049] b) Construction of safety violation knowledge base Knowledge collection: Collect safety violation knowledge from various channels such as relevant laws and regulations, industry standards, historical cases, etc. This knowledge should describe in detail the definition, hazards, and preventive measures of various violations.

[0050] Knowledge organization: Organize the collected knowledge to form a structured knowledge base. The knowledge base can include violation classification, specific description, relevant regulations, case analysis, etc.

[0051] c) Multimodal dataset construction Image and knowledge fusion: On-site violation images are fused with the safety violation knowledge base to form a multimodal dataset. Each data sample includes one or more violation images and the corresponding text information such as violation description, type, hazard, and preventive measures.

[0052] Data enhancement: In order to improve the generalization ability of the model, the data set can be enhanced. For example, operations such as rotating, scaling, cropping, and adding noise to images can be performed to simulate violation scenes under different shooting conditions; synonym replacement and sentence reorganization can be performed on text information to increase the diversity of text data.

[0053] Data cleaning: In the process of building a data set, data cleaning is required to remove duplicate, erroneous or invalid data samples. This helps improve the quality of the data set and the training effect of the model.

[0054] d) Dataset division Training set, validation set, and test set: The multimodal dataset is divided into training set, validation set, and test set. The training set is used for the training process of the multimodal large model; the validation set is used to monitor the performance of the multimodal large model during the training process in order to perform hyperparameter tuning and early stopping; the test set is used to evaluate the final performance of the multimodal large model after the training is completed.

[0055] S2, build a multimodal large model; In the model design phase, the core architecture of the multimodal large model is designed according to the data type and actual application needs. The core idea of ​​the multimodal large model is to map multimodal data to the same feature space, so that images and texts with similar semantics are closer in the feature space.

[0056] The loss function of the multimodal large model is as follows:

[0057] Where D represents the training dataset, (xi,yi) represents a pair of images and texts in the dataset; zi represents the feature vector of image xi, wi represents the feature vector of text yi; s() represents the similarity function; τ is the temperature parameter used to control the smoothness of the similarity; N and M are the total number of texts and images in the dataset, respectively.

[0058] This function contains two parts, image and text, and constructs a two-dimensional space. Each point in the two-dimensional space represents a sample pair containing image and text.

[0059] In the first part, the loss function transforms the sample image z i Pull it closer to its corresponding positive sample w i , and stay away from all other samples w j ; In the second part, the loss function also converts the sample text w i Pull closer to his corresponding positive sample z i , and away from all other samples z k .

[0060] In this way, the loss function can maximize the similarity between positive sample pairs while minimizing the similarity between negative sample pairs.

[0061] S3. Use the preprocessed data to train and optimize the constructed multimodal large model; S301, Training During the training process, the multimodal model will continuously learn from the training data set and update its internal parameters to better fit the data and predict the output of unknown samples; in each cycle, the multimodal model will traverse the entire training data set once, perform forward propagation on each sample to calculate the predicted value, and then calculate the loss (error) between the predicted value and the true value, and back-propagate the loss value back to each layer of the network through the back-propagation algorithm to update parameters such as weights and biases. By repeating this process, the multimodal model gradually learns the inherent laws of the data and optimizes its predictive ability.

[0062] S302. Hyperparameter Tuning Hyperparameters are parameters that need to be set before model training. They have an important impact on the performance of the model and the training process. Through tuning methods such as grid search, we can find the hyperparameter configuration that can minimize the loss on the validation set, thereby improving the generalization ability of the model.

[0063] S4. Use the optimized multimodal large model to output the judgment results of illegal behaviors and realize safety management and control of the work site.

[0064] In yet another embodiment of the present invention, a job site safety management and control system based on a multimodal big model is provided. The system can be used to implement the above-mentioned job site safety management and control method based on a multimodal big model. Specifically, the job site safety management and control system based on a multimodal big model includes a data module, a construction module, a training module and an output module.

[0065] Among them, the data module collects on-site violation data and pre-processes it to obtain multi-modal data; Building modules and constructing large multimodal models; The training module uses the preprocessed multimodal data to train and optimize the multimodal large model; The output module uses the optimized multimodal large model to output the judgment results of illegal behaviors and realize safety management and control of the work site.

[0066] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the job site safety management and control method based on the multimodal large model, including: Collect on-site violation data and preprocess it to obtain multimodal data; build a multimodal large model; use the preprocessed multimodal data to train and optimize the multimodal large model; use the optimized multimodal large model to output the violation behavior judgment results to achieve safety management and control of the work site.

[0067] See also Figure 3 , the terminal device is a computer device, and the computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the method for controlling the safety of the work site based on the multimodal large model in the embodiment is implemented. To avoid repetition, it is not described one by one here. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the system for controlling the safety of the work site based on the multimodal large model in the embodiment are implemented. To avoid repetition, it is not described one by one here.

[0068] The computer device 60 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will appreciate that Figure 3 This is only an example of the computer device 60 and does not constitute a limitation of the computer device 60. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0069] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0070] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 60.

[0071] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.

[0072] See also Figure 4 The terminal device 600 is an electronic device, and the electronic device is in the form of a general computing device. The components of the electronic device may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0073] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 5 Follow the steps shown in .

[0074] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0075] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0076] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0077] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0078] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (Non-VolatileMemory), such as at least one disk memory.

[0079] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the method for safety management and control of a work site based on a multimodal large model in the above-mentioned embodiment; the processor may load and execute the following steps: Collect on-site violation data and preprocess it to obtain multimodal data; build a multimodal large model; use the preprocessed multimodal data to train and optimize the multimodal large model; use the optimized multimodal large model to output the violation behavior judgment results to achieve safety management and control of the work site.

[0080] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0081] Integration and testing Model integration: Clarify the integrated systems and data interaction methods, design the interface protocols and data formats between systems, and ensure smooth interaction between different systems.

[0082] Model integration refers to the process of integrating different software modules or systems into a complete working system according to the established design and architecture. The key to this stage is to ensure the compatibility and interoperability between various components. The specific steps include: System decomposition and module division: First, the entire system needs to be divided into several independent modules according to its functions, and each module is responsible for implementing a specific function.

[0083] Interface design: clarify the data exchange method between modules, and design clear and specific interface specifications, including interface protocols (such as HTTP, WebSocket, etc.), data formats (such as JSON, XML, etc.), and calling methods.

[0084] Dependency management: handle dependencies between modules, ensure the correctness of external dependencies, and avoid integration failures caused by version inconsistencies.

[0085] Integration strategy selection: Choose an appropriate integration strategy, such as top-down integration, bottom-up integration, or function-based integration, to gradually achieve system integration.

[0086] Integration testing: Continuous testing is performed during the integration process to ensure that newly added modules do not disrupt existing functions and to promptly detect and correct interface errors or logic problems.

[0087] System testing: Conduct comprehensive testing on the integrated system, including functional testing, performance testing, security testing, etc., to ensure the reliability and security of the system.

[0088] After the system integration is completed, the entire system is tested in detail and comprehensively to verify whether the system's functions, performance, security, etc. meet the requirements specifications and user expectations. System testing usually includes the following aspects: Functional testing: Verify whether the system implements all the predetermined functions, including whether various business processes, user interactions, data processing, etc. are correct.

[0089] Performance testing: Evaluate the performance of the system under specific load conditions, such as response time, throughput, resource utilization, etc., to ensure that the system can still run stably under high load conditions.

[0090] Security testing: Check whether the system has security loopholes or risks, including data encryption, access control, identity authentication, input verification, etc., to prevent illegal intrusion and data leakage.

[0091] Compatibility testing: Testing the system's operation on different platforms, browsers, and devices to ensure that users have a consistent experience regardless of the device or environment they use.

[0092] Reliability testing: Evaluate the stability of the system by running the test for a long time, and check the system's recovery ability and error handling mechanism when facing abnormal situations.

[0093] User experience testing: From the user's perspective, evaluate the system's ease of use, convenience, and satisfaction to ensure a friendly interface and intuitive operation.

[0094] See also Figure 2 ,The user accesses the multimodal large model through the WEBUI interface, uploads the violation picture, and enters the prompt words to require the large model to output the violation judgment results and reasons for judgment.

[0095] In summary, the present invention provides a method and system for safety control of a work site based on a multimodal large model. By applying a multimodal large model, the false detection rate and missed detection rate of the system are reduced. The specific effects and advantages are reflected in the following aspects: a) Construction of multimodal dataset: The fusion of visible light images and violation knowledge base provides a rich data source for the multimodal large model, which helps the model to propose high-dimensional and informative representations from the original data and more comprehensively reflect the characteristics of violations. b) Construction of similarity calculation algorithm: The defined similarity function can map image features and semantic features into the same space and accurately calculate the similarity between image samples and semantic samples.

[0096] c) Improved generalization ability: Through data augmentation technology and hyperparameter tuning, the model learns more data patterns and features during the training process, improves its generalization ability, and can maintain stable performance when facing unseen data.

[0097] d) High scalability: As the amount of data increases and computing power improves, the model can be further expanded and optimized to improve its performance and accuracy. At the same time, the modular design in the technical solution also facilitates subsequent functional expansion and upgrading.

[0098] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0099] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0100] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0101] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0102] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0103] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0104] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0105] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0106] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0107] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0108] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A method for safety management and control of a work site based on a multimodal large model, characterized in that: The following steps are involved: Collect on-site violation data and pre-process them to obtain multimodal data; Build a large multimodal model; Use preprocessed multimodal data to train and optimize large multimodal models; The optimized multimodal large model is used to output the judgment results of illegal behaviors and realize safety management and control of the work site.

2. The method for safety management and control of a work site based on a multimodal large model according to claim 1 is characterized in that: The data preprocessing is as follows: The on-site violation images are collected and annotated, a safety violation knowledge base is constructed, the on-site violation images are fused with the safety violation knowledge base to form a multimodal dataset, the multimodal dataset is enhanced and cleaned, and then the multimodal dataset is divided into training set, validation set and test set.

3. The method for job site safety management and control based on a multimodal large model according to claim 2 is characterized in that: The annotation content includes the type of violation, location, time, and people or vehicles involved.

4. The method for safety management and control of a work site based on a multimodal large model according to claim 2 is characterized in that: The safety violation knowledge base includes violation classification, specific description, relevant law and regulation references, and case analysis.

5. The method for safety management and control of a work site based on a multimodal large model according to claim 2, characterized in that: Each data sample in the multimodal dataset includes one or more violation images, as well as the corresponding text information such as the violation description, type, hazard, and preventive measures.

6. The method for safety management and control of a work site based on a multimodal large model according to claim 2 is characterized in that: Enhancements include rotation, scaling, cropping, and adding noise.

7. The method for safety management and control of a work site based on a multimodal large model according to claim 1, characterized in that: A large multimodal model is constructed to map multimodal data into the same feature space, so that images and texts with similar semantics are closest to each other in the feature space.

8. The method for safety management and control of a work site based on a multimodal large model according to claim 7 is characterized in that: Loss function for large multimodal models L as follows: Where D represents the training data set, (x i ,y i ) represents a pair of images and text in the dataset; z i Represents image x i The eigenvector of i Represents text y i The feature vector of ; s() represents the similarity function; τ is the temperature parameter used to control the smoothness of the similarity; N and M are the total number of texts and images in the dataset, respectively.

9. The method for safety management and control of a work site based on a multimodal large model according to claim 1, characterized in that: The preprocessed data is used to train the constructed multimodal large model as follows: During the training process, the multimodal model continuously learns from the training dataset and updates its internal parameters, fits the data, and predicts the output of unknown samples; In each cycle, the multimodal model traverses the entire training data set once, performs forward propagation on each sample to calculate the predicted value, then calculates the loss between the predicted value and the true value, and back-propagates the loss value back to each layer of the network through the back-propagation algorithm to update the weight and bias parameters; Through continuous repetition, the multimodal model gradually learns the inherent laws of the data and optimizes its predictive ability.

10. The method for safety management and control of a work site based on a multimodal large model according to claim 9, characterized in that: The optimization process is as follows: Through tuning methods such as grid search, we can find the hyperparameter configuration that can minimize the loss on the validation set and optimize the generalization ability of large multimodal models.

11. A work site safety management and control system based on a multi-modal large model, characterized in that: include: The data module collects on-site violation data and pre-processes it to obtain multi-modal data; Building modules and constructing large multimodal models; The training module uses the preprocessed multimodal data to train and optimize the multimodal large model; The output module uses the optimized multimodal large model to output the judgment results of illegal behaviors and realize safety management and control of the work site.

12. The multi-modal large model-based work site safety management and control system according to claim 11, characterized in that: The data preprocessing is as follows: The on-site violation images are collected and annotated, a safety violation knowledge base is constructed, the on-site violation images are fused with the safety violation knowledge base to form a multimodal dataset, the multimodal dataset is enhanced and cleaned, and then the multimodal dataset is divided into training set, validation set and test set.

13. The multi-modal large model-based work site safety management and control system according to claim 12, characterized in that: The annotation content includes the type of violation, location, time, and persons or vehicles involved; The safety violation knowledge base includes violation classification, specific description, relevant law and regulation references, and case analysis; Each data sample in the multimodal dataset includes one or more violation images, as well as the corresponding text information such as the violation description, type, hazard, preventive measures, etc.; enhancement processing includes rotation, scaling, cropping and adding noise.

14. The multi-modal large model-based work site safety management and control system according to claim 11, characterized in that: Constructing a multimodal large model is used to map multimodal data to the same feature space, so that semantically similar images and texts are closest in the feature space. The loss function of the multimodal large model is L as follows: Where D represents the training data set, (x i ,y i ) represents a pair of images and text in the dataset; z i Represents image x i The eigenvector of i Represents text y i The feature vector of ; s() represents the similarity function; τ is the temperature parameter used to control the smoothness of the similarity; N and M are the total number of texts and images in the dataset, respectively.

15. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 10.

16. A computing device, characterized in that include: One or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include steps for executing the method according to any one of claims 1 to 10.

Images