Workforce community detection method and system using multi-modal data fusion

Abstract

The application discloses a kind of multi-modal data fusion's work personnel community detection method and system, belong to artificial intelligence and graph computing technical field.The application is first to the original data in the process of power grid construction is preprocessed and carries out feature extraction, realizes the deep fusion of multiple data by multi-modal data fusion, and then introduces the improved Louvain algorithm driven by domain knowledge, designs time decay function to strengthen timeliness perception, based on the importance difference of role adjusts weight, to weaken the influence of temporary work personnel, simultaneously introduce ReLU activation function screening abnormal node, complete the community detection of work personnel.The application realizes the high-precision identification of the subcontracting relationship of power grid work personnel, forms clear and intuitive collaboration network, and effectively identifies temporary mobile work personnel, cross-package work risk personnel and abnormal collaborative behavior, provides accurate data support for subcontracting safety control and personnel organization management in construction site.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and graph computing technology, and in particular relates to a method and system for detecting worker communities through multimodal data fusion. Background Technology

[0002] With the continuous expansion of power grid projects and the rapid growth of construction tasks, the subcontracting model has gradually become the main form of construction organization commonly used in power grid projects. This model has played a positive role in improving construction efficiency and reducing management costs, but it has also significantly increased the difficulty of safety management and control, and raised the pressure of safety risks, thus placing higher demands on the safety and quality management of the entire power grid construction process.

[0003] During the current power grid construction process, a large amount of diverse operational data is generated. This data includes project information, construction teams, work permits, personnel information, construction logs, and supervision records, and is characterized by diverse sources and complex structures. Due to the large number of subcontracting teams, high personnel turnover, and insufficient team stability at construction sites, the relevant data exhibits obvious spatiotemporal dynamic characteristics, meaning that elements such as workers, work locations, and work times change dynamically as the project progresses.

[0004] Currently, subcontracting management and personnel attribution analysis in power grid projects typically employ a combined approach of "subcontracting management system + big data linear analysis." At the management level, a subcontracting management system is established to standardize and regulate the construction process. At the data analysis level, work orders are often used as the core data carrier, serving as a link between workers' collaborative relationships. Linear analysis methods are then used to calculate the frequency of collaboration among personnel, thereby identifying the subcontracting unit to which a worker belongs.

[0005] However, this model still has significant technical bottlenecks and limitations: First, it relies solely on a single data type such as the frequency of co-occurrence of work tickets, failing to integrate multimodal information such as textual semantics and safety inspection videos, making it difficult to comprehensively characterize the multidimensional attributes of workers; second, the static graph structure modeling cannot adapt to the characteristics of high worker mobility and dynamic changes in collaborative relationships as the project progresses; third, the lack of integration of construction domain knowledge leads to confusion in the weighting of temporary personnel and core team members, resulting in low accuracy in community segmentation. Therefore, how to efficiently and accurately identify the affiliation of subcontractors for work data with spatiotemporal dynamic characteristics generated during power grid construction has become an urgent technical problem to be solved in the field of power grid construction safety management. Summary of the Invention

[0006] Traditional community detection methods struggle to handle the spatiotemporal dynamics, short-term cross-team collaboration, and significant role differences inherent in power construction data, leading to biases in subcontractor identification. This invention aims to address the problems in power construction where subcontractor identification relies on a single data source, lacks multimodal information fusion, and suffers from low accuracy in community segmentation due to high worker mobility. It provides a worker community detection method and system based on multimodal data fusion. This method, for the first time, deeply integrates heterogeneous information from power construction scenarios, including text records, safety inspection videos, and personnel collaboration graph structures. Simultaneously, it incorporates professional knowledge from the construction field (role weights, process chain relationships, and time decay patterns) to construct a weighted collaboration graph. A domain knowledge-driven edge weight calculation model and a time-series dynamic modularity optimization strategy are designed, and the traditional Louvain algorithm is specifically improved to adapt to the dynamic changes in worker mobility and collaboration patterns, thereby achieving high-precision identification of subcontractor affiliation.

[0007] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for detecting worker communities through multimodal data fusion, comprising the following steps:

[0009] S1. Obtain the raw data during the power grid construction process and format it uniformly. Then, preprocess the missing, duplicate, and outlier values ​​in the formatted data. The raw data includes work order information and safety inspection video data.

[0010] S2. Extract the features of text data and safety inspection video data that reflect the key attribute information of the operators from each preprocessed data to obtain text feature vectors and video feature vectors; model the collaborative relationship between operators as an undirected and unweighted graph, and then extract the topological structure features in the undirected and unweighted graph to obtain graph feature vectors; then perform scale mapping on each feature vector through a linear transformation layer to obtain the mapped feature vectors corresponding to each modality.

[0011] S3. The mapped text feature vector is semantically encoded to generate a context vector. Then, the context vector, the mapped video feature vector, and the mapped graph feature vector are weighted and summed through an attention weighting mechanism to obtain the fusion feature vector corresponding to each preprocessed data.

[0012] S4. The obtained fused feature vector is used to improve the Louvain community detection model in the field of power construction knowledge. After passing through the input layer, the domain knowledge weighting layer, the modularity optimization layer and the community division output layer, the community affiliation label of each worker is obtained, thus completing the community detection of the workers.

[0013] Based on the above scheme, each step can be implemented in the following preferred manner.

[0014] As a preferred embodiment of the first aspect mentioned above, in step S1, the work order information includes the section code, work order number, construction unit, work time, work procedure, and basic personnel information; among which, the basic personnel information includes the personnel's name, personnel identity, personnel's affiliated unit, and personnel role; the work procedure includes the work content, procedure description, risk warning, and required construction equipment.

[0015] As a preferred embodiment of the first aspect mentioned above, in step S2, the text data includes the work content, process description, risk warning, and personnel roles; the safety inspection video data includes the results of the safety behavior inspection of personnel at the construction site; and in the graph structure data, the work duration and collaboration frequency are used as the edge attribute information of the graph.

[0016] Furthermore, in the text feature vector extraction, the text data in each preprocessed data is segmented into words, and important keywords are extracted based on the TF-IDF algorithm and their TF-IDF weights are used to construct the text features of this preprocessed data; then, for each worker, the text features of all preprocessed data associated with him / her within the statistical period are weighted and aggregated to generate the text feature vector of that worker.

[0017] Specifically, in text feature vector extraction, a dictionary-based Chinese word segmentation algorithm is used to segment the text fields in each preprocessed data to obtain a word sequence. Then, stop words are removed and part-of-speech tagging is performed on this word sequence to obtain a processed word sequence. Next, the feature extraction function is called to calculate the TF-IDF weight of each word on the processed word sequence. The K words with the highest TF-IDF weights are taken as keywords, and the TF-IDF weight corresponding to each keyword is taken as the text feature of that keyword. Thus, the text features of a single preprocessed data are composed of the text features of all keywords. For a specific worker, the text features of all preprocessed data associated with that worker within the statistical period are weighted and aggregated to generate the text feature vector of that worker.

[0018] Furthermore, in the video feature vector extraction, the safety detection video data is analyzed by frame extraction, and the safety behavior status of each worker in the image frame is identified by the target detection model. The detection confidence of different safety detection items in each image frame is obtained, and statistical aggregation is performed item by item along the frame-by-frame dimension. The video feature vector of each worker is composed of the inter-frame statistical values ​​of the detection confidence of all safety detection items.

[0019] Specifically, in the video feature vector extraction, the safety detection video data collected at the construction site is frame-by-frame parsing, the target detection model is used to identify the safety behavior status of the workers in the extracted images, the detection confidence of each safety detection item is obtained, and then the detection results of each frame are statistically aggregated in the time dimension to obtain the video statistical feature vector of each worker. Then, the features in the video statistical feature vector are normalized and mapped to the [0,1] interval to obtain the video feature vector.

[0020] Furthermore, in the graph feature vector extraction, each node in the undirected and unweighted graph represents a worker, and each edge represents the collaborative relationship between workers. If two workers participate in the same task on the same work order, an undirected edge is established between the corresponding nodes. The graph extracts three topological features: node degree, clustering coefficient, and betweenness centrality, and forms the graph feature vector corresponding to each worker.

[0021] Specifically, in the graph feature vector extraction, each node in the undirected and unweighted graph represents a worker, and each edge represents the collaborative relationship between workers. If two workers participate in the same task on the same work order, an undirected edge is established between the two corresponding nodes to reflect the degree of collaborative relationship between the two workers. Then, three topological features—node degree, clustering coefficient, and betweenness centrality—are extracted using a graph computing library, and these three topological features are used to form the graph feature vector corresponding to each worker.

[0022] As a preferred embodiment of the first aspect mentioned above, in step S4, the input layer is used to calculate the cosine similarity between two different fused feature vectors; the domain knowledge weighting layer is used to expand the undirected unweighted graph into a weighted collaborative graph through the edge weight matrix, where each edge weight in the edge weight matrix is ​​the weight of each edge, which is obtained by activating the original edge weights using the ReLU activation function, and the original edge weights are calculated based on the cosine similarity between the two nodes connected by the edge; the modularity optimization layer is used to obtain the information of the workers included in each community and use it as the community division result by going through local optimization stage and global optimization stage until the modularity reaches a maximum value; the community division output layer is used to generate and output the community affiliation label of each worker according to the community division result.

[0023] As a preferred option of the first aspect mentioned above, in the domain knowledge weighted layer, the original edge weights between any two different nodes are obtained by multiplying three terms: the first term is the cosine similarity between the fused feature vectors corresponding to the two nodes, the second term is the average role weight of the corresponding workers of the two nodes, and the third term is the time weight that decays over time.

[0024] Furthermore, the original edge weights mentioned above are calculated using the following formula:

[0025]

[0026] in, For the first The node and the first The original edge weights between each node; This indicates the role that the operator plays. At that time, the weight of its role among other workers in the work order; This indicates the role that the operator plays. At that time, the weight of its role among other workers in the work order; The time weights decay over time.

[0027] As a preferred option for the first aspect mentioned above, at the beginning of the local optimization phase, each worker is treated as an independent community. For each node, the communities of all neighboring nodes with which it has a collaborative relationship are scanned sequentially and treated as neighboring communities. The modularity increment generated after the node is moved to each neighboring community is calculated. If the modularity increment is greater than 0, the worker's community affiliation label corresponding to the node is updated to the neighboring community with the largest modularity increment. If the modularity increment is less than or equal to 0, the worker's community affiliation label corresponding to the node remains unchanged. All nodes in the weighted collaboration graph are traversed until no node's position change can make the modularity increment greater than 0, at which point the local optimization phase stops.

[0028] As a preferred embodiment of the first aspect mentioned above, in the global optimization phase, the nodes corresponding to the workers belonging to the same community in the local optimization phase are compressed into a single super node. The self-loop weight between two super nodes is the sum of the edge weights of all nodes within the communities corresponding to these two super nodes, which is used to reflect the cooperation strength between the two communities. For each super node, the communities of all neighboring super nodes with which it has a cooperative relationship are scanned sequentially and treated as new neighboring communities. The modularity increment generated after the super node is moved to each new neighboring community is calculated. If the modularity increment is greater than 0, the community affiliation label of the worker corresponding to the super node is updated to the new neighboring community with the largest modularity increment. If the modularity increment is less than or equal to 0, the community affiliation label of the worker corresponding to the super node remains unchanged. All super nodes in the weighted cooperation graph are traversed until no super node's position change can make the modularity increment greater than 0, at which point the global optimization phase stops.

[0029] Secondly, the present invention provides a multimodal data fusion-based community detection system for workers, comprising:

[0030] The data acquisition module is used to acquire raw data during power grid construction and format it in a unified manner. Then, it preprocesses the formatted data to address issues such as missing values, duplicate values, and outliers. The raw data includes work order information and safety inspection video data.

[0031] The feature extraction module is used to extract features from each preprocessed data, including text data reflecting key attribute information of operators and safety inspection video data, to obtain text feature vectors and video feature vectors. The collaborative relationship between operators is modeled as an undirected and unweighted graph, and then the topological features of the undirected and unweighted graph are extracted to obtain graph feature vectors. Each feature vector is then scaled through a linear transformation layer to obtain the mapped feature vectors corresponding to each modality.

[0032] The feature fusion module is used to semantically encode the mapped text feature vector to generate a context vector, and then use an attention weighting mechanism to perform a weighted sum of the context vector, the mapped video feature vector, and the mapped graph feature vector to obtain the fused feature vector corresponding to each preprocessed data.

[0033] The personnel detection module is used to apply the obtained fused feature vectors to the improved Louvain community detection model based on knowledge in the power construction field. After passing through the input layer, domain knowledge weighting layer, modularity optimization layer and community division output layer, the module obtains the community affiliation label of each worker, thereby completing the community detection of the workers.

[0034] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multimodal data fusion method for detecting worker communities as described in any of the solutions of the first aspect above.

[0035] Fourthly, the present invention provides a computer electronic device, which includes a memory and a processor;

[0036] The memory is used to store computer programs;

[0037] The processor is configured to, when executing the computer program, implement the multimodal data fusion method for detecting worker communities as described in any of the first aspects above.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] This invention improves the Louvain community detection algorithm and introduces a multimodal data fusion framework and an adaptive domain knowledge-driven optimization mechanism to achieve high-precision identification of subcontracting relationships among power grid workers. This method can operate efficiently on large-scale, high-time-sensitivity power grid construction data, exhibiting good real-time performance and scalability. Through the visualization of subcontracting identification, managers can intuitively grasp the affiliation relationships and collaborative networks of each worker, enabling intelligent analysis and dynamic control of personnel organizational structures. This effectively identifies potential risks such as cross-contracting and abnormal collaboration, improving safety supervision and quality management during power grid construction. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method of the present invention;

[0041] Figure 2 This is a flowchart of the improved Louvain community detection model in this invention.

[0042] Figure 3 This is a flowchart illustrating the method of the present invention for detecting real workers.

[0043] Figure 4 This is a system block diagram of the present invention;

[0044] Figure 5 This is a schematic diagram of a computer electronic device provided by the present invention. Detailed Implementation

[0045] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0046] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0047] like Figure 1 As shown, in a preferred embodiment of the present invention, the above-mentioned multimodal data fusion method for detecting worker communities includes the following steps S1 to S4. The specific implementation process of each step will be described in detail below.

[0048] This invention provides a multimodal data fusion method for detecting worker communities. This method constructs a multi-feature map that integrates personnel features, work collaboration relationships, and spatiotemporal attributes, and introduces an advanced model capable of adapting to domain knowledge, thereby achieving efficient identification of subcontracting affiliations among power construction workers.

[0049] S1. Obtain the original data during the power grid construction process and format it uniformly. Then, preprocess the missing values, duplicate values, and outliers in the formatted data. The original data includes work ticket information and safety inspection video data.

[0050] It should be noted that in step S1 of the present invention, the work ticket information includes the section code, work ticket number, construction unit, work time, work procedure and basic personnel information; wherein, the basic personnel information includes the personnel's name, personnel identity, personnel's unit and personnel role (such as person in charge, executor, supervisor, etc.); the work procedure includes the work content, procedure description, risk warning and required construction equipment.

[0051] It should be noted that in step S1 of this invention, the source of the work ticket information file used in this invention is not limited, and it can generally be automatically exported by the construction management system. Preferably, the work ticket information is organized and transmitted in JSON format. This file contains multiple work ticket records, and each record describes work-related elements in the form of key-value pairs, such as work ticket number, construction unit, work time, work duration, work location, work procedure, work type, list of workers and their roles, etc.

[0052] It should be noted that in step S1 of this invention, some of the original data has issues such as missing fields, non-standard formats, and record anomalies. For example, the original data may lack personnel identity fields or construction unit codes, and some original data may contain duplicate records. If such abnormal original data is not processed and directly enters the analysis stage, it will lead to information noise during feature extraction, thereby affecting the accuracy and stability of the model. Therefore, to address the issues of missing values, duplicate values, and outliers in the entered JSON format data, this invention further cleans, labels, enhances, and partitions the original data to improve data reliability and the model's generalization ability, ensuring data integrity and consistency.

[0053] In step S1 of this embodiment, the preprocessing process includes the following four specific steps:

[0054] Data Cleaning: First, perform completeness and logical consistency checks on all fields in the JSON file. For work ticket records missing key fields such as work ticket number, personnel name, and construction unit code, supplementary fields are obtained by cross-referencing other work ticket records from the same time period or project; if this cannot be determined, the record is marked as invalid and removed. Second, for duplicate samples, generate unique identifiers based on "work ticket number + work time + list of personnel corresponding to the work ticket" to remove duplicates. Specifically, for each work ticket record... Calculate its hash value The definition is as follows:

[0055]

[0056] in, This represents a hash function used to convert the input job ticket record into a unique 128-bit hash value; Indicates string concatenation; Indicates the work order number; Indicates the time required for the assignment; This represents the list of personnel corresponding to the work order. If two work order records have the same hash value, they are considered duplicate samples, and only one valid work order record is retained, while the remaining duplicate work order records are discarded.

[0057] Data Annotation: To facilitate subsequent feature extraction and model recognition, this invention performs structured annotation on the cleaned work order records. The annotation content includes work order type, work level, construction procedure category, and personnel role. This process is achieved through a combination of semi-automated rule matching and manual verification.

[0058] Data Augmentation: Considering the significant differences in the number of work order records among different construction units, this invention introduces a data augmentation strategy to avoid sample imbalance during model training. Historical work order records are expanded through time-sliding window sampling (e.g., generating combined samples of work orders from adjacent time periods of the same process), and some scarce types of work order records are generated through simulation, thereby enhancing the model's generalization ability to small sample scenarios.

[0059] Data Split: The preprocessed data is divided into a dataset and then partitioned into training, validation, and test sets according to time sequence or project number. A preferred ratio is 7:2:1 to ensure representativeness and stability during both training and evaluation phases. The partitioned data is stored in a standardized JSON format for direct use in subsequent S2 feature extraction steps.

[0060] The data processed through the above steps has achieved field unification, format standardization, and anomaly removal, providing a data foundation for subsequent feature extraction and multimodal fusion stages.

[0061] S2. Extract the features of text data and safety inspection video data that reflect the key attribute information of the operators from each preprocessed data to obtain text feature vectors and video feature vectors; model the collaborative relationship between operators as an undirected and unweighted graph, and then extract the topological structure features in the undirected and unweighted graph to obtain graph feature vectors; then scale each feature vector through a linear transformation layer to obtain the mapped feature vectors corresponding to each modality.

[0062] It should be noted that the data generated at power construction sites is massive in scale, diverse in data type, and complex in structure. It includes textual semantic information (such as work content, process descriptions, risk warnings, and personnel roles), as well as safety inspection video data (such as the results of personnel safety behavior detection, including helmet wearing and safety belt use). It also implicitly contains collaborative relationships among workers, such as collaboration, repetitive tasks, and the sequence of tasks within the same work order. This data exhibits significant heterogeneity and redundancy. Directly inputting it into a model can easily lead to excessively high feature dimensionality, noise accumulation, and unstable model training. Therefore, this invention designs a multi-level feature extraction step for the preprocessed data in step S2. First, features are extracted from the textual semantic information and safety inspection video data. Then, the collaborative relationships are modeled as graph-structured data. This accurately and efficiently identifies and extracts the core features affecting subcontracting attribution, eliminates redundant, noisy information and irrelevant variables, reduces data dimensionality and computational complexity, and improves the effectiveness of data representation and model computational efficiency.

[0063] In a preferred embodiment, step S2 involves inputting the preprocessed dataset and outputting the corresponding features. The feature extraction process specifically includes the following three parts:

[0064] Text Feature Extraction: A dictionary-based Chinese word segmentation algorithm is used to segment the text data in each preprocessed data entry into words, resulting in a word sequence. The text fields include job content, process description, risk warning, and personnel role. Stop words are then removed and part-of-speech tagging is performed on this word sequence to obtain a processed word sequence. A feature extraction function is then called to calculate the TF-IDF weight of each word in the processed word sequence. The K words with the highest TF-IDF weights are selected as keywords, and the TF-IDF weight corresponding to each keyword is used as the text feature of that keyword. Thus, the text features of a single preprocessed data entry are composed of the text features of all keywords. For a specific worker, the text features of all preprocessed data associated with that worker within the statistical period are weighted and aggregated to generate the worker's text feature vector. This allows the textual features of the preprocessed data to be mapped to the dimension of the operators, thereby constructing a business profile of the operators.

[0065] In this embodiment, the TF-IDF weights are calculated using the following formula:

[0066]

[0067] in, Indicates the first Preprocessed data The Middle Word TF-IDF weights; Represents the feature extraction function; This represents the total number of words in all documents; For including the first The number of documents containing each word.

[0068] It should be noted that in this invention, the aforementioned text feature vector is not specific to a particular work order, but rather a summary of all actions taken by a particular worker within a statistical period; it is a numerical vector. Each position (dimension) in the vector corresponds to a keyword, i.e., a specific power business term (such as "foundation pit," "tower assembly," and "acceptance"), and the value at that position is the corresponding TF-IDF weight. Specifically, after calculating the TF-IDF weights, the extracted global high-frequency keyword library is: {foundation pit, tower assembly, insulation, acceptance, cable}. The text feature vector of worker A is [0.85, 0.12, 0.00, 0.45, 0.00], indicating that worker A frequently participates in foundation pit excavation (0.85) and acceptance (0.45), and almost never does cable work. The text feature vector of worker B is [0.00, 0.00, 0.92, 0.00, 0.78], indicating that worker B is specifically responsible for insulation and cable work. By comparing these two text feature vectors, the model can use cosine similarity in subsequent steps to determine that worker A and worker B are more likely not from the same community because their directions are completely different.

[0069] Video feature extraction: Frame-by-frame analysis is performed on the safety inspection video data collected at the construction site. The target detection model is used to identify the safety behavior status of the workers in the extracted image frames (including wearing safety helmets, using safety belts, and wearing reflective vests, etc.). The detection confidence of each safety inspection item in the image frame is obtained. Then, the detection results of each frame are statistically aggregated along the frame-by-frame dimension to obtain the video statistical feature vector of each worker. The features in the video statistical feature vector are then normalized and mapped to the [0,1] interval to obtain the video feature vector.

[0070] In this embodiment, the safety inspection items include wearing a safety helmet, using a seat belt, and wearing a reflective vest, etc. Item. For the first The number of workers in the statistical period The security detection item, the target detection model in the first... Output the corresponding detection confidence level in the frame. The detection results of all associated frames of the operator are statistically aggregated to obtain the video statistical feature vector. , can be represented as:

[0071]

[0072] in, These represent the functions for calculating the mean, maximum value, and standard deviation, respectively. For the first The first worker in the T-frame The detection confidence sequence corresponding to each safety detection item; This indicates the pass rate of the test item, i.e., the confidence level exceeding the preset test confidence threshold. The number of frames accounts for the total number of frames. The proportion is calculated using the following formula:

[0073]

[0074] in, This is an indicator function that takes the value 1 when the condition is met and 0 otherwise. Therefore, the... The video statistical feature vector of each worker is composed of The statistical feature vectors of each safety inspection item are concatenated. After normalization, the video feature vector is obtained. .

[0075] Graph Feature Extraction: For undirected and unweighted graphs ,in For a set of nodes, each node Representing a worker, Let be a set of edges, where each edge represents a collaborative relationship between workers. If the ... The first worker and the first If multiple workers participate in the same task under the same work order, then the two corresponding nodes will be... and Establish an undirected edge between them This reflects the degree of collaboration between the two workers. Then, three topological features—node degree, clustering coefficient, and betweenness centrality—are extracted using the NetworkX graph computing library. These three topological features are then used to construct a graph feature vector for each worker. .

[0076] In this embodiment, the three topological features described above are used to quantify the position and importance of each worker in the collaborative network. Specifically, the degree represents the number of direct collaborations a worker has; the clustering coefficient represents the density of connections within a worker's neighborhood (a higher clustering coefficient indicates more stable collaborative relationships within the worker's community); and betweenness centrality represents the worker's bridging role between different communities. Taking a single worker as an example, their corresponding node degree Clustering coefficient Intermediation centrality Calculate according to the following formulas:

[0077]

[0078]

[0079]

[0080] in, For elements of the adjacency matrix, when and When a collaborative relationship exists, ; For nodes The actual number of edges in the neighborhood; Indicates from node To the node The number of shortest paths; This indicates that these shortest paths pass through... The number of nodes; For three distinct node indices. Therefore, the... The graphical feature vector of each worker can be represented as: .

[0081] After obtaining the above text feature vectors Video feature vectors Graph feature vectors Subsequently, this embodiment further performs scale mapping on these feature vectors through a linear projection layer to ensure the consistency of dimensions and distribution stability of features across different modalities. The formula is as follows:

[0082]

[0083] in, For modal indexing; and These are the trainable weight matrices and bias terms corresponding to different modalities, used to map the feature vectors of different modalities to a unified dimensional space; It is the feature vector after mapping a certain mode.

[0084] S3. The mapped text feature vector is semantically encoded to generate a context vector. Then, the context vector, the mapped video feature vector, and the mapped graph feature vector are weighted and summed through an attention weighting mechanism to obtain the fused feature vector corresponding to each preprocessed data.

[0085] It should be noted that in this invention, after data preprocessing in step S1 and feature extraction in step S2, a large amount of structured and standardized multi-source data is obtained, including text features, video features, and graph structure features. Since the importance of different modalities varies in different construction scenarios, and to further improve the accuracy and interpretability of data representation, this invention introduces an attention weighting mechanism in step S3. This mechanism performs deep semantic analysis, unified modeling, and fusion representation on the mapped feature vectors of each modality (text, video, and graph) to construct a fusion feature vector that comprehensively reflects the collaborative relationships among workers and has high discriminative power. It is used to support subsequent community detection and personnel attribution inference, and realizes deep fusion expression of multimodal features, rather than simple feature splicing.

[0086] Specifically, this invention introduces an attention weighting mechanism in step S3 to achieve adaptive weight allocation for each modality of information. Then, it performs weighted summation on the features of different modalities to obtain the fused feature vector corresponding to each preprocessed data. :

[0087]

[0088] in, This represents the attention weights for the corresponding modality; for video and graph modalities, that is In other words, these two modalities are directly weighted and fused using the mapped feature vectors. For the text modality, This is the context vector generated after semantic encoding.

[0089] In this embodiment, the enhanced encoding of the aforementioned text features is specifically achieved through the pre-trained language model BERT (Bidirectional Encoder Representations from Transformers). The reason for introducing BERT is that, in the aforementioned text feature extraction, this invention mainly uses TF-IDF weights, but these have limitations. If worker A's work order states "cable laying" and worker B's work order states "cable pulling," TF-IDF weights will consider these two different words with low similarity. BERT, however, maps these two sentences to a high-dimensional real-valued space (semantic space). Since they are extremely similar in power business logic, the numerical vectors they generate will be very close in distance in space. Inspired by this, in the specific implementation of this invention, the text features are derived from the TF-IDF weights in S2, and further generated into context vectors through BERT semantic encoding in S3, enabling the model to capture the implicit semantic relationships in the work order text. The enhanced text semantic features, security detection video features, and graph structure features are dynamically adjusted in S3 through attention weights to strengthen the ability to express key personnel nodes and high-frequency collaborative relationships, and improve the discriminative ability of fused features and the stability of the model.

[0090] S4. The obtained fused feature vector is used to improve the Louvain community detection model in the field of power construction knowledge. After passing through the input layer, the domain knowledge weighting layer, the modularity optimization layer and the community division output layer, the community affiliation label of each worker is obtained, thus completing the community detection of the workers.

[0091] It should be noted that in this invention, after the multimodal feature fusion in step S3, a fused feature vector set can be obtained. , These represent fused feature vectors. To achieve high-precision identification of subcontractor affiliation based on multimodal information, this invention proposes an improved Louvain community detection model (Multimodal Data Fusion and DomainKnowledge-Driven Community Detection Model, MDFDK-CD) in step S4, incorporating knowledge from the power construction domain. This model, based on the traditional Louvain modularity optimization framework, introduces professional knowledge from the power construction domain (including construction time, role weights, process semantics, and historical collaboration strength), and constructs a dynamic weighted collaboration graph by incorporating this knowledge. Dynamic edge weight calculation enhances the structural expressiveness of the weighted collaboration graph, enabling the community partitioning results to more accurately reflect the actual community structure, thereby achieving high-precision identification of subcontractor affiliation. The improved Louvain community detection model mainly includes an input layer, a domain knowledge weighting layer, a modularity optimization layer, and a community partitioning output layer, such as... Figure 2 As shown below, the processing flow within each layer will be explained in detail.

[0092] It should be noted that in step S4 of this invention, the input layer is used to calculate the cosine similarity between two different fused feature vectors; the domain knowledge weighting layer is used to expand the undirected unweighted graph into a weighted collaborative graph through the edge weight matrix, where each edge weight in the edge weight matrix is ​​the weight of each edge, which is obtained by activating the original edge weights using the ReLU activation function, and the original edge weights are calculated based on the cosine similarity between the two nodes connected by the edge; the modularity optimization layer is used to obtain the information of the workers included in each community and use it as the community division result by going through local optimization stage and global optimization stage until the modularity reaches a maximum value; the community division output layer is used to generate and output the community affiliation label of each worker according to the community division result.

[0093] In this embodiment, the input layer first receives the fused feature vector output from step S3. For two different fused feature vectors Then, this layer calculates... and Cosine similarity in multimodal feature space This similarity metric serves as a fundamental measure of the semantic relationships, job attributes, and topological similarity among operators. The similarity is defined as follows:

[0094]

[0095] in, This metric represents the modulus of the metric. It transforms the deep semantic associations captured by BERT in S3, along with the multimodal information adjusted by the attention mechanism, into a computable association strength within the graph structure. Specifically, TF-IDF in S2 is responsible for quickly extracting explicit features from massive amounts of text, while BERT in S3 is responsible for mining implicit semantics. Because the fused feature vector contains BERT semantic encoding from S3, it can identify that two workers, such as those performing "foundation excavation" and "earthwork construction," have extremely high business similarity, thereby increasing the edge weights between them.

[0096] In the domain knowledge weighted layer of this embodiment, there is an undirected, unweighted graph. The graph only contains the connection relationships between nodes and does not include the weight values ​​between nodes. To overcome the problem that this undirected and unweighted graph cannot quantify the differences in cooperation strength and the spatiotemporal dynamics of power construction, this invention introduces an edge weight matrix. This expands the undirected, unweighted graph into a weighted collaboration graph. This further characterizes the weights of the connections between two nodes, building upon the aforementioned undirected and unweighted graph. The original edge weights between any two different nodes are obtained by multiplying three terms: the first term is the cosine similarity between the fused feature vectors of the two nodes; the second term is the mean of the role weights of the workers corresponding to the two nodes; and the third term is the time weight that decays over time. Specifically, it is calculated using the following formula:

[0097]

[0098] in, For the first The node and the first The original edge weights between each node; This indicates the role that the operator plays. At that time, the weight of its role among other workers in the work order; This indicates the role that the operator plays. At that time, the weight of its role among other workers in the work order; The time weights decay over time.

[0099] Furthermore, in this embodiment, the model assigns a higher time weight to recent data, as it more intuitively reflects the current work status of operators and the node relationships between existing teams. The formula for calculating the aforementioned time weight is:

[0100]

[0101] in, The time when two different workers finally collaborate on the work ticket; These represent the earliest and latest times in the preprocessed data, respectively.

[0102] Furthermore, in this embodiment, during the initialization phase before training, the improved Louvain community detection model receives specific values ​​for the weights of each role from domain experts. As training progresses, these role weights dynamically change, updating the original edge weights, ultimately training to obtain the role weights that maximize the module degree value. During training, these role weights are used... For example, it is updated using the following formula:

[0103]

[0104] in, This represents the learning rate, which is 0.01 in this embodiment. Modularity; To find the partial derivative. In the traditional Louvain community detection algorithm, the modularity is defined as:

[0105]

[0106]

[0107] In the first iteration of model training, the original edge weights can be calculated based on the initial values ​​of the role weights. Then, the modularity value for this iteration can be calculated using the above formula. In each subsequent iteration, it will be continuously updated during model training to obtain an optimal value. The total weight of the weighted collaboration graph; and All are weighted degrees of the nodes. This is used to characterize the level of activity of the operator within the entire power grid cooperation network; For indicator functions, Indicates the first The community to which the workers belong Indicates the first The community to which the workers belong. When the... The first worker and the first When all workers belong to the same community, the indicator function is set to 1; otherwise, it is set to 0. A higher modularity indicates a more accurate community structure.

[0108] After this weighted mapping, each edge in the edge set corresponds to a weight, thus providing a more expressive input structure for the subcontracting model. Therefore, this invention, by introducing the partial derivative algorithm shown in the following formula, can, while attempting to divide the community, reverse-engineer the impact of the initial role weights given by experts on the subcontracting results, and then correct the parameters through partial derivatives. This parameter self-learning mechanism based on modularity feedback ensures that the model of this invention does not solely rely on expert experience when calculating the subcontracting results. Specifically, the above partial derivative algorithm is as follows:

[0109]

[0110] in, Represents the original edge weights in relation to the modularity. The impact, This indicates the impact of role weights on the original edge weights.

[0111] It should be noted that, in order to further filter out nodes with weak connections to other nodes (which may be outliers), this invention introduces the ReLU activation function to adjust the original edge weights. Activate the output to obtain the final edge weights. This process constructs an edge weight matrix. After this operation, nodes connected by edges with lower weights will have their influence significantly weakened during subsequent community partitioning, and may even be considered to have almost no connection with other nodes, thus being marked as outliers. These outliers may represent temporary staff, newly hired employees, or former employees.

[0112] The modularity optimization layer in this embodiment is the core computational unit for improving the Louvain community detection model. It is responsible for clustering workers based on the structural information of the weighted collaboration graph to maximize the modularity index of the improved Louvain community detection algorithm, identifying closely collaborating and structurally stable communities. Specifically, this layer employs a hierarchical search process, not completing the partitioning all at once. Instead, it repeats local optimization and global optimization phases until the entire model structure reaches its most stable state (i.e., the modularity reaches its maximum value), ultimately outputting the optimal community partitioning result with maximum modularity. The local optimization phase, or node migration, is the lowest-level micro-adjustment, aiming to help each worker find the most suitable small community. The global optimization phase, or community reconstruction, treats each community partitioned in the local optimization phase as a whole, compressing it into a new supernode.

[0113] At the beginning of the local optimization phase, each worker is treated as an independent community; for each node... Scan the edges that have cooperative relationships with it in sequence (i.e., edge weights). The community where all neighboring nodes of ) are located Then, considering the neighboring communities, the module degree increment generated after transferring the node to each neighboring community is calculated. ,like Then, update the community affiliation label of the workers corresponding to that node to the neighboring community with the largest module degree increment. If the community affiliation label of the operator corresponding to the node remains unchanged, then traverse all nodes in the weighted collaboration graph until no node's position change can make the modularity increment greater than 0, and stop the local optimization phase.

[0114] Furthermore, the formula for calculating the above-mentioned module degree increment is:

[0115]

[0116] in, Indicates the current community The sum of all collaboration strengths among internal nodes is used to characterize the current level of business collaboration within the community; To represent nodes The sum of edge weights connected to other nodes in the current community is used to measure the increase in collaboration strength brought about by incorporating the personnel to be assigned into the community. Indicates the current community The sum of collaboration strengths generated by all nodes within the community and the entire weighted collaboration graph is used to characterize the current macro-level business scale of the community. Because... It includes business semantic similarity extracted by BERT. At this stage, the present invention will prioritize planning operators with similar business and experience in cooperation into a community.

[0117] In the global optimization phase, nodes belonging to the same community as workers in the local optimization phase are compressed into a single super node. The self-loop weight between two super nodes is the sum of the edge weights of all nodes within the communities corresponding to these two super nodes, reflecting the collaboration strength between the two communities. For each super node, the communities of all neighboring super nodes with which it has a collaborative relationship are scanned sequentially and treated as new neighboring communities. The modularity increment generated after moving the super node to each new neighboring community is calculated. If the modularity increment is greater than 0, the community affiliation label of the worker corresponding to the super node is updated to the new neighboring community with the largest modularity increment. If the modularity increment is less than or equal to 0, the community affiliation label of the worker corresponding to the super node remains unchanged. All super nodes in the weighted collaboration graph are traversed until no super node's position change can result in a modularity increment greater than 0, at which point the global optimization phase stops.

[0118] As explained above, the global optimization phase addresses the macro-level organizational issue of whether the community size is reasonable, and the penalized items... The larger the community, the higher the barrier to entry. This is because the present invention assumes that a large community, due to its size, inherently has a higher probability of randomly encountering another person. When the community expands to a certain extent, the marginal benefits (increased collaboration intensity) brought by joining new members will not offset the size penalty.

[0119] In summary, this invention improves upon the traditional Louvain community detection algorithm, proposing an improved Louvain community detection model, MDFDK-CD. Its main innovation lies in:

[0120] First, considering the characteristics of collaborative relationships among workers in power construction scenarios, such as high time sensitivity, significant role differences, prominent process dependencies, and dynamic changes in collaboration intensity with project stages, this invention constructs a domain knowledge-driven edge weight matrix. This matrix comprehensively calculates the weight values ​​of collaborative edges based on multiple dimensions, including construction time, personnel roles, semantic similarity of processes, and historical collaboration intensity. Specifically, time decay weights enhance the credibility of recent collaborative behavior; role weights differentiate the influence of different positions such as supervisors, executors, and supervisors in collaborative relationships; and semantic similarity of process texts strengthens the collaborative association between personnel in the same or similar processes. These weights, after normalization, are superimposed to form dynamic edge weights, achieving a fine-grained characterization of the strength of collaborative relationships.

[0121] Secondly, edge weights are applied to the original undirected and unweighted graph to form a semantically enhanced weighted collaboration graph that simultaneously reflects temporal attributes, role characteristics, process semantics, and collaboration stability. Compared with traditional collaboration graphs constructed solely based on work ticket co-occurrence relationships, the weighted collaboration graph constructed in this invention better reflects the actual evolution of the organizational structure of workers and has stronger expressive power.

[0122] Finally, based on the aforementioned weighted collaboration graph, this invention introduces a dynamic edge weight adjustment mechanism in the local optimization stage of the Louvain algorithm, enabling the community partitioning process to adapt to the actual characteristics of the continuous changes in collaboration edges as the project progresses; and introduces time weight and role weight into the modularity function to improve the sensitivity of the modularity maximization process to domain-specific structural features.

[0123] In addition, this invention utilizes ReLU nonlinearity to suppress weak connections, enabling automatic identification of abnormal nodes with low collaboration, discontinuous processes, or frequent cross-group jumps, thereby effectively distinguishing between temporary mobile workers, cross-package risk personnel, and personnel who have left the project.

[0124] Additionally, it should be noted that when the method of this invention is tested on actual workers, such as... Figure 3As shown, firstly, a query request for the desired data is sent to the platform database, and raw data is retrieved from the platform database according to a set date range. This data includes at least work order information, personnel information, and construction logs. Subsequently, the retrieved raw data undergoes cleaning, standardization, and structured preprocessing, including at least missing value completion, duplicate record removal, outlier removal, unified encoding of time fields, and normalization of personnel identifiers, resulting in preprocessed data. Based on this, the present invention divides the preprocessed data into two processing paths: collaboration information and personnel information. First, based on the collaboration information, graph data is constructed, with workers as nodes and relationships involving participation in the same work order or construction task as edges, forming an undirected, unweighted graph structure. Graph topological features such as node degree, clustering coefficient, and betweenness centrality are further extracted. Second, text and video features are extracted based on personnel information, work order text, and related attribute fields. The text fields undergo word segmentation, stop word removal, and TF-IDF processing to obtain initial text feature vectors. In the multimodal fusion stage, semantic encoding is further performed using BERT to obtain context vectors. Safety detection video data extracts safety behavior detection features such as helmet wearing, seatbelt use, and reflective vest wearing. Then, the graph feature vectors, text feature vectors, and video feature vectors are fused together in the multimodal data fusion process. A unified dimension mapping and attention weighting mechanism are used to generate a fused feature vector for the workers. Based on the fused feature vector, worker community detection is performed to complete subcontracting attribution identification or work group community division. Finally, the integrated model output is correlated with the original data to form a data partitioning and correlation result, which is then written back to the platform database for subsequent querying, tracing, and business management.

[0125] The present invention will now demonstrate the application effect of the multimodal data fusion worker community detection method described in S1 to S4 of the above embodiments on a specific dataset through a specific example, so as to facilitate understanding of the essence of the present invention.

[0126] Example

[0127] The implementation process of the multimodal data fusion method for detecting worker communities used in this embodiment is as described above and will not be repeated here.

[0128] To verify the effectiveness and superiority of the multimodal data fusion-based worker community detection method proposed in this invention, experiments were conducted on multiple power construction datasets. In a preferred embodiment of this invention, the datasets used in the experiments originated from actual data collection in power construction scenarios, covering datasets from three representative construction units. Due to data confidentiality, the dataset names are labeled SG_01, SG_02, and SG_03, respectively, and their specific data quantity distribution details are shown in Table 1.

[0129] Table 1. Distribution of Data Quantity

[0130] To verify the performance of the proposed method, this embodiment selects five community detection algorithms for comparative experiments. Modularity is used as the evaluation index to verify the feasibility and application value of the proposed method in actual power construction scenarios. The selected comparison algorithms are all classic and widely used community detection algorithms, including Girvan-Newman (GN), Fast Newman (FN), Infomap, Label Propagation (LAP), and Leiden. The first four comparison algorithms are classic and widely used community detection algorithms, while Leiden is an advanced community detection algorithm based on modularity. The experimental data of different algorithms are shown in Table 2, where the gray label represents the optimal result of the comparison algorithm in this instance.

[0131] Table 2. Results of different algorithms

[0132] The comparative results show that the GN algorithm requires significantly more time and memory when dealing with large-scale data, even experiencing memory overflow issues during the experiments, making it difficult to solve real-world large-scale problems. For example, the GN algorithm failed to run properly when processing large datasets such as SG_02 and SG_03, while other algorithms were able to successfully complete community partitioning. The method proposed in this invention achieved the best results in all three instances and significantly outperformed the selected comparative algorithms.

[0133] It should also be noted that the multimodal data fusion method for detecting worker communities in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a multimodal data fusion system for detecting worker communities, corresponding to the multimodal data fusion method for detecting worker communities provided in the above embodiments, such as... Figure 4 As shown, it includes:

[0134] The data acquisition module is used to acquire raw data during power grid construction and format it in a unified manner. Then, it preprocesses the formatted data to address issues such as missing values, duplicate values, and outliers. The raw data includes work order information and safety inspection video data.

[0135] The feature extraction module is used to extract features from each preprocessed data, including text data reflecting key attribute information of operators and safety inspection video data, to obtain text feature vectors and video feature vectors. The collaborative relationship between operators is modeled as an undirected and unweighted graph, and then the topological features of the undirected and unweighted graph are extracted to obtain graph feature vectors. Each feature vector is then scaled through a linear transformation layer to obtain the mapped feature vectors corresponding to each modality.

[0136] The feature fusion module is used to semantically encode the mapped text feature vector to generate a context vector, and then use an attention weighting mechanism to perform a weighted sum of the context vector, the mapped video feature vector, and the mapped graph feature vector to obtain the fused feature vector corresponding to each preprocessed data.

[0137] The personnel detection module is used to apply the obtained fused feature vectors to the improved Louvain community detection model based on knowledge in the power construction field. After passing through the input layer, domain knowledge weighting layer, modularity optimization layer and community division output layer, the module obtains the community affiliation label of each worker, thereby completing the community detection of the workers.

[0138] It is understood that the multimodal data fusion-based worker community detection method described in S1-S4 above can essentially be implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the multimodal data fusion-based worker community detection method provided in the above embodiments, such as... Figure 5 As shown, it includes a memory and a processor;

[0139] The memory is used to store computer programs;

[0140] The processor is configured to implement the multimodal data fusion method for detecting workers in communities as described in the above embodiments when executing the computer program.

[0141] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0142] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the multimodal data fusion worker community detection method provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can realize the multimodal data fusion worker community detection method in the above embodiments.

[0143] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0144] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0145] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0146] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for detecting worker communities using multimodal data fusion, characterized in that, Includes the following steps: S1. Obtain the raw data during the power grid construction process and format it uniformly. Then, preprocess the missing, duplicate, and outlier values ​​in the formatted data. The raw data includes work order information and safety inspection video data. S2. Extract the features of text data and safety inspection video data that reflect the key attribute information of the operators from each preprocessed data to obtain text feature vectors and video feature vectors; model the collaborative relationship between operators as an undirected and unweighted graph, and then extract the topological structure features in the undirected and unweighted graph to obtain graph feature vectors; then perform scale mapping on each feature vector through a linear transformation layer to obtain the mapped feature vectors corresponding to each modality. S3. The mapped text feature vector is semantically encoded to generate a context vector. Then, the context vector, the mapped video feature vector, and the mapped graph feature vector are weighted and summed through an attention weighting mechanism to obtain the fusion feature vector corresponding to each preprocessed data. S4. The obtained fused feature vector is used to improve the Louvain community detection model in the field of power construction knowledge. After passing through the input layer, the domain knowledge weighting layer, the modularity optimization layer and the community division output layer, the community affiliation label of each worker is obtained, thus completing the community detection of the workers.

2. The multimodal data fusion-based community detection method for workers as described in claim 1, characterized in that, In S1, the work order information includes the section code, work order number, construction unit, work time, work procedure, and basic personnel information; among which, the basic personnel information includes the personnel's name, personnel identity, personnel's unit, and personnel role; the work procedure includes the work content, procedure description, risk warning, and required construction equipment.

3. The multimodal data fusion-based community detection method for workers as described in claim 2, characterized in that, In S2, text data includes job content, process description, risk warnings, and personnel roles; safety inspection video data includes the results of safety behavior inspections of personnel at the construction site; in graph structure data, job duration and collaboration frequency are used as edge attribute information of the graph.

4. The multimodal data fusion-based community detection method for workers as described in claim 3, characterized in that, In text feature vector extraction, the text data in each preprocessed data is segmented into words, and important keywords are extracted based on the TF-IDF algorithm and their TF-IDF weights are used to construct the text features of this preprocessed data. For each worker, the text features of all preprocessed data associated with them within the statistical period are weighted and aggregated to generate a text feature vector for that worker. In video feature vector extraction, the safety detection video data is analyzed by frame extraction, and the safety behavior status of each worker in the image frame is identified using the object detection model. The detection confidence of different safety detection items in each image frame is obtained, and statistical aggregation is performed item by item along the frame-by-frame dimension. The video feature vector of each worker is composed of the inter-frame statistical values ​​of the detection confidence of all safety detection items. In graph feature vector extraction, each node in the undirected and unweighted graph represents a worker, and each edge represents the collaborative relationship between workers. If two workers participate in the same task on the same work order, an undirected edge is established between the corresponding nodes. Three topological features, namely node degree, clustering coefficient, and betweenness centrality, are extracted from the graph and used to form the graph feature vector corresponding to each worker.

5. The multimodal data fusion method for detecting worker communities as described in claim 1, characterized in that, In S4, the input layer is used to calculate the cosine similarity between two different fused feature vectors; the domain knowledge weighting layer is used to expand the undirected unweighted graph into a weighted collaboration graph through the edge weight matrix. Each edge weight in the edge weight matrix is ​​the weight of each edge, which is obtained by activating the original edge weights using the ReLU activation function. The original edge weights are calculated based on the cosine similarity between the two nodes connected by the edge; the modularity optimization layer is used to obtain the information of the workers included in each community and use it as the community division result through local optimization stage and global optimization stage until the modularity reaches a maximum value; the community division output layer is used to generate and output the community affiliation label of each worker based on the community division result.

6. The multimodal data fusion method for detecting worker communities as described in claim 5, characterized in that, In the domain knowledge weighted layer, the original edge weights between any two different nodes are obtained by multiplying three terms: the first term is the cosine similarity between the fused feature vectors of the two nodes, the second term is the mean of the role weights of the workers corresponding to the two nodes, and the third term is the time weight that decays over time.

7. The multimodal data fusion-based community detection method for workers as described in claim 5, characterized in that, At the beginning of the local optimization phase, each worker is regarded as an independent community. For each node, the communities of all neighboring nodes with which it has a cooperative relationship are scanned in turn and regarded as neighboring communities. Then, the modularity increment generated after the node is transferred to each neighboring community is calculated. If the modularity increment is greater than 0, the worker community affiliation label corresponding to the node is updated to the neighboring community with the largest modularity increment. If the modularity increment is less than or equal to 0, the worker community affiliation label corresponding to the node remains unchanged. Traverse all nodes in the weighted collaboration graph until no node's position change can cause the module degree increment to be greater than 0, at which point the local optimization phase stops.

8. The multimodal data fusion method for detecting worker communities as described in claim 5, characterized in that, In the global optimization phase, the nodes corresponding to workers belonging to the same community in the local optimization phase are compressed into a single super node. The self-loop weight between two super nodes is the sum of the edge weights of all nodes within the communities corresponding to these two super nodes, which reflects the cooperation strength between the two communities. For each super node, the communities of all neighboring super nodes with which it has a cooperative relationship are scanned sequentially and taken as new neighboring communities. Then, the modularity increment generated after the super node is transferred to each new neighboring community is calculated. If the modularity increment is greater than 0, the community affiliation label of the worker corresponding to the super node is updated to the new neighboring community with the largest modularity increment. If the modularity increment is less than or equal to 0, the community affiliation label of the worker corresponding to the super node remains unchanged. Traverse all super nodes in the weighted collaboration graph until no change in the position of any super node can result in a module degree increment greater than 0, at which point the global optimization phase stops.

9. A multimodal data fusion-based community detection system for workers, characterized in that, include: The data acquisition module is used to acquire raw data during power grid construction and format it in a unified manner. Then, it preprocesses the formatted data to address issues such as missing values, duplicate values, and outliers. The raw data includes work order information and safety inspection video data. The feature extraction module is used to extract features from each preprocessed data, including text data reflecting key attribute information of operators and safety inspection video data, to obtain text feature vectors and video feature vectors. The collaborative relationship between operators is modeled as an undirected and unweighted graph, and then the topological features of the undirected and unweighted graph are extracted to obtain graph feature vectors. Each feature vector is then scaled through a linear transformation layer to obtain the mapped feature vectors corresponding to each modality. The feature fusion module is used to semantically encode the mapped text feature vector to generate a context vector, and then use an attention weighting mechanism to perform a weighted sum of the context vector, the mapped video feature vector, and the mapped graph feature vector to obtain the fused feature vector corresponding to each preprocessed data. The personnel detection module is used to apply the obtained fused feature vectors to the improved Louvain community detection model based on knowledge in the power construction field. After passing through the input layer, domain knowledge weighting layer, modularity optimization layer and community division output layer, the module obtains the community affiliation label of each worker, thereby completing the community detection of the workers.

10. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the multimodal data fusion method for detecting worker communities as described in any one of claims 1 to 8.

Images