A Federated Learning-Based Method for Segmenting Chemical Production Operations

Abstract

This invention discloses a method for segmenting chemical production operation processes based on federated learning. The method aggregates the datasets of each training node using a federated learning strategy to train a usable action segmentation model, which is then deployed to an IoT system cloud server. When an edge server detects an action segmentation requirement, it captures video of the chemical production operation and sends it to the edge processor. The edge processor preprocesses the video to obtain a feature sequence and sends the sequence to the cloud server. The cloud server uses the action segmentation model to identify the feature sequence and obtain a confidence sequence. The confidence sequence is then processed to obtain the actual chemical production operation process, which is compared and analyzed with a standard process to generate an analysis report. This invention uses an action segmentation model to accurately classify the actions in each frame of the video data, enabling action classification for each frame in the entire uncropped long video, significantly reducing manpower waste.

Description

Technical Field

[0001] This invention belongs to the field of chemical production and relates to a method for segmenting chemical production operation processes based on federated learning. Background Technology

[0002] Chemical production operations are inherently risky. Errors in the process can easily lead to accidents such as substandard product quality, equipment damage, and material leaks. Since chemical production processes consist of various operational steps, identifying these steps from operational videos to derive the actual chemical production process, comparing it with standard operating procedures, and conducting analysis is crucial for ensuring safe chemical production operations.

[0003] After acquiring videos of chemical production operations, typical action recognition algorithms require cropping videos containing multiple actions into shorter videos containing only one action before recognition. However, cropping videos is labor-intensive, limiting the practical value of such methods. This invention employs an operation action segmentation algorithm designed to identify multiple continuous actions in uncropped long videos, making it more suitable for real-world applications.

[0004] Applying action segmentation algorithms to chemical production operations helps identify specific processes, thereby improving safety. However, due to data privacy and the interests of data owners, data silos are often difficult to aggregate, creating data islands. This is especially true in industries with high privacy requirements, such as chemical production. Federated learning methods effectively avoid this data silo problem. Through distributed learning, while ensuring data privacy, security, and legal compliance, collaborative modeling is achieved, improving the recognition performance of action segmentation models.

[0005] By employing a federated learning-based action segmentation algorithm, a model with strong generalization capabilities can be obtained while ensuring data privacy and security. Furthermore, leveraging IoT technology, it is possible to achieve ubiquitous connectivity between things and between things and people by collecting information in real time through various sensors and accessing various possible networks. Currently, due to the iterative upgrades of sensors, the information collected by sensors is becoming increasingly accurate, but this has also led to an excessively large amount of data, posing a severe challenge to the communication and computing capabilities of IoT devices. Thanks to the development of 5G mobile communication technology and edge computing, IoT devices can migrate some computing tasks to edge processors. This cloud-edge-device collaborative processing of computing tasks effectively solves the problem of insufficient computing power in IoT devices. Summary of the Invention

[0006] To overcome the above problems, the purpose of this invention is to provide a method for segmenting chemical production operation processes based on federated learning, which can analyze the operational processes of chemical production operators.

[0007] One aspect of the present invention provides a method for segmenting chemical production operation processes based on federated learning, the method comprising the following steps:

[0008] S1: The central server distributes the global model of the action segmentation algorithm to multiple training nodes;

[0009] S2: Each training node acquires a local set of chemical production operation videos;

[0010] The local chemical production operation video set is preprocessed and input into a pre-trained action recognition model to extract the feature sequence set;

[0011] S3: Using the feature sequence set obtained in step S2, the action segmentation model is trained using the federated learning method, and finally a high-generalization action segmentation model is obtained and deployed to the Internet of Things system.

[0012] S4: The IoT system records chemical production operation videos in real time on the end devices and sends them to the edge processor. The edge processor preprocesses the video data to obtain feature sequences and sends them to the cloud server.

[0013] S5: The cloud server uses the action segmentation model deployed in step S3 to perform action segmentation and recognition on the feature sequence sent by the side processor in step S4. The recognition results are processed to obtain the actual operation process and compared and analyzed with the standard operation process. Finally, an analysis report is output.

[0014] Furthermore, the specific process of step S1 is as follows:

[0015] S1.1: Use the action segmentation algorithm ASFormer, and obtain the global model w after initialization. global ;

[0016] S1.2: Obtain the global model w global Distribute to each training node;

[0017] S1.3: Each training node will use the global model w global Load the local model.

[0018] Furthermore, the specific process of step S2 is as follows:

[0019] S2.1: Training node K i Get local chemical production operation video set B i ={b d};

[0020] S2.2: Transfer local video set B i The pixel size of each frame in each video is adjusted to obtain video set M. i ={m d}, d = 1, 2, 3, ..., D, where d is the video sequence number and D is the total number of videos;

[0021] S2.3: For each video m d The video is segmented chronologically using a sliding window method to obtain T short videos of length s frames each. t ;

[0022] S2.4: Each short video o t Input a pre-trained action recognition model to extract feature vectors vec t ;

[0023] S2.5: Extract all feature vectors vec from each video t Synthesize a feature sequence v in chronological order. d Feature sequence v d Constitutes the feature sequence set V i .

[0024] Furthermore, the specific process of step S3 is as follows:

[0025] S3.1: K of each training node i The processed feature sequence set V i As a dataset, the local action segmentation model is trained, and iterative optimization is performed using an optimization algorithm and loss function. After running a specified number of training epochs l, the updated local model w is obtained. node,i ;

[0026] S3.2: Each training node will update its local model w node,i and the size n of the training data. i Uploaded to the central server;

[0027] S3.3: The central server receives the updated local model w from each training node. node,i and data size n i For each training node, the amount of video data n i The corresponding training node weights w are obtained through processing. i ;

[0028] Through each local model w node,i With corresponding weight w i The weighted summation yields the updated global model w on the central server. global ;

[0029] S3.4: The central server will update the global model w global Then it is sent to each of the training nodes for training;

[0030] S3.5: Repeat steps S3.1-S3.4 for a specified number of rounds e. During the multiple training and aggregation cycles, the central server records the accuracy, loss, and F1 score of the aggregation model for each iteration. After training for the specified number of rounds, an action segmentation model with strong generalization ability is finally obtained.

[0031] S3.6: Deploy the trained action segmentation model to the cloud server in the Internet of Things system;

[0032] Furthermore, the specific process of step S4 is as follows:

[0033] S4.1: Real-time detection of whether chemical operators need to perform chemical production process analysis;

[0034] S4.2: Upon detecting the analysis requirement, the terminal device begins real-time video recording and continuously determines whether the operation process is complete.

[0035] S4.3: The operation process has been completed, and the recorded video b is transferred. d Send to the edge processor;

[0036] S4.4: The side processor receives video b sent by the end device. d Then, the preprocessing procedure in step S2 is executed to obtain the feature sequence v. d And then send it to the cloud server.

[0037] Furthermore, the specific process of step S5 is as follows:

[0038] S5.1: Using the action segmentation model deployed in step S3, process the feature sequence v uploaded by the endpoint server in step S5. d Perform action segmentation and recognition to obtain the feature sequence v d A reliability sequence x of the same length that contains reliability for all action types. d ;

[0039] S5.2: Transform the reliability sequence x d The action category with the highest confidence value within each confidence vector is determined as the action category of the frame represented by that confidence vector;

[0040] S5.3: Group the confidence vectors of the same action category and arrange them together into one action segment, and record the sequence number from the start frame to the end frame in each segment to calculate the duration of the segment;

[0041] S5.4: Arrange all action steps in sequence to obtain the actual operation process and compare it with the standard operation process to generate an analysis report.

[0042] Furthermore, the analysis report includes: whether the actual operation process sequence conforms to the safe production operation specifications, whether there are any missing links in the actual operation process, and whether the operation time of each link is reasonable.

[0043] Another aspect of the present invention provides an Internet of Things (IoT) system, including an end device, an edge processor, a cloud server, and a program deployed and running on the edge processor and the cloud server, wherein the edge processor and the cloud server execute the program to implement the method described above.

[0044] The beneficial effects of this invention are:

[0045] 1. This invention employs an action segmentation model to accurately classify the actions in each frame of a video image. Compared to other action recognition algorithms, this invention can classify the actions of each frame in an entire uncropped long video, greatly reducing the waste of manpower.

[0046] 2. This invention uses a Video-Swin-Transformer pre-trained model for feature extraction. Compared with other video feature extraction methods, this invention makes full use of the transformer model structure and adopts a local attention mechanism, which effectively improves the feature extraction capability while reducing the amount of computation and memory consumption.

[0047] 3. This invention adopts a federated learning strategy, which can perform distributed learning of the model while ensuring data privacy and protecting the interests of the data ownership institutions. This effectively improves the generalization performance of the model and avoids overfitting.

[0048] 4. This invention adopts a cloud-edge-device IoT architecture, which can allocate tasks such as feature extraction to edge processors for execution, effectively alleviating the problem of insufficient computing power in IoT devices. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments or prior art, the accompanying drawings used in the embodiments will be briefly introduced below. The features and advantages of this article will be more clearly understood by referring to the accompanying drawings. The accompanying drawings are schematic and should not be construed as limiting this article in any way. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] in:

[0051] Figure 1 This is a block diagram illustrating the system principle of the present invention;

[0052] Figure 2 This is a flowchart of the federated learning process of this invention;

[0053] Figure 3 This is a flowchart of the video acquisition module of the present invention;

[0054] Figure 4 This is a flowchart of the feature extraction module of the present invention;

[0055] Figure 5 This is a flowchart of the action segmentation and analysis module of the present invention;

[0056] Figure 6 This is a flowchart illustrating the method of the present invention;

[0057] Figure 7 This is a diagram showing the running results of the action segmentation algorithm used in this invention. Detailed Implementation

[0058] The technical solution of this application will be further explained below with reference to the accompanying drawings.

[0059] Please see Figure 1 This is a system principle block diagram of a chemical production operation process segmentation method based on federated learning provided in this application.

[0060] This application provides a method for segmenting chemical production operation processes based on federated learning, which includes two parts: a federated learning part and an Internet of Things (IoT) system. The IoT system consists of a video acquisition (end) module, a feature extraction (edge) module, and an action segmentation (cloud) module.

[0061] The federated learning component described in this application mainly includes four steps: defining and initializing the model, processing local data, training the model, and aggregating the model; the central server defines the global model w. global Then, the global model is initialized using the model initialization method and distributed to each training node K. i Training node K i For local video set B i After preprocessing, the input to the action recognition model Video-Swin-Transformer finally yields the feature sequence set V. i The training node will use the feature sequence set V i As a training set for the local model w node,i Training will proceed, and after a specified number of rounds, each training node will transfer its local model w. node,i The data is sent back to the central server, which then sends the model data uploaded by each training node back to the central server. node,i The aggregation is performed and then redeployed to each training node, repeating the previous steps until the specified training round number e is reached, ultimately resulting in a global model w with strong generalization capabilities. global .

[0062] The video acquisition (end) module described in this application includes two steps: detecting the analysis requirement and acquiring and sending video data. When the end device detects the need to analyze the current chemical production operation, the camera starts recording video in real time. After receiving the operation completion signal, the end device ends recording and sends the video to the side processor. The feature extraction (side) module includes two steps: preprocessing the video data and extracting a feature sequence set. The side processor first processes the video data b sent by the end device. d Modify the pixel size, then change the video size. d Extracting short video segments and inputting them into a pre-trained action recognition model yields feature sequences v. d Finally, the feature sequence v d Send to the cloud server; the action segmentation (cloud) module includes the feature sequence v d The process involves two steps: action segmentation and recognition, and analysis of the results to generate an analysis report. The cloud server receives the feature sequence v sent by the edge processor. d Then, it is input into the deployed action segmentation model to identify the reliability sequence. The reliability sequence is processed to obtain the actual chemical production operation process. The actual chemical production operation process is compared and analyzed with the standard process, and an analysis report is output.

[0063] Please see Figure 2 This is a federated learning flowchart of a chemical production operation process segmentation method based on federated learning provided by the present invention. Its workflow is as follows:

[0064] 1. The central server defines the action segmentation model and initializes the global model w using the initialization method. global ;

[0065] 2. The central server will define the global model w global Distribute to each training node K i ;

[0066] 3. K of each training node i First, analyze the local video dataset B. i Preprocessing is performed on the local video dataset B. i Each video in b d The pixel size is changed to h×j, and then the modified video data m d The video is segmented into T short videos of length s using a sliding window of size s. t All short videos are input into the Video-Swin-Transformer action recognition model in chronological order to obtain the feature sequence set V. i ;

[0067] 4. K of each training nodei The processed feature sequence set V i The model w, sent to the central server as part of the training set, global After training for a specified number of training epochs (l), the updated local model w is obtained. node,i ;

[0068] 5. K of each training node i The local video dataset size n i and the updated model w node,i Send to the central server;

[0069] 6. The central server calculates the data volume n for each training node. i Generate the training weights w for each training node i And based on this, we can analyze each model w. node,i Perform weighted aggregation to obtain the updated global model w. global ;

[0070] 7. Determine if the training epochs have reached the specified epoch number e. If they have, update the global model w obtained from the last update. global Save; otherwise, repeat process 2-6.

[0071] Please see Figure 3 This is a video acquisition (end) flowchart of a chemical production operation process segmentation method based on federated learning provided in this embodiment. Its workflow is as follows:

[0072] 1. Turn on the terminal device (camera) and determine in real time whether there is a need for chemical production operation process analysis;

[0073] 2. If there is no demand, it will enter a waiting state; otherwise, it will call the camera to start recording video.

[0074] 3. Submit the completed chemical production operation video. d Saved to the local hard drive;

[0075] 4. Save the complete chemical production operation video locally. d Upload to the edge processor.

[0076] Please see Figure 4 This is a feature extraction (edge) flowchart of a chemical production operation process segmentation method based on federated learning provided in this embodiment. Its workflow is as follows:

[0077] 1. The edge processor receives chemical production operation video b transmitted from the end device. d ;

[0078] 2. Adjust the pixel size of each frame in the chemical production operation video to h×j;

[0079] 3. Divide the adjusted video into T smaller videos of length s frames using a sliding window of size s. t ;

[0080] 4. Divide each of the short videos into smaller segments. t The input is fed into the pre-trained action recognition model Video-Swin-Transformer to extract a feature vector vec of size q×p. t ;

[0081] 5. Transfer the feature vector vec t Synthesize the feature sequence v in chronological order d ;

[0082] 6. Combine the synthesized feature sequence v d Send to the cloud server;

[0083] Please see Figure 5 This is an action segmentation (cloud) flowchart of a chemical production operation process segmentation method based on federated learning provided in this embodiment. Its workflow is as follows:

[0084] 1. Receive the feature sequence v sent by the side processor. d ;

[0085] 2. The feature sequence v d The input is fed into an action segmentation model trained through federated learning for recognition, resulting in a confidence sequence x with the same length as the feature sequence and containing confidence scores for all action types. d ;

[0086] 3. Transform the confidence sequence x d The action category with the highest confidence value in each confidence vector is determined as the action category of the frame represented by that confidence vector. Confidence vectors of the same action category are grouped into one action segment. The sequence number from the start frame to the end frame in each segment is recorded to calculate the duration of that segment.

[0087] 4. Arrange all action steps in sequence to obtain the actual operation flow, and compare and analyze it with the standard action flow, including judging the flow order, the flow completeness, and the duration of each step.

[0088] 5. Compile an analysis report containing information such as the accuracy of the process sequence, the completeness of the process, and the duration of each step, to assess the safety of the chemical production operation process. The analysis report includes: whether the actual operation process sequence conforms to the safe production operation specifications, whether there are any missing steps in the actual operation process, and whether the operation time of each step is reasonable.

[0089] Please see Figure 6This is a flowchart of a chemical production operation process segmentation method based on federated learning provided in this embodiment, which includes the following steps:

[0090] S1: The central server will use the global model w of the ASFormer action segmentation algorithm. global Distribute to training node K i , i = 1, 2, 3, ..., N, where i is the training node number and N is the total number of nodes;

[0091] S2: Training node K i Get local chemical production operation video set B i ={b d}, d = 1, 2, 3, ..., D, where d is the video sequence number and D is the total number of videos; b d ={img t}, t=1,2,3,...,T, where t is the video frame number and T is the total number of video frames.

[0092] Local chemical production operation video collection B i Preprocessing is performed and the result is input into a pre-trained action recognition model, Video-Swin-Transformer, to extract the feature sequence set V. i ={v d}, d = 1, 2, 3, ..., D, where d is the video frame number and D is the total number of video frames, d = 1, 2, 3, ..., D, v d ={vec t}, t=1,2,3,...,T, where t is the video frame number and T is the total number of video frames;

[0093] S3: Using the feature sequence set V obtained in step S2 i The action segmentation model is trained using a federated learning method, ultimately resulting in an action segmentation model w with strong generalization ability. global And deploy it into the Internet of Things (IoT) system;

[0094] S4: The IoT system records real-time video of chemical production operations via end devices (cameras). d The data is then sent to the side processor, which preprocesses the video data using the data preprocessing method from step S2 to obtain the feature sequence v. d And send it to the cloud server;

[0095] S5: The cloud server uses the action segmentation model deployed in step S5 to process the feature sequence v sent by the side processor in step S6. d Action segmentation and recognition are performed, the recognition results are processed to obtain the actual operation process, and compared and analyzed with the standard operation process. Finally, an analysis report is output.

[0096] In one embodiment, step S1 further specifically includes the following steps:

[0097] S1.1: Use the action segmentation model network ASFormer and initialize the network model using common model initialization methods to obtain the global model w. global

[0098] S1.2: Obtain the global model w global Distribute to each training node;

[0099] S1.3: Each training node loads the global model into its local model.

[0100] In one embodiment, step S2 further specifically includes the following steps:

[0101] S2.1: Training node K i Get local chemical production operation video set B i ={b d};

[0102] S2.2: Set up each video b in the local video cluster. d The pixel size of each frame is adjusted to h×j, resulting in video set M. i ={m d}, d = 1, 2, 3, ..., D, where d is the video sequence number and D is the total number of videos;

[0103] S2.3: Adjust each video m after the process. d The video is segmented using a sliding window of size s, resulting in T short videos of length s frames. t ;

[0104] S2.4: Divide each of the small videos from step S2.3 into segments. t Input a pre-trained action recognition model, Video-Swin-Transformer, to extract a feature vector vec of size q×p. t ;

[0105] S2.5: Extract all feature vectors vec from each video t Synthesize a feature sequence v sequentially. d Then combine all the feature sequences v d Constitutes the feature sequence set V i ;

[0106] In one embodiment, step S3 further includes the following steps:

[0107] S3.1: K of each training node i The processed feature sequence set V iThe local action segmentation model is trained using the dataset, and iterative optimization is performed using common optimization algorithms and loss functions. After running a specified number of training epochs l, the updated local model w is obtained. node,i ;

[0108] S3.2: Each training node will update its local model w node,i and the size of the data n i Uploaded to the central server;

[0109] S3.3: The central server receives the updated local model w from each training node. node,i and data size n i First, the video data volume n of each training node... i The weights w of each training node are obtained by processing using formula (1). i :

[0110]

[0111] Then, using formula (2), each local model w node,i With corresponding weight w i The weighted summation yields the global update model set w on the central server. global ;

[0112]

[0113] S3.4: The central server will update the global model w global Then it is sent to each of the training nodes for training;

[0114] S3.5: Repeat steps S3.1-S3.4 for a specified number of rounds e. In multiple training and aggregation cycles, the central server records the accuracy, loss, and F1 score of the aggregation model for each round. After training for a specified number of rounds, an action segmentation model with strong generalization ability is finally obtained.

[0115]

[0116] Where p is the precision and r is the recall, p is calculated by the following formula:

[0117]

[0118] Where TP represents true positives, FP represents false positives, and r is calculated by the following formula:

[0119]

[0120] FN represents false negatives. After training for a specified number of rounds, a strong generalization action segmentation model is finally obtained.

[0121] S3.6: Deploy the trained action segmentation model to the cloud server in the Internet of Things system;

[0122] In one embodiment, step S4 further includes the following steps:

[0123] S4.1: Real-time detection of whether chemical operators need to perform chemical production process analysis;

[0124] S4.2: When the analysis requirement is detected, the system begins to execute the process analysis task. The end device (camera) begins to record video in real time and determines in real time whether the operation process is completed.

[0125] S4.3: The operation process has been completed. The recorded video b is now ready. d Send to the edge processor;

[0126] S4.4: The side processor receives video b sent by the end device. d Then, the preprocessing procedure in step S2 is executed to obtain the feature sequence v. d And send it to the cloud server;

[0127] In one embodiment, step S5 further includes the following steps:

[0128] S5.1: Using the action segmentation model deployed in step S3, process the feature sequence v uploaded by the endpoint server in step S5. d Perform action segmentation and recognition to obtain a confidence sequence x with the same length as the feature sequence and containing confidence scores for all action types. d ;

[0129] S5.2: Transform the reliability sequence x d The action category with the highest confidence value within each confidence vector is determined as the action category of the frame represented by that confidence vector;

[0130] S5.3: Group the confidence vectors of the same action category into one action segment, and record the sequence number from the start frame to the end frame in each segment to calculate the duration of the segment;

[0131] S5.4: Arrange all action steps in sequence to obtain the actual operation process and compare it with the standard operation process to generate an analysis report; the analysis report includes: whether the sequence of the actual operation process conforms to the safety production operation specifications, whether there are any missing steps in the actual operation process, and whether the operation time of each step is reasonable.

[0132] The specific application scenarios of the present invention will be further described below with reference to specific embodiments:

[0133] Scene 1:

[0134] The central server uses the action segmentation model Asformer, with model w global The central server distributes the model to three training nodes K. i i = 1, 2, 3; Each training node uses the local pump start / stop video dataset B. i Preprocessing is performed by first adjusting the local video data to a pixel size of 320×240, and then processing the adjusted video dataset M... i The video was segmented using a sliding window with a window size of 16, resulting in multiple short videos of 16 frames each. t Finally, each short video is input into the pre-trained action recognition model Video-Swin-Transformer in chronological order to extract a feature vector vec of size 1×1024. t vec of all feature vectors extracted from each video t Synthesize a feature sequence v sequentially. d Then combine all the feature sequences v d Constitutes the feature sequence set V i .

[0135] Each training node will use model w global Deploy locally to obtain the local model w node,i and the local feature sequence set V i As a dataset for the local model w node,i The model is trained using the Adam optimization algorithm and cross-entropy loss function, and the updated local model w is obtained after 50 iterations. node,i Each training node will update its local model w node,i and data size n i The data is sent to the central server, which then uses formula (1) to determine the data size n. i The weight w corresponding to each training node is calculated. i =[0.37,0.42,0.21], where the model w sent to each training node is weighted according to the corresponding weights of each training node. node,i The updated global model w is obtained by weighted aggregation using formula (2). global Repeat the above steps 30 times to obtain an operation action segmentation model w with strong generalization ability. global It is then deployed to the cloud server of the Internet of Things system.

[0136] In actual pump start-up and shutdown operation scenarios, the cloud detects a need for operation action segmentation and recognition, and begins recording video of the pump start-up and shutdown operation using a camera. Once the cloud detects a completion signal, recording ends and the video is saved to the local hard drive. The video is 31 seconds long and contains 944 frames. The saved video is then sent to an edge processor. The edge processor first preprocesses the video data, adjusting the video pixel size to 320×240. Then, the adjusted video data is divided into smaller videos based on each frame being separated from its nearest neighbor by less than 8 frames. These smaller videos are sequentially input into a pre-trained action recognition model, Video-Swin-Transformer. Each smaller video extracts a 1×1024 feature vector. All feature vectors are then aggregated in chronological order into a feature sequence v. d Send to the cloud server, the cloud server receives the feature sequence v d The input is then fed into the deployed action segmentation model, and the confidence sequence x is obtained after model recognition. d Then the confidence sequence x d The action category with the highest confidence value within each confidence vector is determined as the action category of the frame represented by that confidence vector. Confidence vectors with the same action category arranged together are grouped into one action segment. Simultaneously, the sequence number from the start frame to the end frame within each segment is recorded to calculate the duration of that segment, thus obtaining the actual pump start-up and shutdown operation procedure. The actual pump start-up and shutdown operation procedure is then compared and analyzed with the standard pump start-up and shutdown operation procedure, comparing whether the procedure is missing, the procedure order, and the standard procedure time, and finally outputting an analysis report.

[0137] Please see Figure 7 This is a flowchart illustrating the results of a federated learning-based chemical production operation segmentation method provided in this embodiment. The results in the flowchart are obtained by identifying pump shutdown videos captured on-site in the specific application scenario described above using an action segmentation model trained through federated learning. The coordinate axes represent frame numbers; the black line represents the standard pump shutdown operation flow, and the line below the black line represents the actual pump shutdown operation flow identified by the action segmentation model. Dark red areas represent transitional stages, including personnel movement and non-chemical operations such as background noise. Dark blue areas represent the pump outlet valve closure stage, purple areas represent the pump shutdown stage, and red areas represent the pump inlet valve closure stage.

[0138] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented, in whole or in part, as a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0139] The above description is merely a specific implementation of the embodiments of this specification. These embodiments further illustrate the technical solution of the present invention, enabling those skilled in the art to understand the technical concept of the present invention. Methods and terms not explicitly disclosed, such as algorithms, can be implemented using conventional techniques in the art, and no specific limitations are imposed on them. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles of the embodiments of this specification, and these improvements and modifications should also be considered within the protection scope of the embodiments of this specification.

Claims

1. A method for segmenting chemical production operation processes based on federated learning, characterized in that... The method includes the following steps: S1: The central server distributes the global model of the action segmentation algorithm to multiple training nodes; S2: Each training node acquires a local set of chemical production operation videos; The local chemical production operation video set is preprocessed and input into a pre-trained action recognition model to extract the feature sequence set; S3: Using the feature sequence set obtained in step S2, the action segmentation model is trained using the federated learning method, and finally a high-generalization action segmentation model is obtained and deployed to the Internet of Things system. S4: The IoT system records chemical production operation videos in real time on the end devices and sends them to the edge processor. The edge processor preprocesses the video data to obtain feature sequences and sends them to the cloud server. S5: The cloud server uses the action segmentation model deployed in step S3 to perform action segmentation and recognition on the feature sequence sent by the side processor in step S4, processes the recognition results to obtain the actual operation process and compares and analyzes it with the standard operation process, and finally outputs an analysis report. The specific process of step S3 is as follows: S3.1: K of each training node i The processed feature sequence set V i As a dataset, the local action segmentation model is trained, and iterative optimization is performed using an optimization algorithm and loss function. After running a specified number of training epochs l, the updated local model w is obtained. node,i ; S3.2: Each training node will update its local model w node,i and the size n of the training data. i Uploaded to the central server; S3.3: The central server receives the updated local model w from each training node. node,i and data size n i For each training node, the video data size n i The corresponding training node weights w are obtained through processing. i ; Through each local model w node,i With corresponding weight w i The weighted summation yields the updated global model w on the central server. global ; S3.4: The central server will update the global model w global Then it is sent to each of the training nodes for training; S3.5: Repeat steps S3.1-S3.4 for a specified number of rounds e. In multiple training and aggregation cycles, the central server records the accuracy, loss value and F1 score of the aggregation model for each round. After training for a specified number of rounds, an action segmentation model with strong generalization ability is finally obtained. S3.6: Deploy the trained action segmentation model to the cloud server in the Internet of Things system.

2. The chemical production operation process segmentation method based on federated learning according to claim 1, characterized in that, The specific process of step S1 is as follows: S1.1: Use the action segmentation algorithm ASFormer, and obtain the global model w after initialization. global ; S1.2: Obtain the global model w global Distribute to each training node; S1.3: Each training node will use the global model w global Load the local model.

3. The chemical production operation process segmentation method based on federated learning according to claim 1, characterized in that, The specific process of step S2 is as follows: S2.1: Training node K i Get local chemical production operation video set B i = , where b d Indicates video; S2.2: Transfer local video set B i The pixel size of each frame in each video is adjusted to obtain video set M. i = d=1,2,3,...,D, where d is the video sequence number and D is the total number of videos; S2.3: For each video The video is segmented chronologically using a sliding window method, resulting in T short videos of length s frames. ; S2.4: Each short video Input a pre-trained action recognition model to extract feature vectors ; S2.5: Extract all feature vectors from each video Synthesize a feature sequence v in chronological order. d Feature sequence v d Constitutes the feature sequence set V i .

4. The chemical production operation process segmentation method based on federated learning according to claim 1, characterized in that, The specific process of step S4 is as follows: S4.1: Real-time detection of whether chemical operators need to perform chemical production process analysis; S4.2: Upon detecting the analysis requirement, the terminal device begins real-time video recording and continuously determines whether the operation process is complete. S4.3: The operation process has been completed, and the recorded video b is transferred. d Send to the edge processor; S4.4: The side processor receives video b sent by the end device. d Then, the preprocessing procedure in step S2 is executed to obtain the feature sequence v. d And then send it to the cloud server.

5. The chemical production operation process segmentation method based on federated learning according to claim 1, characterized in that, The specific process of step S5 is as follows: S5.1: Using the action segmentation model deployed in step S3, process the feature sequence v uploaded by the endpoint server in step S5. d Perform action segmentation and recognition to obtain the feature sequence v d A reliability sequence x of the same length that contains reliability for all action types. d ; S5.2: Transform the reliability sequence x d The action category with the highest confidence value within each confidence vector is determined as the action category of the frame represented by that confidence vector; S5.3: Group the confidence vectors of the same action category that are arranged together into one action segment, and record the sequence number from the start frame to the end frame in each segment to calculate the duration of the segment; S5.4: Arrange all action steps in sequence to obtain the actual operation process and compare it with the standard operation process to generate an analysis report.

6. The chemical production operation process segmentation method based on federated learning according to claim 1, characterized in that, The analysis report includes: whether the actual operation process sequence conforms to the safe production operation specifications, whether there are any missing links in the actual operation process, and whether the operation time of each link is reasonable.

7. An Internet of Things (IoT) system, comprising an end device, an edge processor, a cloud server, and a program deployed on the edge processor and the cloud server, wherein the edge processor and the cloud server execute the program to implement the method as described in any one of claims 1-6.

Images