Multi-modal large model training method, nuclear power plant field digital work supervision and guidance system and method

Abstract

The invention aims to provide a multi-modal large model training method, and a nuclear power plant field digital work supervision and guidance system and method. The training method comprises the following steps: acquiring information in a pre-training database; information in the pre-training database comprises a personnel information base, an equipment information base, a regulation information base containing regulation activity information and regulation content, and a standard activity base containing standard operation activity information; inputting the information in the pre-training database into the multi-modal large model for initialization training, so as to obtain a first multi-modal large model with the capability of identifying the activity and the equipment state of the standard operating personnel. According to the invention, the appropriate training database is established for training the multi-modal large model according to the uniqueness of the field work of the nuclear power plant, and the multi-modal large model is utilized to realize the field digital work supervision and guidance of the nuclear power plant, so that the technical problem that the real-time supervision and guidance function of the activity of the field operation personnel of the nuclear power plant cannot be realized in the prior art is solved.

Description

Technical Field

[0001] The present application belongs to the field of nuclear power technology, and specifically relates to a training method for a multi-modal large model, and a nuclear power plant on-site digital work supervision and guidance system and method. Background Art

[0002] On-site work at nuclear power plants is safety-critical, and effective prevention of operator errors is crucial to the conduct of on-site work at nuclear power plants.

[0003] At present, there are many technologies that can realize the supervision and guidance functions of similar on-site work, such as the intelligent monitoring system that uses video communication equipment to realize management personnel supervision (CN202111128274.5, CN202022945939.9), the specific task indicator supervision method and system based on simple algorithm (CN202011228110.5), and the remote guidance system and method based on augmented reality technology for modular operation display (CN202110951488.6). However, the above systems only realize the transmission of information such as pictures, sounds, and operation models, and still rely on people for supervision and guidance.

[0004] With the development of artificial intelligence technology, some devices, systems or methods that use machine vision to replace human supervision and guidance have also emerged, such as operation guidance equipment based on machine vision to realize intelligent operation review (CN202211670413.1), safety analysis method based on artificial intelligence analysis of on-site video recognition of action and step relationship (CN202110878934.5), etc. Invention Patent 1 (CN202211670413.1) describes an intelligent operation and maintenance operation guidance device for maintenance scenarios, which uses an artificial intelligence computing unit to identify devices and texts, and takes pictures of the maintenance components after the workflow steps are completed to review and guide the operation process, that is, only supervise the status of the components. Invention Patent 2 (CN202110878934.5) collects video image information of the oil well operation area and uses artificial intelligence technology to supervise and warn the operation specifications of the operators. However, the above methods, equipment or systems do not provide real-time supervision and guidance for the operation activities of operators in the complex on-site operation environment and strict safety behavior specifications of nuclear power plants.

[0005] In summary, mistakes made by nuclear power site operators during their tasks may lead to significant losses, so supervision and guidance of operators is necessary. Currently, there are many methods and technologies to achieve supervision and guidance of operator activities. However, previous technologies either only provide a communication method, and the actual supervision task still relies on off-site personnel, or rely on artificial intelligence technology to review and warn operators' activities. These technologies cannot achieve real-time supervision and guidance of the activities of nuclear power plant site operators. The participation of off-site personnel will also introduce new forms of human error, and the reliability of supervision and guidance is difficult to guarantee. Summary of the invention

[0006] In view of this, the present application is committed to providing a training method for a multimodal large model, a system and method for supervising and guiding digital work on site in a nuclear power plant, and establishing a suitable training database to train the multimodal large model based on the uniqueness of on-site work in a nuclear power plant. The multimodal large model is used to realize digital supervision and guidance of digital work on site in a nuclear power plant, so as to solve the technical problem that the existing technology cannot realize the real-time supervision and guidance function of the activities of on-site operators in a nuclear power plant.

[0007] The first aspect of the present application provides a training method for a multimodal large model for on-site digital work supervision and guidance of a nuclear power plant, and the training method for a multimodal large model for on-site digital work supervision and guidance of a nuclear power plant comprises: obtaining information in a pre-training database. The information in the pre-training database includes a personnel information database, an equipment information database, a procedure information database containing procedure activity information and procedure content, and a standard activity database containing standard operation activity information; the information in the pre-training database is input into the multimodal large model for initialization training to obtain a first multimodal large model capable of identifying standard operation personnel activities and equipment status.

[0008] In a specific embodiment of the present application, after the information in the pre-trained database is input into the multimodal large model for initialization training to obtain a first multimodal large model capable of identifying standard operating personnel activities and equipment status, the training method of the multimodal large model for on-site digital work supervision and guidance of nuclear power plants also includes: acquiring information in a dynamic collection database from the terminal device in real time. The information in the dynamic collection database includes an operation activity library and an evaluation result library, and the operation activity library contains audio and video information of the operation activities collected during the application process. The evaluation result library contains evaluation data information on whether the operation activities meet the requirements of the specifications and procedures; the information in the dynamic collection database is input into the first multimodal large model for continuous adversarial learning training to obtain a second multimodal large model.

[0009] The second aspect of the present application provides a cloud device, which includes an acquisition module, an initialization training module, a calculation module and an output module. The acquisition module is used to acquire information from a pre-training database. The information in the pre-training database includes a personnel information library, an equipment information library, a procedure information library containing procedure activity information and procedure content, and a standard activity library containing standard operation activity information. The initialization training module is used to input the information in the pre-training database into a multimodal large model for initialization training to obtain a first multimodal large model with the ability to identify standard operator activities and equipment status. The calculation module is used to calculate and output audio and video guidance information through the first multimodal large model according to the real-time audio and video data transmitted by the terminal device. The real-time audio and video data transmitted by the terminal device includes audio and video information of the operator's face, audio and video information of the operator's actions, audio and video information of the operating equipment status, operation task information and use procedure information. The audio and video guidance information includes judgment information on whether the operator's activities comply with the specifications and procedures, judgment information on whether the equipment status meets expectations, and guidance information on the operator's activities. The output module is used to transmit the audio and video guidance information to the terminal device.

[0010] In a specific embodiment of the present application, the cloud device also includes a continuous adversarial learning training module. The acquisition module is also used to acquire information in the dynamic collection database in real time. The information in the dynamic collection database includes a job activity library and an evaluation result library. The job activity library contains audio and video information of the job activities collected during the application process. The evaluation result library contains evaluation data information on whether the job activities meet the requirements of the specifications and procedures. The continuous adversarial learning training module is used to input the information in the dynamic collection database into the first multimodal large model for continuous adversarial learning training to obtain a second multimodal large model. The operation module is also used to output audio and video guidance information through the second multimodal large model operation based on the pre-processed real-time audio and video data transmitted by the terminal device, and transmit the audio and video guidance information to the terminal device.

[0011] The third aspect of the present application provides a terminal device, which includes a fixed or mobile personnel activity monitoring device, a wearable audio and video acquisition and receiving conversion device worn by the operator, and an edge computing device. The personnel activity monitoring device and the audio and video acquisition and receiving conversion device are used by the operator to collect the information data of the operator, the audio and video data of the operator's activities, the status data of the operating equipment, and the audio and video data of the task scene in real time during the execution of the operation task. The audio and video acquisition and receiving conversion device is also used to receive the result data of supervision and guidance transmitted by the cloud device, convert the result data of supervision and guidance transmitted by the cloud device into audio and video information and transmit it to the operator. The edge computing device is connected to the personnel activity monitoring device and the audio and video acquisition and receiving conversion device respectively, and the edge computing device is used to pre-process the collected audio and video data, transmit the pre-processed audio and video data to the cloud device, transmit the task description and selected procedure data to the cloud device, and output the description of the operation activity, deviation and corrected audio and video data to the operator.

[0012] The fourth aspect of the present application provides a nuclear power plant on-site digital work supervision and guidance system, which includes a cloud device and a terminal device that are communicatively connected. The terminal device of the third aspect of the present application is used to collect information data of operators, audio and video data of operators' activities, status data of operating equipment, and audio and video data of task scenes in real time and pre-process them, transmit the pre-processed audio and video data to the cloud device provided in the embodiment of the present application, and receive the results of supervision and guidance transmitted by the cloud device. The cloud device of the second aspect of the present application is used to process multimodal large model training and calculation, and calculates the results of supervision and guidance based on the pre-processed audio and video data transmitted by the terminal device and transmits them to the terminal device.

[0013] The fifth aspect of the present application provides a method for supervising and guiding digital work on site in a nuclear power plant, which includes: when the operating personnel are performing their operating tasks, the personnel activity monitoring equipment and the audio and video acquisition and receiving conversion equipment in the terminal device collect the operating personnel's information data, audio and video data of the operating personnel's activities, status data of the operating equipment and audio and video data of the task scene in real time; the edge computing device in the terminal device pre-processes the collected audio and video data, and then transmits the pre-processed data to the cloud device; the cloud device calls a multimodal large model to analyze and calculate the input audio and video data, obtains the supervision and guidance result data, and outputs the supervision and guidance result data to the terminal device.

[0014] In a specific embodiment of the present application, when the operating personnel are performing the operating tasks, the personnel activity monitoring equipment and the audio and video acquisition and receiving conversion equipment in the terminal device collect the operating personnel's information data, audio and video data of the operating personnel's activities, status data of the operating equipment and audio and video data of the task scene in real time. Previously, the nuclear power plant on-site digital work supervision and guidance method also included: the terminal device identifies the operating personnel, obtains and verifies the identity information from the personnel information library to collect the operating personnel's information data; the terminal device identifies the operating equipment, and obtains the status data of the operating equipment from the equipment information library; the terminal device selects the procedure, and the cloud device calls the corresponding information in the procedure activity library.

[0015] While the operators are performing their tasks, the personnel activity monitoring equipment and the audio and video acquisition and receiving conversion equipment in the terminal device collect the operators' information data, audio and video data of the operators' activities, status data of the operating equipment, and audio and video data of the task scene in real time; the edge computing device in the terminal device pre-processes the collected audio and video data, and then transmits the pre-processed data to the cloud device, including: after the start of the operation, for each operation step of the operator, the terminal device collects the operators' information data, audio and video data of the operators' activities, status data of the operating equipment, and audio and video data of the task scene in real time and pre-processes them.

[0016] The cloud device calls the multimodal large model to analyze and calculate the input audio and video data, obtains the supervision and guidance result data, and outputs the supervision and guidance result data to the terminal device, including: the cloud device receives the pre-processed data transmitted by the terminal device; the cloud device calls the first multimodal large model or the second multimodal large model to compare the pre-processed data with the standard activity library and the procedure activity library to obtain the supervision and guidance result data; the cloud device sends the supervision and guidance result data to the terminal device, and transmits warning guidance information to the operator in real time. According to the warning guidance information, the operator continues the operation, or corrects the activity according to the warning guidance information.

[0017] In a specific implementation of the present application, the cloud device sends the result data of supervision and guidance to the terminal device, and transmits warning guidance information to the operator in real time. According to the warning guidance information, the operator continues the operation, or corrects the activity according to the warning guidance information, including: the operator identifies the operating equipment, the terminal device collects audio and video data and uploads it to the cloud device after preprocessing, the cloud device issues the analysis result after analysis, and the terminal device transmits the analysis result to the operator, that is, whether the operating equipment meets the work requirements; the operator orally states the operation target, the terminal device collects audio and video data and uploads it to the cloud device after preprocessing, the cloud device issues the analysis result after analysis, and the terminal device transmits the analysis result to the operator, that is, whether the operation that the operator is about to complete meets the requirements; the operator completes the current operation step, the terminal device collects audio and video data and uploads it to the cloud device after preprocessing, the cloud device issues the analysis result after analysis, and the terminal device transmits the analysis result to the operator, that is, whether the state reached by the operating equipment meets the requirements and whether the operator can proceed to the next step.

[0018] In a specific embodiment of the present application, the method for supervising and guiding digital work on site of a nuclear power plant also includes: the terminal device uploads the activity data of the operating personnel to the operating activity database of the cloud device; the cloud device uses a multimodal large model to evaluate whether the activities of the operating personnel meet the requirements of the specifications and the expectations of the procedures; the auditor reviews and corrects the evaluation results of the cloud device and then uploads the evaluation results to the evaluation result library; the cloud device regularly uses the data in the operating activity database and the evaluation result library for adversarial learning training.

[0019] The beneficial effects of the technical solution of the present application are: in view of the uniqueness of on-site work in nuclear power plants, a pre-training database is established, and the information in the pre-training database includes a personnel information database, an equipment information database, a procedure information database containing procedure activity information and procedure content, and a standard activity database containing standard operation activity information, and the information in the pre-training database is input into the multimodal large model for initialization training, so that the first multimodal large model has the ability to identify standard operating personnel activities and equipment status, and can determine whether the operating personnel, equipment and equipment status, on-site operating activities, etc. meet the operating requirements, status expectations, behavioral norms, procedure requirements, etc. The embodiment of the present application establishes and uses an audio and video database of standard activities of operating personnel that conform to the on-site behavioral specifications of nuclear power plants to train a multimodal large model, thereby realizing the functions of accurately identifying and analyzing the standard activities of operating personnel and providing prompts and guidance when the conformity between the operating personnel activities and the standard activities is low; establishes and uses a personnel and equipment information database to train a multimodal large model, thereby realizing the accurate identification and analysis of on-site operating personnel of nuclear power plants and the status of operating equipment during operations, thereby confirming whether the on-site operating personnel meet the qualification requirements and whether the status of operating equipment meets the requirements; establishes and uses a database of on-site operating procedure information of nuclear power plants to train a multimodal large model, thereby realizing the accurate identification and analysis of procedure activities and providing prompts and guidance when the conformity between the operating personnel activities and the procedure activities is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The figure is a flow chart of a training method for a multimodal large model for on-site digital work supervision and guidance in a nuclear power plant provided in one embodiment of the present application.

[0021] Figure 2 Shown is a structural schematic diagram of a nuclear power plant on-site digital work supervision and guidance system provided by an embodiment of the present application.

[0022] Figure 3 Shown is a technical roadmap for a method for on-site digital work supervision and guidance of a nuclear power plant provided in one embodiment of the present application.

[0023] Figure 4 The figure is a flow chart of a method for supervising and guiding digital work on-site in a nuclear power plant provided in one embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] At least one embodiment of the present application provides a training method for a multi-modal large model for on-site digital work supervision and guidance of a nuclear power plant, referring to Figure 1 ,The training method of the multi-modal large model for on-site digital work supervision and guidance of nuclear power plants includes the following steps.

[0026] S20: Acquire information in the pre-training database. The information in the pre-training database includes a personnel information library 5, an equipment information library 6, a procedure information library 7 including procedure activity information and procedure content, and a standard activity library 8 including standard operation activity information.

[0027] It should be noted that the pre-training database is a fixed database, which can also be called a fixed standard operation activity audio and video database. The information in the pre-training database can be collectively referred to as standard nuclear power plant on-site operation process information, including but not limited to the action information, sound information and equipment information of the operators in the standard nuclear power plant on-site operation process.

[0028] The personnel information database contains personnel information. The equipment information database contains equipment information such as equipment functions and equipment operation interfaces.

[0029] The procedure information base containing procedure activity information and procedure content can also be called a fixed procedure standard operation base, or a database using nuclear power plant on-site operation procedure information. Nuclear power plant on-site operation procedure information includes but is not limited to procedure activity audio and video data and the text data of the procedure itself.

[0030] The standard activity library containing standard operating activity information may also be referred to as a standard nuclear power plant personnel activity library, or may also be referred to as a standard activity library containing activity information that complies with on-site operating behavior specifications.

[0031] S40: Input the information in the pre-training database into the multimodal big model for initialization training to obtain a first multimodal big model capable of recognizing standard operator activities and equipment status.

[0032] According to the technical solution provided in the embodiment of the present application, in view of the uniqueness of on-site work in nuclear power plants, a pre-training database is first established, and the information in the pre-training database is set to include a personnel information library 5, an equipment information library 6, a procedure information library 7 containing procedure activity information and procedure content, and a standard activity library 8 containing standard operation activity information. The information in the pre-training database is input into the multimodal large model for initialization training, so that the first multimodal large model has the ability to identify standard operating personnel activities and equipment status, and can determine whether the operating personnel, equipment and equipment status, on-site operating activities, etc. meet the operating requirements, status expectations, behavioral norms, procedure requirements, etc. The embodiment of the present application establishes and uses an audio and video database of standard activities of operating personnel that conform to the on-site behavioral specifications of nuclear power plants to train a multimodal large model, thereby realizing the functions of accurately identifying and analyzing the standard activities of operating personnel and providing prompts and guidance when the conformity between the operating personnel activities and the standard activities is low; establishes and uses a personnel and equipment information database to train a multimodal large model, thereby realizing the accurate identification and analysis of on-site operating personnel of nuclear power plants and the status of operating equipment during operations, thereby confirming whether the on-site operating personnel meet the qualification requirements and whether the status of operating equipment meets the requirements; establishes and uses a database of on-site operating procedure information of nuclear power plants (audio and video data of procedure activities and text data of the procedures themselves) to train a multimodal large model, thereby realizing the accurate identification and analysis of procedure activities and providing prompts and guidance when the conformity between the operating personnel activities and the procedure activities is low.

[0033] In at least one embodiment of the present application, after the above S40, the training method of the multimodal large model for on-site digital work supervision and guidance of nuclear power plants also includes S60 and S80.

[0034] S60: Acquire information in the dynamic collection database from the terminal device 3 in real time. The information in the dynamic collection database includes an operation activity database and an evaluation result database. The operation activity database contains audio and video information of the operation activities collected during the application process. The evaluation result database contains whether the operation activities meet the requirements of the specifications and the expectations of the regulations.

[0035] Specifically, a dynamic collection database is established by continuously collecting audio and video data of operators' work activities in real time during on-site operations at nuclear power plants, and manually determining whether the work activities are in line with expectations.

[0036] The dynamic collection database can be classified into a correct activity library and an incorrect activity library through identification. The information in the pre-training database can be classified into a standard activity library through identification. In this way, by establishing a correct activity library and an incorrect activity library during use, the large model has the ability to accurately judge the correctness of the activity and give guidance and suggestions.

[0037] The dynamic collection database can be called a dynamic update database, a dynamic activity library, or a real-time collection of audio and video databases of operation activities. The information in the dynamic collection database can be collectively referred to as the actual on-site operation process information of the nuclear power plant. The actual on-site operation process information of the nuclear power plant includes but is not limited to the action information, sound information and equipment information of the operators in the actual on-site operation process of the nuclear power plant, as well as the correct and incorrect label data classified by the evaluators.

[0038] S80: Input the information in the dynamically collected database into the first multimodal large model for continuous adversarial learning training to obtain a second multimodal large model.

[0039] In an embodiment of the present application, by acquiring information in a dynamic collection database in real time, and then using the information in the dynamic collection database to perform continuous adversarial learning training on the first multimodal large model, the stability of the large model is further improved, and accurate judgment of correct and incorrect operator activities is achieved.

[0040] The embodiment of the present application provides a training method for a multimodal large model for on-site digital work supervision and guidance in a nuclear power plant. By using the multimodal large model technology, a pre-training database is established for model pre-training, and a dynamic collection database is established for stabilization training of the model. This enables the large model to have the function of real-time recognition, understanding and analysis of the activities of operators through audio and video information, and ultimately realizes the function of real-time supervision and guidance of the activities of operators.

[0041] At least one embodiment of the present application further provides a cloud device, which is used to perform multimodal large model training and calculation. The cloud device 2 includes an acquisition module, an initialization training module, a calculation module and an output module.

[0042] The acquisition module is used to acquire information in the pre-training database. The information in the pre-training database includes a personnel information library 5, an equipment information library 6, a procedure information library 7 containing procedure activity information and procedure content, and a standard activity library 8 containing standard operation activity information.

[0043] The initialization training module is used to input the information in the pre-training database into the multimodal large model for initialization training to obtain a first multimodal large model capable of recognizing standard operator activities and equipment status.

[0044] The operation module is used to output audio and video guidance information through the first multimodal large model operation according to the real-time audio and video data transmitted by the terminal device 3. The real-time audio and video data transmitted by the terminal device 3 includes audio and video information of the face of the operator 1, audio and video information of the action of the operator 1, audio and video information of the state of the operating device 4, operation task information and use procedure information. The audio and video guidance information includes the determination information of whether the activities of the operator are in compliance with the specifications and procedures, the determination information of whether the equipment status is in compliance with the expectations, and the guidance information for the activities of the operator.

[0045] The output module is used to transmit the audio and video guidance information to the terminal device 3.

[0046] For example, the operation module can read the audio and video information of the face of the operator 1 transmitted by the terminal device 3, use the first multimodal large model or the second multimodal large model to identify the identity information of the personnel, and check whether the operator meets the job qualification requirements, so as to obtain the judgment information whether the activities of the operator comply with the specifications and procedures.

[0047] The operation module can read the audio and video information of the status of the operating device 4 transmitted by the terminal device 3, and use the first multimodal large model or the second multimodal large model to identify the device and its status to obtain judgment information whether the device status meets expectations.

[0048] The operation module can analyze the deviation of the activity of the operator 1 by reading the audio and video information of the operator 1's face, the audio and video information of the operator 1's movements, and the audio and video information of the state of the operating device 4 transmitted by the terminal device 3 in real time. If there is no deviation, the output module outputs an instruction to continue the operation activity to the terminal device 3; if there is a deviation, the output module outputs a warning and guidance information to the terminal device 3.

[0049] The operation module can generate a standard operation activity supervision and guidance program by reading the operation task information and usage procedure information transmitted by the terminal device 3 to obtain guidance information for the activities of the operators.

[0050] In at least one embodiment of the present application, the cloud device 2 also includes a continuous adversarial learning training module. The acquisition module is also used to acquire information in the dynamic collection database in real time. The information in the dynamic collection database includes a work activity library and an evaluation result library. The work activity library contains audio and video information of the work activities collected during the application process. The evaluation result library contains whether the work activities meet the requirements of the specifications and the expectations of the procedures. The continuous adversarial learning training module is used to input the information in the dynamic collection database into the first multimodal large model for continuous adversarial learning training to obtain a second multimodal large model. The operation module is also used to output audio and video guidance information through the second multimodal large model operation based on the pre-processed real-time audio and video data transmitted by the terminal device 3, and transmit the audio and video guidance information to the terminal device 3.

[0051] It should be noted that the audio and video information of the operator 1's face and the audio and video information of the operator's 1 actions may also be referred to as activity information of the operator 1. The audio and video information of the state of the operating device 4 may also be referred to as state information of the operating device 4.

[0052] At least one embodiment of the present application also provides a terminal device, which is connected to the cloud device 2 in communication and is used to perform data collection, preprocessing, data transmission and conversion. The terminal device 3 includes a fixed or mobile personnel activity monitoring device, a wearable audio and video acquisition and receiving conversion device worn by the operator, and an edge computing device. The personnel activity monitoring device and the audio and video acquisition and receiving conversion device are used by the operator 1 to collect the information data of the operator 1, the audio and video data of the operator 1's activities, the status data of the operating device 4, and the audio and video data of the task scene in real time during the execution of the operation task. The audio and video acquisition and receiving conversion device is also used to receive the result data of supervision and guidance transmitted by the cloud device 2, convert the result data of supervision and guidance transmitted by the cloud device 2 into audio and video information and transmit it to the operator 1. The edge computing device is connected to the personnel activity monitoring device and the audio and video acquisition and receiving conversion device respectively, and the edge computing device is used to preprocess the collected audio and video data, transmit the preprocessed audio and video data to the cloud device 2, transmit the task description and selected procedure data to the cloud device 2, and output the description of the operation activity, deviation and corrected audio and video data to the operator 1.

[0053] It should be noted that the preprocessing includes but is not limited to formatting or feature extraction. The audio and video data collected by the personnel activity monitoring device and the audio and video collection and receiving conversion device may contain overlapping or complementary information.

[0054] At least one embodiment of the present application provides a nuclear power plant on-site digital work supervision and guidance system, referring to Figure 2 and Figure 3 , the nuclear power plant on-site digital work supervision and guidance system includes a cloud device 2 and a terminal device 3 that are communicatively connected. The terminal device 3 provided in the embodiment of the present application is used to collect information data of the operator 1, audio and video data of the operator 1 activities, status data of the operating device 4, and audio and video data of the task scene in real time and pre-process them, transmit the pre-processed audio and video data to the cloud device 2 provided in the embodiment of the present application, and receive the supervision and guidance results transmitted by the cloud device 2. The cloud device 2 provided in the embodiment of the present application is used to process multi-modal large model training and calculation, and calculates the supervision and guidance results based on the pre-processed audio and video data transmitted by the terminal device 3 and transmits them to the terminal device 3.

[0055] It should be noted that the cloud device 2 may be a cloud server system, and the terminal device 3 may be a terminal device system. The operator 1 may also be referred to as a field operator.

[0056] At least one embodiment of the present application also provides a method for supervising and guiding digital work on-site in a nuclear power plant. The method for supervising and guiding digital work on-site in a nuclear power plant includes the following steps.

[0057] S100: When the operator 1 is performing the task, the personnel activity monitoring equipment and the audio and video acquisition and receiving conversion equipment in the terminal device 3 collect the information data of the operator 1, the audio and video data of the operator's 1 activities, the status data of the operating device 4 and the audio and video data of the task scene in real time; the edge computing device in the terminal device 3 pre-processes the collected audio and video data, and then transmits the pre-processed data to the cloud device 2.

[0058] S200: The cloud device 2 calls the multimodal large model to analyze and calculate the input audio and video data, obtains the supervision and guidance result data, and outputs the supervision and guidance result data to the terminal device 4.

[0059] It should be noted that, further, the terminal device 4 can convert the result data of supervision and guidance into audio and video information and transmit it to the operator 1 to realize the supervision and guidance function.

[0060] According to the technical solution provided in the embodiment of the present application, real-time supervision and guidance of the work activities of on-site operators in nuclear power plants are realized based on audio and video data acquisition technology, multimodal large model technology, edge computing technology and cloud computing technology; the multimodal large model is trained using audio and video data of operator activities, equipment status and task scenarios so that it can correctly understand the interactive activities between operators and equipment in task scenarios. The embodiment of the present application realizes real-time judgment of whether the operator's actions, sounds and the status of the operation object meet expectations during the operator's task execution. Relying on the model's understanding of standard operation, it gives reminders and guides the operator to perform correct operations when the operator's activities are wrong, thereby significantly reducing the operator's mistakes and avoiding huge economic losses.

[0061] In at least one embodiment of the present application, reference Figure 4 , before the above S100, it also includes S1 to S3. S4 is a specific implementation of the above S100. S5 to S7 are a specific implementation of the above S200. S1 to S3 are the preparation stage. S4 to S7 are the operation stage.

[0062] S1: The terminal device 3 identifies the operator 1, obtains and verifies the identity information from the personnel information database 5, and collects the information data of the operator 1.

[0063] S2: The terminal device 3 identifies the operating device 4 and obtains the status data of the operating device 4 from the device information library 6.

[0064] S3: The terminal device 3 selects a procedure, and the cloud device 2 retrieves the corresponding information in the procedure activity library 7.

[0065] S4: After the operation starts, for each operation step of the operator 1, the terminal device 3 collects the information data of the operator 1, the audio and video data of the operator 1 activities, the status data of the operating device 4 and the audio and video data of the task scene in real time and performs preprocessing.

[0066] S5: Cloud device 2 receives the pre-processed data transmitted by terminal device 3.

[0067] The preprocessed data may be audio and video data of the operation activity in the current step.

[0068] S6: The cloud device 2 calls the first multimodal large model or the second multimodal large model to compare the preprocessed data with the standard activity library 8 and the procedure activity library 7 to obtain the result data of supervision and guidance.

[0069] S7: The cloud device 2 sends the result data of the supervision and guidance to the terminal device 3, and transmits the warning guidance information to the operator 1 in real time. According to the warning guidance information, the operator 1 continues the operation or corrects the activity according to the warning guidance information.

[0070] If the activities of operator 1 are in line with expectations, an instruction to continue the work is sent; if the activities of operator 1 are not in line with expectations, such as not meeting the requirements of the code of conduct or not meeting the requirements of the regulations, a warning guidance message is given.

[0071] The embodiment of the present application develops an on-site digital work supervision and guidance method based on a multimodal large model, which realizes intelligent supervision and guidance of the task activities of nuclear power plant operators, thereby helping to improve the standardization of operators' behavior and reduce human errors in the operation process. Compared with previous methods, the method based on the multimodal large model can identify personnel behavior activities and monitor equipment status, thereby realizing real-time understanding and correction of operators' work activities.

[0072] In at least one embodiment of the present application, S71 to S73 are a specific implementation of the above-mentioned S7.

[0073] S71: Operator 1 identifies operating device 4, terminal device 3 collects audio and video data and uploads it to cloud device 2 after pre-processing, cloud device 2 issues analysis results after analysis, and terminal device 3 transmits the analysis results to operator 1, that is, whether the operating device 4 meets the work requirements.

[0074] S72: Operator 1 orally states the operation target, terminal device 3 collects audio and video data and uploads it to cloud device 2 after pre-processing, cloud device 2 issues analysis results after analysis, and terminal device 3 transmits the analysis results to operator 1, that is, whether the operation that operator 1 is about to complete meets the requirements.

[0075] S73: Operator 1 completes the current operation step, terminal device 3 collects audio and video data and uploads it to cloud device 2 after pre-processing, cloud device 2 issues analysis results after analysis, and terminal device 3 transmits the analysis results to operator 1, that is, whether the state reached by operating device 1 meets the requirements and whether operator 1 can proceed to the next step.

[0076] In at least one embodiment of the present application, the method for supervising and guiding on-site digital work of a nuclear power plant further includes the following steps S8 to S11. S8 to S11 are feedback stages.

[0077] S8: The terminal device 3 uploads the worker activity data to the work activity database 9 of the cloud device 2.

[0078] S9: Cloud device 2 uses a multimodal large model to evaluate whether the activities of operators comply with regulatory requirements and procedural expectations.

[0079] S10: The reviewer reviews and corrects the evaluation results of the cloud device 2 and uploads the evaluation results to the evaluation result library 10.

[0080] S11: The cloud device 2 regularly uses the data in the operation activity database 9 and the evaluation result database 10 to perform adversarial learning training.

[0081] According to the technical solution provided by the embodiment of the present application, by collecting audio and video information of personnel activities, equipment changes, task scene changes, etc. during the operation process and preprocessing the audio and video data (formatting, feature extraction, etc.), and receiving the audio and video guidance information transmitted by the cloud device; the cloud device receives the preprocessed real-time audio and video data transmitted by the terminal device, and outputs the judgment information of whether the activities of the operators meet the specifications and procedures, the judgment information of whether the equipment status meets the expectations, and the guidance information for the activities of the operators through the large model calculation; the above information is then transmitted from the cloud device to the terminal device, and the operating activities of the operators are prompted and guided by audio and video. In addition, the evaluation results are formed by manually reviewing whether the operating activities of the operators meet the expectations, and the activity data and the corresponding evaluation data are used for adversarial learning training to strengthen the stability of the large model. The embodiment of the present application proposes to use multimodal large model technology to realize the real-time understanding and analysis of the activities of the operators on the site of the nuclear power plant, and make real-time feedback based on the analysis results, so as to realize the function of supervising and guiding the behavior norms of the operators, the implementation of the operating procedures, the operation process and the results.

[0082] It should be noted that the combination of the various technical features in the embodiments of the present application is not limited to the combination described in the embodiments of the present application or the combination described in the specific embodiments, and all technical features described in the present application can be freely combined or combined in any way unless there is a contradiction between them.

[0083] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the term "comprising" only indicates that the steps and elements that have been clearly identified are included, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0084] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0085] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A training method for a multimodal large model for on-site digital work supervision and guidance in nuclear power plants, characterized in that: include: Acquire information in a pre-training database, the information in the pre-training database includes a personnel information database, an equipment information database, a procedure information database including procedure activity information and procedure content, and a standard activity database including standard operation activity information; The information in the pre-training database is input into the multimodal big model for initial training to obtain a first multimodal big model capable of recognizing standard operator activities and equipment status.

2. The training method according to claim 1, characterized in that: After inputting the information in the pre-training database into the multimodal big model for initialization training to obtain a first multimodal big model capable of recognizing standard operator activities and equipment status, the method further includes: Acquire information in the dynamic collection database from the terminal device in real time. The information in the dynamic collection database includes the operation activity database and the evaluation result database. The operation activity database contains the audio and video information of the operation activities collected during the application process, and the evaluation result database contains the evaluation data information on whether the operation activities meet the requirements of the specifications and the requirements of the regulations; The information in the dynamically collected database is input into the first multimodal large model for continuous adversarial learning training to obtain the second multimodal large model.

3. A cloud device, characterized in that: It includes acquisition module, initialization training module, operation module and output module. The acquisition module is used to acquire information in the pre-training database, the information in the pre-training database includes a personnel information database, an equipment information database, a procedure information database including procedure activity information and procedure content, and a standard activity database including standard operation activity information; The initialization training module is used to input the information in the pre-training database into the multimodal large model for initialization training to obtain a first multimodal large model capable of recognizing standard operator activities and equipment status; The operation module is used to output audio and video guidance information through the first multimodal large model operation according to the real-time audio and video data transmitted by the terminal device, the real-time audio and video data transmitted by the terminal device includes audio and video information of the face of the operator, audio and video information of the operator's actions, audio and video information of the operating equipment status, operation task information and use procedure information, and the audio and video guidance information includes determination information of whether the activities of the operator are in compliance with the specifications and procedures, determination information of whether the equipment status is in compliance with expectations, and guidance information for the activities of the operator; The output module is used to transmit the audio and video guidance information to the terminal device.

4. The cloud device according to claim 3, characterized in that: The cloud device also includes a continuous adversarial learning training module. The acquisition module is also used to acquire information in the dynamic collection database in real time. The information in the dynamic collection database includes an operation activity database and an evaluation result database. The operation activity database contains audio and video information of the operation activities collected during the application process, and the evaluation result database contains evaluation data information on whether the operation activities meet the requirements of the specifications and the expectations of the regulations. The continuous adversarial learning training module is used to input the information in the dynamic collection database into the first multimodal large model for continuous adversarial learning training to obtain the second multimodal large model; The operation module is also used to output audio and video guidance information through the second multimodal large model operation based on the pre-processed real-time audio and video data transmitted by the terminal device, and transmit the audio and video guidance information to the terminal device.

5. A terminal device, characterized in that: Including fixed or mobile personnel activity monitoring equipment, wearable audio and video acquisition and receiving conversion equipment worn by operators, and edge computing equipment. Personnel activity monitoring equipment and audio and video acquisition and receiving conversion equipment are used to collect information data of operators, audio and video data of operators' activities, status data of operating equipment and audio and video data of task scenes in real time when operators are performing their tasks; The audio and video acquisition and receiving conversion device is also used to receive the result data of supervision and guidance transmitted by the cloud device, convert the result data of supervision and guidance transmitted by the cloud device into audio and video information and transmit it to the operating personnel; The edge computing devices are connected to the personnel activity monitoring devices and the audio and video acquisition and receiving conversion devices respectively. The edge computing devices are used to preprocess the collected audio and video data, transmit the preprocessed audio and video data to the cloud devices, transmit the task description and selected procedure data to the cloud devices, and output the description of the work activities, deviations and corrected audio and video data to the operators.

6. A digital work supervision and guidance system for a nuclear power plant, characterized in that: Including cloud devices and end devices connected by communication, The terminal device described in claim 5 is used to collect information data of operators, audio and video data of operators' activities, status data of operating equipment, and audio and video data of task scenes in real time and pre-process them, transmit the pre-processed audio and video data to the cloud device provided in the embodiment of the present application, and receive the results of supervision and guidance transmitted by the cloud device; The cloud device described in claim 3 or 4 is used to process multimodal large model training and calculation, and transmits the results of supervision and guidance of the pre-processed audio and video data transmitted by the end device to the end device.

7. A method for supervising and guiding digital work on site of a nuclear power plant, characterized in that: include: When operators are performing tasks, the personnel activity monitoring equipment and audio and video acquisition and receiving conversion equipment in the terminal device collect the information data of the operators, the audio and video data of the operators' activities, the status data of the operating equipment, and the audio and video data of the task scene in real time. The edge computing device in the terminal device pre-processes the collected audio and video data, and then transmits the pre-processed data to the cloud device. The cloud device calls the multimodal large model to analyze and calculate the input audio and video data, obtains the supervision and guidance result data, and outputs the supervision and guidance result data to the end device.

8. A method for supervising and guiding digital work on site of a nuclear power plant according to claim 7, characterized in that: The personnel activity monitoring equipment and the audio and video acquisition and receiving conversion equipment in the terminal equipment collect the information data of the operating personnel, the audio and video data of the operating personnel's activities, the status data of the operating equipment and the audio and video data of the task scene in real time, and also include: The terminal device identifies the operator, obtains and verifies the identity information from the personnel information database, and collects the information data of the operator; The terminal device identifies the operating device and obtains the status data of the operating device from the device information library; The end device selects the procedure, and the cloud device retrieves the corresponding information from the procedure activity library; The cloud device calls the multimodal large model to analyze and calculate the input audio and video data, obtains the supervision and guidance result data, and outputs the supervision and guidance result data to the end device, including: The cloud device receives the pre-processed data transmitted by the terminal device; The cloud device calls the first multimodal large model or the second multimodal large model to compare the preprocessed data with the standard activity library and the procedure activity library to obtain result data of supervision and guidance; The cloud device sends the result data of supervision and guidance to the terminal device, and transmits warning and guidance information to the operator in real time. Based on the warning and guidance information, the operator continues the operation or corrects the activity according to the warning and guidance information.

9. A method for supervising and guiding digital work on site of a nuclear power plant according to claim 8, characterized in that: The cloud device sends the result data of supervision and guidance to the terminal device, and transmits warning guidance information to the operator in real time. According to the warning guidance information, the operator continues the operation or corrects the activity according to the warning guidance information, including: The operator identifies the operating equipment, the terminal device collects audio and video data and uploads it to the cloud device after pre-processing. The cloud device issues the analysis results after analysis, and the terminal device transmits the analysis results to the operator, that is, whether the operating equipment meets the work requirements; The operator verbally states the operation target, the terminal device collects audio and video data and uploads it to the cloud device after pre-processing. The cloud device issues the analysis result after analysis, and the terminal device transmits the analysis result to the operator, that is, whether the operation to be completed by the operator meets the requirements; The operator completes the current operation steps, the terminal device collects audio and video data and uploads it to the cloud device after preprocessing. The cloud device issues the analysis results after analysis, and the terminal device transmits the analysis results to the operator, that is, whether the state reached by the operating device meets the requirements and whether the operator can proceed to the next step.

10. A method for supervising and guiding digital work on site of a nuclear power plant according to claim 7, characterized in that: Also includes: The terminal device uploads the activity data of the operator to the operation activity database of the cloud device; Cloud devices use multimodal large models to evaluate whether the activities of operators meet the requirements of regulations and expectations of procedures; After reviewing and calibrating the cloud device evaluation results, the auditor uploads the evaluation results to the evaluation result database; The cloud device regularly uses the data in the job activity database and evaluation result database for adversarial learning training.

Images