Method and device for predicting remaining completion time of multi-modal production line

Abstract

The invention discloses a method and device for predicting the remaining completion time of a multi-modal production line, and the method comprises the steps: collecting production progress data, which comprises the internal state feature data of the production line and the state feature data between cross-production lines; wherein the state characteristic data across the production lines is used for representing the dependency and competition relationship between the production lines; preprocessing the production progress data, and generating feature data through feature engineering; training an MMoE model by using the feature data, wherein the number of experts of the model is equal to the number of production lines; the trained MMoE model is used for predicting the remaining completion time of the whole process. According to the method, the cooperative work scene of all workshops or production lines of the factory is considered, the factory is used as a whole for fusion prediction, the predicted remaining completion time of the factory is obtained, and the prediction accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for predicting the remaining completion time of a multimodal production line. Background Art

[0002] Current production monitoring systems generally use programmable logic controllers (PLCs) to collect data in real time and rely on host computers for data visualization. However, with the upgrading of intelligent manufacturing requirements, traditional monitoring models can no longer meet the management needs of complex production scenarios and have the following problems:

[0003] Existing monitoring systems lack the ability to collaboratively predict the progress of processes across multiple workshops. Existing methods can predict the progress or remaining time of a single production line or workshop, but modern factories often consist of multiple workshops or production lines working together. Different production lines compete for resources, rely on each other, and interact with each other, all of which can affect the estimated completion time of the entire factory. For example, the busy or idle status of the shared robotic arm, the current task type, whether there are production lines waiting in line to use the robotic arm, and possible failures will directly affect the output speed of the chip and lens production lines, as well as the connection between the material flow and the warehouse. Existing methods cannot solve these problems.

[0004] The interaction form is rigid, relying on fixed patterns or keyword triggers, and the system cannot interact with users using natural language. The display form is single, limited to numerical display, and cannot achieve multimodal feedback (such as voice broadcast, digital human display, etc.);

[0005] In some factories that have introduced large language models (LLMs), the semantic understanding and conversation capabilities in industrial applications have not yet been optimized for specific scenarios, and there is a lack of fine-tuning data and prompt word design for production progress descriptions. Summary of the Invention

[0006] The purpose of the present invention is to provide a method and device for predicting the remaining completion time of a multimodal production line in order to address the deficiencies of the existing technology.

[0007] The objective of the present invention is achieved through the following technical solution: A method for predicting the remaining completion time of a multimodal production line, comprising:

[0008] Collect production progress data, including status feature data within the production line and status feature data between production lines. The status feature data between production lines is used to characterize the dependency and competition relationships between production lines.

[0009] Preprocess production progress data and generate feature data through feature engineering;

[0010] Use feature data to train the MMoE model, where the number of experts in the model is equal to the number of production lines;

[0011] The trained MMoE model is used to predict the remaining completion time of the entire process.

[0012] Furthermore, the feature engineering includes Z-score standardization, continuous value discretization, discrete feature one-hot encoding, sliding window mean, first-order difference, historical progress change rate, and construction of cross features.

[0013] Furthermore, it also includes:

[0014] Clean, organize, and compile historical production progress data; use Matplotlib, Seaborn, and ptitprince libraries to generate various charts for easy analysis and optimization.

[0015] Furthermore, it also includes natural language interaction based on a large language model; specifically:

[0016] Collect historical production progress data and question-and-answer corpus from the production process, including text and audio; construct the text portion into a corresponding question-and-answer dataset;

[0017] Fine-tune a large language model using a question-answering dataset;

[0018] Natural language interaction includes:

[0019] Obtain real-time production progress data and use the trained MMoE model to predict the remaining completion time of the entire process;

[0020] Prompt words are generated based on the question text entered by the user or the question text converted from voice, real-time production progress data, and the remaining completion time of the entire process. These are input into the fine-tuned large language model and the response text is output. The user input format is voice or text. If the user input is voice, the voice is converted to text.

[0021] Furthermore, it also includes the progress report of the digital human, specifically:

[0022] Convert the response text output by the large language model into a speech frequency file wave and obtain the audio duration t;

[0023] Call the MuseV model to generate a character dynamic video v with a duration of t based on the character image used for the digital human broadcast;

[0024] Call the MuseTalk model, load the video v and audio wave, and obtain the audio-driven digital human voice progress broadcast video video.

[0025] Furthermore, the audio part of the question-answering corpus is used as a fine-tuning training dataset;

[0026] Use the VQGAN weights of the Fish Speech 1.5 model to extract semantic tokens from the fine-tuning training dataset;

[0027] Keep the low-rank matrix r, freeze the remaining weights of the model, and use LORA for fine-tuning;

[0028] The fine-tuned Fish Speech 1.5 model is used to convert the response text output by the large language model into a speech frequency file.

[0029] The present invention also provides a device for predicting the remaining completion time of a multimodal production line, comprising:

[0030] The data acquisition module is used to collect production progress data, including status characteristic data within the production line and status characteristic data between production lines. The status characteristic data between production lines is used to characterize the dependency and competition relationship between production lines.

[0031] Feature generation module, used to preprocess production progress data and generate feature data through feature engineering;

[0032] The model training module is used to train the MMoE model using feature data, where the number of experts in the model is equal to the number of production lines;

[0033] The prediction module predicts the remaining completion time of the entire process through the trained MMoE model.

[0034] The present invention also provides an electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned method for predicting the remaining completion time of a multimodal production line.

[0035] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for predicting the remaining completion time of a multimodal production line.

[0036] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned method for predicting the remaining completion time of a multi-modal production line.

[0037] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0038] 1. Monitor the production status of each workshop in real time through intranet services and dynamically adjust the monitoring configuration.

[0039] 2. Conduct collaborative data analysis on the production progress of multiple workshops, visualize the data and display it on the system front end.

[0040] 3. Considering the collaborative work scenarios of all workshops or production lines in the factory, the factory is integrated and predicted as a whole to obtain the estimated remaining completion time of the factory, improve the accuracy of the prediction, and display it on the front-end interface of the system.

[0041] 4. Train the LLM model for the factory production context so that the system can understand and use natural language to interact with users.

[0042] 5. Train a TTS model for factory production contexts so that the system can use natural and fluent voice functions to report progress.

[0043] 6. Use customizable AI digital humans to broadcast factory progress and display it on a large screen, avoiding missing key information due to a single progress display method.

[0044] 7. This method is compatible with any factory that uses PLC as a controller. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 A schematic flow chart of a method for predicting the remaining completion time of a multimodal production line provided in Example 1 of the present invention;

[0047] Figure 2 A schematic diagram of a method flow diagram when a user asks a question is provided for an embodiment of the present invention;

[0048] Figure 3 This is a product application example diagram;

[0049] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be described in detail below with reference to the accompanying drawings. Unless there is any conflict, the features of the following embodiments and implementations may be combined with each other.

[0051] A method for predicting the remaining completion time of a multimodal production line of the present invention comprises the following steps:

[0052] (1) Collecting production progress data, including status feature data within the production line and status feature data across production lines; the status feature data across production lines is used to characterize the dependency and competition relationships between production lines;

[0053] (2) preprocessing the production progress data and generating first feature data through feature engineering;

[0054] (3) using the first feature data to train the MMoE model, where the number of experts in the model is equal to the number of production lines;

[0055] (4) The trained MMoE model is used to predict the remaining completion time of the entire process.

[0056] In one embodiment, the feature engineering includes Z-score normalization, continuous value discretization, discrete feature one-hot encoding, sliding window mean, first-order difference, historical progress change rate, and construction of cross features.

[0057] In one embodiment, it further includes:

[0058] Clean, organize, and compile historical production progress data; use Matplotlib, Seaborn, and ptitprince libraries to generate various charts for easy analysis and optimization.

[0059] In one embodiment, natural language interaction is also performed based on a large language model; specifically:

[0060] Collect historical production progress data and question-and-answer corpus from the production process, including text and audio; construct the text portion into a corresponding question-and-answer dataset;

[0061] Fine-tune a large language model using a question-answering dataset;

[0062] Natural language interaction includes:

[0063] If the user input is voice, convert the voice into text;

[0064] Obtain real-time production progress data and use the trained MMoE model to predict the remaining completion time of the entire process;

[0065] Prompt words are generated based on the question text entered by the user or the question text converted from voice, real-time production progress data, and the remaining completion time of the entire process. These are input into the fine-tuned large language model and the response text is output.

[0066] In one embodiment, the digital human progress report is also included, specifically:

[0067] Convert the response text output by the large language model into a speech frequency file wave and obtain the audio duration t;

[0068] Call the MuseV model to generate a character dynamic video v with a duration of t based on the character image used for the digital human broadcast;

[0069] Call the MuseTalk model, load the video v and audio wave, and obtain the audio-driven digital human voice progress broadcast video video.

[0070] In one embodiment, the audio portion of the question-answering corpus is used as a fine-tuning training dataset;

[0071] Use the VQGAN weights of the Fish Speech 1.5 model to extract semantic tokens from the fine-tuning training dataset;

[0072] Keep the low-rank matrix r, freeze the remaining weights of the model, and use LORA for fine-tuning;

[0073] The fine-tuned Fish Speech 1.5 model is used to convert the response text output by the large language model into a speech frequency file.

[0074] Example 1:

[0075] A method for predicting the remaining completion time of a multi-modal production line in this embodiment is described in detail in the following sections. Figure 1 , including the following steps:

[0076] Step 1: Production line maintenance and data collection.

[0077] 1. The user dynamically enters the number of processes and their names on the PLC touch screen. For example, enter the number "3," followed by "Chip Workshop," "Lens Workshop," and "Warehouse Workshop." During this process, the touch screen dynamically generates a text input box based on the number of processes entered by the user.

[0078] 2. Status characteristic data within each production line. The progress data of each process is stored in the PLC internal data block (DB). The PLC program updates the status (percentage, stage status, single-step time) in real time according to the actual production situation and writes the information to the PLC through triggering events. For example:

[0079] (1) Accurate timestamp records.

[0080] (2) The time taken for each step and the time taken for the current active step.

[0081] (3) Operation status of key equipment / production lines (operating / idle / faulty).

[0082] (4) Current work order / product information, etc.

[0083] 3. Status collection across production lines. For example, a 6-axis robotic arm is responsible for grabbing chip modules from a certain output point in the chip workshop and placing them in a shared intermediate cache / loading station. At the same time, it is also responsible for grabbing lens components from the output point of the lens workshop and placing them in the same cache / loading station. The AGV or conveyor belt in the warehouse will eventually pick up the parts from this station. In the above process, there are dependencies and competition relationships between the shared robotic arms, shared conveyor belts, shared intermediate warehouses, shared AGV fleets, etc. between the production lines, and their feature data needs to be collected. The following is a detailed explanation using a shared robotic arm as an example:

[0084] (1) DB100.DBX0.0 (BOOL): Arm_Busy (Whether the robotic arm is executing a task)

[0085] (2)DB100.DBW2(INT):Arm_Status(0:Idle,1:MovingToChipPickup,2:PickingChip,3:MovingToBufferFromChip,4:PlacingChip,5:MovingToLensPickup ,6:PickingLens,7:MovingToBufferFromLens,8:PlacingLens,9:MovingEmpty,10:Error,11:WaitingForBufferSpace,12:WaitingForPart...) - Use integer enumeration status

[0086] (3) DB100.DBW4(INT): Arm_Current_Task_Source(0:None,1:ChipLine,2:LensLine....)-Indicates the current service object

[0087] (4) DB100.DBX0.1 (BOOL): ChipLine_Requesting_Arm (Whether the chip line has material waiting for the robot arm to process)

[0088] (5) DB100.DBX0.2 (BOOL): LensLine_Requesting_Arm (Whether there is material in the lens line waiting for the robot arm to process)

[0089] (6) DB100.DBX0.3 (BOOL): Buffer_Available_For_Arm (whether there is space for the shared buffer station)

[0090] (7)DB100.DBW6(INT):Arm_Error_Code(0:NoError, other values indicate specific faults)

[0091] (8) DB100.DBD8 (DINT): Arm_Cycle_Time_Last_ms (last complete task cycle time, milliseconds)

[0092] Step 2: Server-side data polling and file management.

[0093] 1. Build an intranet python backend service to communicate with various industrial production line PLCs in real time through the S7 protocol.

[0094] 2. The backend service polls the progress data of each process in the PLC DB and writes the data to the file system. Each record includes a timestamp, process name, current progress, and status identifier.

[0095] 3. The file management module supports file rotation and archiving to ensure the integrity of historical progress data and facilitate subsequent statistical and predictive analysis.

[0096] Step 3: Historical data analysis and progress forecast.

[0097] Data Analysis and Visualization:

[0098] 1. Use the Python data analysis library Pandas to clean, organize and compile historical progress data.

[0099] 2. Use Matplotlib, Seaborn, and ptitprince libraries to generate various charts:

[0100] (1) Scatter plot: shows the time consumption details of each process;

[0101] (2) Violin plot: shows progress distribution and fluctuation;

[0102] (3) Box plot: shows the time distribution of each process.

[0103] 3. Visualize the charts, update them in real time and display them on the front-end interface of the system, so as to trace and display the historical details of the entire process of multiple workshops in the factory, and facilitate analysis and optimization.

[0104] Progress prediction algorithm:

[0105] 1. Data preprocessing: remove outliers and fill in null values.

[0106] 2. Feature Engineering: Z-score normalization, continuous value discretization, discrete feature one-hot encoding, sliding window mean, first-order difference, historical progress change rate, construction of cross-features, etc. The following uses a shared robotic arm as an example to illustrate the feature engineering method:

[0107] (1)Arm_Busy: directly used as a 0 / 1 feature.

[0108] (2) Arm_Status: One-hot encoding is performed and converted into multiple binary features, such as Arm_Status_Is_Idle, Arm_Status_Is_MovingChip, Arm_Status_Is_MovingLens, Arm_Status_Is_Error, etc. This allows the model to distinguish the impact of different states.

[0109] (3) Arm_Current_Task_Source: One-Hot encoding (Arm_Servicing_Chip, Arm_Servicing_Lens)

[0110] (4) Resource_Contention: ChipLine_Requesting_Arm AND LensLine_Requesting_Arm. Create a new Boolean feature to represent resource contention.

[0111] (5) Waiting_States: Combining Arm_Status (such as 11, 12) and *_Requesting_Arm signals, the waiting time can be calculated (this needs to be combined with historical data). For example, if ChipLine_Requesting_Arm is True and Arm_Status is not the state of the service chip (1-4), the chip line is waiting. Record the duration of this state.

[0112] (6) Arm_Error_Code: One-hot encoding is performed. The Arm_In_Error (BOOL) feature can be generated by Arm_Error_Code!=0.

[0113] (7) Arm_Cycle_Time_Last_ms: Converted into a Z-score as a numerical feature. Calculate its moving average and use it as a feature.

[0114] 3. Based on the above characteristic data, train the MMoE model. Set the number of experts in the model to the number of production lines. For the entire factory, predict the remaining time for the entire process and obtain the predicted result, time. With the characteristic data of inter-line collaboration as input, the MMoE model will better understand the mutual impact and resource bottlenecks between production lines, thereby more accurately predicting the total remaining time of the factory, taking into account parallelization and synergy effects.

[0115] 4. The predicted remaining time is displayed in real time on the front-end interface of the system, enabling forward-looking control of the overall progress of multiple workshops in the factory.

[0116] See also Figure 2 , also includes: Step 4: LLM natural language interaction.

[0117] 1. Collect historical production progress data and the factory-site question-and-answer corpus (including audio and text). Construct the text portion into a corresponding question-and-answer dataset in the following format:

[0118] {

[0119] "System":"Please answer the questions based on the following latest progress report (chip workshop 20%, lens workshop 10%, and an estimated remaining 93)",

[0120] "User":"What is the overall progress now?",

[0121] "Factory": "The chip workshop is 20% complete, the lens workshop is 10% complete, and the overall estimated completion time is 93 minutes."

[0122] }、

[0123] {

[0124] "System":"Please answer the questions based on the following latest progress report (chip workshop 20%, lens workshop 10%, expected remaining 51)",

[0125] "User":"How is process 2 going?",

[0126] "Factory": "The chip workshop is currently 20% complete, with an estimated overall completion time of 51 minutes."

[0127] }

[0128] 2. Load the tokenizer from the DeepSeek LLM base model to obtain the vocabulary corresponding to the model.

[0129] 3. Use the above vocabulary to map the text in the dataset to input_ids and generate the attention mask matrix.

[0130] 4. Using the above JSON format data as training samples, fine-tune the base model using the LORA method. During the fine-tuning process, it is necessary to freeze most of the parameters in the base model and only retain the low-rank matrix r. After training is completed, Model 1 is obtained.

[0131] 5. Deploy the trained model on the intranet server and execute the following steps every time a user asks a question:

[0132] (1) If the user input is voice, execute this step, otherwise skip it: Use the FunASR pre-trained model to convert the user input audio into text to obtain the user question text.

[0133] (2) Extract the real-time progress information from the file system in step 2 and save it in memory.

[0134] (3) Call the progress prediction algorithm in step 3 to predict the remaining completion time of the production line and save it in memory.

[0135] (4) The user's question text, the real-time progress information in the memory, and the remaining completion time are combined into the following prompt format:

[0136] {

[0137] "System":"Please answer the questions based on the following latest progress report (real-time progress information, estimated remaining completion time of the production line)",

[0138] "User": "User question text",

[0139] "Factory":}

[0140] (5) Input the above prompt format data into the fine-tuned model for prediction, save the response text generated by the model in memory, and name the result text.

[0141] Step 5: TTS model fine-tuning and prediction.

[0142] 1. Use the factory-site speech data collected in step 1 of step 4 as the fine-tuning training dataset.

[0143] 2. Use the VQGAN weights of the Fish Speech 1.5 model to extract semantic tokens from the above dataset

[0144] 3. Freeze most of the model weights, retaining only the low-rank matrix r, and use LORA for fine-tuning to obtain Model 2.

[0145] 4. Deploy model 2 on the intranet server. Whenever a user asks about the progress, input the text output by model 1 in step 4 (5) into model 2, output the audio file, and save it in the memory. Name the result wave.

[0146] Step 6: Digital human progress report.

[0147] 1. The system administrator uploads the character image f used for the digital human broadcast in advance.

[0148] 2. Load and deploy the pre-trained MuseV model and MuseTalk model on the intranet server.

[0149] 3. Whenever a user asks a question, perform the following steps:

[0150] (1) Load the audio file wave obtained in step 4 of step 5 and obtain its audio duration t.

[0151] (2) Call the MuseV model, load the image f and generate a character dynamic video v of length t.

[0152] (3) Call the MuseTalk model, load the video v and audio wave, and obtain the digital human voice progress broadcast video video driven by audio.

[0153] (4) Real-time video rendering on the system front-end interface, see Figure 3 .

[0154] The present invention also provides a device for predicting the remaining completion time of a multimodal production line, comprising:

[0155] The data acquisition module is used to collect production progress data, including status characteristic data within the production line and status characteristic data between production lines. The status characteristic data between production lines is used to characterize the dependency and competition relationship between production lines.

[0156] Feature generation module, used to preprocess production progress data and generate feature data through feature engineering;

[0157] The model training module is used to train the MMoE model using feature data, where the number of experts in the model is equal to the number of production lines;

[0158] The prediction module predicts the remaining completion time of the entire process through the trained MMoE model.

[0159] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment. You can refer to the content of the above method embodiment and will not repeat it here.

[0160] Figure 4This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 4 The electronic device provided in this embodiment includes: a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions. When the program instructions are loaded and executed by the processor, a method for predicting the remaining completion time of a multi-modal production line of the present invention is implemented. It should be noted that, in addition to Figure 4 In addition to the memory and processor shown, the electronic device may also include other hardware according to its actual functions, which will not be described in detail.

[0161] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for predicting the remaining completion time of a multimodal production line.

[0162] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned method for predicting the remaining completion time of a multi-modal production line.

[0163] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0164] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0165] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0166] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0168] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A method for predicting the remaining completion time of a multimodal production line, characterized in that: include: Collect production progress data, including status feature data within the production line and status feature data between production lines. The status feature data between production lines is used to characterize the dependency and competition relationships between production lines. Preprocess production progress data and generate feature data through feature engineering; Use feature data to train the MMoE model, where the number of experts in the model is equal to the number of production lines; The trained MMoE model is used to predict the remaining completion time of the entire process.

2. The method according to claim 1, characterized in that The feature engineering includes Z-score standardization, continuous value discretization, discrete feature one-hot encoding, sliding window mean, first-order difference, historical progress change rate, and construction of cross features.

3. The method according to claim 1, characterized in that Also includes: Clean, organize and compile historical production progress data; Matplotlib, Seaborn, and ptitprince libraries are used to generate various charts for easy analysis and optimization.

4. The method according to claim 1, wherein It also includes natural language interaction based on a large language model; specifically: Collecting historical production progress data and question-and-answer corpus from the production process, including text and audio; Construct the text part into the corresponding question-answering dataset; Fine-tune a large language model using a question-answering dataset; Natural language interaction includes: Obtain real-time production progress data and use the trained MMoE model to predict the remaining completion time of the entire process; Prompt words are generated based on the question text entered by the user or the question text converted from voice, real-time production progress data, and the remaining completion time of the entire process. These are input into the fine-tuned large language model and the response text is output. The user input format is voice or text. If the user input is voice, the voice is converted to text.

5. The method according to claim 4, characterized in that It also includes the progress report of the digital human, specifically: Convert the response text output by the large language model into a speech frequency file wave and obtain the audio duration t; Call the MuseV model to generate a character dynamic video v with a duration of t based on the character image used for the digital human broadcast; Call the MuseTalk model, load the video v and audio wave, and obtain the audio-driven digital human voice progress broadcast video video.

6. The method according to claim 5, characterized in that Use the audio part of the question-answering corpus as a fine-tuning training dataset; Use the VQGAN weights of the Fish Speech 1.5 model to extract semantic tokens from the fine-tuning training dataset; Keep the low-rank matrix r, freeze the remaining weights of the model, and use LORA for fine-tuning; The fine-tuned Fish Speech 1.5 model is used to convert the response text output by the large language model into a speech frequency file.

7. A device for predicting the remaining completion time of a multimodal production line, characterized in that: include: The data acquisition module is used to collect production progress data, including status characteristic data within the production line and status characteristic data between production lines. The status characteristic data between production lines is used to characterize the dependency and competition relationship between production lines. Feature generation module, used to preprocess production progress data and generate feature data through feature engineering; The model training module is used to train the MMoE model using feature data, where the number of experts in the model is equal to the number of production lines; The prediction module predicts the remaining completion time of the entire process through the trained MMoE model.

8. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the remaining completion time prediction method for a multimodal production line as described in any one of claims 1-6 above.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a method for predicting the remaining completion time of a multimodal production line as described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the method for predicting the remaining completion time of a multimodal production line as described in any one of claims 1 to 6.

Images