An intelligent factory multi-modal data processing system based on edge computing

Abstract

The application discloses an intelligent factory multi-modal data processing system based on edge computing, which comprises a multi-protocol data acquisition module, a multi-modal fusion engine, an edge inference engine and a dynamic optimization module; the multi-protocol data acquisition module is used for acquiring multi-modal data of the intelligent factory; the multi-modal fusion engine is used for modal feature extraction and determination of the task priority of a to-be-decided task; the dynamic optimization module is used for dynamically adjusting the allocation strategy of NPU and GPU operation resources; and the edge inference engine is used for cooperative analysis of the modal features associated with the task to obtain a decision result of the current decision task. The multi-modal data processing system can dynamically adjust the allocation strategy of NPU and GPU operation resources according to the task priority, so that the emergency can be quickly decided and processed, and the decision efficiency of edge computing is ensured; and the task-related modal features can be associated with each other through task association, so that the comprehensiveness of decision evaluation is ensured.

Description

Technical Field

[0001] This invention relates to the field of industrial Internet of Things and edge computing technology, and in particular to a smart factory multimodal data processing system based on edge computing. Background Technology

[0002] Currently, traditional factories suffer from problems such as silos of multi-source heterogeneous data, insufficient real-time performance, and lack of cross-modal semantic correlation. The problem of multi-source heterogeneous data silos arises because factory operations generate diverse data types, such as sensor time-series data, visual images, and equipment logs, which traditional systems struggle to process uniformly. Insufficient real-time performance stems from cloud dependency; centralized cloud computing leads to high latency (>100ms), failing to meet the real-time control requirements of production lines, such as the <10ms response time required for robotic arm motion control. The lack of cross-modal semantic correlation arises from a semantic gap; multimodal data (such as vibration spectra + thermal imaging images) lacks cross-modal correlation analysis, resulting in low fault diagnosis accuracy (false alarm rate >25% with traditional methods).

[0003] In existing technologies, edge computing nodes have supported lightweight data processing (such as TensorFlow Lite model deployment), but their multimodal fusion capabilities are limited. Industrial protocol conversion technologies (such as OPC UA to MQTT) have achieved device interconnection, but have not solved the problem of data semantic alignment. Factory simulation systems based on digital twins can simulate production processes, but their dynamic data-driven capabilities are still insufficient. Summary of the Invention

[0004] The purpose of this invention is to provide a smart factory multimodal data processing system based on edge computing, which can process multimodal data in real time and improve the decision-making efficiency of edge control.

[0005] Technical solution: The edge computing-based smart factory multimodal data processing system of the present invention includes a multi-protocol data acquisition module, a multimodal fusion engine, an edge inference engine, and a dynamic optimization module;

[0006] The multi-protocol data acquisition module is used to acquire multimodal data from the smart factory in real time. The multimodal data consists of time-series data, image data, and text data.

[0007] The multimodal fusion engine is used to extract modal features from the acquired multimodal data and determine the task priority of the current task to be decided based on the extracted modal features.

[0008] The dynamic optimization module is used to dynamically adjust the allocation strategy of NPU and GPU computing resources according to task priority;

[0009] The edge inference engine is used to associate the acquired modal features with tasks, and to perform collaborative analysis on the associated modal features to obtain the decision results of the current decision task.

[0010] Furthermore, the multi-protocol data acquisition module includes a data acquisition unit, a preprocessing unit, and a data storage unit;

[0011] The data acquisition unit is used to acquire time-series data, image data, and text data, and to time-align the time-series data, image data, and text data.

[0012] The preprocessing unit is used to segment the time-aligned time series data, image data and text data according to the set time interval size, and obtain the segmented multimodal data in sequence. Each multimodal data includes the time series data, image data and text data in the corresponding time interval.

[0013] The data storage unit is used to store the preprocessed multimodal data.

[0014] Furthermore, the preprocessing unit is also used to perform anomaly screening on the time-series data, image data, and text data in each multimodal dataset, and output the anomaly-processed multimodal data. The specific steps for anomaly screening are as follows:

[0015] Step a1: Extract each multimodal data in the order of acquisition time, and determine whether there is abnormal data in each time series data, image data and text data in the multimodal data according to the anomaly type judgment rules. If there is abnormal data, proceed to step a2; if there is no abnormal data, proceed to step a5.

[0016] Step a2: Determine whether each abnormal data is recoverable. If the abnormal data is recoverable, proceed to step a3; if the abnormal data is not recoverable, proceed to step a4.

[0017] Step a3: Perform recovery processing on the various abnormal data in the time series data, image data, and text data, restore the abnormal data to normal data, and then proceed to step a5;

[0018] Step a4: Replace each unrecoverable abnormal data in the time series data, image data, and text data by replacing the current unrecoverable abnormal data with the previous normal data before proceeding to step a5.

[0019] Step a5: Output each normal multimodal data, the restored multimodal data, and / or the replaced multimodal data in the order of acquisition time.

[0020] Furthermore, in step a1, the specific steps for determining whether there is abnormal data in the multimodal data according to the anomaly type judgment rules are as follows:

[0021] Step a1.1: Set the exception type judgment rules for judging abnormal data. The exception type judgment rules include time series judgment rules for judging time series data exceptions, graphic judgment rules for judging graphic data exceptions, and text judgment rules for judging text data exceptions.

[0022] Step a1.2: Read each time series data in the multimodal data sequentially according to the time sequence, and use the time series judgment rules to judge the anomalies of the time series data. If the current time series data has partial missing, data mutation, or complete missing, it is determined that the current time series data is abnormal.

[0023] Step a1.3: Read each image data in the multimodal data sequentially according to the time sequence, and use the image judgment rules to judge the image data for anomalies. If the current image data has image deformation, blurring or missing images, it is determined that the current image data has an anomaly.

[0024] Step a1.4: Read each text data in the multimodal data sequentially according to the time sequence, and use the text judgment rules to judge the anomalies of the text data. If the current text data has syntax errors, disordered format or missing text, it is determined that the current text data is abnormal.

[0025] Furthermore, in step a2, the specific steps for determining whether each abnormal data point can be recovered are as follows:

[0026] Step a2.1: Sequentially obtain each abnormal data, and then determine the data type of each abnormal data. If the data type is time series data, proceed to step a2.2; if the data type is image data, proceed to step a2.3; if the data type is text data, proceed to step a2.4.

[0027] Step a2.2: Use the timing judgment rules to determine whether the timing data is partially missing or has a data mutation. If it is partially missing or has a data mutation, the timing data is determined to be recoverable; otherwise, the timing data is determined to be unrecoverable.

[0028] Step a2.3: Use the graphic judgment rules to determine whether the image data is distorted or blurred. If it is distorted or blurred, the image data is determined to be recoverable; otherwise, the image data is determined to be unrecoverable.

[0029] Step a2.4: Use text judgment rules to determine whether the text data has a syntax error or a disordered format. If it has a syntax error or a disordered format, the text data is determined to be recoverable; otherwise, the text data is determined to be unrecoverable.

[0030] Furthermore, in step a3, the specific steps for restoring abnormal data to normal data are as follows:

[0031] Step a3.1: Determine the recoverable anomaly types of abnormal data. The recoverable anomaly types of abnormal time series data include partial missing data and data mutation. The recoverable anomaly types of abnormal image data include image distortion and image blurring. The recoverable anomaly types of abnormal text data include syntax errors and garbled text.

[0032] Step a3.2: Select the appropriate recovery processing model according to the recoverable anomaly type of the abnormal data, and use the corresponding recovery processing model to perform recovery processing on each abnormal data to obtain abnormal recovery data;

[0033] Step a3.3: Replace the corresponding abnormal data in the time series data, image data, and text data with the abnormal recovery data, thereby restoring the abnormal data to normal data.

[0034] Furthermore, in step a3.2, the recovery processing model includes a partial missing data processing model, a data mutation processing model, an image deformation processing model, an image blurring processing model, a syntax error processing model, and a text garbled code processing model.

[0035] The partial missing data processing model is used to recover partially missing abnormal time-series data; the data mutation processing model is used to recover abnormal time-series data with data mutations; the image deformation processing model is used to recover abnormal image data with image deformation; the image blurring processing model is used to recover abnormal image data with blurred images; the syntax error processing model is used to recover abnormal text data with syntax errors; and the text garbled character processing model is used to recover abnormal text data with garbled characters.

[0036] Furthermore, the multimodal fusion engine includes a feature extraction unit and a task allocation unit;

[0037] The feature extraction unit is used to extract features from multimodal data to obtain modal features at each time step. The modal features include time-series features extracted from time-series data, image features extracted from image data, and text features extracted from text data.

[0038] The specific steps to obtain modal features are as follows:

[0039] For time series features, statistical analysis is performed on the collected data in each time series data according to the time series order, and the emergency situation is judged according to the emergency data threshold library. If it is determined that there is an emergency situation in the current collected data, the collected data at the current time is recorded as the time series feature; otherwise, it is determined that there is no emergency situation in the current collected data, and the time series feature at the current time is recorded as empty.

[0040] For image features, target objects are identified sequentially in each image data according to the time sequence. Then, the identified target objects are analyzed. Based on the emergency event database, it is determined whether there is an emergency for the target object. If it is determined that there is an emergency in the current image, the target object identified at the current moment is recorded as an image feature. Otherwise, it is determined that there is no emergency in the current image, and the image feature at the current moment is recorded as empty.

[0041] For text features, semantic recognition is performed on the text information in each text data in chronological order. Then, it is determined whether there is an emergency based on the emergency semantic event database. If it is determined that there is an emergency in the current semantic, the text information at the current moment is recorded as the text feature. Otherwise, it is determined that there is no emergency in the current semantic and the text feature at the current moment is recorded as empty.

[0042] The task allocation unit is used to identify emergency events based on the extracted modal features, and to determine the task priority of the current task to be decided based on the identified emergency events. The specific steps are as follows:

[0043] First, the modal features at each time step are obtained in chronological order, and then the temporal features, image features, and text features are extracted from the modal features at each time step.

[0044] Then, the temporal features, image features, and text features extracted at each time point are analyzed. If all of the temporal features, image features, and text features extracted at the current time point are non-empty, it is determined that a Level 1 emergency event has occurred at the current time point. If any of the temporal features, image features, and text features extracted at the current time point are empty, it is determined that a Level 2 emergency event has occurred at the current time point. If two of the temporal features, image features, and text features extracted at the current time point are empty, it is determined that a Level 3 emergency event has occurred at the current time point. If all of the temporal features, image features, and text features extracted at the current time point are empty, the modal features of the current time point are deleted.

[0045] Then, the task priority of the decision-making task corresponding to the Level 1 emergency is set as the first priority, the task priority of the decision-making task corresponding to the Level 2 emergency is set as the second priority, and the task priority of the decision-making task corresponding to the Level 3 emergency is set as the third priority.

[0046] Finally, the modal features corresponding to Level 1 emergency events are stored in the first data cache unit as the first priority task to be decided, the modal features corresponding to Level 2 emergency events are stored in the second data cache unit as the second priority task to be decided, and the modal features corresponding to Level 3 emergency events are stored in the third data cache unit as the third priority task to be decided. At the same time, the number of tasks to be decided in the first, second, and third data cache units is updated in real time.

[0047] Furthermore, the dynamic optimization module includes a strategy setting unit and a resource allocation unit;

[0048] The strategy setting unit is used to obtain the task priority of the current task to be decided, and then set the allocation strategy of NPU and GPU computing resources according to the task priority. The specific steps are as follows:

[0049] First, a first decision thread, a second decision thread, and a third decision thread are established to perform initial decision processing on each task to be decided in the first data cache unit, the second data cache unit, and the third data cache unit, respectively. Then, a fourth decision thread is established to perform comprehensive decision processing.

[0050] Then, based on the number of tasks to be decided in the first, second, and third data cache units at the current time, E1, E2, and E3, the resource allocation state is determined: when E1 > W11, E1 > W21 and E2 > W22, or E1 > W31, E2 > W32 and E3 > W33, where W11 > W21 > W31 and W22 > W32, then the resource allocation state is determined; when E1 < W12, E1 < W21 > W22, E2 > W32, the resource allocation state is determined; when E1 < W12, E2 > W22, E2 > W32, the resource allocation state is determined. When W23 and E2 < W24, or when E1 < W34, E2 < W35 and E3 < W36, where W12 > W23 > W34 and W24 > W35, it is determined that the resource internal control state is entered. Then, the resource call state and the resource internal control state are set as follows: W11 > W12, W21 > W23, W22 > W24, W31 > W34, W32 > W35, and W33 > W36. At the same time, the initial resource allocation state is set as the resource internal control state.

[0051] Finally, the allocation strategy for NPU and GPU computing resources is set as follows:

[0052] At the start, it enters the initial resource allocation state, that is, the resource internal control state;

[0053] After determining that the system has entered a resource control state, the base values ​​for NPU computing resources allocated to the first, second, third, and fourth decision threads are set as J1, J2, J3, and J4, respectively. A dynamically allocated resource of size J0 is then assigned, where J1 > J2 > J3, and J0 + J1 + J2 + J3 + J4 = N, where N is the computing resource value used by the NPU for decision-making computation. The computing resource allocation values ​​for the first, second, and third decision threads under the resource control state are then calculated as follows: , as well as ;

[0054] After determining that the resource call state has been entered, the base values ​​for NPU computing resources allocated to the first and fourth decision threads are set to D1 and D4, respectively. The base values ​​for GPU computing resources allocated to the second and third decision threads are set to D2 and D3, respectively. A dynamically allocated resource of size D5 for GPU computing resources is then set, where D1 + D4 = N, N is the computing resource value used by the NPU for decision-making, D2 > D3, and D2 + D3 + D5 = G, where G is the computing resource value used by the GPU for decision-making. The computing resource allocation values ​​for the second and third decision threads in the resource call state are then calculated as follows: as well as ;

[0055] The resource allocation unit is used to allocate NPU and GPU computing resources according to the set allocation strategy. The specific steps are as follows:

[0056] First, obtain the number of tasks to be decided in the first data cache unit, the second data cache unit, and the third data cache unit at the current moment, namely E1, E2, and E3;

[0057] Then, the current resource allocation status is determined based on the amount of tasks to be decided, E1, E2, and E3.

[0058] Finally, resource allocation is performed based on the resource allocation status: if the current time is under resource control, then according to... , , The computational resource allocation values ​​of J4 allocate the NPU's computational resources to the first decision thread, the second decision thread, the third decision thread, and the fourth decision thread, respectively. If the current time is in a resource call state, the NPU's computational resources are allocated to the first and fourth decision threads according to the computational resource allocation values ​​of D1 and D4, respectively. as well as The computing resource allocation value allocates the GPU's computing resources to the second decision thread and the third decision thread respectively.

[0059] Furthermore, the edge inference engine includes a task association unit and a collaborative analysis unit;

[0060] The task association unit is used to associate the acquired modal features with tasks, enabling the current decision task to collect all relevant modal features. The specific steps are as follows:

[0061] First, the first decision thread, the second decision thread, and the third decision thread retrieve the modal features stored in the first data cache unit, the second data cache unit, and the third data cache unit respectively. The three decision threads retrieve only one modal feature at a time according to the temporal order in the corresponding data cache unit.

[0062] Then, the first decision thread, the second decision thread, and the third decision thread synchronously perform the initial decision processing:

[0063] The first decision thread performs initial decision processing on the modal features retrieved from the first data cache unit, and determines whether the three emergency situations among the temporal features, image features, and text features of the modal features are consistent. If the emergency situations are consistent, the current modal feature is stored in the fourth data cache unit. If the emergency situations are inconsistent, the current modal feature is deleted.

[0064] The second decision thread performs initial decision processing on the modal features retrieved from the second data cache unit, and determines whether the two emergency situations in the temporal features, image features, and text features of the modal features are consistent. If the emergency situations are consistent, the current modal feature is stored in the fourth data cache unit. If the emergency situations are inconsistent, the current modal feature is deleted.

[0065] The third decision thread performs initial decision processing on the modal features retrieved from the third data cache unit, obtains an emergency situation from the temporal features, image features, and text features of the modal features, and stores the current modal features in the fourth data cache unit;

[0066] Finally, the fourth decision thread sorts the modal features in the fourth data cache unit according to time sequence, and then extracts the modal features that are sequential and have the same urgency to construct the task association dataset, thus obtaining the task association datasets belonging to different decision tasks.

[0067] The collaborative analysis unit is used to perform collaborative analysis on the modal features after task association to obtain the decision result of the current task to be decided. The specific steps are as follows:

[0068] First, read the task-related datasets in the order they were built, and obtain the emergency situations corresponding to the task-related datasets. Then, use the obtained emergency situations as the emergency situations of the current task to be decided.

[0069] Then, based on the obtained emergency situation, query the task decision table to obtain the decision result corresponding to the current emergency situation, and use the obtained decision result as the final decision result of the current task to be decided;

[0070] Finally, delete the task-related dataset corresponding to the current task to be decided.

[0071] Compared with existing technologies, the advantages of this invention are: by dynamically adjusting the allocation strategy of NPU and GPU computing resources according to task priority through the dynamic optimization module, emergency situations can be quickly decided and processed, ensuring the decision-making efficiency of edge computing; by associating tasks through the edge inference engine, task-related modal features can be correlated with each other, thereby ensuring the comprehensiveness of decision evaluation. Attached Figure Description

[0072] Figure 1 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0073] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the embodiments described.

[0074] like Figure 1 As shown, the edge computing-based smart factory multimodal data processing system disclosed in this invention includes: a multi-protocol data acquisition module, a multimodal fusion engine, an edge inference engine, and a dynamic optimization module;

[0075] The multi-protocol data acquisition module is used to acquire multimodal data from the smart factory in real time. The multimodal data consists of time-series data, image data, and text data.

[0076] The multimodal fusion engine is used to extract modal features from the acquired multimodal data and determine the task priority of the current task to be decided based on the extracted modal features.

[0077] The dynamic optimization module is used to dynamically adjust the allocation strategy of NPU and GPU computing resources according to task priority;

[0078] The edge inference engine is used to associate the acquired modal features with tasks, and to perform collaborative analysis on the associated modal features to obtain the decision results of the current decision task.

[0079] The dynamic optimization module dynamically adjusts the allocation strategy of NPU and GPU computing resources based on task priority, enabling rapid decision-making in emergency situations and ensuring the decision-making efficiency of edge computing. The edge inference engine associates tasks, linking task-related modal features to each other, thereby ensuring the comprehensiveness of decision evaluation.

[0080] Furthermore, the multi-protocol data acquisition module includes a data acquisition unit, a preprocessing unit, and a data storage unit;

[0081] The data acquisition unit is used to acquire time-series data, image data, and text data, and to time-align the time-series data, image data, and text data according to the acquisition time.

[0082] The preprocessing unit is used to segment the time-aligned time series data, image data and text data according to the set time interval size, and obtain the segmented multimodal data in sequence. Each multimodal data includes the time series data, image data and text data in the corresponding time interval.

[0083] The data storage unit is used to store the preprocessed multimodal data.

[0084] Time alignment is performed using the data acquisition unit to ensure that the data from each modality refers to the same event, thus ensuring the reliability of the decision. The preprocessing unit performs data segmentation to obtain multimodal data, thereby centralizing the data and facilitating subsequent multimodal data recognition and processing.

[0085] Furthermore, the preprocessing unit is also used to perform anomaly screening on the time-series data, image data, and text data in each multimodal dataset, and output the anomaly-processed multimodal data. The specific steps for anomaly screening are as follows:

[0086] Step a1: Extract each multimodal data in the order of acquisition time, and determine whether there is abnormal data in each time series data, image data and text data in the multimodal data according to the anomaly type judgment rules. If there is abnormal data, proceed to step a2; if there is no abnormal data, proceed to step a5.

[0087] Step a2: Determine whether each abnormal data is recoverable. If the abnormal data is recoverable, proceed to step a3; if the abnormal data is not recoverable, proceed to step a4.

[0088] Step a3: Perform recovery processing on the various abnormal data in the time series data, image data, and text data, restore the abnormal data to normal data, and then proceed to step a5;

[0089] Step a4: Replace each unrecoverable abnormal data in the time series data, image data, and text data by replacing the current unrecoverable abnormal data with the previous normal data before proceeding to step a5.

[0090] Step a5: Output each normal multimodal data, the restored multimodal data, and / or the replaced multimodal data in the order of acquisition time.

[0091] Anomaly screening is performed by a preprocessing unit to promptly identify abnormal data and ensure the reliability of subsequent multimodal data identification decisions. Recovery processing is performed on recoverable anomaly data to ensure data integrity and accuracy. Replacement processing is performed on unrecoverable anomaly data to effectively ensure data integrity and reduce the risk of misjudgment.

[0092] Furthermore, in step a1, the specific steps for determining whether there is abnormal data in the multimodal data according to the anomaly type judgment rules are as follows:

[0093] Step a1.1: Set the exception type judgment rules for judging abnormal data. The exception type judgment rules include time series judgment rules for judging time series data exceptions, graphic judgment rules for judging graphic data exceptions, and text judgment rules for judging text data exceptions.

[0094] Step a1.2: Read each time series data in the multimodal data sequentially according to the time sequence, and use the time series judgment rules to judge the anomalies of the time series data. If the current time series data has partial missing, data mutation, or complete missing, it is determined that the current time series data is abnormal.

[0095] Step a1.3: Read each image data in the multimodal data sequentially according to the time sequence, and use the image judgment rules to judge the image data for anomalies. If the current image data has image deformation, blurring or missing images, it is determined that the current image data has an anomaly.

[0096] Step a1.4: Read each text data in the multimodal data sequentially according to the time sequence, and use the text judgment rules to judge the anomalies of the text data. If the current text data has syntax errors, disordered format or missing text, it is determined that the current text data is abnormal.

[0097] By setting time series judgment rules, image judgment rules, and text judgment rules, it is possible to quickly identify and judge anomalies in time series data, image data, and text data, thereby quickly determining the abnormal data in multimodal data.

[0098] Furthermore, in step a2, the specific steps for determining whether each abnormal data point can be recovered are as follows:

[0099] Step a2.1: Sequentially obtain each abnormal data, and then determine the data type of each abnormal data. If the data type is time series data, proceed to step a2.2; if the data type is image data, proceed to step a2.3; if the data type is text data, proceed to step a2.4.

[0100] Step a2.2: Use the timing judgment rules to determine whether the timing data is partially missing or has a data mutation. If it is partially missing or has a data mutation, the timing data is determined to be recoverable; otherwise, the timing data is determined to be unrecoverable.

[0101] Step a2.3: Use the graphic judgment rules to determine whether the image data is distorted or blurred. If it is distorted or blurred, the image data is determined to be recoverable; otherwise, the image data is determined to be unrecoverable.

[0102] Step a2.4: Use text judgment rules to determine whether the text data has a syntax error or a disordered format. If it has a syntax error or a disordered format, the text data is determined to be recoverable; otherwise, the text data is determined to be unrecoverable.

[0103] By identifying and judging recoverable data, it is possible to restore abnormal data in multimodal data as much as possible, ensuring the reliability of multimodal data and providing a guarantee for subsequent decision-making.

[0104] Furthermore, in step a3, the specific steps for restoring abnormal data to normal data are as follows:

[0105] Step a3.1: Determine the recoverable anomaly types of abnormal data. The recoverable anomaly types of abnormal time series data include partial missing data and data mutation. The recoverable anomaly types of abnormal image data include image distortion and image blurring. The recoverable anomaly types of abnormal text data include syntax errors and garbled text.

[0106] Step a3.2: Select the appropriate recovery processing model according to the recoverable anomaly type of the abnormal data, and use the corresponding recovery processing model to perform recovery processing on each abnormal data to obtain abnormal recovery data;

[0107] Step a3.3: Replace the corresponding abnormal data in the time series data, image data, and text data with the abnormal recovery data, thereby restoring the abnormal data to normal data.

[0108] By selecting appropriate recovery processing models to process each abnormal data, abnormal recovery data is obtained. Then, the abnormal recovery data is used to replace the corresponding abnormal data, thereby achieving rapid recovery of abnormal events.

[0109] Furthermore, in step a3.2, the recovery processing model includes a partial missing data processing model, a data mutation processing model, an image deformation processing model, an image blurring processing model, a syntax error processing model, and a text garbled code processing model.

[0110] The partial missing data handling model is used to recover partially missing anomalous time-series data. The specific steps are as follows:

[0111] Step b1: Obtain the previous normal time series data and the next normal time series data of the current abnormal time series data, and find the normal data in the previous normal time series data and the next normal time series data that are in the same position as the missing data in the current abnormal time series data, so as to provide the corresponding position data for subsequent replacement.

[0112] Step b2: Determine whether the two identified normal data parts are the same. If the two normal data parts are the same, proceed to step b3; otherwise, proceed to step b4.

[0113] Step b3: The found normal data is used to fill the missing data position in the current abnormal time series data. Since the two normal data are the same, it indicates that the data is in the smooth acquisition stage and can be directly replaced. Then proceed to step b6.

[0114] Step b4: Obtain the first three normal time series data points and the last three normal time series data points of the current abnormal time series data. Find the normal data points in the first three normal time series data points and the last three normal time series data points that are in the same position as the missing data in the current abnormal time series data. Then calculate the first trend value based on the first three normal data points. The first trend value is the average of the two adjacent differences of the first three normal data points. Then calculate the second trend value based on the last three normal data points. The second trend value is the average of the two adjacent differences of the last three normal data points. By obtaining more normal data to calculate the trend value, the data in the changed state can be replaced. Then proceed to step b5.

[0115] Step b5: Calculate the first estimated value using the first trend value and the previous normal data. The first estimated value is the sum of the first trend value and the previous normal data. Calculate the second estimated value using the second trend value and the next normal data. The second estimated value is the sum of the second trend value and the next normal data. Then, average the first and second estimated values ​​to obtain a partial time-series estimated value. Calculate the partial time-series estimated value using the first and second estimated values ​​to maintain the data trend as much as possible, making the replaced data closer to the true value. Then, supplement the missing data positions in the current abnormal time-series data with the partial time-series estimated value, and then proceed to step b6.

[0116] Step b6: Use the supplemented time series data as the abnormal recovery data of the current abnormal time series data to complete the recovery processing of the abnormal time series data;

[0117] The data mutation handling model is used to recover abnormal time-series data that has undergone data mutations. The abnormal data that has mutated is generally 3 to 5 times larger than the normal data, preferably 3 times larger. The specific steps are as follows:

[0118] Step c1: Obtain the previous normal data and the next normal data in the current abnormal time series data, and determine whether the previous normal data and the next normal data are the same. If the previous normal data and the next normal data are the same, proceed to step c2; otherwise, proceed to step c3.

[0119] Step c2: Replace the current abnormal time series data with the previous normal data. Since the two normal data are the same, it indicates that the data is in the smooth acquisition stage and can be directly replaced. Then proceed to step c4.

[0120] Step c3: Obtain the two normal time series data points before and after the current abnormal time series data point in the time series data. Then calculate the average time series value of the four normal time series data points. The average time series value can be quickly obtained by direct averaging. The average time series value is then used to directly replace the current abnormal time series data point. Then proceed to step c4.

[0121] Step c4: Use the time series data formed after replacement as the abnormal recovery data of the current abnormal time series data to complete the recovery processing of abnormal time series data with data mutation.

[0122] The image deformation processing model is used to restore deformed image data. The specific steps are as follows:

[0123] Step d1: Select geometrically deformed images that can be restored from the abnormal image data of image deformation. Such geometrically deformed images can be directly repaired by image restoration tools.

[0124] Step d2: Use AI repair tools to restore the selected geometrically deformed images. For example, use AILabTools to repair the geometrically deformed images and obtain the restored normal images.

[0125] Step d3: Use the restored normal image as the abnormal recovery data of the current abnormal image data to complete the recovery processing of the abnormal image data of the current image deformation;

[0126] The image blurring processing model is used to restore blurry and abnormal image data. The specific steps are as follows:

[0127] Step e1: Select jittery blurred images and out-of-focus blurred images that can be restored from the abnormal image data with blurred images. Such jittery blurred images and out-of-focus blurred images can be directly restored using existing algorithms.

[0128] Step e2: Select the corresponding blur restoration method for the selected jittery and out-of-focus blur images. For example, for jittery and blur images (<30 pixel displacement), use Adobe Photoshop's image stabilization filter for restoration processing. For out-of-focus blur images, use a trained CNN model (such as SRCNN or DeblurGAN) to directly predict the clear image. Use the corresponding blur restoration method to restore the jittery and out-of-focus blur images to obtain the restored normal image.

[0129] Step e3: Use the restored normal image as the abnormal recovery data of the current deformed image to complete the recovery processing of the blurred abnormal image data of the current screen.

[0130] The syntax error handling model is used to recover abnormal text data containing syntax errors. The specific steps are as follows:

[0131] Step f1: Build a grammar rule base using correct text data from historical text data. The grammar rule base sets the prescribed words and corresponding grammar rules for the text data. By building the grammar rule base using historical text data, it is possible to ensure that the grammar rule base has high matching efficiency when used.

[0132] Step f2: Obtain abnormal text data, locate sentences with grammatical errors from the abnormal text data based on semantic recognition, perform word segmentation on the sentences with grammatical errors, and obtain each word in the abnormal text data.

[0133] Step f3: Match each obtained word segment with the corresponding specified words in the grammar rule base, and then recombine the matched specified words according to the grammar rules in the grammar rule base to output the recombined text data.

[0134] Step f4: Use the recombined text data as the abnormal recovery data of the current abnormal text data with syntax errors to complete the recovery process of the current abnormal text data.

[0135] The text encoding processing model is used to recover abnormal text data containing garbled characters. The specific steps are as follows:

[0136] Step j1: Obtain the abnormal text data containing garbled characters, and obtain the storage format of the abnormal text data containing garbled characters.

[0137] Step j2: Open the abnormal text data using a text editor (such as Notepad++ or VS Code) and convert the storage format to normal display text.

[0138] Step j3: Display the normal text as abnormal recovery data for the current abnormal text data containing garbled characters, and complete the recovery process for the current abnormal text data.

[0139] Furthermore, the multimodal fusion engine includes a feature extraction unit and a task allocation unit;

[0140] The feature extraction unit is used to extract features from multimodal data to obtain modal features at each time step. The modal features include time-series features extracted from time-series data, image features extracted from image data, and text features extracted from text data.

[0141] The specific steps to obtain modal features are as follows:

[0142] For time series features, statistical analysis is performed on the collected data in each time series data according to the time sequence, and emergency situations are judged according to the emergency data threshold library. The emergency data threshold library stores the emergency judgment thresholds for each type of collected data. When making a judgment, the type of collected data is first determined, such as temperature data, humidity data, air pressure data, etc. The corresponding emergency judgment threshold is set in the emergency data threshold library, such as temperature threshold, etc. The corresponding emergency judgment threshold is found according to the type of collected data. The collected data in the current time series data is compared with the corresponding emergency judgment threshold. If it exceeds the emergency judgment threshold, it is judged that there is an emergency. If it is determined that there is an emergency in the current collected data, the collected data at the current moment is recorded as the time series feature; otherwise, it is determined that there is no emergency in the current collected data, and the time series feature at the current moment is recorded as empty.

[0143] For image features, target objects are identified sequentially in each image data according to the time sequence, and then image analysis is performed on the identified target objects. In this embodiment, a neural network is used to identify target objects to determine whether there are corresponding target objects in the acquired images, such as whether there are cracks or fires. The emergency event database is used to determine whether there is an emergency for the target objects. The emergency event database stores various target objects for target matching, such as cracks in various scenarios. If the identified target object matches the corresponding target object in the emergency event database, it can be determined that there is an emergency. If it is determined that there is an emergency in the current acquired image, the target object identified at the current moment is recorded as an image feature; otherwise, it is determined that there is no emergency in the current acquired image, and the image feature at the current moment is recorded as empty.

[0144] For text features, semantic recognition is performed on the text information in each text data in chronological order. Then, the existence of an emergency is determined according to the emergency semantic event database. The emergency semantic event database stores various emergency semantics for semantic matching, such as "fire". If the recognized semantic matches the corresponding emergency semantic in the emergency semantic event database, it can be determined that there is an emergency. If it is determined that there is an emergency in the current semantic, the text information at the current moment is recorded as the text feature. Otherwise, it is determined that there is no emergency in the current semantic and the text feature at the current moment is recorded as empty.

[0145] The task allocation unit is used to identify emergency events based on the extracted modal features, and to determine the task priority of the current task to be decided based on the identified emergency events. The specific steps are as follows:

[0146] First, the modal features at each time step are obtained in chronological order, and then the temporal features, image features, and text features are extracted from the modal features at each time step.

[0147] Then, the temporal features, image features, and text features extracted at each time point are analyzed. If all three features extracted at the current time point are non-empty, indicating an emergency, then a Level 1 emergency event is determined to have occurred at the current time point. If one of the three features extracted at the current time point is empty, indicating an emergency, then a Level 2 emergency event is determined to have occurred at the current time point. If two of the three features extracted at the current time point are empty, indicating an emergency, then a Level 3 emergency event is determined to have occurred at the current time point. If all three features extracted at the current time point are empty, then the modal features of the current time point are deleted.

[0148] Then, the task priority of the decision-making task corresponding to the Level 1 emergency is set as the first priority, the task priority of the decision-making task corresponding to the Level 2 emergency is set as the second priority, and the task priority of the decision-making task corresponding to the Level 3 emergency is set as the third priority.

[0149] Finally, the modal features corresponding to Level 1 emergency events are stored in the first data cache unit as the first priority task to be decided, the modal features corresponding to Level 2 emergency events are stored in the second data cache unit as the second priority task to be decided, and the modal features corresponding to Level 3 emergency events are stored in the third data cache unit as the third priority task to be decided. At the same time, the number of tasks to be decided in the first, second, and third data cache units is updated in real time.

[0150] Furthermore, the dynamic optimization module includes a strategy setting unit and a resource allocation unit;

[0151] The strategy setting unit is used to obtain the task priority of the current task to be decided, and then set the allocation strategy of NPU and GPU computing resources according to the task priority. The specific steps are as follows:

[0152] First, a first decision thread, a second decision thread, and a third decision thread are established to perform initial decision processing on each task to be decided in the first data cache unit, the second data cache unit, and the third data cache unit, respectively. Then, a fourth decision thread is established to perform comprehensive decision processing.

[0153] Then, based on the number of tasks to be decided in the first, second, and third data cache units at the current moment, E1, E2, and E3, the resource allocation state is determined: when E1 > W11, E1 > W21 and E2 > W22, or E1 > W31, E2 > W32 and E3 > W33, where W11 > W21 > W31 and W22 > W32, for example, setting W11=100, W21=95, W22=90, W31=85, W32=80, and W33=75, then the resource call state is determined; when E1 < W12, E1 < W23 and E2 < W24, or E1 < W34 and E2 < W35 When E3 < W36, where W12 > W23 > W34 and W24 > W35, for example, setting W12=70, W23=65, W24=60, W34=55, W35=50, and W36=45, then it is determined to enter the resource internal control state; then set the resource call state and the resource internal control state as W11 > W12, W21 > W23, W22 > W24, W31 > W34, W32 > W35, and W33 > W36, and set the initial resource allocation state as the resource internal control state. Then, based on the amount of tasks to be decided E1, E2, and E3, it is determined whether to switch to the resource call state. If no switch is needed, it remains in the resource internal control state.

[0154] By setting the resource allocation state based on the amount of tasks to be decided (E1, E2, and E3), the computing resources of the NPU and GPU can be rationally allocated according to the amount of tasks to be decided. By setting E1 > W11, E1 > W21 and E2 > W22, and E1 > W31, E2 > W32 and E3 > W33 as judgment conditions, the system switches to the resource call state only under these three call conditions. These three call conditions allow for comprehensive state evaluation, utilizing the combined computing resources of the NPU and GPU to complete large amounts of data processing and ensuring processing efficiency. By setting E1 < W12, E1 < W23 and E2 < W24, and E1 < W34, E2 < W35 and E3 < W36 as judgment conditions, the system switches to the resource internal control state only under these three internal control conditions. These three internal control conditions allow for comprehensive state evaluation, utilizing the computing resources of the NPU to complete the current data processing, thus avoiding the waste of computing resources.

[0155] Finally, the allocation strategy for NPU and GPU computing resources is set as follows:

[0156] At the start, it enters the initial resource allocation state, that is, the resource internal control state;

[0157] After determining that the system has entered a resource control state, the base values ​​for NPU computing resources allocated to the first, second, third, and fourth decision threads are set as J1, J2, J3, and J4, respectively. For example, J1, J2, J3, and J4 can be set to 30%N, 15%N, 10%N, and 35%N, respectively. Then, a dynamically allocated resource of size J0 (e.g., 10%N) is set. By setting the dynamically allocated resource, the resource size can be dynamically adjusted within a certain range according to the amount of tasks to be decided, satisfying the small-range adaptive change of computing resources. J1 > J2 > J3, and J0 + J1 + J2 + J3 + J4 = N, where N is the computing resource value used by the NPU for decision-making calculations. N is less than or equal to the total computing resource value of the NPU; for example, it can be set to 80% of the total computing resource value of the NPU. Then, the computing resource allocation values ​​for the first, second, and third decision threads under the resource control state are calculated as follows: , as well as ;

[0158] After determining that a resource call state has been entered, the base values ​​for NPU computing resources allocated to the first and fourth decision threads are set to D1 and D4, respectively, with D1 > J1 + J0. For example, D1 and D4 are set to 55%N and 45%N, respectively. By setting a higher base value of D1 for NPU computing resources, the first decision thread can efficiently handle various first-level emergency events. The base values ​​for GPU computing resources allocated to the second and third decision threads are set to D2 and D3, respectively, with D2 > J2 + J0 and D3 > J3 + J0. A dynamically allocated resource of size D5 is then set, for example, D2, D3, and D5 are set to 35%G, 25%G, and 20%G, respectively. By setting the dynamically allocated resource, the resource size can be dynamically adjusted within a certain range according to the amount of tasks to be decided, satisfying the small-range adaptive change of computing resources, and D1 + D4 =N, where N is the computing resource value used by the NPU for decision-making computation. N is less than or equal to the total computing resource value of the NPU, for example, it can be set to 80% of the total computing resource value of the NPU. D2 > D3, D2 + D3 + D5 = G, where G is the computing resource value used by the GPU for decision-making computation. G is less than or equal to the total computing resource value of the GPU, for example, it can be set to 60% of the total computing resource value of the GPU. Then, the computing resource allocation values ​​of the second and third decision threads under the resource call state are calculated as follows. as well as ;

[0159] The resource allocation unit is used to allocate NPU and GPU computing resources according to the set allocation strategy. The specific steps are as follows:

[0160] First, obtain the number of tasks to be decided in the first data cache unit, the second data cache unit, and the third data cache unit at the current moment, namely E1, E2, and E3;

[0161] Then, the current resource allocation status is determined based on the amount of tasks to be decided, E1, E2, and E3.

[0162] Finally, resource allocation is performed based on the resource allocation status: if the current time is under resource control, then according to... , , The computational resource allocation values ​​of J4 allocate the NPU's computational resources to the first decision thread, the second decision thread, the third decision thread, and the fourth decision thread, respectively. If the current time is in a resource call state, the NPU's computational resources are allocated to the first and fourth decision threads according to the computational resource allocation values ​​of D1 and D4, respectively. as well as The computing resource allocation value allocates the GPU's computing resources to the second decision thread and the third decision thread respectively.

[0163] Furthermore, the edge inference engine includes a task association unit and a collaborative analysis unit;

[0164] The task association unit is used to associate the acquired modal features with tasks, enabling the current decision task to collect all relevant modal features. The specific steps are as follows:

[0165] First, the first decision thread, the second decision thread, and the third decision thread retrieve the modal features stored in the first data cache unit, the second data cache unit, and the third data cache unit respectively. The three decision threads retrieve only one modal feature at a time according to the temporal order in the corresponding data cache unit.

[0166] Then, the first decision thread, the second decision thread, and the third decision thread synchronously perform the initial decision processing:

[0167] The first decision thread performs initial decision processing on the modal features retrieved from the first data cache unit, and determines whether the three emergency situations in the temporal features, image features, and text features of the modal features point to the same situation. If they point to the same situation, it means that they point to the same emergency situation. For example, if the temporal data indicates that the collected voltage data is abnormal, the image data indicates that there is a fine weld crack, and the text data indicates that the log file shows a welding abnormality, then all three emergency situations point to a welding abnormality. If the emergency situations point to the same situation, the current modal feature is stored in the fourth data cache unit. If the emergency situations point to different situations, the current modal feature is deleted.

[0168] The second decision thread performs initial decision processing on the modal features retrieved from the second data cache unit, determining whether two emergency situations among the temporal features, image features, and text features of the modal features point to the same emergency situation. If they point to the same emergency situation, it means that they point to the same emergency situation. For example, if the temporal data indicates that the collected temperature data is abnormally high, the image data indicates that there is a flame or smoke, and the text data is empty, then two of the three emergency situations point to a fire. If the emergency situations point to the same emergency situation, the current modal feature is stored in the fourth data cache unit. If the emergency situations point to different emergency situations, the current modal feature is deleted.

[0169] The third decision thread performs initial decision processing on the modal features retrieved from the third data cache unit, and obtains an emergency situation in the temporal features, image features, and text features of the modal features. For example, the temporal data indicates that the collected temperature data is abnormally high, the image data is empty, and the text data is empty. The current modal features are then stored in the fourth data cache unit.

[0170] Finally, the fourth decision thread sorts the modal features in the fourth data cache unit according to time sequence, and then extracts the modal features that are sequential and have the same urgency to construct the task association dataset, thus obtaining the task association datasets belonging to different decision tasks.

[0171] By placing temporally consecutive modal features with the same urgency into a task-related dataset, modal features belonging to the same decision-making task are grouped together, ensuring the comprehensiveness of the decision-making task evaluation.

[0172] The collaborative analysis unit is used to perform collaborative analysis on the modal features after task association to obtain the decision result of the current task to be decided. The specific steps are as follows:

[0173] First, read the datasets associated with each task in the order they were built, and obtain the emergency situation corresponding to the datasets associated with each task as the emergency situation of the task to be decided.

[0174] Then, based on the acquired emergency situation, the task decision table is queried. The task decision table stores the correspondence between emergency situations and decisions. The correspondence is established with historical experience data. Therefore, by querying the task decision table, the decision result corresponding to the current emergency situation can be obtained, and the obtained decision result can be used as the final decision result of the current task to be decided.

[0175] Finally, delete the task-related dataset corresponding to the current task to be decided.

[0176] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A multimodal data processing system for smart factories based on edge computing, characterized in that: It includes a multi-protocol data acquisition module, a multi-modal fusion engine, an edge inference engine, and a dynamic optimization module; The multi-protocol data acquisition module is used to acquire multimodal data from the smart factory in real time. The multimodal data consists of time-series data, image data, and text data. The multimodal fusion engine is used to extract modal features from the acquired multimodal data and determine the task priority of the current task to be decided based on the extracted modal features. The dynamic optimization module is used to dynamically adjust the allocation strategy of NPU and GPU computing resources according to task priority; The edge inference engine is used to associate the acquired modal features with tasks, and to perform collaborative analysis on the associated modal features to obtain the decision results of the current decision task. The multimodal fusion engine includes a feature extraction unit and a task allocation unit; The feature extraction unit is used to extract features from multimodal data to obtain modal features at each time step. The modal features include time-series features extracted from time-series data, image features extracted from image data, and text features extracted from text data. The specific steps to obtain modal features are as follows: For time series features, statistical analysis is performed on the collected data in each time series data according to the time series order, and the emergency situation is judged according to the emergency data threshold library. If it is determined that there is an emergency situation in the current collected data, the collected data at the current time is recorded as the time series feature; otherwise, it is determined that there is no emergency situation in the current collected data, and the time series feature at the current time is recorded as empty. For image features, target objects are identified sequentially in each image data according to the time sequence. Then, the identified target objects are analyzed. Based on the emergency event database, it is determined whether there is an emergency for the target object. If it is determined that there is an emergency in the current image, the target object identified at the current moment is recorded as an image feature. Otherwise, it is determined that there is no emergency in the current image, and the image feature at the current moment is recorded as empty. For text features, semantic recognition is performed on the text information in each text data in chronological order. Then, it is determined whether there is an emergency based on the emergency semantic event database. If it is determined that there is an emergency in the current semantic, the text information at the current moment is recorded as the text feature. Otherwise, it is determined that there is no emergency in the current semantic and the text feature at the current moment is recorded as empty. The task allocation unit is used to identify emergency events based on the extracted modal features, and to determine the task priority of the current task to be decided based on the identified emergency events. The specific steps are as follows: First, the modal features at each time step are obtained in chronological order, and then the temporal features, image features, and text features are extracted from the modal features at each time step. Then, the temporal features, image features, and text features extracted at each time point are analyzed. If all of the temporal features, image features, and text features extracted at the current time point are non-empty, it is determined that a Level 1 emergency event has occurred at the current time point. If any of the temporal features, image features, and text features extracted at the current time point are empty, it is determined that a Level 2 emergency event has occurred at the current time point. If two of the temporal features, image features, and text features extracted at the current time point are empty, it is determined that a Level 3 emergency event has occurred at the current time point. If all of the temporal features, image features, and text features extracted at the current time point are empty, the modal features of the current time point are deleted. Then, the task priority of the decision-making task corresponding to the Level 1 emergency is set as the first priority, the task priority of the decision-making task corresponding to the Level 2 emergency is set as the second priority, and the task priority of the decision-making task corresponding to the Level 3 emergency is set as the third priority. Finally, the modal features corresponding to Level 1 emergency events are stored in the first data cache unit as the first priority task to be decided, the modal features corresponding to Level 2 emergency events are stored in the second data cache unit as the second priority task to be decided, and the modal features corresponding to Level 3 emergency events are stored in the third data cache unit as the third priority task to be decided. At the same time, the number of tasks to be decided in the first, second, and third data cache units is updated in real time. The dynamic optimization module includes a strategy setting unit and a resource allocation unit; The strategy setting unit is used to obtain the task priority of the current task to be decided, and then set the allocation strategy of NPU and GPU computing resources according to the task priority. The specific steps are as follows: First, a first decision thread, a second decision thread, and a third decision thread are established to perform initial decision processing on each task to be decided in the first data cache unit, the second data cache unit, and the third data cache unit, respectively. Then, a fourth decision thread is established to perform comprehensive decision processing. Then, based on the number of tasks to be decided in the first, second, and third data cache units at the current time, E1, E2, and E3, the resource allocation state is determined: when E1 > W11, E1 > W21 and E2 > W22, or E1 > W31, E2 > W32 and E3 > W33, where W11 > W21 > W31 and W22 > W32, then the resource allocation state is determined; when E1 < W12, E1 < W21 > W22, E2 > W32, the resource allocation state is determined; when E1 < W12, E2 > W22, E2 > W32, the resource allocation state is determined. When W23 and E2 < W24, or when E1 < W34, E2 < W35 and E3 < W36, where W12 > W23 > W34 and W24 > W35, it is determined that the resource internal control state is entered. Then, the resource call state and the resource internal control state are set as follows: W11 > W12, W21 > W23, W22 > W24, W31 > W34, W32 > W35, and W33 > W36. At the same time, the initial resource allocation state is set as the resource internal control state. Finally, the allocation strategy for NPU and GPU computing resources is set as follows: At the start, it enters the initial resource allocation state, that is, the resource internal control state; After determining that the system has entered a resource control state, the base values ​​for NPU computing resources allocated to the first, second, third, and fourth decision threads are set as J1, J2, J3, and J4, respectively. A dynamically allocated resource of size J0 is then assigned, where J1 > J2 > J3, and J0 + J1 + J2 + J3 + J4 = N, where N is the computing resource value used by the NPU for decision-making computation. The computing resource allocation values ​​for the first, second, and third decision threads under the resource control state are then calculated as follows: , as well as ; After determining that the resource call state has been entered, the base values ​​for NPU computing resources allocated to the first and fourth decision threads are set to D1 and D4, respectively. The base values ​​for GPU computing resources allocated to the second and third decision threads are set to D2 and D3, respectively. A dynamically allocated resource of size D5 for GPU computing resources is then set, where D1 + D4 = N, N is the computing resource value used by the NPU for decision-making, D2 > D3, and D2 + D3 + D5 = G, where G is the computing resource value used by the GPU for decision-making. The computing resource allocation values ​​for the second and third decision threads in the resource call state are then calculated as follows: as well as ; The resource allocation unit is used to allocate NPU and GPU computing resources according to the set allocation strategy. The specific steps are as follows: First, obtain the number of tasks to be decided in the first data cache unit, the second data cache unit, and the third data cache unit at the current moment, namely E1, E2, and E3; Then, the current resource allocation status is determined based on the amount of tasks to be decided, E1, E2, and E3. Finally, resource allocation is performed based on the resource allocation status: if the current time is under resource control, then according to... , , The computational resource allocation values ​​of J4 allocate the NPU's computational resources to the first decision thread, the second decision thread, the third decision thread, and the fourth decision thread, respectively. If the current time is in a resource call state, the NPU's computational resources are allocated to the first and fourth decision threads according to the computational resource allocation values ​​of D1 and D4, respectively. as well as The computing resource allocation value allocates the GPU's computing resources to the second decision thread and the third decision thread respectively.

2. The edge computing-based smart factory multimodal data processing system according to claim 1, characterized in that: The multi-protocol data acquisition module includes a data acquisition unit, a preprocessing unit, and a data storage unit; The data acquisition unit is used to acquire time-series data, image data, and text data, and to time-align the time-series data, image data, and text data. The preprocessing unit is used to segment the time-aligned time series data, image data and text data according to the set time interval size, and obtain the segmented multimodal data in sequence. Each multimodal data includes the time series data, image data and text data in the corresponding time interval. The data storage unit is used to store the preprocessed multimodal data.

3. The edge computing-based smart factory multimodal data processing system according to claim 1, characterized in that: The preprocessing unit is also used to perform anomaly screening on the time series data, image data, and text data in each multimodal dataset, and output the anomaly-processed multimodal data. The specific steps for anomaly screening are as follows: Step a1: Extract each multimodal data in the order of acquisition time, and determine whether there is abnormal data in each time series data, image data and text data in the multimodal data according to the anomaly type judgment rules. If there is abnormal data, proceed to step a2; if there is no abnormal data, proceed to step a5. Step a2: Determine whether each abnormal data is recoverable. If the abnormal data is recoverable, proceed to step a3; if the abnormal data is not recoverable, proceed to step a4. Step a3: Perform recovery processing on the various abnormal data in the time series data, image data, and text data, restore the abnormal data to normal data, and then proceed to step a5; Step a4: Replace each unrecoverable abnormal data in the time series data, image data, and text data by replacing the current unrecoverable abnormal data with the previous normal data before proceeding to step a5. Step a5: Output each normal multimodal data, the restored multimodal data, and / or the replaced multimodal data in the order of acquisition time.

4. The edge computing-based smart factory multimodal data processing system according to claim 3, characterized in that: In step a1, the specific steps for determining whether there is abnormal data in the multimodal data according to the anomaly type judgment rules are as follows: Step a1.1: Set the exception type judgment rules for judging abnormal data. The exception type judgment rules include time series judgment rules for judging time series data exceptions, graphic judgment rules for judging graphic data exceptions, and text judgment rules for judging text data exceptions. Step a1.2: Read each time series data in the multimodal data sequentially according to the time sequence, and use the time series judgment rules to judge the anomalies of the time series data. If the current time series data has partial missing, data mutation, or complete missing, it is determined that the current time series data is abnormal. Step a1.3: Read each image data in the multimodal data sequentially according to the time sequence, and use the image judgment rules to judge the image data for anomalies. If the current image data has image deformation, blurring or missing images, it is determined that the current image data has an anomaly. Step a1.4: Read each text data in the multimodal data sequentially according to the time sequence, and use the text judgment rules to judge the anomalies of the text data. If the current text data has syntax errors, disordered format or missing text, it is determined that the current text data is abnormal.

5. The edge computing-based smart factory multimodal data processing system according to claim 4, characterized in that: In step a2, the specific steps for determining whether each abnormal data can be recovered are as follows: Step a2.1: Sequentially obtain each abnormal data, and then determine the data type of each abnormal data. If the data type is time series data, proceed to step a2.2; if the data type is image data, proceed to step a2.3; if the data type is text data, proceed to step a2.

4. Step a2.2: Use the timing judgment rules to determine whether the timing data is partially missing or has a data mutation. If it is partially missing or has a data mutation, the timing data is determined to be recoverable; otherwise, the timing data is determined to be unrecoverable. Step a2.3: Use the graphic judgment rules to determine whether the image data is distorted or blurred. If it is distorted or blurred, the image data is determined to be recoverable; otherwise, the image data is determined to be unrecoverable. Step a2.4: Use text judgment rules to determine whether the text data has a syntax error or a disordered format. If it has a syntax error or a disordered format, the text data is determined to be recoverable; otherwise, the text data is determined to be unrecoverable.

6. The edge computing-based smart factory multimodal data processing system according to claim 5, characterized in that: In step a3, the specific steps to restore the abnormal data to normal data are as follows: Step a3.1: Determine the recoverable anomaly types of abnormal data. The recoverable anomaly types of abnormal time series data include partial missing data and data mutation. The recoverable anomaly types of abnormal image data include image distortion and image blurring. The recoverable anomaly types of abnormal text data include syntax errors and garbled text. Step a3.2: Select the appropriate recovery processing model according to the recoverable anomaly type of the abnormal data, and use the corresponding recovery processing model to perform recovery processing on each abnormal data to obtain abnormal recovery data; Step a3.3: Replace the corresponding abnormal data in the time series data, image data, and text data with the abnormal recovery data, thereby restoring the abnormal data to normal data.

7. The edge computing-based smart factory multimodal data processing system according to claim 6, characterized in that: In step a3.2, the recovery processing model includes a partial missing data processing model, a data mutation processing model, an image deformation processing model, an image blurring processing model, a syntax error processing model, and a text garbled code processing model. The partial missing data handling model is used to recover partially missing anomalous time-series data; The data mutation handling model is used to recover abnormal time-series data that has undergone data mutation. Image deformation processing models are used to restore deformed image data. The image blur processing model is used to restore blurry and abnormal image data; The syntax error handling model is used to recover abnormal text data containing syntax errors; The text encoding processing model is used to recover abnormal text data containing garbled characters.

8. The edge computing-based smart factory multimodal data processing system according to claim 1, characterized in that: The edge inference engine includes a task association unit and a collaborative analysis unit; The task association unit is used to associate the acquired modal features with tasks, enabling the current decision task to collect all relevant modal features. Specific steps include... for: First, the first decision thread, the second decision thread, and the third decision thread retrieve the modal features stored in the first data cache unit, the second data cache unit, and the third data cache unit respectively. The three decision threads retrieve only one modal feature at a time according to the temporal order in the corresponding data cache unit. Then, the first decision thread, the second decision thread, and the third decision thread synchronously perform the initial decision processing: The first decision thread performs initial decision processing on the modal features retrieved from the first data cache unit, and determines whether the three emergency situations among the temporal features, image features, and text features of the modal features are consistent. If the emergency situations are consistent, the current modal feature is stored in the fourth data cache unit. If the emergency situations are inconsistent, the current modal feature is deleted. The second decision thread performs initial decision processing on the modal features retrieved from the second data cache unit, and determines whether the two emergency situations in the temporal features, image features, and text features of the modal features are consistent. If the emergency situations are consistent, the current modal feature is stored in the fourth data cache unit. If the emergency situations are inconsistent, the current modal feature is deleted. The third decision thread performs initial decision processing on the modal features retrieved from the third data cache unit, obtains an emergency situation from the temporal features, image features, and text features of the modal features, and stores the current modal features in the fourth data cache unit; Finally, the fourth decision thread sorts the modal features in the fourth data cache unit according to time sequence, and then extracts the modal features that are sequential and have the same urgency to construct the task association dataset, thus obtaining the task association datasets belonging to different decision tasks. The collaborative analysis unit is used to perform collaborative analysis on the modal features after task association to obtain the decision result of the current task to be decided. The specific steps are as follows: First, read the task-related datasets in the order they were built, and obtain the emergency situations corresponding to the task-related datasets. Then, use the obtained emergency situations as the emergency situations of the current task to be decided. Then, based on the obtained emergency situation, query the task decision table to obtain the decision result corresponding to the current emergency situation, and use the obtained decision result as the final decision result of the current task to be decided; Finally, delete the task-related dataset corresponding to the current task to be decided.

Images