Intelligent data generation processing method and system based on artificial intelligence

Abstract

The embodiment of the specification discloses an intelligent data processing method and system based on artificial intelligence. The intelligent data processing method comprises the following steps: obtaining to-be-executed production task information and multi-modal data; extracting personnel state features including at least action standard degree, action frequency and fatigue index corresponding to video data based on a pre-connected first artificial intelligence model; extracting equipment health state features corresponding to sensor time series data based on a pre-connected second artificial intelligence model; fusing the personnel state features, the equipment health state features, material inventory data and personnel skill profile data based on a pre-connected multi-modal feature fusion model to obtain an efficiency index; and generating a prediction result according to the efficiency index and the to-be-executed production task information based on a pre-connected scheduling effect prediction model. The embodiment of the specification can not only deeply fuse multi-modal production data, but also accurately quantitatively predict future scheduling effects.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to an intelligent data processing method and system based on artificial intelligence. Background Technology

[0002] In modern manufacturing, especially in small-batch, multi-model production models, the complexity of production scheduling is increasing daily. Traditional scheduling methods rely heavily on the personal experience of managers, lacking precise quantitative analysis of dynamic data from the production floor. Although data on personnel, equipment, and materials can be collected using sensors and other means, this data often exists in isolation, failing to achieve deep integration and intelligent prediction. Traditional data processing technologies have several shortcomings: in personnel assessment, they cannot combine real-time work status with static skill levels, making it difficult to accurately predict changes in efficiency under new tasks; in equipment management, most systems are limited to monitoring the current status and cannot predict failure risks based on future production loads, resulting in delayed warnings; in scheduling decisions, managers cannot foresee personnel status, equipment risks, and task completion times under different scheduling schemes, leading to a lack of forward-looking data support for decisions, resulting in low production efficiency, suboptimal resource allocation, and unstable production plan achievement rates. Therefore, there is an urgent need for an AI-based intelligent data generation processing method that can deeply integrate multimodal production data and accurately quantify and predict future scheduling effects. Summary of the Invention

[0003] This specification provides an intelligent data generation processing method and system based on artificial intelligence, the technical solution of which is as follows:

[0004] In a first aspect, embodiments of this specification provide an intelligent data processing method based on artificial intelligence, comprising: acquiring information on production tasks to be executed and multimodal data, wherein the multimodal data includes at least video data, sensor time-series data, material inventory data, and personnel skill profile data; extracting personnel status features corresponding to the video data, including at least motion standardization, motion frequency, and fatigue index, based on a pre-accessed first artificial intelligence model; extracting equipment health status features corresponding to the sensor time-series data based on a pre-accessed second artificial intelligence model; fusing personnel status features, equipment health status features, material inventory data, and personnel skill profile data based on a pre-accessed multimodal feature fusion model to obtain performance indicators, wherein the performance indicators include comprehensive personnel efficiency, comprehensive equipment status, and material availability status; and generating prediction results based on a pre-accessed scheduling effect prediction model, wherein the prediction results include predicted personnel status values, predicted equipment failure probability values, and predicted production task completion time.

[0005] Secondly, embodiments of this specification provide an intelligent data processing system based on artificial intelligence, including a data acquisition module for acquiring information on production tasks to be executed and multimodal data, the multimodal data including at least video data, sensor time-series data, material inventory data, and personnel skill profile data; a first feature module for extracting personnel status features corresponding to the video data, including at least action standard, action frequency, and fatigue index, based on a pre-accessed first artificial intelligence model; a second feature module for extracting equipment health status features corresponding to the sensor time-series data, based on a pre-accessed second artificial intelligence model; a feature fusion module for fusing personnel status features, equipment health status features, material inventory data, and personnel skill profile data based on a pre-accessed multimodal feature fusion model to obtain performance indicators, including comprehensive personnel efficiency, comprehensive equipment status, and material availability status; and a prediction module for generating prediction results based on a pre-accessed scheduling effect prediction model, according to the performance indicators and information on production tasks to be executed, the prediction results including predicted personnel status values, predicted equipment failure probability values, and predicted production task completion time.

[0006] Thirdly, embodiments of this specification provide an electronic device, including a processor and a memory;

[0007] The processor is connected to the memory; the memory is used to store executable program code; the processor runs the program corresponding to the executable program code by reading the executable program code stored in the memory, so as to perform steps such as in an artificial intelligence-based intelligent data generation processing method.

[0008] Fourthly, embodiments of this specification provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of an artificial intelligence-based intelligent data generation processing method.

[0009] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0010] This embodiment of the specification can extract personnel status features corresponding to video data, including at least motion standardization, motion frequency, and fatigue index, based on a first artificial intelligence model; it can also extract equipment health status features corresponding to sensor time-series data based on a pre-accessed second artificial intelligence model; then, based on a pre-accessed multimodal feature fusion model, it can fuse personnel status features, equipment health status features, material inventory data, and personnel skill profile data to obtain performance indicators; this embodiment of the specification can also generate prediction results based on a pre-accessed scheduling effect prediction model, according to performance indicators and information on production tasks to be executed, including predicted personnel status values, predicted equipment failure probability values, and predicted production task completion times. This specification's embodiments, based on a multimodal feature fusion model, can integrate isolated production factor information such as personnel, equipment, and materials. It can also intelligently assess the mutual influence between different production factor information, obtaining performance indicators including overall personnel efficiency, overall equipment status, and material availability. Furthermore, by using comprehensive performance indicators that better align with actual production patterns, it provides a comprehensive and reliable basis for scheduling. Specifically, by constructing a scheduling effect prediction model, it predicts personnel status curves, equipment failure probability values, and production task completion times. This specification's embodiments not only deeply integrate multimodal production data but also accurately quantify and predict future scheduling effects, enabling managers to accurately anticipate various potential problems before scheduling execution. This specification's embodiments improve the utilization efficiency of production resources and the robustness of the entire production system. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram illustrating an application scenario of an intelligent data generation and processing method based on artificial intelligence, as provided in this manual.

[0013] Figure 2 This is a flowchart illustrating an intelligent data generation and processing method based on artificial intelligence, as provided in this manual.

[0014] Figure 3 This is a flowchart illustrating the process of extracting personnel status features provided in this manual.

[0015] Figure 4 This is a flowchart illustrating the process of extracting device health status characteristics corresponding to sensor time-series data, as provided in this manual.

[0016] Figure 5 This is a schematic diagram of the structure of an intelligent data generation and processing system based on artificial intelligence, as provided in this specification.

[0017] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this specification. Detailed Implementation

[0018] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings.

[0019] The terms "first," "second," etc., in the description, claims, and accompanying drawings are used to distinguish different objects and not to describe a particular order. Furthermore, the term "comprising" and any variations thereof are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0020] The artificial intelligence-based intelligent data generation processing method provided in several embodiments of this specification can be executed by the artificial intelligence-based intelligent data generation processing system provided in the embodiments of this invention.

[0021] Before this specification elaborates on the AI-based intelligent data generation processing method in conjunction with one or more embodiments, it first introduces the application scenarios of this AI-based intelligent data generation processing method.

[0022] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of an artificial intelligence-based intelligent data generation processing method provided in an embodiment of the present invention. In this embodiment, the artificial intelligence-based intelligent data generation processing system 100 can be integrated into an electronic device, such as a terminal or a server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer (PC); the server can be a single server or a server cluster composed of multiple servers.

[0023] In some embodiments, the intelligent data generation processing system 100 may also be integrated into multiple electronic devices. For example, the intelligent data generation processing system 100 may be integrated into multiple servers, and the artificial intelligence-based intelligent data generation processing method of this application may be implemented by multiple servers.

[0024] In some embodiments, the server may also be implemented as a terminal. The terminal may be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer (PC), etc. The terminal includes a central processing unit (CPU), a graphics processing unit (GPU), memory, storage devices, a network communication module, sensors, a display screen, a battery and power management module, etc.

[0025] For example, refer to Figure 1 The electronic device may include a server 110, a storage terminal 120, etc. The storage terminal 120 stores multimodal data, etc. The multimodal monitoring data may include at least video data, sensor time series data, material inventory data, and personnel skill file data, etc. The server 110 and the storage terminal 120 communicate with each other, which will not be described in detail here.

[0026] The server 110 may include a processor and memory. The server 110 can acquire information about production tasks to be executed and multimodal data, including at least video data, sensor time-series data, material inventory data, and personnel skill profile data. Based on a pre-accessed first artificial intelligence model, it extracts personnel status features corresponding to the video data, including at least motion standardization, motion frequency, and fatigue index. Based on a pre-accessed second artificial intelligence model, it extracts equipment health status features corresponding to the sensor time-series data. Based on a pre-accessed multimodal feature fusion model, it fuses personnel status features, equipment health status features, material inventory data, and personnel skill profile data to obtain performance indicators, including overall personnel efficiency, overall equipment status, and material availability status. Based on a pre-accessed scheduling effect prediction model, it generates prediction results based on the performance indicators and information about production tasks to be executed, including predicted personnel status values, predicted equipment failure probability values, and predicted production task completion times.

[0027] It should be noted that, Figure 1 The schematic diagram of an AI-based intelligent data generation processing system shown is merely an example. The AI-based intelligent data generation processing system and scenario described in this embodiment are intended to more clearly illustrate the technical solutions of this embodiment and do not constitute a limitation on the technical solutions provided by this embodiment. As those skilled in the art will know, with the evolution of AI-based intelligent data generation processing systems and the emergence of new scenarios, the technical solutions provided by this embodiment are also applicable to similar technical problems.

[0028] Please see Figure 2 , Figure 2 This is a flowchart illustrating an artificial intelligence-based intelligent data generation processing method provided in an embodiment of the present invention. This artificial intelligence-based intelligent data generation processing method can be... Figure 1 The intelligent data generation processing system 100 shown is executed, and the intelligent data generation processing system 100 may include servers, etc. This artificial intelligence-based intelligent data generation processing method may include at least the following steps:

[0029] 200. Obtain information on production tasks to be executed and multimodal data;

[0030] 210. Based on the pre-accessed first artificial intelligence model, extract personnel state characteristics corresponding to video data, including at least action standard, action frequency, and fatigue index;

[0031] 220. Based on the pre-accessed second artificial intelligence model, extract the device health status features corresponding to the sensor time series data;

[0032] 230. Based on the pre-accessed multimodal feature fusion model, personnel status features, equipment health status features, material inventory data and personnel skill file data are fused to obtain performance indicators, which include personnel comprehensive efficiency, equipment comprehensive status and material availability status.

[0033] 240. Based on the pre-access scheduling effect prediction model, the prediction results are generated according to the efficiency indicators and the production task information to be executed. The prediction results include the predicted value of personnel status, the predicted value of equipment failure probability, and the predicted time of production task completion.

[0034] In this embodiment, multimodal data may include at least video data, sensor time-series data, material inventory data, and personnel skill profile data.

[0035] This embodiment can acquire real-time video streams of personnel operations by using visual sensors deployed on the production site to obtain video data; this embodiment can also acquire high-dimensional time-series data of equipment operation by using IoT sensors integrated into the production equipment to obtain sensor time-series data; this embodiment can also acquire dynamic change data of material inventory and structured data corresponding to personnel skill files in real time from the manufacturing execution system.

[0036] This embodiment, based on a multimodal feature fusion model, can integrate isolated production factor information such as personnel, equipment, and materials, and intelligently assess the mutual influence between different production factor information to obtain performance indicators including comprehensive personnel efficiency, comprehensive equipment status, and material availability status. Furthermore, by using comprehensive performance indicators that better align with actual production patterns, it provides a comprehensive and reliable basis for scheduling. Specifically, by constructing a scheduling effect prediction model, it predicts personnel status curves, equipment failure probability predictions, and production task completion times. This embodiment not only deeply integrates multimodal production data but also accurately quantifies and predicts future scheduling effects, enabling managers to accurately anticipate various potential problems before scheduling execution, thereby improving the utilization efficiency of production resources and the robustness of the entire production system.

[0037] In some embodiments, please refer to Figure 3 , Figure 3 This is a flowchart illustrating the extraction of personnel state features according to an embodiment of the present invention. Based on a pre-accessed first artificial intelligence model, personnel state features corresponding to video data are extracted, including at least action standardization, action frequency, and fatigue index, including:

[0038] 300. Obtain the keypoint coordinate sequence corresponding to the video data;

[0039] 310. Based on the trained temporal autoencoder neural network, calculate the root mean square error between the key point coordinate sequence corresponding to the video data and the reconstructed output sequence. The root mean square error is the motion standard.

[0040] 320. Based on preset core action key points, determine the action frequency of the core action key points within a unit of time according to the key point coordinate sequence corresponding to the video data;

[0041] 330. Obtain facial region images based on video data, perform ROI analysis on the facial region images to obtain ROI analysis data, and determine the fatigue index corresponding to the ROI analysis data based on a pre-trained fatigue assessment model.

[0042] In this embodiment, Region of Interest (ROI) analysis involves selecting a facial region image and then further selecting from the facial region image to obtain a sub-region, such as the facial skin region or the eye region. The fatigue index can be the output data of a pre-trained fatigue assessment model after performing fatigue assessment analysis on the ROI analysis data. The fatigue assessment model can be a classification model, etc.

[0043] In some embodiments, based on a trained temporal autoencoder neural network, the root mean square error (RMSE) between the keypoint coordinate sequence corresponding to the video data and the reconstructed output sequence is calculated, where the RMSE is the action standard. This includes: acquiring a standard operation video and extracting the keypoint coordinate sequence corresponding to the standard operation video and the keypoint coordinate sequence corresponding to the video data using a convolutional neural network; training the temporal autoencoder neural network based on the keypoint coordinate sequence corresponding to the standard operation video to obtain a trained temporal autoencoder neural network; inputting the keypoint coordinate sequence corresponding to the video data into the trained temporal autoencoder neural network to obtain a reconstructed output sequence; and calculating the root mean square error (RMSE) between the keypoint coordinate sequence corresponding to the video data and the reconstructed output sequence, where the RMSE is the action standard.

[0044] This embodiment can extract features from video data using a convolutional neural network to obtain the key point coordinate sequence corresponding to the human body in the video data; the trained temporal autoencoder neural network in this embodiment can be determined by training it based on a standard operation video dataset.

[0045] In some embodiments, determining the motion frequency of core motion key points within a unit duration based on preset core motion key points and the key point coordinate sequence corresponding to the video data includes: acquiring displacement signals of core motion key points based on video data; performing autocorrelation analysis on the displacement signals and determining the periodic time of the motion by locating the peak value of the autocorrelation function; performing autocorrelation analysis on the displacement signals to obtain an autocorrelation function and determining the periodic peak value in the autocorrelation function that satisfies a preset threshold; determining the position of the periodic peak value and determining the periodic duration of the core motion key points based on the position of the periodic peak value; determining the number of cycles of core motion key points within a preset time window based on the periodic duration of the core motion key points; calculating the ratio between the number of cycles of core motion key points within the preset time window and the preset time window, where the ratio is the motion frequency of the core motion key points within a unit duration.

[0046] In this embodiment, autocorrelation analysis can be performed by first determining the translation signal after the displacement signal undergoes time-delay translation, and then calculating the similarity measure between the displacement signal and the translation signal to obtain the autocorrelation function.

[0047] In some embodiments, facial region images are acquired based on video data, ROI analysis is performed on the facial region images to obtain ROI analysis data, and a fatigue index corresponding to the ROI analysis data is determined based on a pre-trained fatigue assessment model. This includes: determining facial region images corresponding to several consecutive frames based on video data, and determining the average pixel intensity change of the skin region based on the facial region images corresponding to the several consecutive frames to obtain a photoplethysmography (PPG) signal; performing spectral analysis on the PPG signal to determine the frequency ratio of low-frequency power to high-frequency power corresponding to the PPG signal, where the frequency ratio is a heart rate variability index; determining the number of blinks per unit time, the average blink duration, and the proportion of eyelid closure time based on the facial region images corresponding to the several consecutive frames; inputting the heart rate variability index, the number of blinks per unit time, the average blink duration, and the proportion of eyelid closure time into a pre-trained fatigue assessment model, and the pre-trained fatigue assessment model outputting the fatigue index corresponding to the ROI analysis data.

[0048] In this embodiment, the heart rate variability index can be the frequency ratio of low-frequency power to high-frequency power corresponding to the photoplethysmography signal; the number of blinks per unit time, the average blink duration, and the proportion of eyelid closure time can be determined based on facial region images corresponding to several consecutive frames, and by determining the change in the pupil center coordinates; the proportion of eyelid closure time can be the ratio of the eyelid closure duration to the preset duration within a preset duration; the average blink duration can be the average of the blink durations of several blink actions calculated after statistically analyzing the blink duration from the start to the end of each blink action within a preset duration. In this embodiment, the heart rate variability index, the number of blinks per unit time, the average blink duration, and the proportion of eyelid closure time can be input into a pre-trained fatigue assessment model, and the pre-trained fatigue assessment model can output a physiological fatigue index.

[0049] In some embodiments, please refer to Figure 4 , Figure 4 This is a schematic flowchart illustrating the process of extracting device health status features corresponding to sensor time-series data according to an embodiment of the present invention. Based on a pre-accessed second artificial intelligence model, the extraction of device health status features corresponding to sensor time-series data includes:

[0050] 400. Preprocess the sensor time series data to obtain preprocessed sensor time series data;

[0051] 410. Determine the statistical characteristics of the preprocessed sensor time series data through feature engineering. The statistical characteristics include time-domain statistical characteristics and frequency-domain statistical characteristics.

[0052] 420. Input the statistical features into the second artificial intelligence model. The second artificial intelligence model outputs the classification probability of the equipment health status. The classification probability of the equipment health status is the equipment health status feature.

[0053] In this embodiment, the second artificial intelligence model can be an ensemble learning classifier based on a tree model, etc. The input of the second artificial intelligence model is statistical features constructed through feature engineering, and the output of the second artificial intelligence model is the classification probability of the device health status; the sensor time series data can include at least the device vibration amplitude, device temperature, and device current, etc.

[0054] In this embodiment, the multimodal feature fusion model may include a feature embedding layer, an attention-based fusion layer, and a parallel multi-task output layer. The parallel multi-task output layer may include at least three fully connected networks. The feature embedding layer is used to map personnel status features, equipment health status features, material inventory data, and personnel skill profile data to a unified feature space to obtain several modal features. The attention-based fusion layer is used to generate an attention weight for each modal feature. Each modal feature is multiplied by its corresponding attention weight to obtain a weighted feature vector. All weighted feature vectors are summed to obtain a summed fusion vector. The parallel multi-task output layer includes at least three fully connected networks, which are used to regress three performance indicators—personnel comprehensive efficiency, equipment comprehensive status, and material availability status—based on the summed fusion vector.

[0055] In some embodiments, based on a pre-accessed scheduling effect prediction model, a prediction result is generated according to performance indicators and information on production tasks to be executed. This includes: determining a sequence of production tasks to be executed based on the information on production tasks to be executed, the sequence including several sub-tasks, each sub-task including at least task type, task intensity, task duration, task equipment, and corresponding equipment load information; encoding each sub-task to obtain a feature vector corresponding to each sub-task; concatenating the overall personnel efficiency with the feature vectors corresponding to each sub-task to obtain several concatenated vectors; inputting these concatenated vectors into a pre-trained personnel status prediction model in chronological order, the personnel status prediction model outputs predicted personnel status values ​​corresponding to each sub-task; and concatenating the equipment load information and overall equipment status corresponding to each sub-task into a pre-trained equipment fault prediction model to obtain a concatenated equipment feature vector; the equipment feature vector is used as input to the equipment fault prediction model, the equipment fault prediction model outputs predicted equipment fault probability values.

[0056] In this embodiment, the scheduling effect prediction model may include a personnel status prediction model and an equipment failure prediction model, etc.; the personnel status prediction model is used to determine the personnel status prediction value based on the feature vectors corresponding to each sub-task and the overall personnel efficiency; the equipment failure prediction model is used to determine the equipment failure probability prediction value based on the overall equipment status and the equipment load information corresponding to the sub-task.

[0057] In some embodiments, based on a pre-accessed scheduling effect prediction model, prediction results are generated according to efficiency indicators and information on production tasks to be executed, including: determining the average comprehensive efficiency value of the personnel corresponding to each sub-task; based on a pre-trained task completion time prediction model, determining the completion prediction time corresponding to each sub-task according to the task hours of the sub-task and the average comprehensive efficiency value of the personnel corresponding to the sub-task; and determining the production task completion prediction time according to the completion prediction time corresponding to each sub-task.

[0058] In this embodiment, the scheduling effect prediction model also includes a task completion time prediction model, etc. The task completion time prediction model is used to determine the completion prediction time of each sub-task based on the task hours of the sub-task and the average comprehensive efficiency value of the personnel corresponding to the sub-task. Then, the completion prediction times of each sub-task are added together to determine the production task completion prediction time.

[0059] In this embodiment, the personnel status prediction model can be a time-series prediction model based on long short-term memory networks, etc.; the equipment failure prediction model can be a probability classification model based on gradient boosting trees, etc.; and the task completion time prediction model can be a regression model based on gradient boosting trees, etc.

[0060] This embodiment of the specification can extract personnel status features corresponding to video data, including at least motion standardization, motion frequency, and fatigue index, based on a first artificial intelligence model; it can also extract equipment health status features corresponding to sensor time-series data based on a pre-accessed second artificial intelligence model; then, based on a pre-accessed multimodal feature fusion model, it can fuse personnel status features, equipment health status features, material inventory data, and personnel skill profile data to obtain performance indicators; this embodiment of the specification can also generate prediction results based on a pre-accessed scheduling effect prediction model, according to performance indicators and information on production tasks to be executed, including predicted personnel status values, predicted equipment failure probability values, and predicted production task completion times. This specification's embodiments, based on a multimodal feature fusion model, can integrate isolated production factor information such as personnel, equipment, and materials. It can also intelligently assess the mutual influence between different production factor information, obtaining performance indicators including overall personnel efficiency, overall equipment status, and material availability. Furthermore, by using comprehensive performance indicators that better align with actual production patterns, it provides a comprehensive and reliable basis for scheduling. Specifically, by constructing a scheduling effect prediction model, it predicts personnel status curves, equipment failure probability values, and production task completion times. This specification's embodiments not only deeply integrate multimodal production data but also accurately quantify and predict future scheduling effects, enabling managers to accurately anticipate various potential problems before scheduling execution. This specification's embodiments improve the utilization efficiency of production resources and the robustness of the entire production system.

[0061] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0062] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an intelligent data generation system based on artificial intelligence, provided as an embodiment of this specification.

[0063] like Figure 5 As shown, the AI-based intelligent data processing system may include at least a data acquisition module 500, a first feature module 510, a second feature module 520, a feature fusion module 530, and a prediction module 540, wherein:

[0064] The data acquisition module 500 is used to acquire information on production tasks to be executed and multimodal data. The multimodal data includes at least video data, sensor time-series data, material inventory data, and personnel skill profile data.

[0065] The first feature module 510 is used to extract personnel state features corresponding to video data, including at least action standard, action frequency and fatigue index, based on the pre-accessed first artificial intelligence model.

[0066] The second feature module 520 is used to extract device health status features corresponding to sensor time series data based on the pre-accessed second artificial intelligence model.

[0067] The feature fusion module 530 is used to fuse personnel status features, equipment health status features, material inventory data and personnel skill file data based on the pre-accessed multimodal feature fusion model to obtain performance indicators, which include personnel comprehensive efficiency, equipment comprehensive status and material availability status.

[0068] The prediction module 540 is used to generate prediction results based on the pre-accessed scheduling effect prediction model, according to the efficiency indicators and the production task information to be executed. The prediction results include personnel status prediction values, equipment failure probability prediction values, and production task completion prediction times.

[0069] In some embodiments, the first feature module 510 includes a feature extraction module, which is used to: acquire the key point coordinate sequence corresponding to the video data; calculate the root mean square error between the key point coordinate sequence corresponding to the video data and the reconstructed output sequence based on a trained temporal autoencoder neural network, wherein the root mean square error is the motion standard; determine the motion frequency of the core motion key points within a unit time period based on preset core motion key points and the key point coordinate sequence corresponding to the video data; acquire a facial region image based on the video data, perform ROI analysis on the facial region image to obtain ROI analysis data, and determine the fatigue index corresponding to the ROI analysis data based on a pre-trained fatigue assessment model.

[0070] In some embodiments, the feature extraction module includes a standardization determination module, which is used to: acquire a standard operation video, and extract the key point coordinate sequence corresponding to the standard operation video and the key point coordinate sequence corresponding to the video data based on a convolutional neural network; train a temporal autoencoder neural network based on the key point coordinate sequence corresponding to the standard operation video to obtain a trained temporal autoencoder neural network; input the key point coordinate sequence corresponding to the video data into the trained temporal autoencoder neural network to obtain a reconstructed output sequence; and calculate the root mean square error between the key point coordinate sequence corresponding to the video data and the reconstructed output sequence, wherein the root mean square error is the action standardization.

[0071] In some embodiments, the feature extraction module includes an action frequency module, which is used to: acquire displacement signals of core action key points based on video data; perform autocorrelation analysis on the displacement signals and determine the periodicity of the action by locating the peak value of the autocorrelation function; perform autocorrelation analysis on the displacement signals to obtain an autocorrelation function and determine the periodic peak value in the autocorrelation function that satisfies a preset threshold; determine the position of the periodic peak value and determine the periodic duration of the core action key point based on the position of the periodic peak value; determine the number of periods of the core action key point in the preset time window based on the periodic duration of the core action key point; calculate the ratio between the number of periods of the core action key point in the preset time window and the preset time window, the ratio being the action frequency of the core action key point within a unit time duration.

[0072] In some embodiments, the feature extraction module includes a fatigue index module, which is used to: determine facial region images corresponding to several consecutive frames based on video data, and determine the average pixel intensity change of the skin region based on the facial region images corresponding to several consecutive frames to obtain a photoplethysmography (PPG) signal; perform spectral analysis on the PPG signal to determine the frequency ratio of low-frequency power to high-frequency power corresponding to the PPG signal, where the frequency ratio is a heart rate variability index; determine the number of blinks per unit time, the average blink duration, and the proportion of eyelid closure time based on the facial region images corresponding to several consecutive frames; input the heart rate variability index, the number of blinks per unit time, the average blink duration, and the proportion of eyelid closure time into a pre-trained fatigue assessment model, and the pre-trained fatigue assessment model outputs the fatigue index corresponding to the ROI analysis data.

[0073] In some embodiments, the second artificial intelligence model is an ensemble learning classifier based on a tree model, the input of the second artificial intelligence model is statistical features constructed through feature engineering, and the output of the second artificial intelligence model is the classification probability of the device health status; the sensor time series data includes at least the device vibration amplitude, device temperature, and device current.

[0074] In some embodiments, the second feature module 520 includes a second feature submodule, which is used to: preprocess the sensor time-series data to obtain preprocessed sensor time-series data; determine the statistical features corresponding to the preprocessed sensor time-series data through feature engineering, the statistical features including time-domain statistical features and frequency-domain statistical features; input the statistical features into a second artificial intelligence model, the second artificial intelligence model outputs the classification probability of the device health status, and the classification probability of the device health status is the device health status feature.

[0075] In some embodiments, the multimodal feature fusion model includes: a feature embedding layer for mapping personnel status features, equipment health status features, material inventory data, and personnel skill profile data to a feature space of a unified dimension to obtain several modal features; an attention-based fusion layer for generating an attention weight for each modal feature; multiplying each modal feature with its corresponding attention weight to obtain a weighted feature vector; summing all weighted feature vectors to obtain a fusion vector after summing; and a parallel multi-task output layer including at least three fully connected networks, which are respectively used to regress three performance indicators: overall personnel efficiency, overall equipment status, and material availability status, based on the fusion vector after summing.

[0076] In some embodiments, the scheduling effect prediction model includes a personnel status prediction model and an equipment failure prediction model; the prediction module 540 includes a first prediction module, which is used to: determine a sequence of production tasks to be executed based on the information of the production tasks to be executed, the sequence of production tasks to be executed includes several sub-tasks, and each sub-task includes at least task type, task intensity, task working hours, task equipment, and corresponding equipment load information; obtain feature vectors corresponding to each sub-task by encoding each sub-task; concatenate the overall personnel efficiency with the feature vectors corresponding to each sub-task to obtain several concatenated vectors; input the several concatenated vectors into the personnel status prediction model in chronological order based on the pre-trained personnel status prediction model, and the personnel status prediction model outputs the personnel status prediction value corresponding to each sub-task; and concatenate the equipment load information and overall equipment status corresponding to the sub-tasks based on the pre-trained equipment failure prediction model to obtain a concatenated equipment feature vector; the equipment feature vector is used to input the equipment failure prediction model, and the equipment failure prediction model outputs the equipment failure probability prediction value.

[0077] In some embodiments, the scheduling effect prediction model further includes a task completion time prediction model; the prediction module 540 includes a second prediction module, which is used to: determine the average comprehensive efficiency value of the personnel corresponding to each sub-task; determine the completion prediction time corresponding to each sub-task based on the pre-trained task completion time prediction model, according to the task hours of the sub-task and the average comprehensive efficiency value of the personnel corresponding to the sub-task; and determine the production task completion prediction time based on the completion prediction time corresponding to each sub-task.

[0078] Based on the content of an AI-based intelligent data processing system in several embodiments of this specification, it can be seen that the embodiments of this specification can integrate isolated production factor information such as personnel, equipment, and materials based on a multimodal feature fusion model, and can intelligently evaluate the mutual influence between different production factor information to obtain performance indicators including comprehensive personnel efficiency, comprehensive equipment status, and material availability status. Furthermore, by using comprehensive performance indicators that are more in line with actual production patterns, a comprehensive and reliable basis for scheduling can be provided. Specifically, by constructing a scheduling effect prediction model, the system predicts personnel status prediction curves, equipment failure probability prediction values, and production task completion prediction times. The embodiments of this specification not only deeply integrate multimodal production data but also accurately quantify and predict future scheduling effects, enabling managers to accurately anticipate various potential problems before scheduling execution. These embodiments improve the utilization efficiency of production resources and the robustness of the entire production system.

[0079] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of the AI-based intelligent data generation processing system are relatively simple in description because they are fundamentally similar to the AI-based intelligent data generation processing method embodiments; relevant parts can be referred to the descriptions in the method embodiments.

[0080] Please see Figure 6 The diagram shown is a structural schematic of an electronic device for an intelligent data processing system based on artificial intelligence, as provided in an embodiment of this specification.

[0081] like Figure 6 As shown, the electronic device 600 may include at least one processor 610, at least one network interface 640, a user interface 630, a memory 650, and at least one communication bus 620.

[0082] The communication bus 620 can be used to realize the connection and communication of the above components.

[0083] The user interface 630 may include buttons, and the optional user interface may also include a standard wired interface or a wireless interface.

[0084] The network interface 640 may include, but is not limited to, Bluetooth modules, NFC modules, ZigBee modules, and UWB modules.

[0085] The processor 610 may include one or more processing cores. The processor 610 connects to various parts within the electronic device 600 using various interfaces and lines. It performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 650, and by calling data stored in the memory 650. Optionally, the processor 610 may be implemented using at least one hardware form selected from DSP, FPGA, and PLA. The processor 610 may integrate one or more combinations of CPU and GPU. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display on the screen.

[0086] The memory 650 may include RAM or ROM. Optionally, the memory 650 may include a non-transitory computer-readable medium. The memory 650 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 650 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory 650 may also be at least one storage device located remotely from the aforementioned processor 610. As a computer storage medium, the memory 650 may include an operating system, a communication module, a user interface module, and an AI-based intelligent data generation processing application. The processor 610 may be used to call the AI-based intelligent data generation processing application stored in the memory 650 and execute the AI-based intelligent data generation processing steps mentioned in the foregoing embodiments.

[0087] This specification also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform the above-described instructions. Figures 2-5 One or more steps in the illustrated embodiment. If the constituent modules of the above-described electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0088] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this specification is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0089] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation can be combined arbitrarily.

[0090] The above embodiments are merely preferred embodiments described in this specification and are not intended to limit the scope of this specification. Any modifications and improvements made by those skilled in the art to the technical solutions of this specification without departing from the spirit of this specification should fall within the protection scope defined by the claims of this specification.

Claims

1. A data processing method based on artificial intelligence, characterized in that, include: Acquire information on production tasks to be executed and multimodal data, wherein the multimodal data includes at least video data, sensor time-series data, material inventory data, and personnel skill profile data; Based on the pre-accessed first artificial intelligence model, extract the personnel state characteristics corresponding to the video data, including at least action standard, action frequency, and fatigue index. Based on the pre-accessed second artificial intelligence model, the device health status features corresponding to the sensor time-series data are extracted; Based on the pre-accessed multimodal feature fusion model, the personnel status features, the equipment health status features, the material inventory data and the personnel skill file data are fused to obtain performance indicators, which include personnel comprehensive efficiency, equipment comprehensive status and material availability status. Based on the pre-accessed scheduling effect prediction model, prediction results are generated according to the efficiency indicators and the production task information to be executed. The prediction results include personnel status prediction values, equipment failure probability prediction values, and production task completion prediction times. The scheduling effectiveness prediction model includes a personnel status prediction model and an equipment failure prediction model; the pre-access-based scheduling effectiveness prediction model generates prediction results based on the performance indicators and the production task information to be executed, including: The production task sequence to be executed is determined based on the production task information to be executed. The production task sequence to be executed includes several sub-tasks. Each sub-task includes at least the task type, task intensity, task duration, task equipment, and corresponding equipment load information. By encoding each subtask, we obtain the feature vector corresponding to each subtask. The overall efficiency of personnel is concatenated with the feature vectors corresponding to each subtask to obtain several concatenated vectors; Based on the pre-trained personnel status prediction model, the several concatenated vectors are input into the personnel status prediction model in time sequence, and the personnel status prediction model outputs the personnel status prediction value corresponding to each sub-task. Based on the pre-trained equipment fault prediction model, the equipment load information and overall equipment status corresponding to the sub-task are concatenated to obtain the concatenated equipment feature vector; the equipment feature vector is used as input to the equipment fault prediction model, and the equipment fault prediction model outputs the predicted equipment fault probability value. The scheduling effectiveness prediction model also includes a task completion time prediction model; the pre-access-based scheduling effectiveness prediction model generates prediction results based on the performance indicators and the production task information to be executed, including: Determine the average overall personnel efficiency value for the personnel performing each sub-task; Based on the pre-trained task completion time prediction model, the completion prediction time of each sub-task is determined according to the task time of the sub-task and the average comprehensive efficiency value of the personnel corresponding to the sub-task. The production task completion forecast time is determined based on the completion forecast time corresponding to each sub-task.

2. The method according to claim 1, characterized in that, The first artificial intelligence model based on pre-access extracts personnel state features corresponding to the video data, including at least action standardization, action frequency, and fatigue index, including: Obtain the key point coordinate sequence corresponding to the video data; Based on the trained temporal autoencoder neural network, the root mean square error between the key point coordinate sequence corresponding to the video data and the reconstructed output sequence is calculated, and the root mean square error is the motion standard. Based on preset core action key points, the action frequency of the core action key points within a unit of time is determined according to the key point coordinate sequence corresponding to the video data. Facial region images are obtained based on the video data, ROI analysis is performed on the facial region images to obtain ROI analysis data, and the fatigue index corresponding to the ROI analysis data is determined based on a pre-trained fatigue assessment model.

3. The method according to claim 2, characterized in that, The trained temporal autoencoder neural network calculates the root mean square error (RMSE) between the keypoint coordinate sequence corresponding to the video data and the reconstructed output sequence. The RMSE is the motion standard deviation, including: Obtain a standard operation video, and extract the key point coordinate sequence corresponding to the standard operation video and the key point coordinate sequence corresponding to the video data based on a convolutional neural network; Based on the key point coordinate sequence corresponding to the standard operation video, the temporal autoencoder neural network is trained to obtain the trained temporal autoencoder neural network. The key point coordinate sequence corresponding to the video data is input into the trained temporal autoencoder neural network to obtain the reconstructed output sequence. Calculate the root mean square error between the key point coordinate sequence corresponding to the video data and the reconstructed output sequence, where the root mean square error is the motion standard.

4. The method according to claim 2, characterized in that, The method of determining the motion frequency of the core motion key points within a unit time period based on preset core motion key points and the key point coordinate sequence corresponding to the video data includes: Based on the video data, obtain the displacement signals of the key points of the core action; Autocorrelation analysis is performed on the displacement signal, and the period of the action is determined by locating the peak value of the autocorrelation function; Autocorrelation analysis is performed on the displacement signal to obtain the autocorrelation function, and periodic peak values ​​that satisfy a preset threshold in the autocorrelation function are determined. Determine the position of the periodic peak, and determine the periodic duration of the key point of the core action based on the position of the periodic peak; Based on a preset time window, the number of cycles of the core action key points in the preset time window is determined according to the cycle duration of the core action key points. Calculate the number of cycles of the core action key points within the preset time window and the ratio between the preset time windows. The ratio is the action frequency of the core action key points within a unit of time.

5. The method according to claim 2, characterized in that, The process of acquiring facial region images based on the video data, performing ROI analysis on the facial region images to obtain ROI analysis data, and determining the fatigue index corresponding to the ROI analysis data based on a pre-trained fatigue assessment model includes: Based on the video data, facial region images corresponding to several consecutive frames are determined, and the average pixel intensity change of the skin region is determined according to the facial region images corresponding to the several consecutive frames to obtain the photoplethysmography signal. Spectral analysis is performed on the photoplethysmography signal to determine the frequency ratio of the low-frequency power to the high-frequency power of the photoplethysmography signal, and the frequency ratio is a heart rate variability index. The number of blinks per unit time, the average blink duration, and the proportion of eyelid closure time are determined based on the facial region images corresponding to the several consecutive frames. The heart rate variability index, blink count per unit time, average blink duration, and eyelid closure time percentage are input into a pre-trained fatigue assessment model, which outputs the fatigue index corresponding to the ROI analysis data.

6. The method according to claim 1, characterized in that, The second artificial intelligence model is an ensemble learning classifier based on a tree model. The input of the second artificial intelligence model is statistical features constructed through feature engineering, and the output of the second artificial intelligence model is the classification probability of the device health status. The sensor time-series data includes at least the device vibration amplitude, device temperature, and device current. The second artificial intelligence model based on pre-access extracts device health status features corresponding to the sensor time-series data, including: The sensor time series data is preprocessed to obtain preprocessed sensor time series data; The statistical features corresponding to the preprocessed sensor time-series data are determined by feature engineering. The statistical features include time-domain statistical features and frequency-domain statistical features. The statistical features are input into the second artificial intelligence model, and the second artificial intelligence model outputs the classification probability of the device health status. The classification probability of the device health status is the device health status feature.

7. The method according to claim 1, characterized in that, The multimodal feature fusion model includes: The feature embedding layer maps the personnel status features, equipment health status features, material inventory data, and personnel skill profile data to a unified feature space to obtain several modal features. The attention-based fusion layer generates an attention weight for each modal feature. Each modal feature is multiplied by its corresponding attention weight to obtain a weighted feature vector. All weighted feature vectors are summed to obtain a fusion vector. The parallel multi-task output layer includes at least three fully connected networks, which are used to regress three performance indicators—personnel overall efficiency, equipment overall status, and material availability—based on the fusion vector.

8. An intelligent data processing system based on artificial intelligence, characterized in that, include: The data acquisition module is used to acquire information on production tasks to be executed and multimodal data, including at least video data, sensor time-series data, material inventory data, and personnel skill profile data. The first feature module is used to extract personnel state features corresponding to the video data, including at least action standard, action frequency, and fatigue index, based on the pre-accessed first artificial intelligence model. The second feature module is used to extract device health status features corresponding to the sensor time-series data based on the pre-accessed second artificial intelligence model. The feature fusion module is used to fuse the personnel status features, the equipment health status features, the material inventory data and the personnel skill file data based on the pre-accessed multimodal feature fusion model to obtain performance indicators, which include personnel comprehensive efficiency, equipment comprehensive status and material availability status. The prediction module is used to generate prediction results based on the pre-accessed scheduling effect prediction model, according to the efficiency indicators and the production task information to be executed. The prediction results include personnel status prediction values, equipment failure probability prediction values, and production task completion prediction times. The scheduling effectiveness prediction model includes a personnel status prediction model and an equipment failure prediction model; the pre-access-based scheduling effectiveness prediction model generates prediction results based on the performance indicators and the production task information to be executed, including: The production task sequence to be executed is determined based on the production task information to be executed. The production task sequence to be executed includes several sub-tasks. Each sub-task includes at least the task type, task intensity, task duration, task equipment, and corresponding equipment load information. By encoding each subtask, we obtain the feature vector corresponding to each subtask. The overall efficiency of personnel is concatenated with the feature vectors corresponding to each subtask to obtain several concatenated vectors; Based on the pre-trained personnel status prediction model, the several concatenated vectors are input into the personnel status prediction model in time sequence, and the personnel status prediction model outputs the personnel status prediction value corresponding to each sub-task. Based on the pre-trained equipment fault prediction model, the equipment load information and overall equipment status corresponding to the sub-task are concatenated to obtain the concatenated equipment feature vector; the equipment feature vector is used as input to the equipment fault prediction model, and the equipment fault prediction model outputs the predicted equipment fault probability value. The scheduling effectiveness prediction model also includes a task completion time prediction model; the pre-access-based scheduling effectiveness prediction model generates prediction results based on the performance indicators and the production task information to be executed, including: Determine the average overall personnel efficiency value for the personnel performing each sub-task; Based on the pre-trained task completion time prediction model, the completion prediction time of each sub-task is determined according to the task time of the sub-task and the average comprehensive efficiency value of the personnel corresponding to the sub-task. The production task completion forecast time is determined based on the completion forecast time corresponding to each sub-task.

Images