AI enterprise operation intelligent system and method based on multi-modal fusion

Abstract

The invention relates to the technical field of intelligent operation management, and discloses an AI enterprise operation intelligent system and method based on multi-modal fusion, and the method comprises the steps: obtaining original enterprise operation data, and carrying out the preprocessing of the original enterprise operation data, and obtaining a multi-modal enterprise operation data set; carrying out feature extraction processing on the multi-modal enterprise operation data set by adopting a lightweight model; performing modal fusion processing on the depth feature representation by adopting a hierarchical attention mechanism; analyzing and processing the fused multi-modal features by adopting a knowledge graph construction algorithm; performing decision optimization processing by adopting meta-learning and reinforcement learning algorithms; analyzing and processing the intelligent decision scheme by adopting a natural language processing technology; a micro-service architecture is adopted for packaging, and a final enterprise operation intelligent system is obtained; according to the invention, by deploying the intelligent enterprise operation system, automatic management of multiple business links of an enterprise is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent operation management, more specifically, it relates to an AI enterprise operation intelligent system and method based on multi-modal fusion. BACKGROUND

[0002] With the advancement of global digital transformation, small and medium-sized manufacturing enterprises are facing unprecedented challenges in operation management. These enterprises usually involve the collaborative work of multiple business departments, including business department, project department, R&D department, procurement department, planning department, production department, quality department, administrative department, financial department, etc. Different information systems are used by each department for daily operation management. Traditional enterprise management methods mainly rely on manual supervision and manual operation, which has the problems of low efficiency, easy to make mistakes, information island, etc. The execution and tracking of tasks in each department mainly rely on manual supervision, which is easy to miss, leading to a decrease in efficiency, and even affecting customer experience. The implementation and follow-up of work goals also face the risk of omission under manual supervision, with declining efficiency, affecting customer trust.

[0003] Although the existing enterprise resource planning system and office automation system have improved management efficiency to some extent, there are still certain technical limitations. These systems mainly process structured data, and lack effective processing capabilities for a large amount of unstructured data generated in enterprise operation, such as customer emails, meeting records, product drawings, quality inspection photos, etc. Secondly, there is a lack of effective data integration mechanism between different systems, forming a serious information island problem, leading to information transmission relying on manual transcription, low efficiency and easy to make mistakes. In addition, the existing system lacks intelligent decision support function, and key business decisions still mainly rely on the experience judgment of management personnel, which cannot fully utilize the large amount of operation data accumulated by the enterprise.

[0004] The specific technical problems existing in the current technical background include: insufficient multi-modal data fusion processing capability, unable to effectively integrate text, image, voice and other different types of enterprise operation data; lack of cross-system data integration technology, difficult to realize seamless data interaction between heterogeneous systems; weak intelligent decision execution link, lack of closed-loop mechanism from data analysis to automatic execution; limited natural language interaction capability, users need to master complex system operation methods; enterprise knowledge management is scattered, and personnel flow easily leads to loss of key information. Therefore, an enterprise operation intelligent system capable of effectively fusing multi-modal data, providing intelligent decision support, and realizing natural language interaction is urgently needed to solve the technical bottlenecks of traditional management methods and improve enterprise operation efficiency and competitiveness. SUMMARY

[0005] The application provides an AI enterprise operation intelligent system and method based on multi-modal fusion, which solves the technical problems of information island, low efficiency of artificial supervision, and scattered knowledge management of traditional enterprise management systems in related technologies.

[0006] The application provides an AI enterprise operation intelligent method based on multi-modal fusion, which comprises: Obtaining original enterprise operation data, preprocessing the original enterprise operation data to obtain a multi-modal enterprise operation data set; Performing feature extraction processing on the multi-modal enterprise operation data set using a lightweight model to obtain deep feature representations of three modalities of text, image, and voice; Performing modal fusion processing on the deep feature representations using a hierarchical attention mechanism to obtain fused multi-modal features; Performing analysis and processing on the fused multi-modal features using a knowledge graph construction algorithm to obtain a real-time updated enterprise knowledge graph; Performing decision optimization processing on the real-time updated enterprise knowledge graph using a meta-learning and reinforcement learning algorithm to obtain an intelligent decision scheme; Performing analysis and processing on the intelligent decision scheme using natural language processing technology to obtain a natural language system response result; Encapsulating the multi-modal enterprise operation data set, the deep feature representations, the fused multi-modal features, the enterprise knowledge graph, the intelligent decision scheme, and the natural language system response result using a micro-service architecture, ensuring system operation through load balancing technology, and obtaining the final enterprise operation intelligent system.

[0007] In a preferred embodiment, the original enterprise operation data is obtained specifically by: Real-time acquisition of order numbers, customer information, product specifications, and delivery requirement structured data fields in the ERP system through a standard SQL interface; Using Apache POI technology to parse production plan tables in Excel format and extracting key plan nodes, responsible persons, and completion status information; Using a multi-threaded crawler technology to extract customer emails from enterprise mailboxes and using regular expressions to identify order numbers, product models, and technical requirement key information in the emails.

[0008] In a preferred embodiment, the lightweight model performs feature extraction processing specifically by: Obtaining text data in the multi-modal enterprise operation data set, performing word segmentation and encoding processing on the text data source using a lightweight BERT-Base-Chinese model to obtain a 768-dimensional text feature vector; Image data from a multimodal enterprise operation dataset was acquired, and features were extracted from the image data using the MobileNetV3-Large architecture. Fine-tuning was performed on the ImageNet pre-trained model to obtain a 1000-dimensional image feature vector. We acquire speech data from a multimodal enterprise operation dataset, perform Mel-spectrum transformation on the speech data, and use a lightweight Wav2Vec2 model to extract features from the transformed Mel-spectrum to obtain a 512-dimensional speech feature vector.

[0009] In a preferred embodiment, the hierarchical attention mechanism includes two stages: intramodal self-attention and cross-modal attention, specifically including: The intramodal self-attention stage calculates self-attention weights based on feature sequences from three modalities: text, image, and speech. The cross-modal attention stage calculates the cross-modal attention weights among text, image, and speech modalities using a learnable weight matrix; By integrating self-attention weights and cross-modal attention weights through weighted summation, fused multimodal features are obtained.

[0010] In a preferred embodiment, the knowledge graph construction algorithm analysis and processing specifically includes: The fused multimodal features are obtained, and the named entity recognition model based on CRF is used to perform entity recognition processing on the fused multimodal features to obtain an entity set; A relation extraction process based on an attention mechanism is used to extract relations from the entity set. Link entities between the entity set and existing entities in the historical enterprise knowledge graph; Based on the results of relation extraction and entity linking, a version control mechanism is used to update the enterprise knowledge graph, resulting in a real-time updated enterprise knowledge graph.

[0011] In a preferred embodiment, the decision optimization process performed by the meta-learning and reinforcement learning algorithms specifically includes: To acquire real-time updated enterprise knowledge graphs and current business status data, the meta-learning framework uses the MAML algorithm to analyze and process the knowledge graphs. By training the learning model on five related tasks to initialize parameters, an adaptive decision model is obtained. Based on the adaptive decision-making model, the near-end policy optimization (PPO) algorithm is used for reinforcement learning decision optimization. The current task queue, personnel status, and resource usage are used as the three dimensions of the state space to obtain the decision state representation. Based on the decision state representation, a discrete decision action space is defined, including three types: task allocation, resource scheduling, and priority adjustment. The optimal decision action is selected through a policy network to obtain the decision scheme. The decision-making scheme is evaluated using a reward function. The overall reward value is calculated by comprehensively considering three factors: decision-making effect, execution cost, and risk assessment, thus obtaining an intelligent decision-making scheme.

[0012] In a preferred embodiment, the natural language processing technology for response generation specifically includes: It acquires intelligent decision-making solutions and user natural language commands, and adopts an end-to-end dialogue system architecture, including four core components: natural language understanding component, dialogue management component, task execution component, and natural language generation component; The natural language understanding component uses a BERT-based intent recognition model and slot filling model for natural language understanding, and employs the BIO tagging scheme and conditional random field model for entity annotation. The dialogue management component employs a dialogue state tracking method based on graph neural networks and a dialogue management strategy based on reinforcement learning. The task execution component maps natural language intents to system operations via the API registry; The natural language generation component uses a combination of templates and neural networks to generate natural language responses, thus obtaining the natural language system response results.

[0013] In a preferred embodiment, the encapsulation using a microservice architecture specifically includes: To obtain the response results from the natural language system, a microservice architecture is adopted to encapsulate and deploy each processing module. The functional modules are encapsulated into independent service modules, including a data acquisition service module, a feature extraction service module, a modality fusion service module, a knowledge graph service module, a decision optimization service module, and a response generation service module, resulting in a set of microservice components. The microservice component set is registered and configured, and an inter-service communication mechanism and API gateway are established to obtain a distributed service architecture. Based on a distributed service architecture, load balancing technology is used to allocate and schedule traffic for each microservice component, thereby achieving elastic scaling and failover of services and obtaining a service cluster. By unifying the monitoring and management of service clusters and using service mesh technology to achieve secure communication and performance optimization between services, an intelligent enterprise operation system can be obtained.

[0014] In a preferred embodiment, the microservice architecture encapsulation also includes implementing a monitoring and logging system using the ELK technology stack, collecting and analyzing system logs in real time, monitoring system performance indicators through Prometheus, triggering an alarm mechanism when indicators are abnormal, realizing distributed tracing, and obtaining the final enterprise operation intelligent system.

[0015] This invention provides an AI-based enterprise operation intelligence system based on multimodal fusion, used to execute the aforementioned AI-based enterprise operation intelligence method based on multimodal fusion, comprising: The data acquisition module is used to acquire raw enterprise operation data and preprocess the raw enterprise operation data to obtain a multimodal enterprise operation dataset. The feature extraction module is used to perform feature extraction processing on the multimodal enterprise operation dataset using a lightweight model to obtain deep feature representations of three modalities: text, image, and speech. The multimodal fusion module is used to perform modal fusion processing on deep feature representations using a hierarchical attention mechanism to obtain fused multimodal features; The knowledge graph construction module is used to analyze and process the fused multimodal features using knowledge graph construction algorithms to obtain a real-time updated enterprise knowledge graph; The intelligent decision-making module is used to perform decision optimization processing on the real-time updated enterprise knowledge graph using meta-learning and reinforcement learning algorithms to obtain intelligent decision-making solutions; The natural language interaction module is used to analyze and process intelligent decision-making schemes using natural language processing technology to obtain the response results of the natural language system. The system architecture module is used to encapsulate multimodal enterprise operation datasets, deep feature representations, fused multimodal features, enterprise knowledge graphs, intelligent decision-making solutions, and natural language system response results using a microservice architecture. Load balancing technology ensures system operation, resulting in the final intelligent enterprise operation system.

[0016] The beneficial effects of this invention are as follows: by deploying an intelligent enterprise operation system and adopting multimodal data fusion technology and intelligent analysis and decision-making mechanism, the automated management of multiple business links of the enterprise is realized, which improves the overall operational efficiency and reduces the operating cost; Through automated email content extraction and historical data similarity analysis technology, we have achieved intelligent understanding of project requirements and automatic recommendation of project templates. Through natural language interaction and anomaly detection algorithms, real-time tracking and risk warning of project progress were achieved, improving project management efficiency and reducing the risk of project delays; and through intelligent integration technology of ERP data, customer communication records and production plan information, automatic tracking of the entire order lifecycle was achieved. Through production capacity analysis and intelligent scheduling algorithms, accurate prediction of delivery commitments and dynamic adjustment of production plans have been achieved, improving order processing efficiency and customer satisfaction. Furthermore, through real-time acquisition and fusion analysis of multimodal data, digital modeling of the production process has been realized. Through equipment status monitoring and defect detection algorithms, automatic early warning and handling of production anomalies have been achieved, improving production efficiency and product quality.

[0017] Through comprehensive analysis of employee data and intelligent matching technology, automated management of personnel allocation and task assignment has been achieved; through skills assessment and training recommendation algorithms, continuous optimization of team capabilities has been achieved, improving human resource management efficiency and employee job satisfaction. Attached Figure Description

[0018] Figure 1 This is a flowchart of an AI-based intelligent enterprise operation method based on multimodal fusion in this invention; Figure 2 This is a module diagram of an AI-based intelligent enterprise operation system based on multimodal fusion in this invention. Detailed Implementation

[0019] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0020] At least one embodiment of the present invention discloses an AI-based enterprise operation intelligence method based on multimodal fusion, such as... Figure 1 As shown, it includes: Step 1: Obtain raw enterprise operation data and preprocess the raw enterprise operation data to obtain a multimodal enterprise operation dataset; Input: Raw data from the enterprise's internal ERP system, WeChat Work, email system, document management system, etc.; The system connects to various enterprise information systems through configurable data adapters, categorizing raw data into four main types based on data source and format characteristics: For structured data, such as order information in an ERP system, the system obtains data fields such as order number, customer information, product specifications, and delivery requirements in real time through a standard SQL interface. After processing by the data adapter, it is mainly transformed into numerical data, categorical data, and time information; For semi-structured data, such as production plan tables in Excel format, the system uses Apache... POI technology parses the table structure, extracting key information such as plan nodes, responsible persons, and completion status. After parsing, it forms structured text data, numerical data, and time information. For unstructured data processing, the system uses multi-threaded web crawling technology to extract customer emails from the company's email inbox, using regular expressions to identify key information such as order numbers, product models, and technical requirements. Simultaneously, the system obtains group chat records and work reports through the WeChat API interface, using natural language processing technology to extract business information such as task assignments, progress updates, and problem feedback. After initial processing, this is mainly converted into text data. For image data, the system automatically scans CAD drawings, product photos, quality inspection reports, and other files in a specified folder, using the OpenCV library for image format conversion and size standardization. The system uniformly converts the images to RGB format, adjusts the resolution to 224×224 pixels, and performs histogram equalization to enhance image contrast. For voice data, the system supports obtaining call recordings and conference recordings from telephone and conferencing systems, using voice activity detection technology to remove silent segments, using Mel-spectrum transform to extract audio features, and converting the voice into text data using a pre-trained speech recognition model.

[0021] After processing by the configurable data adapter, the system obtains four main data types: text data, numerical data, categorical data, and time information. The system establishes unified data standards and metadata description specifications, and performs standardized preprocessing on these four types of data: For text data (from email content, chat logs, voice-to-text transcription, etc.), encoding detection and conversion are performed, uniformly adopting the UTF-8 encoding format, Chinese word segmentation is performed, a custom dictionary is set to improve the accuracy of professional terminology recognition, and HanLP is used for part-of-speech tagging and named entity recognition to extract key entities such as names of people, organizations, products, time, and location; For numerical data (from ERP systems such as order amounts, production quantities, and quality indicators), the system uses the Z-score standardization method for normalization, converting the values ​​into a standard normal distribution with a mean of 0 and a standard deviation of 1; For categorical data (from product types, department names, task status, etc.), the system uses one-hot encoding to convert it into numerical vectors; For time information (from order dates, task deadlines, meeting times, etc.), the system uniformly converts it to ISO8601 format for standardized storage and extracts time features such as year, month, day, and hour.

[0022] Output: Multimodal Enterprise Operations Dataset.

[0023] Step 2: Use a lightweight model to extract features from the multimodal enterprise operation dataset to obtain deep feature representations of text, image, and speech modalities. Input: The multimodal enterprise operations dataset generated in step 1, after being cleaned, format-converted, and standardized; Text feature extraction employs a lightweight BERT-Base-Chinese model, with 110M parameters, reducing computation by 70% compared to the BERT-Large model. The input text is segmented and encoded, generating word ID sequences and positional codes. The text is processed through a 12-layer Transformer encoder, each layer containing 12 attention heads, with a hidden layer dimension of 768. To further reduce computational complexity, the system uses knowledge distillation to train the lightweight model. The knowledge distillation loss function consists of two parts: the first is the standard cross-entropy loss, which calculates the cross-entropy between the true label and the predicted probability for each sample, multiplies the logarithm of the predicted probability by the true label, sums the results over all samples, and takes a negative value, with a weight of 0.7 to maintain the model's task performance; the second is the distillation loss, with a weight of 0.3, which transfers knowledge by minimizing the KL divergence of the output distributions of the student and teacher models.

[0024] When calculating the distillation loss, the outputs of the teacher model and the student model need to be preprocessed by temperature scaling. The original logits are divided by the temperature parameter (usually set to 4), and then converted into a probability distribution through the softmax function to ensure that the probability distributions output by the two models have the same smoothness, which facilitates knowledge transfer.

[0025] The system ultimately extracts a 768-dimensional text feature vector for each text as its text representation.

[0026] Image feature extraction employs the MobileNetV3-Large architecture, specifically optimized for mobile devices to reduce computation while maintaining accuracy. The network input is a 224×224×3 RGB image, and feature extraction is performed using depthwise separable convolutions and inverted residual blocks. The system is fine-tuned using enterprise product image data based on a pre-trained ImageNet model. To adapt to enterprise scenarios, specific classification heads such as product category, quality level, and defect type are added. The final output is a 1000-dimensional image feature vector.

[0027] Speech feature extraction converts the audio signal into a Mel spectrogram using a 40-Mel filter bank, a frame length of 25 milliseconds, and a frame shift of 10 milliseconds. Mel spectrogram transformation is performed on the speech data, and a lightweight Wav2Vec2 model is used to extract features from the transformed Mel spectrogram, resulting in a 512-dimensional speech feature vector. The lightweight Wav2Vec2 model employs a combination architecture of a convolutional neural network encoder and a Transformer decoder. To reduce computational complexity, the system performs 8-bit quantization on the model, converting floating-point operations into integer operations.

[0028] To improve feature extraction efficiency, the system implements a feature caching mechanism. For data with similarity exceeding a threshold, the system directly reads the already calculated features from the cache, avoiding duplicate calculations. Similarity calculation employs the Locality Sensitive Hashing (LSH) algorithm. The system employs Locality Sensitive Hashing (LSH) to calculate data similarity. It calculates the dot product of two feature vectors, then calculates the L2 norm (the square root of the sum of the squares of the elements) of each vector. The dot product is divided by the product of the two L2 norms to obtain the similarity value. High-dimensional feature vectors are mapped to low-dimensional hash values ​​through random projection and bucket width parameters, enabling fast similarity retrieval. Two data points are considered to have high similarity when their hash values ​​are the same.

[0029] Output: Obtain deep feature representations for three modalities: text, image, and speech, including 768-dimensional text features, 1000-dimensional image features, and 512-dimensional speech features.

[0030] Furthermore, data augmentation techniques based on generative adversarial networks (GANs) can be employed. Since enterprise operational data often suffers from class imbalance and sample scarcity, the system uses GANs to generate synthetic training data. Specifically, the generator network learns the distribution characteristics of real data and generates synthetic samples similar to the real data. The generator employs a deep convolutional neural network architecture, taking a random noise vector as input and outputting synthetic samples with the same dimensions as the real data. The discriminator network is responsible for distinguishing between real and generated data, and trains adversarially against the generator. The discriminator also uses a convolutional neural network architecture, outputting binary classification probabilities, representing the probability that the input data is real data.

[0031] The generator and discriminator are optimized through an adversarial training mechanism. The generator aims to generate realistic synthetic samples to deceive the discriminator, while the discriminator strives to distinguish between real and generated data. Both continuously improve their performance during the game.

[0032] To ensure the quality and diversity of the generated data, the system also incorporates techniques such as conditional generation and progressive generation. Conditional generation adds class label information to the noise vector, enabling the generator to generate samples of the corresponding class based on the specified class label, thereby achieving targeted data augmentation.

[0033] Progressive generation starts training at low resolution, gradually increasing the resolution and complexity of the generated samples to improve training stability and generation quality. Synthetic data is used to augment the training set, particularly focusing on enhancing scarce classes to improve model performance in scenarios with few samples. This alternative embodiment aims to address the issues of class imbalance and sample scarcity in enterprise operational data by generating high-quality synthetic data to enhance the model's generalization ability.

[0034] Step 3: Apply a hierarchical attention mechanism to the deep feature representation for modal fusion processing to obtain fused multimodal features; Input: Obtain deep feature representations of three modalities: text, image, and speech; Multimodal feature fusion employs a hierarchical attention mechanism, first performing intramodal self-attention, then cross-modal attention. In the intramodal self-attention stage, the system performs self-attention calculations on the feature sequences of each modality (text, image, speech), performs linear transformations on the multimodal feature vectors of orders, projects, and tasks to unify the representation space, captures contextual dependencies and long-range associations within the same modality, and obtains an enhanced single-modal representation. In the cross-modal attention stage, cross-modal attention weights are calculated to align and weightedly aggregate multimodal semantic cues, thereby obtaining fused multimodal features.

[0035] Establish a semantic association model for enterprise operational data, construct an enterprise operational business ontology, and define core business entities and business relationships: Core business entities include order entities (including attributes such as order number, customer information, product specifications, delivery time requirements, priority, etc.), project entities (including attributes such as project number, person in charge, milestones, resource requirements, risk level, etc.), task entities (including attributes such as task ID, executor, dependencies, completion status, quality standards, etc.), and department entities (including attributes such as department code, functional scope, staffing, and collaboration relationships); Business relationships include: order-project mapping relationship, project-task decomposition relationship, task-personnel allocation relationship, department-collaboration dependency relationship, resource-requirement matching relationship, etc.

[0036] In the cross-modal attention phase, order-project-task association fusion, departmental collaboration relationship fusion, and resource-demand matching fusion are performed, specifically as follows: The order-project-task association is fused using multimodal feature fusion based on business semantics. Order change semantics (such as delivery date adjustments and technical requirements) are extracted from deep text features, visual cues of design drawings and quality inspection are obtained from deep image features, and acoustic key points of meetings and calls are extracted from deep speech features (aligned with the semantic features of the transcribed text). Semantic association computation based on graph attention networks is employed, with the following steps: attention scores between nodes are calculated by combining business relationship embeddings; nonlinear activation and normalization are applied to the scores to obtain attention weights; the attention weights are used to weighted aggregate the multimodal features of adjacent nodes and transformed using a nonlinear function to obtain the fused representation of the order-project-task association. The calculation process for business dependency strength involves: linearly transforming the multimodal features of nodes, concatenating them with the business relationship embedding vector, processing them through attention vector dot product and LeakyReLU, and then performing exponential mapping and normalization to obtain the dependency strength weights.

[0037] The departmental collaboration relationship fusion uses deep text features to characterize the semantics of inter-departmental communication and task handover, and combines deep image features (such as flowcharts / report screenshots) and deep speech features (such as acoustic cues from meeting discussions) to identify collaboration patterns and information flow. The steps of collaboration relationship fusion are as follows: degree normalization of collaboration adjacency relationships; weighted aggregation and linear transformation of multimodal features of departments according to adjacency relationships; obtaining updated collaboration representations through nonlinear activation, and iterating in multi-layer networks to capture broader collaboration structure information; the collaboration strength evaluation process is as follows: the historical collaboration frequency, task success rate, and communication efficiency are standardized and then weighted and summed to obtain a collaboration strength score; the weights can be dynamically adjusted according to business priorities.

[0038] When matching and fusing resources and demands, the skills and workload of human resources are represented as textual deep features, the status and quality images of equipment are represented as image deep features, and the real-time status clues in voice reports are represented as voice deep features, forming a dynamic resource profile. Attention-based resource allocation optimization is used for matching and fusing. The specific steps are as follows: using the multimodal features of project demands as queries, the multimodal features of resources are scored for similarity and normalized to obtain attention weights; then, the multimodal features of resources are weighted and aggregated using these attention weights to generate matching results. The resource matching degree comprehensively considers skill matching degree, time availability, and cost-effectiveness: these three indicators are standardized and then weighted and summed to obtain the final matching degree score; the weights can be dynamically adjusted according to business priorities.

[0039] Fusion Results Convergence: After the fusion of order-project-task association, departmental collaboration relationship, and resource-demand matching, the three types of fusion representations are aligned in a unified representation space; then, their weights are normalized according to business importance; subsequently, weighted aggregation is performed and transformed through a nonlinear function to obtain fused multimodal features, which serve as input for subsequent knowledge graph construction.

[0040] A contrastive learning mechanism based on enterprise operation scenario awareness is adopted (used for metric constraints and weight calibration during the training phase, without directly generating or replacing fused multimodal features). The system constructs positive and negative sample pairs for business scenarios based on trimodal deep features: positive samples are multimodal data from different stages of the same business process (such as text features of demand analysis for the same order, image features of corresponding drawings, and voice features of related meetings), while negative samples are data from different business types or abnormal situations. The contrast measurement of scenario awareness is implemented in logical steps: first, the business similarity between the anchor sample and the positive sample is calculated (fusing semantic similarity and business relevance); then, the similarity between the anchor and all samples is normalized and summarized on the same scale; finally, the difference is measured by the normalized ratio of the positive sample to all samples, and the logarithm is taken to obtain the contrast loss. The two components of business similarity include: cosine similarity (calculated by trimodal fusion features) and business relevance (based on the entity relationship path and weight of the business ontology), and the two are weighted to obtain the comprehensive similarity.

[0041] Output: Fusion of multimodal features.

[0042] Furthermore, an improved multimodal fusion method based on the Transformer architecture can be adopted. This method uses an adaptive attention mechanism to dynamically adjust the weights of different modalities, automatically selecting the most relevant modal information based on the characteristics of the current task and data quality. In specific implementation, a quality score for each modality is calculated, which is derived from a comprehensive calculation based on factors such as data integrity, noise level, and semantic consistency. Data integrity is assessed through the proportion of missing values ​​and data format standardization; noise level is assessed through signal-to-noise ratio and outlier detection results; and semantic consistency is assessed through intermodal correlation analysis. The quality score is calculated using a weighted summation method, comprising three components: the data integrity score multiplied by its corresponding weight, the reciprocal of the noise level (i.e., 1 minus the noise level) multiplied by its corresponding weight, and the semantic consistency score multiplied by its corresponding weight. The three parts are added together to obtain the final quality score.

[0043] Before calculating the quality score, each indicator needs to be standardized and preprocessed: the completeness score is calculated by the proportion of missing values, and the proportion of missing values ​​needs to be linearly mapped from the [0,1] interval to the [0,1] interval of the completeness score; the noise level is calculated by the signal-to-noise ratio, and the signal-to-noise ratio needs to be converted from dB units to standardized values ​​in the [0,1] interval; the semantic consistency is calculated by the intermodal correlation coefficient, and the correlation coefficient needs to be mapped from the [-1,1] interval to the [0,1] interval.

[0044] The system adjusts attention weights based on quality scores, employing softmax normalization to calculate adaptive attention weights. The system multiplies the quality score by the base attention score, then performs an exponential transformation and normalization to ensure higher weights are assigned to high-quality modal data and lower weights to low-quality modal data. This adaptive mechanism maintains the stability of the fusion effect even with uneven data quality, improving the system's adaptability to environmental changes. This alternative embodiment aims to address the performance instability of traditional fixed-weight fusion methods when facing data quality fluctuations by dynamically adjusting modal weights to enhance the robustness of the fusion effect. Experimental results show that, compared to fixed-weight fusion methods, the adaptive attention mechanism improves overall performance, with a more significant improvement in scenarios with large differences in data quality.

[0045] Step 4: The fused multimodal features are analyzed and processed using a knowledge graph construction algorithm to obtain a real-time updated enterprise knowledge graph; Input: Fusion of multimodal features and historical enterprise knowledge graph (from the previous version snapshot of the enterprise knowledge graph historical version repository, including timestamp and confidence); The enterprise knowledge graph construction adopts an incremental update strategy to avoid the high computational cost of full reconstruction. Entities and relationships are identified from fused features, compared with the existing knowledge graph, and only the changed parts are updated.

[0046] In the entity recognition phase, the system uses a named entity recognition model based on a Conditional Random Field (CRF) model. The label set includes people, organizations, products, projects, time, and location. CRF calculates the conditional probability of the label sequence under a given input sequence, and combines feature functions and weight parameters to determine the optimal label sequence, identifying entities related to enterprise operations and obtaining the entity set. The system solves for the optimal label sequence using the Viterbi algorithm, calculating the maximum probability path for each label at each location using dynamic programming. It calculates and records the optimal path step-by-step from left to right, and obtains the complete optimal label sequence through backtracking.

[0047] Relation extraction employs an attention-based relation classification model. For candidate entity pairs, the attention mechanism is used to calculate the contextual representations between entities, and the attention weights are normalized using softmax to ensure that the sum of the weights is 1. The probability distribution of relation classification is calculated through the following steps: the contextual representation vector of the entity pair is multiplied by the weight matrix of the relation classifier, a bias vector is added, and the output is converted into a probability distribution using a softmax activation function. This yields the probability distribution of various relations for a given entity pair, identifies the relationships between entity pairs, and forms the triples obtained from relation extraction.

[0048] Entity linking employs a semantic similarity-based matching algorithm. For newly identified entities, the system calculates a similarity score between the newly identified entity and existing entities in the knowledge graph. The similarity score consists of two parts: the first part is semantic similarity, calculated by the cosine similarity of the two entity vector representations, with a weighting coefficient of 0.7; the second part is contextual similarity, calculated based on the TF-IDF vectors of the surrounding words, with a weighting coefficient of 0.3. The entity's vector representation function is learned through the TransE model.

[0049] The specific implementation of the entity vector representation function is as follows: the entity name is converted into an initial vector representation through a pre-trained word vector model, and then trained and optimized on the knowledge graph through the TransE (Translating Embeddings for Modeling Multi-relational Data) model to learn the low-dimensional dense vector representation of the entity in the relation space. The vector dimension is set to 128 dimensions.

[0050] The specific calculation process of the context similarity function is as follows: extract the context window (5 words before and after) in which the two entities appear in the text, construct feature vectors, and calculate the cosine similarity between the two vectors as the context similarity score.

[0051] When the similarity score exceeds the threshold of 0.8, it is considered that the two entities point to the same object, and the triples obtained by relation extraction are integrated with the triples in the existing knowledge graph.

[0052] The knowledge graph updates employ a version control mechanism, with the system maintaining a timestamp and confidence score for each triple. The new triple confidence score is calculated using a weighted average, combining the model's prediction confidence and the data source's reliability score. Weight parameters, ranging from 0 to 1, balance the importance of model prediction and data source reliability.

[0053] Before calculating the confidence score, the model prediction score and the data source reliability need to be normalized and preprocessed: the model prediction score is converted into a probability distribution by the softmax function and the maximum value is taken to ensure that the value is in the range [0,1]; the data source reliability is comprehensively evaluated based on the historical accuracy, update frequency and authority of the data source, and a standardized score is calculated by weighted averaging.

[0054] When the confidence level of a new triplet is higher than that of an existing triplet, the system updates the knowledge graph content.

[0055] To improve query efficiency, the system uses the graph database Neo4j to store the knowledge graph and establishes a multi-level index structure. Entity indexes are built based on entity type and attributes, while relation indexes are built based on relation type and direction. The system also implements a distributed storage scheme, storing large-scale knowledge graph shards across multiple nodes and using a consistent hashing algorithm for load balancing.

[0056] On top of the enterprise knowledge graph, entities, relationships, and constraints related to decision-making are extracted to construct an enterprise operational decision-making knowledge graph (decision subgraph). The relationship between the two is as follows: they share entity and relationship types, and attributes are aligned through mapping rules; the enterprise knowledge graph provides facts and context, while the decision subgraph provides strategies, processes, and triggering conditions. Preliminary decision generation adopts a combination of rule-based and graph query methods: based on the business ontology and threshold strategies, target entities and constraints are filtered from the graph to generate preliminary decision suggestions for specific scenarios (such as priority ranking, collaboration suggestions, resource matching, and contingency plans) for subsequent optimization.

[0057] Through entity recognition, relation extraction, entity linking, and version updates, a real-time updated enterprise knowledge graph is obtained. Based on the real-time updated enterprise knowledge graph, an enterprise operation decision knowledge graph is constructed and aligned. Through rules and graph queries, a set of intelligent decision-making solutions is generated as input for subsequent decision optimization and interaction.

[0058] Output: A real-time updated enterprise knowledge graph and intelligent decision-making solution set, including structured knowledge representations of entities, relationships, attributes, etc., as well as decision suggestions for specific business scenarios.

[0059] Step 5: Apply meta-learning and reinforcement learning algorithms to the real-time updated enterprise knowledge graph for decision optimization to obtain an intelligent decision-making scheme; Inputs: Real-time updated enterprise knowledge graph, enterprise operation decision knowledge graph (exported from step 4), current business status data, historical enterprise operation decision samples and preliminary decision suggestions; A three-layer structure for decision generation is constructed, comprising a strategic layer (long-term planning, market positioning, resource allocation), a tactical layer (project management, departmental coordination, process optimization), and an operational layer (daily task allocation, resource scheduling, and exception handling). Each layer has corresponding objectives, constraints, and evaluation indicators. Decision-making tasks are organized according to task families (e.g., order processing, production management, human resources, customer service), and transferable general strategies are learned. New tasks are rapidly adapted using a small number of support samples, and context encoding and historical decision sequences are combined to improve cold start and transfer effectiveness. The enterprise operating environment is modeled as a decision-making process with strategic, tactical, and operational states. Actions include task assignment, resource scheduling, and process adjustment. Rewards are weighted scores based on comprehensive efficiency, quality, cost, satisfaction, and risk. A deep Q-network is used for strategy learning, combined with priority experience replay and a double Q-network to reduce overestimation and improve convergence stability.

[0060] The specific enterprise operation decision-making process includes: intelligent order processing decisions, which comprehensively consider customer type, product complexity, delivery time, and profit characteristics, combined with current capacity and resource status, and take into account order dependence and resource competition, using dynamic programming to select the processing solution with the greatest comprehensive value among feasible actions; collaborative production scheduling decisions, which optimize multi-production line planning and equipment scheduling under constraints such as capacity, demand, and maintenance windows, aiming at minimizing total cost and plan feasibility, and improving scheduling quality by combining heuristics and precise solutions; dynamic human resource allocation decisions, which allocate personnel to projects based on skill matching, learning effects, and collaboration effects, calculate allocation scores, and select the optimal allocation plan, balancing short-term delivery and long-term capacity building; supply chain risk management decisions, which comprehensively assess risks based on historical performance and environmental changes, set trigger thresholds, and generate mitigation strategies (such as supplier switching, inventory adjustment, and contract renegotiation), continuously monitoring and revising them; and intelligent customer relationship management decisions, which assess customer lifetime value based on discounted cumulative net income over time, and use this information to manage and optimize service priorities, resource allocation, and communication frequency.

[0061] The system employs an online learning mechanism to continuously optimize decision-making strategies. By collecting decision execution results and business feedback, the system continuously updates model parameters and decision rules. The update process uses incremental learning logic: based on existing parameters, the loss of the current decision is evaluated using new feedback data; the directional adjustment of the parameters by the loss is calculated, and the adjustment step size is controlled by a set learning rate; a small update is performed on the parameters along this direction to obtain the new parameter configuration; this process only uses new data for local updates to avoid destroying existing knowledge; simultaneously, the system maintains decision performance evaluation indicators, including decision accuracy, execution efficiency, and business improvement degree, which serve as the basis for the next round of optimization.

[0062] Through rapid adaptation of meta-learning and strategy optimization of reinforcement learning, a set of intelligent decision-making solutions for enterprise operations is obtained, which is used to drive subsequent natural language interaction and system encapsulation.

[0063] Output: A collection of intelligent decision-making solutions for enterprise operations, including complete information such as hierarchical decision recommendations, execution sequence, resource requirements, risk assessment, and expected results.

[0064] Furthermore, a dynamic resource scheduling algorithm based on reinforcement learning can be employed. This algorithm models the enterprise operating environment as a Markov decision process. The state space includes the current task queue, personnel status, and equipment usage, while the action space includes task priority adjustment, personnel reallocation, and equipment scheduling. In practice, the state vector contains key information such as task queue length, urgency of each task, personnel skill matching, and equipment utilization. The action vector represents the scheduling decisions the system can execute in the current state, including tasks reordering, personnel reassignment, and equipment allocation.

[0065] The reward function comprehensively considers multiple objectives such as efficiency, quality, cost, and customer satisfaction, and is calculated using a weighted summation method. The reward value comprises four components: task completion rate (number of completed tasks divided by total number of tasks) multiplied by its corresponding weight, average quality score multiplied by its corresponding weight, resource usage cost multiplied by its corresponding weight (as a negative term), and delay penalty multiplied by its corresponding weight (as a negative term). The weight coefficients for each component are set according to the company's business priorities.

[0066] Before calculating the reward function, the indicators need to be standardized and preprocessed: the resource usage cost is processed by the maximum and minimum value normalization method, and the actual cost value is mapped to the [0,1] interval; the delay penalty is calculated based on the delay days and the importance of the task, and mapped to the [0,1] interval through the sigmoid function; the quality score needs to be weighted and averaged from the scores of different evaluation dimensions and normalized to the [0,1] interval.

[0067] The system uses a Deep Q-Network (DQN) to learn the optimal scheduling policy. The Q-function update process is as follows: calculate the target Q-value, which is the current reward plus a discount factor multiplied by the maximum Q-value of all possible actions in the next state; calculate the temporal difference error, which is the difference between the target Q-value and the current Q-value; add the learning rate to the current Q-value and multiply by the temporal difference error to obtain the updated Q-value. The discount factor is used to balance immediate and long-term rewards, and the learning rate controls the step size of parameter updates.

[0068] To improve learning efficiency and stability, the system also employs techniques such as priority experience replay and dual-Q networks. Priority experience replay determines the importance of samples based on the magnitude of temporal difference errors, prioritizing training on samples with larger errors. The dual-Q network uses two independent Q-networks for action selection and value evaluation respectively, reducing overestimation problems. This alternative embodiment aims to address the problem that traditional static scheduling rules cannot adapt to dynamically changing environments, achieving adaptive resource scheduling optimization through reinforcement learning. In practical applications, reinforcement learning scheduling algorithms can dynamically adjust resource allocation based on real-time conditions, resulting in improved overall operational efficiency compared to traditional static rule methods, with more pronounced advantages in complex and ever-changing business scenarios.

[0069] Step 6: Analyze and process the intelligent decision-making scheme using natural language processing technology to obtain the natural language system response results; Input: User's natural language commands, current system status, and dialogue history; The natural language interaction module adopts an end-to-end dialogue system architecture, which includes four core components: natural language understanding, dialogue management, task execution, and natural language generation.

[0070] In the natural language understanding component, the system uses a BERT-based intent recognition model and a slot-filling model. The intent recognition model classifies user input into predefined intent categories, such as query progress, task assignment, and report generation. The probability distribution of intent classification is calculated as follows: Intent recognition uses a BERT-based classification model, achieving high-precision intent understanding through fine-tuning the pre-trained model. The system encodes user utterances using BERT, then uses a linear classifier and softmax normalization to obtain the probability distribution of each intent category, selecting the category with the highest probability as the recognition result.

[0071] Slot filling employs a sequence labeling method, using the BIO (Browser Injection) labeling scheme to identify entity boundaries. The system calculates the probability of a label sequence under a given input sequence using a Conditional Random Field (CRF) model, and combines feature functions and weight parameters to determine the optimal entity labeling sequence.

[0072] The dialogue management component employs a graph neural network-based approach, modeling multi-turn dialogues as a dynamic graph structure. Each dialogue turn corresponds to a node in the graph, and edges represent dependencies between turns. The system updates the dialogue state through the graph neural network, comprehensively considering historical states, system actions, and current user input. The state update process includes three steps: message aggregation, state fusion, and nonlinear transformation. Message aggregation uses attention-weighted summation, state fusion uses a gating mechanism, and the nonlinear transformation is implemented through a two-layer fully connected network.

[0073] The task execution component invokes the corresponding business functions based on the understood results. The system maintains an API registry, mapping natural language intents to specific system operations. The execution process includes parameter validation, permission checks, business logic processing, and result return. For complex combined operations, the system uses a workflow engine for task orchestration.

[0074] The natural language generation component employs a combination of template-based and neural network-based approaches. For common response types, the system uses predefined templates for rapid generation; for complex, personalized responses, the system uses a GPT-based generative model. Generation quality is evaluated using metrics such as BLEU and ROUGE.

[0075] To enhance the interactive experience, the system also implements voice input and output functionality. Speech recognition employs an end-to-end Transformer model, supporting mixed Chinese and English recognition. Speech synthesis utilizes the FastSpeech2 model to generate natural and fluent speech output. The system aggregates and checks the consistency between the template-generated text response and the synthesized speech response, resulting in a natural and fluent system response that includes confirmation of the execution result and suggestions for subsequent operations. Output: A smooth and natural system response, confirmation of execution results, and suggestions for follow-up operations.

[0076] Furthermore, an intelligent question-answering system based on knowledge graph reasoning can be employed to answer complex user queries about business operations, such as "Which projects were delayed last month, and what were the main reasons?". In practice, semantic parsing of user questions is performed, using a BERT-based question understanding model to identify query intent and key entities. Question parsing includes three sub-tasks: intent classification, entity recognition, and relation extraction. Intent classification categorizes questions into different types, such as query, analysis, and prediction. Entity recognition extracts key business entities from the question, such as project name, personnel name, and time range. Relation extraction identifies semantic relationships between entities, such as "responsible for," "participating in," and "influencing."

[0077] The system performs multi-hop reasoning on the knowledge graph, finding relevant information through path search and subgraph matching. The multi-hop reasoning algorithm employs a breadth-first search strategy, starting from the query entity and expanding along the relational edges in the knowledge graph. The score of a reasoning path is calculated by multiplying the weight of each step on the path by the confidence of the triple, with the weight decreasing as the path length increases. The system retains the top K reasoning paths with the highest scores as candidate answers.

[0078] The confidence of the inference path is described using logical steps: the confidence of each triple on the path is logarithmically transformed and averaged, and then exponentially restored to obtain the overall confidence of the path.

[0079] The confidence level of the inference path is calculated by the geometric mean of the confidence levels of each triple along the path, which is to multiply the confidence levels of all triples and take the Lth root to ensure that the confidence level is within a reasonable range.

[0080] The system converts the reasoning results into natural language answers and provides a chain of evidence for user verification. Answer generation employs a combination of template matching and neural generation; predefined templates are used for common question types, while a GPT-based generative model is used for complex questions. The purpose of this alternative embodiment is to enhance the system's intelligent question-answering capabilities, enabling it to handle complex analytical queries and provide enterprise managers with in-depth business insights.

[0081] Step 7: Using a microservice architecture, encapsulate the multimodal enterprise operation dataset, deep feature representation, fusion of multimodal features, enterprise knowledge graph, intelligent decision-making scheme, and natural language system response results. Ensure system operation through load balancing technology to obtain the final intelligent enterprise operation system. Inputs: Outputs of each functional module (including multimodal enterprise operation datasets, deep feature representations, fused multimodal features, enterprise knowledge graphs, intelligent decision-making solutions, and system response results) and system operation status data; A microservice architecture is adopted to encapsulate and deploy each processing module, and the functional modules are encapsulated into independent service modules, including data acquisition service module, feature extraction service module, modality fusion service module, knowledge graph service module, decision optimization service module, and response generation service module, resulting in a set of microservice components.

[0082] The microservice component set is registered and configured, and an inter-service communication mechanism and API gateway are established to obtain a distributed service architecture. Based on a distributed service architecture, load balancing technology is used to allocate and schedule traffic for each microservice component, thereby achieving elastic scaling and failover of services and obtaining a service cluster. By unifying the monitoring and management of service clusters and using service mesh technology to achieve secure communication and performance optimization between services, an intelligent enterprise operation system can be obtained.

[0083] Services communicate with each other via REST APIs and message queues, supporting horizontal scaling and fault isolation. The system is deployed using Docker containers and orchestrated and managed with Kubernetes.

[0084] The load balancing strategy combines round-robin and weighted round-robin, dynamically adjusting weights based on service response time and resource usage. When a service instance fails, the system automatically forwards traffic to healthy instances, ensuring high service availability.

[0085] For database optimization, the system adopts a read-write separation architecture, with write operations using the master database and read operations using the slave database. For frequently queried data, the system uses Redis caching for acceleration. Knowledge graph data is stored using the Neo4j graph database, with appropriate indexes created to improve query performance.

[0086] Model inference optimization employs model compression and quantization techniques. The system uses TensorRT to optimize deep learning models, converting floating-point operations into low-precision integer operations to improve inference speed while maintaining accuracy. For edge device deployment, the system uses the ONNX format for model conversion to ensure cross-platform compatibility.

[0087] The monitoring and logging system utilizes the ELK (Elasticsearch, Logstash, Kibana) technology stack to collect and analyze system logs in real time. Prometheus is used to monitor system performance metrics, triggering an alert mechanism when metrics become abnormal. The system also implements distributed tracing to help quickly locate and resolve performance issues, ultimately resulting in an intelligent enterprise operational system.

[0088] Output: The final intelligent system for enterprise operations.

[0089] Furthermore, a federated learning framework can be used to protect enterprise data privacy. Federated learning allows multiple enterprises to jointly train a model without sharing the original data; each enterprise only needs to upload the updated gradients of the model parameters. In practice, each participating enterprise trains its model locally and uploads the gradient update information of the model parameters to the federated learning server. The server aggregates the gradient information of all enterprises, calculates the global model parameter update, and then distributes the updated model parameters to each enterprise. The parameter update process of the federated averaging algorithm is as follows: calculate the weighted average of the local gradients of all participating enterprises, with the weights being the proportion of each enterprise's sample size to the total sample size; subtract the learning rate from the current global model parameters and multiply by the weighted average gradient to obtain the global model parameters for the next round.

[0090] Before aggregating federated learning parameters, the gradients uploaded by each enterprise need to be preprocessed: the gradients are pruned to limit the gradient norm to a preset threshold to prevent gradient explosion; the gradients are standardized to ensure that the gradients of different enterprises have the same magnitude; and the gradients are weighted according to the data quality and credibility of each enterprise.

[0091] The system uses differential privacy techniques to add noise to gradient information to prevent the original data from being inferred from the gradient information. Before uploading the gradient, the system adds calibrated Gaussian noise to the gradient, and the noise intensity is calculated based on the privacy budget and failure probability. The choice of noise intensity requires a trade-off between privacy protection and model performance; a larger noise intensity provides stronger privacy protection but may reduce model accuracy.

[0092] Federated learning employs the FedAvg algorithm for model parameter aggregation. In each training round, each participant trains its model on its local data and uploads the model parameters to the coordinating server. The server-side parameter aggregation process involves a weighted average of the local model parameters from all participants, with the weights being the proportion of each participant's data sample size to the total sample size. This ensures that participants with larger datasets contribute more to the global model, improving overall model performance.

[0093] Through a federated learning framework, the system can improve the generalization performance and robustness of the model by leveraging multi-party data while protecting sensitive corporate information. This alternative embodiment aims to address privacy and security issues related to data sharing among enterprises, enabling multiple enterprises to collectively benefit from collaborative learning without disclosing sensitive data.

[0094] An AI-based intelligent system for enterprise operations based on multimodal fusion, such as Figure 2 As shown, an AI-based enterprise operation intelligence method based on multimodal fusion, as described above, includes: The data acquisition module is used to acquire raw enterprise operation data and preprocess the raw enterprise operation data to obtain a multimodal enterprise operation dataset. The raw enterprise operation data includes structured data obtained from the ERP system, semi-structured data obtained from Excel-formatted production plan sheets, unstructured data obtained from enterprise email and WeChat, as well as image data obtained from the document management system and voice data obtained from the telephone system and conferencing system. The feature extraction module is used to perform feature extraction processing on the multimodal enterprise operation dataset using a lightweight model to obtain deep feature representations of three modalities: text, image, and speech. The multimodal fusion module is used to perform modal fusion processing on deep feature representations using a hierarchical attention mechanism to obtain fused multimodal features; The knowledge graph construction module is used to analyze and process the fused multimodal features using knowledge graph construction algorithms to obtain a real-time updated enterprise knowledge graph; it extracts enterprise entity and relationship information from the fused multimodal features through entity recognition, relation extraction, and entity linking technologies; The intelligent decision-making module is used to perform decision optimization processing on the real-time updated enterprise knowledge graph using meta-learning and reinforcement learning algorithms to obtain intelligent decision-making solutions; The natural language interaction module is used to analyze and process intelligent decision-making schemes using natural language processing technology to obtain the response results of the natural language system. The system architecture module is used to encapsulate multimodal enterprise operation datasets, deep feature representations, fused multimodal features, enterprise knowledge graphs, intelligent decision-making solutions, and natural language system response results using a microservice architecture. Load balancing technology ensures system operation, resulting in the final intelligent enterprise operation system.

[0095] In one embodiment of the present invention, an application example of an AI-based enterprise operation intelligence system and method based on multimodal fusion is provided: A 90-day field test was conducted in an electronics manufacturing industrial park in District B of City A. During the test, a complete multimodal fusion AI enterprise operation intelligence system was deployed, covering three small and medium-sized manufacturing enterprises, with a test area of ​​approximately 15,000 square meters and involving a total of 450 employees. The test environment included production workshops, office areas, warehouse areas, and quality inspection laboratories, comprehensively verifying the system's data acquisition and processing capabilities in a real enterprise operation environment.

[0096] Table 1 shows an example of how data is retrieved from an ERP system. Table 1: Example of Data Acquisition from ERP System As can be seen from Table 1, ERP data collection covers core business fields such as orders, inventory, production plans, quality inspection, and finance. The interface methods and collection frequencies range from real-time to daily, and the data scale is sufficient to support real-time analysis and closed-loop management.

[0097] Table 2 shows an example of data acquisition from an enterprise communication platform: Table 2: Examples of Data Acquisition by Enterprise Communication Platforms As can be seen from Table 2, enterprise communication and collaboration channels can extract event, personnel and time elements in a structured manner through standard interfaces and push / polling methods, and can timely gather key information such as order changes, approvals and meetings, providing a basis for anomaly identification and cross-departmental collaboration.

[0098] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. An AI-based intelligent enterprise operation method based on multimodal fusion, characterized in that, include: Obtain raw enterprise operation data and preprocess the raw enterprise operation data to obtain a multimodal enterprise operation dataset; A lightweight model was used to extract features from the multimodal enterprise operation dataset to obtain deep feature representations of text, image, and speech modalities. A hierarchical attention mechanism is used to perform modal fusion processing on the deep feature representation to obtain fused multimodal features; The fused multimodal features are analyzed and processed using a knowledge graph construction algorithm to obtain a real-time updated enterprise knowledge graph; Meta-learning and reinforcement learning algorithms are used to optimize decision-making processes on real-time updated enterprise knowledge graphs to obtain intelligent decision-making solutions. The intelligent decision-making scheme is analyzed and processed using natural language processing technology to obtain the response results of the natural language system. A microservice architecture is adopted to encapsulate multimodal enterprise operation datasets, deep feature representations, and integrate multimodal features, enterprise knowledge graphs, intelligent decision-making schemes, and natural language system response results. Load balancing technology is used to ensure system operation, resulting in the final intelligent enterprise operation system.

2. The AI-based enterprise operation intelligence method based on multimodal fusion according to claim 1, characterized in that, The acquisition of raw enterprise operational data specifically includes: The system can obtain structured data fields such as order number, customer information, product specifications, and delivery requirements from the ERP system in real time through a standard SQL interface. The Apache POI technology is used to parse the production plan in Excel format and extract key planning nodes, responsible persons, and completion status information. Multi-threaded web crawling technology is used to extract customer emails from corporate mailboxes, and regular expressions are used to identify key information such as order numbers, product models, and technical requirements in the emails.

3. The AI-based enterprise operation intelligence method based on multimodal fusion according to claim 1, characterized in that, The lightweight model performs feature extraction processing specifically including: Text data from a multimodal enterprise operation dataset is obtained, and a lightweight BERT-Base-Chinese model is used to segment and encode the text data source to obtain a 768-dimensional text feature vector. Image data from a multimodal enterprise operation dataset was acquired, and features were extracted from the image data using the MobileNetV3-Large architecture. Fine-tuning was performed on the ImageNet pre-trained model to obtain a 1000-dimensional image feature vector. We acquire speech data from a multimodal enterprise operation dataset, perform Mel-spectrum transformation on the speech data, and use a lightweight Wav2Vec2 model to extract features from the transformed Mel-spectrum to obtain a 512-dimensional speech feature vector.

4. The AI-based enterprise operation intelligence method based on multimodal fusion according to claim 1, characterized in that, The hierarchical attention mechanism comprises two stages: intramodal self-attention and cross-modal attention. Specifically, it includes: The intramodal self-attention stage calculates self-attention weights based on feature sequences from three modalities: text, image, and speech. The cross-modal attention stage calculates the cross-modal attention weights among text, image, and speech modalities using a learnable weight matrix; By integrating self-attention weights and cross-modal attention weights through weighted summation, fused multimodal features are obtained.

5. The AI-based enterprise operation intelligence method based on multimodal fusion according to claim 1, characterized in that, The knowledge graph construction algorithm analysis and processing specifically includes: The fused multimodal features are obtained, and the named entity recognition model based on CRF is used to perform entity recognition processing on the fused multimodal features to obtain an entity set; A relation extraction process based on an attention mechanism is used to extract relations from the entity set. Link entities between the entity set and existing entities in the historical enterprise knowledge graph; Based on the results of relation extraction and entity linking, a version control mechanism is used to update the enterprise knowledge graph, resulting in a real-time updated enterprise knowledge graph.

6. The AI-based enterprise operation intelligence method based on multimodal fusion according to claim 1, characterized in that, The decision optimization process performed by the meta-learning and reinforcement learning algorithms specifically includes: To acquire real-time updated enterprise knowledge graphs and current business status data, the meta-learning framework uses the MAML algorithm to analyze and process the knowledge graphs. By training the learning model on five related tasks to initialize parameters, an adaptive decision model is obtained. Based on the adaptive decision-making model, the near-end policy optimization (PPO) algorithm is used for reinforcement learning decision optimization. The current task queue, personnel status, and resource usage are used as the three dimensions of the state space to obtain the decision state representation. Based on the decision state representation, a discrete decision action space is defined, including three types: task allocation, resource scheduling, and priority adjustment. The optimal decision action is selected through a policy network to obtain the decision scheme. The decision-making scheme is evaluated using a reward function. The overall reward value is calculated by comprehensively considering three factors: decision-making effect, execution cost, and risk assessment, thus obtaining an intelligent decision-making scheme.

7. The AI-based enterprise operation intelligence method based on multimodal fusion according to claim 1, characterized in that, The natural language processing technology for response generation specifically includes: It acquires intelligent decision-making solutions and user natural language commands, and adopts an end-to-end dialogue system architecture, including four core components: natural language understanding component, dialogue management component, task execution component, and natural language generation component; The natural language understanding component uses a BERT-based intent recognition model and slot filling model for natural language understanding, and employs the BIO tagging scheme and conditional random field model for entity annotation. The dialogue management component employs a dialogue state tracking method based on graph neural networks and a dialogue management strategy based on reinforcement learning. The task execution component maps natural language intents to system operations via the API registry; The natural language generation component uses a combination of templates and neural networks to generate natural language responses, thus obtaining the natural language system response results.

8. The AI-based enterprise operation intelligence method based on multimodal fusion according to claim 1, characterized in that, The encapsulation using a microservice architecture specifically includes: To obtain the response results from the natural language system, a microservice architecture is adopted to encapsulate and deploy each processing module. The functional modules are encapsulated into independent service modules, including a data acquisition service module, a feature extraction service module, a modality fusion service module, a knowledge graph service module, a decision optimization service module, and a response generation service module, resulting in a set of microservice components. The microservice component set is registered and configured, and an inter-service communication mechanism and API gateway are established to obtain a distributed service architecture. Based on a distributed service architecture, load balancing technology is used to allocate and schedule traffic for each microservice component, thereby achieving elastic scaling and failover of services and obtaining a service cluster. By unifying the monitoring and management of service clusters and using service mesh technology to achieve secure communication and performance optimization between services, an intelligent enterprise operation system can be obtained.

9. The AI-based enterprise operation intelligence method based on multimodal fusion according to claim 1, characterized in that, The microservice architecture encapsulation also includes using the ELK technology stack to implement a monitoring and logging system, collecting and analyzing system logs in real time, monitoring system performance indicators through Prometheus, triggering an alarm mechanism when indicators are abnormal, realizing distributed tracing, and finally obtaining an intelligent enterprise operation system.

10. An AI-based intelligent system for enterprise operation based on multimodal fusion, characterized in that, An AI-based enterprise operation intelligence method based on multimodal fusion, as described in any one of claims 1-9, comprises: The data acquisition module is used to acquire raw enterprise operation data and preprocess the raw enterprise operation data to obtain a multimodal enterprise operation dataset. The feature extraction module is used to perform feature extraction processing on the multimodal enterprise operation dataset using a lightweight model to obtain deep feature representations of three modalities: text, image, and speech. The multimodal fusion module is used to perform modal fusion processing on deep feature representations using a hierarchical attention mechanism to obtain fused multimodal features; The knowledge graph construction module is used to analyze and process the fused multimodal features using knowledge graph construction algorithms to obtain a real-time updated enterprise knowledge graph; The intelligent decision-making module is used to perform decision optimization processing on the real-time updated enterprise knowledge graph using meta-learning and reinforcement learning algorithms to obtain intelligent decision-making solutions; The natural language interaction module is used to analyze and process intelligent decision-making schemes using natural language processing technology to obtain the response results of the natural language system. The system architecture module is used to encapsulate multimodal enterprise operation datasets, deep feature representations, fused multimodal features, enterprise knowledge graphs, intelligent decision-making solutions, and natural language system response results using a microservice architecture. Load balancing technology ensures system operation, resulting in the final intelligent enterprise operation system.

Images