Log generation method and device based on multi-modal process data and electronic equipment

Abstract

The application discloses a log generation method and device based on multi-modal process data and electronic equipment. The method comprises the following steps: performing semantic analysis on a target log by using a first machine learning model to obtain a first business content and a business semantic distribution, obtaining a plurality of process feature vectors by using a second machine learning model, determining a candidate corresponding relationship between each process feature vector and each log segment, then determining a target process feature vector, a to-be-completed part and a corresponding relationship, and generating an updated log meeting preset time sequence constraints and business logic constraints; performing semantic analysis on the updated log again to obtain a second business content, and determining the contribution degree of newly added log content to the completeness of the target business based on the difference between the second business content and the benchmark business content, and outputting the updated log when the similarity and the contribution degree meet the preset threshold. By using the application, the accuracy, time sequence rationality and business consistency of log completion and updating can be improved.

Description

Technical Field

[0001] This application relates to the field of large model technology, and in particular to a log generation method, apparatus and electronic device based on multimodal process data. Background Technology

[0002] With the widespread application of information systems in industrial production, business processing, government workflows, and customer service, the recording, tracking, auditing, and analysis of business process execution typically rely on process logs generated by these systems. Existing process logs are generally structured text logs, used to record event names, execution times, executing entities, status information, and relevant business parameters during process execution. Related process analysis, anomaly detection, process mining, and auditing are also primarily based on these text logs.

[0003] However, in actual business operations, factors such as system failures, manual omissions, network transmission anomalies, interface interruptions, or equipment malfunctions often lead to issues in process logs, including missing events, missing attributes, omitted key nodes, or incomplete temporal relationships. Existing technologies typically rely solely on the contextual information of existing text logs, statistical patterns from historical log samples, or simple machine learning methods for completion. Because this completion method primarily depends on the text logs themselves and lacks utilization of process information such as images, audio, video, operation records, and execution data during business execution, the completion results are prone to inconsistencies with the actual business execution scenario, making it difficult to accurately reconstruct the true business execution trajectory.

[0004] Furthermore, in many practical application scenarios, key information reflecting business execution status and process progress is not only found in text logs, but may also exist in device status images, on-site operation videos, voice commands, business document images, terminal operation records, and other execution process data. Current technologies, based on single text logs for process recording and analysis, struggle to effectively integrate this multimodal process data. This makes it difficult to accurately identify process information related to the current business content from the multimodal process data when logs are incomplete or distorted, and also makes it difficult to supplement and update logs based on this process information.

[0005] Meanwhile, even existing technologies that process multimodal data such as images, voice, and video mostly focus on general recognition or understanding tasks, lacking processing mechanisms adapted to business log analysis scenarios. In particular, they lack a closed-loop processing mechanism that first infers business content from existing target logs, then combines multimodal process data to determine process features related to that business content, and finally verifies the consistency between the updated logs and the target business. Therefore, existing technologies struggle to balance the temporal rationality of log completion results, business logic consistency, and the degree of matching with the target business baseline content, resulting in relatively low accuracy in log completion and low reliability in subsequent process analysis.

[0006] Therefore, existing technologies need technical solutions that can effectively complete and update target logs by combining multimodal process data during the execution of the target business, even when the target logs are missing, incomplete, or inconsistent with the actual business execution trajectory. Summary of the Invention

[0007] This application provides a log generation method, apparatus, and electronic device based on multimodal process data to solve the problem that existing technologies mainly rely on single text records and are difficult to accurately reconstruct the real business execution trajectory when logs have missing events, missing attributes, missing key nodes, or incomplete time sequence relationships.

[0008] This application provides a log generation method based on multimodal process data. In this method, process data during the execution of the target business and the target log corresponding to the business identifier are obtained based on the business identifier of the target business. Baseline business content corresponding to the target business is obtained from a preset business database based on the business identifier. A first machine learning model is used to perform semantic analysis on the target log to obtain first business content. Based on the first business content, the business semantic distribution corresponding to each log segment in the target log is determined. The first business content is used to characterize the target business execution content inferred from the target log. A second machine learning model is used to extract features from the process data to obtain multiple process feature vectors. Based on the distribution relationship of each process feature vector in the time dimension, candidate correspondences between each process feature vector and each log segment are determined. Based on the business semantic distribution corresponding to each log segment and the candidate correspondences between each process feature vector and each log segment, the correlation between each process feature vector and the first business content is calculated, and target process feature vectors with a correlation greater than a preset correlation threshold are determined. Based on the candidate correspondence between each target process feature vector and each log segment, the correlation between each target process feature vector and the first business content is calculated. Then, based on the correlation between each target process feature vector and the first business content, the parts to be completed in the target log and the target process feature vectors corresponding to each part to be completed are determined. Based on the target process feature vectors corresponding to each part to be completed, the correlation between each target process feature vector and the first business content, and the target log, an updated log is generated. The newly added log content in the updated log satisfies preset temporal constraints and business logic constraints corresponding to the business identifier in the original log content of the target log. A first machine learning model is used to perform semantic analysis on the updated log to obtain the second business content. Based on the difference between the second business content and the baseline business content, the contribution of each newly added log content in the updated log to the completeness of the target business is determined. When the similarity between the second business content and the baseline business content is greater than a preset similarity threshold and the contribution of each newly added log content to the completeness of the target business is greater than a preset contribution threshold, the updated log is output. The above method enables us to first infer the current business content based on the target log, then combine multimodal process data to filter process features related to the current business content, and use the relevant process features to complete and update the target log. At the same time, the update results are verified based on the baseline business content, thereby improving the accuracy of log generation results and business consistency.

[0009] According to an embodiment of this application, a first machine learning model is used to perform semantic analysis on a target log to obtain first business content, and the business semantic distribution corresponding to each log segment in the target log is determined based on the first business content. This includes: semantically encoding multiple text units in the target log to obtain multiple semantic representation vectors; dividing the target log into segments based on the multiple semantic representation vectors to obtain multiple log segments; performing business semantic recognition on each log segment to obtain the business semantic distribution corresponding to each log segment; and determining the first business content corresponding to the target log based on the business semantic distribution corresponding to each log segment. By performing fragmented semantic analysis on the target log, the ability to characterize the relationship between the local semantics and the overall business content of the target log can be improved, providing a foundation for matching features with log segments in subsequent processes.

[0010] According to an embodiment of this application, a second machine learning model is used to extract features from process data to obtain multiple process feature vectors. Based on the distribution relationship of each process feature vector in the time dimension, candidate correspondences between each process feature vector and each log segment are determined. This includes: extracting features from data of different modalities in the process data to obtain multiple modal features; performing timestamp calculation and semantic mapping on the multiple modal features to obtain multiple process feature vectors; and matching each process feature vector with each log segment based on at least one of the time position, duration, and event triggering order to determine candidate correspondences between each process feature vector and each log segment. By uniformly representing different modal data and establishing candidate correspondences between them and log segments, the alignment between multimodal process data and target logs can be improved.

[0011] According to the embodiments of this application, the correlation between each process feature vector and the first business content is calculated, including: based on the business semantic distribution corresponding to each log segment and the candidate correspondence between each process feature vector and each log segment, calculating the degree of matching between each process feature vector and the first business content in at least one of the semantic dimension, time dimension, and business rule dimension; and determining the correlation between each process feature vector and the first business content based on the calculated degree of matching. Determining the correlation through a multi-dimensional matching method can reduce the interference of irrelevant process features on the log completion results.

[0012] According to an embodiment of this application, the method for determining the parts to be completed in the target log and the target process feature vectors corresponding to each part to be completed is based on the correlation between each target process feature vector and the first business content. This includes: determining the target log segment corresponding to each target process feature vector based on the candidate correspondence between each target process feature vector and each log segment; determining the parts to be completed based on the correlation between each target process feature vector and the first business content and the log completeness of each target log segment; and determining the corresponding target process feature vector based on the determined parts to be completed. By first determining the target log segment and then combining the correlation and log completeness to locate the parts to be completed, the targeting of the identification of the parts to be completed can be improved.

[0013] According to an embodiment of this application, an updated log is generated based on the target process feature vectors corresponding to each part to be completed, the correlation between each target process feature vector and the first business content, and the target log. This includes: converting the target log into a log vector; generating a completed log vector based on the target process feature vectors corresponding to each part to be completed; determining the update position of each completed log vector in the target log based on the correlation between each target process feature vector and the first business content; and updating the log vectors to generate the updated log. Using a vectorized update method for log completion improves the structure and computability of the log generation process.

[0014] According to an embodiment of this application, the method further includes: when the similarity between the second business content and the baseline business content is less than a preset similarity threshold and / or the contribution of each newly added log content to the target business integrity is less than a preset contribution threshold, performing semantic analysis again based on the updated log to obtain new first business content; redetermining the candidate correspondence between each process feature vector and each log segment, the correlation between each process feature vector and the new first business content, and the parts to be completed based on the new first business content; updating the updated log again based on the redetermined parts to be completed, until the similarity between the second business content and the baseline business content is greater than a preset similarity threshold and the contribution of each newly added log content to the target business integrity is greater than a preset contribution threshold, or the number of iterations reaches a preset number threshold. Through the iterative update mechanism, the matching degree between the updated log and the target business baseline content can be gradually improved.

[0015] According to embodiments of this application, process data includes at least two of image data, audio data, video data, text data, and execution data. By combining process data from at least two different modalities, the completeness of the representation of the business execution process can be improved.

[0016] According to embodiments of this application, the preset timing constraints are used to limit at least one of the following relationships: the sequential relationship, the time interval relationship, and the node connection relationship between the newly added log content in the updated log and the original log content in the target log. By constraining the timing relationship between the newly added log content and the original log content, the rationality of the updated log in the time dimension can be improved.

[0017] According to embodiments of this application, business logic constraints are used to limit at least one of the following relationships between the newly added log content in the updated log and the baseline business content corresponding to the business identifier: process dependency, node triggering relationship, and business rule consistency relationship. By introducing business logic constraints, the consistency between the updated log and the actual execution logic of the target business can be improved.

[0018] This application also provides a log generation device based on multimodal process data, the device comprising: a data acquisition module, a first analysis module, a feature extraction module, a correlation determination module, a completion determination module, a log generation module, a second analysis module, and an output module. The data acquisition module is used to acquire process data during the execution of the target business and the target log corresponding to the business identifier based on the business identifier of the target business, and to retrieve the baseline business content corresponding to the target business from the preset business database based on the business identifier. The first analysis module is used to perform semantic analysis on the target log using a first machine learning model to obtain the first business content, and to determine the business semantic distribution corresponding to each log segment in the target log based on the first business content. The feature extraction module is used to extract features from the process data using a second machine learning model to obtain multiple process feature vectors, and to determine the candidate correspondence between each process feature vector and each log segment based on the distribution relationship of each process feature vector in the time dimension. The correlation determination module is used to calculate the correlation between each process feature vector and the first business content based on the business semantic distribution corresponding to each log segment and the candidate correspondence between each process feature vector and each log segment, and to determine the target process feature vector with a correlation greater than a preset correlation threshold. The completion determination module is used to determine the candidate correspondence between each target process feature vector and each log segment. The system employs a correspondence-based approach, calculating the correlation between each target process feature vector and the first business content. Based on this correlation, it determines the parts of the target log to be completed and the corresponding target process feature vectors. A log generation module generates an updated log based on the target process feature vectors corresponding to each part to be completed, the correlation between these vectors and the first business content, and the target log itself. The updated log content satisfies preset temporal constraints and business logic constraints corresponding to the business identifier, while the newly added log content in the updated log satisfies preset temporal constraints and business logic constraints corresponding to the business identifier. A second analysis module performs semantic analysis on the updated log using a first machine learning model to obtain the second business content. Based on the difference between the second business content and the baseline business content, it determines the contribution of each newly added log content in the updated log to the completeness of the target business. An output module outputs the updated log when the similarity between the second business content and the baseline business content is greater than a preset similarity threshold and the contribution of each newly added log content to the completeness of the target business is greater than a preset contribution threshold.

[0019] According to an embodiment of this application, the first analysis module is specifically used for: semantically encoding multiple text units in the target log to obtain multiple semantic representation vectors; dividing the target log into segments based on the multiple semantic representation vectors to obtain multiple log segments; performing business semantic recognition on each log segment to obtain the business semantic distribution corresponding to each log segment; and determining the first business content corresponding to the target log based on the business semantic distribution corresponding to each log segment.

[0020] According to the embodiments of this application, the feature extraction module is specifically used to: extract features from data of different modalities in the process data to obtain multiple modal features; perform timestamp calculation and semantic mapping on the multiple modal features to obtain multiple process feature vectors; and match each process feature vector with each log segment based on at least one of the time position, duration, and event triggering order to determine the candidate correspondence between each process feature vector and each log segment.

[0021] According to an embodiment of this application, the correlation determination module is specifically used to: calculate the degree of matching between each process feature vector and the first business content in at least one of the semantic dimension, time dimension, and business rule dimension, based on the business semantic distribution corresponding to each log segment and the candidate correspondence between each process feature vector and each log segment; and determine the correlation between each process feature vector and the first business content based on the calculated degree of matching.

[0022] According to an embodiment of this application, the module for determining the part to be completed is specifically used to: determine the target log segment corresponding to each target process feature vector based on the candidate correspondence between each target process feature vector and each log segment; determine the part to be completed based on the correlation between each target process feature vector and the first business content and the log completeness of each target log segment; and determine the corresponding target process feature vector based on the determined part to be completed.

[0023] According to an embodiment of this application, the log generation module is specifically used to: convert the target log into a log vector; generate a completed log vector based on the target process feature vector corresponding to each part to be completed; determine the update position of each completed log vector in the target log based on the correlation between each target process feature vector and the first business content; and update the log vector to generate an updated log.

[0024] According to an embodiment of this application, the device further includes an iterative update module, which is used to: when the similarity between the second business content and the baseline business content is less than a preset similarity threshold and / or the contribution of each newly added log content to the target business integrity is less than a preset contribution threshold, perform semantic analysis again based on the updated log to obtain new first business content; redetermine the candidate correspondence between each process feature vector and each log segment, the correlation between each process feature vector and the new first business content, and the parts to be completed based on the new first business content; update the updated log again based on the redetermined parts to be completed, until the similarity between the second business content and the baseline business content is greater than a preset similarity threshold and the contribution of each newly added log content to the target business integrity is greater than a preset contribution threshold, or the number of iterations reaches a preset number threshold.

[0025] According to embodiments of this application, process data includes at least two of the following: image data, audio data, video data, text data, and execution data.

[0026] This application also provides an electronic device, including one or more processors and a memory, the memory being used to store one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the above-described log generation method based on multimodal process data.

[0027] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described log generation method based on multimodal process data.

[0028] This application provides a log generation method, apparatus, and electronic device based on multimodal process data. It obtains process data, target logs, and corresponding baseline business content during the execution of a target business based on the business identifier of that business. First, it uses a first machine learning model to perform semantic analysis on the target logs to obtain the first business content representing the business execution content reflected in the current logs and the business semantic distribution corresponding to each log segment. Then, it uses a second machine learning model to extract features from the process data to obtain multiple process feature vectors. Based on the candidate correspondence between the process feature vectors and each log segment, and combined with the business semantic distribution corresponding to each log segment, it calculates the relationship between each process feature vector and the first... The correlation and relevance between business content are analyzed to determine the parts to be completed in the target log and the target process feature vectors corresponding to each part. Based on the target process feature vectors, relevance, and target log, an updated log that meets preset time constraints and business logic constraints is generated. Then, semantic analysis is performed on the updated log to obtain the second business content. The contribution of the new log content to the target business integrity is determined based on the difference between the second business content and the baseline business content. When the similarity between the second business content and the baseline business content is greater than a preset similarity threshold and the contribution of each new log content to the target business integrity is greater than a preset contribution threshold, the updated log is output. Compared to existing technologies that rely solely on single text logs for log completion or process analysis, which struggle to effectively utilize multimodal process information such as images, audio, video, text, and execution data during business execution—leading to problems like difficulty in accurately locating missing log portions, inconsistencies between completed content and actual business scenarios, lack of temporal rationality and business logic consistency in completed results, difficulty in verifying update results, and difficulty in accurately reconstructing the true business execution trajectory—this application addresses these issues by associating and collaboratively utilizing the semantic analysis results of the target log with the feature extraction results of multimodal process data. Furthermore, it combines this with baseline business content to perform dual verification of similarity and contribution of the updated log. This effectively improves the accuracy of identifying the parts to be completed, enhances the matching degree between the newly added log content and the actual execution process of the target business, strengthens the rationality of the updated log in terms of temporal sequence and business logic, and improves the ability of the log generation results to represent the integrity of the target business. This, in turn, enhances the accuracy, reliability, and usability of log completion and updates, providing a more reliable data foundation for subsequent process analysis, business auditing, anomaly identification, and risk management.

[0029] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0030] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart of an embodiment of the log generation method based on multimodal process data provided in this application; Figure 2 A schematic diagram of the structure of an embodiment of the log generation device based on multimodal process data provided in this application; Figure 3 A schematic diagram of the structure of an embodiment of the electronic device provided in this application. Detailed Implementation

[0031] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0032] It should be noted that in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish different objects and should not be construed as limiting the order of importance, sequence, or quantity. Unless otherwise expressly stated, the terms "comprising," "including," etc., are intended to cover non-exclusive inclusion, such that a process, method, apparatus, or system that comprises a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, apparatus, or system.

[0033] The business identifier involved in this application embodiment can be used to uniquely identify a target business, target process, target task, or target event instance; the target log can be a text log, structured log, or event log that records the execution process of the target business; the process data can be multimodal data collected, recorded, or generated during the execution of the target business; the baseline business content can be a standard business description, standard process content, standard node information, or standard business semantic template pre-stored in a preset business database and corresponding to the business identifier. The first machine learning model and the second machine learning model mentioned in this application can be a neural network model, a deep learning model, a large language model, a multimodal model, or other models capable of completing the corresponding analysis tasks, and this application does not limit them.

[0034] In this embodiment, the business identifier can be a single number or a combined identifier. The combined identifier can be formed by combining business type identifiers, process instance identifiers, timestamp identifiers, subject identifiers, terminal identifiers, order identifiers, or other information capable of jointly locating the target business. Electronic devices can use multiple fields in the combined identifier to aggregate and associate target logs and process data scattered across different data sources, thereby determining the data content belonging to the same target business execution process.

[0035] In this embodiment, the target log can be a single log file, a collection of multiple log files, a set of log records in a log database, or a collection of log fragments corresponding to the target business extracted from a complete log. The target log can be in plain text format, structured field format, semi-structured event format, or a hybrid format combining both. The log fragments mentioned in this application can be divided according to time windows, event boundaries, semantic boundaries, field boundaries, or line break separation methods. For a target log containing multiple log record items, a log fragment can correspond to one log record item or a local segment composed of multiple consecutive log record items.

[0036] In this embodiment, the first business content characterizes the target business execution content inferred from the current record content of the target log, reflecting the business semantic result from the perspective of the original log. The second business content characterizes the target business execution content obtained from the analysis of the updated log, reflecting the business semantic result after introducing new log content. The baseline business content characterizes the business content that the target business should possess in a standard execution state, a preset template state, or a rule-constrained state. Any one of the first business content, the second business content, and the baseline business content can be represented in the form of business semantic tags, a set of business sub-contents, a sequence of process nodes, a set of structured fields, a semantic vector, a set of knowledge graph nodes, or any combination thereof.

[0037] In this embodiment, the business semantic distribution can be used to characterize the degree to which a certain log segment belongs to multiple candidate business semantic categories, business nodes, or business sub-contents. The business semantic distribution can be determined by the classification probability, clustering probability, attention weight, semantic similarity score, or confidence score output by the first machine learning model, and can be represented as a probability set, weight set, or score set corresponding to multiple business semantic labels.

[0038] In this embodiment, candidate correspondences are used to characterize a preliminary matching relationship between process feature vectors and log fragments, indicating a possible correspondence. Candidate correspondences can be one-to-one, one-to-many, many-to-one, or many-to-many relationships, and can include at least one of the following indicators: temporal overlap, sequential matching, semantic proximity, and candidate confidence. Electronic devices can establish candidate correspondences based on time alignment results, event sequence consistency, contextual interval overlap, or preliminary semantic matching results.

[0039] In the embodiments of this application, correlation and relevance can correspond to different technical meanings in different processing stages. Correlation is used to determine whether a certain process feature vector belongs to the candidate features related to the first business content, mainly used for screening process feature vectors; relevance is used to determine the strength of the completion effect of the target process feature vector on a specific part to be completed, mainly used for log generation after the part to be completed is determined and updated. That is, correlation is more inclined to the screening judgment of whether it is related, while relevance is more inclined to the judgment of how high the degree of relevance is and how strong the support for completion is in the update utilization.

[0040] In this embodiment, the part to be completed can be at least one of the following: the location corresponding to a missing event in the target log, the location corresponding to a missing attribute field, the time breakpoint between log segments, the location of an event timing error, the location of an incomplete node state, or the location of a broken business context. The part to be completed can be located within a single log segment or in the gap between multiple log segments.

[0041] In this embodiment, log completeness can be determined based on at least one of the following: event completeness rate, attribute completeness rate, temporal continuity, contextual coherence, process closure, and degree of business rule fulfillment. Electronic devices can determine a log completeness score by comparing the current target log segment with baseline business content, historical complete samples, or a preset log template, and identify log segments with completeness scores below a preset completeness threshold as candidate parts to be completed.

[0042] In this embodiment, timing constraints can originate from baseline business content, historical business samples, rule configuration files, business knowledge bases, or system preset rules. Timing constraints can be used to limit at least one of the following relationships: sequential order of different log content, minimum or maximum time interval, node connection, and trigger time window. Business logic constraints can also originate from baseline business content, rule engines, process templates, or knowledge graphs. Business logic constraints can be used to limit at least one of the following relationships: dependency of preceding nodes, node triggering relationships, role and permission relationships, state transition relationships, field consistency relationships, and business rule consistency relationships.

[0043] In this embodiment, the contribution score is used to characterize the degree to which the newly added log content improves the integrity of the target business. The contribution score can be determined based on at least one of the following: the degree to which the newly added log content supplements missing business processes, the degree to which it improves the log's temporal integrity, the degree to which it improves the consistency of business logic, the amount of increase in the similarity between the second business content and the baseline business content, and the weighted contribution to the recovery of key business nodes. The contribution score can be represented by a score, probability value, weight value, gain value, or level value.

[0044] In this embodiment, the preset similarity threshold and preset contribution threshold can be determined based on historical sample training results, manual experience setting results, business rule configuration results, statistical results of different business types, or dynamic threshold adjustment strategies. Different preset similarity thresholds and preset contribution thresholds can be set for target businesses of different business types, different business scenarios, or different risk levels. The preset similarity threshold and preset contribution threshold can be fixed thresholds, interval thresholds, graded thresholds, or dynamically updated thresholds.

[0045] In this embodiment, the iterative update mechanism can terminate not only by reaching a preset number of times threshold, but also by at least one of the following conditions: the similarity increase of logs after two adjacent updates is less than a preset change threshold; the contribution change of newly added log content in two adjacent updates is less than a preset change threshold; no new parts to be completed are identified after reanalysis; or the updated logs have met all preset output conditions. By setting the above termination conditions, the consumption of computing resources and processing latency can be controlled while ensuring the quality of log generation.

[0046] First Embodiment In industrial production, government affairs processing, financial transactions, customer service, equipment maintenance, and other applications that rely on information systems for process management, systems typically record the execution of business processes through logs and use these logs for process tracking, business auditing, anomaly detection, compliance analysis, and risk identification. In existing technologies, logs usually record business events, time information, executing entities, and business status in text or structured field format. Therefore, subsequent analysis of business processes typically relies primarily on the text or structured information already recorded in the logs.

[0047] However, in practical applications, log recording is often affected by factors such as system failures, human error, network fluctuations, interface anomalies, and device interruptions, leading to situations where logs contain missing events, missing attributes, missing time sequences, missing key nodes, or incomplete records. Existing technologies typically rely on existing log context, historical log samples, statistical patterns, or simple machine learning models for completion to address these issues. Because such solutions primarily rely on inferences based on the text logs themselves, they struggle to effectively utilize process information such as images, audio, video, text, and execution data generated during business execution. Consequently, the resulting completion results are prone to deviating from the actual business execution scenario.

[0048] Furthermore, in many real-world business scenarios, information that truly reflects the business execution status, process trajectory, and node changes exists not only in text logs but also extensively in process data, such as on-site operation videos, equipment operation images, voice commands, document images, screen recordings, and terminal execution data. Existing technologies struggle to accurately identify information relevant to the current business content from this multimodal process data when the target log is incomplete or distorted. They also find it difficult to supplement and update the logs with temporal consistency and business logic based on this information. Therefore, log completion results often fail to accurately reconstruct the true business trajectory, thus affecting the accuracy of subsequent process analysis, audit judgment, and anomaly identification.

[0049] To address the aforementioned issues, this application provides a log generation scheme based on multimodal process data. The scheme obtains process data and corresponding target logs from the execution of the target service based on its business identifier. It then retrieves baseline business content from a pre-defined business database based on the business identifier. Next, a first machine learning model is used to perform semantic analysis on the target logs to obtain the first business content and the business semantic distribution corresponding to each log segment. A second machine learning model is then used to extract features from the process data to obtain multiple process feature vectors, and candidate correspondences are established between each process feature vector and each log segment. Based on the business semantic distribution and candidate correspondences, target process feature vectors related to the first business content are determined, and the parts to be completed in the target log are identified accordingly to generate an updated log. Finally, semantic analysis is performed on the updated logs to obtain the second business content. The updated logs are then validated and output based on the similarity between the second business content and the baseline business content, as well as the contribution of the newly added log content to the completeness of the target service.

[0050] The above approach first determines the business content reflected in the current log based on the target log, then filters out process features related to the business content by combining multimodal process data, thereby enabling targeted completion and updating of the parts to be completed. At the same time, the updated log is double-verified by the baseline business content and contribution, thereby improving the matching degree between the log generation results and the actual business execution process, improving the rationality of the updated log in terms of time sequence and business logic, enhancing the log's ability to represent the integrity of the target business, and providing a more reliable data foundation for subsequent process analysis, business auditing, anomaly identification, and risk control.

[0051] Second Embodiment Figure 1 A flowchart illustrating a log generation method based on multimodal process data according to an embodiment of this application is shown. This method can be executed by an electronic device, which can be a server, cloud platform device, edge computing device, terminal device, or other device with data processing capabilities. Figure 1 As shown in the embodiments of this application, the log generation method may include: S101: Based on the business identifier of the target business, obtain the process data during the execution of the target business and the target log corresponding to the business identifier, and obtain the baseline business content corresponding to the target business from the preset business database based on the business identifier.

[0052] In this embodiment, the business identifier can be a business number, process instance number, order number, task number, case number, work order number, or other information capable of identifying the target business instance. The electronic device can, based on the business identifier, obtain process data and target logs related to the execution process of the target business from a business system, log system, database, object storage, file system, message queue, monitoring system, or other data source. The target logs may include at least some event information, time information, subject information, and status information recorded during the execution of the target business.

[0053] Process data can include multimodal data generated or collected during the execution of the target business, such as image data, audio data, video data, text data, and execution data. Image data can include images of business documents, equipment status images, and interface screenshots; audio data can include voice commands, dialogue voice, and broadcast voice; video data can include on-site video, monitoring video, and screen recordings; text data can include instant message text, explanatory text, and annotation text; and execution data can include operation records, equipment operating parameters, click trajectories, control commands, and sensor data.

[0054] The pre-defined business database can store baseline business content corresponding to multiple business types or business instances. This baseline business content can be used to characterize the business nodes, business semantics, process sequence, logical dependencies, and completed content that the target business should possess under normal or standard execution conditions. After obtaining the business identifier, the electronic device can further query and read the baseline business content corresponding to that business identifier from the pre-defined business database for subsequent verification of the updated logs.

[0055] In step S101, target logs, process data, and baseline business content can be acquired simultaneously, which can establish a unified data foundation for subsequent log semantic analysis, multimodal feature extraction, determination of parts to be completed, and log verification after update, so that the entire processing flow revolves around the same target business.

[0056] Furthermore, in this embodiment of the application, the process data obtained in step S101 may include at least two of the following: image data, audio data, video data, text data, and execution data.

[0057] For example, process data is not limited to a single modality but can include at least two different types of data to improve the completeness of the representation of the target business execution process. For instance, in industrial scenarios, process data can simultaneously include equipment operation videos and control command data; in government processing scenarios, process data can simultaneously include document images and business dialogue audio; and in service scenarios, process data can simultaneously include screen recording videos and interactive text. By introducing at least two different modalities of data, the actual business execution situation can be more fully reflected, thereby improving the accuracy of subsequent log generation.

[0058] S102, use the first machine learning model to perform semantic analysis on the target log to obtain the first business content, and determine the business semantic distribution corresponding to each log fragment in the target log based on the first business content.

[0059] In step S102, the first machine learning model can be used to perform semantic understanding and business content inference on the text content, structured fields, or event sequences in the target log. The first machine learning model can be a language model, sequence analysis model, classification model, or clustering model trained on business corpus. The target log can be input into the first machine learning model to obtain the first business content reflected in the current target log. The first business content can be understood as the business execution content or business status description inferred based on existing log records.

[0060] Furthermore, the electronic device can perform semantic localization and semantic association analysis on each log segment within the target log based on the first business content, in order to determine the business semantic distribution corresponding to each log segment. The business semantic distribution can characterize the probability distribution, confidence distribution, or weight distribution of a log segment across multiple candidate business semantic categories, business nodes, or business sub-contents. Through this business semantic distribution, the relationship between local segments of the target log and the overall business content can be described in a more granular manner.

[0061] Since it is necessary to associate the multimodal process features with the log fragments in the future, the business semantic distribution at the log fragment level is obtained first in S102. This helps to improve the matching accuracy between the subsequent process features and the log content, and also helps to improve the accuracy of locating the parts to be completed.

[0062] For example, the business semantic distribution output by the first machine learning model can be a probability vector for multiple preset business semantic categories. For instance, for the same log segment, the first machine learning model can output the probability values ​​of the log segment belonging to multiple business sub-contents such as application submission, information verification, approval processing, and result notification. The electronic device can take the business sub-content with the highest probability value as the main semantic category of the log segment, or it can retain multiple business sub-contents and their corresponding probability values ​​as the complete business semantic distribution of the log segment.

[0063] In this embodiment, the first business content corresponding to the target log can be determined based on the business semantic distribution corresponding to multiple log segments, using methods such as weighted aggregation, sequence inference, probabilistic fusion, or graph structure reasoning. For example, the main semantic categories of all log segments can be statistically analyzed, and the business topic with the highest cumulative probability value can be selected as the first business content; alternatively, a sequence of business sub-contents can be constructed based on the chronological order of the log segments, and the first business content can be output by the sequence model. This approach helps improve the ability of the first business content to represent the overall business state of the target log.

[0064] Furthermore, in this embodiment of the application, using a first machine learning model to perform semantic analysis on the target log to obtain the first business content, and determining the business semantic distribution corresponding to each log segment in the target log based on the first business content may further include: semantically encoding multiple text units in the target log to obtain multiple semantic representation vectors; dividing the target log into segments based on the multiple semantic representation vectors to obtain multiple log segments; performing business semantic recognition on each log segment to obtain the business semantic distribution corresponding to each log segment; and determining the first business content corresponding to the target log based on the business semantic distribution corresponding to each log segment.

[0065] For example, the target log can first be split into multiple text units, such as words, phrases, sentences, event description fields, or structured record items, and then these text units can be semantically encoded using a first machine learning model to obtain multiple semantic representation vectors. Subsequently, the target log can be segmented based on semantic similarity, temporal adjacency, event boundaries, or business field distribution to form multiple log fragments.

[0066] For each log segment, a first machine learning model can be invoked to perform business semantic recognition, thereby obtaining the business semantic distribution corresponding to that log segment. For example, a log segment may correspond to one or more different business sub-contents such as application submission, document verification, approval processing, and result feedback, each with different probability values. Finally, the business semantic distributions corresponding to all log segments can be combined to determine the first business content corresponding to the target log. This approach can simultaneously consider the semantics of local segments and the inference of the overall business content, thus improving the stability of the semantic analysis results.

[0067] S103, use the second machine learning model to extract features from the process data to obtain multiple process feature vectors, and determine the candidate correspondence between each process feature vector and each log segment based on the distribution relationship of each process feature vector in the time dimension.

[0068] In step S103, a second machine learning model can be used to extract features from the process data. This second machine learning model can be a multimodal feature extraction model or a multimodal fusion model, used to extract feature vectors with a unified representation from process data of different modalities. For different modalities of data, the second machine learning model can employ different sub-models or encoders. For example, a visual encoding model can be used for image data, a speech recognition model or an audio encoding model can be used for audio data, a temporal visual model can be used for video data, and a temporal modeling model or a structured data encoding model can be used for execution data.

[0069] After obtaining multiple process feature vectors, their distribution along the time dimension can be further analyzed. This time dimension can include acquisition time, occurrence time, duration, triggering order, time interval, or event timeline position. By matching the time information of the process feature vectors with the time information, sequence information, or contextual segments of each log segment in the target log, candidate correspondences between each process feature vector and each log segment can be determined. A candidate correspondence can be understood as a preliminary matching relationship where a certain process feature vector may be associated with a certain log segment.

[0070] By first establishing a candidate correspondence between process feature vectors and log fragments in the time dimension, the scope of subsequent correlation calculations can be narrowed, and the interference of irrelevant process features on subsequent completion processing can be reduced.

[0071] In one alternative implementation, candidate correspondences are not limited to unique mappings. The same process feature vector can correspond to multiple log segments simultaneously; similarly, the same log segment can correspond to multiple process feature vectors simultaneously. The electronic device can assign candidate confidence scores to each set of candidate correspondences. Candidate confidence scores can be determined by at least one of temporal overlap, sequential consistency, semantic proximity, and contextual matching.

[0072] For example, when the time interval of a video segment overlaps with the time windows corresponding to two consecutive log segments, and the business actions reflected in the video are related to the context of both log segments, the electronic device can simultaneously establish candidate correspondences between the process feature vector corresponding to the video segment and the two log segments, and assign different candidate confidence levels to each. Subsequently, when calculating correlation and relevance, the candidate confidence levels can be further used to filter or weight different candidate correspondences.

[0073] Furthermore, in this embodiment, using a second machine learning model to extract features from the process data to obtain multiple process feature vectors, and determining the candidate correspondence between each process feature vector and each log segment based on the distribution relationship of each process feature vector in the time dimension, may further include: extracting features from data of different modalities in the process data to obtain multiple modal features; performing timestamp calculation and semantic mapping on the multiple modal features to obtain multiple process feature vectors; and matching each log segment with each process feature vector based on at least one of the time position, duration, and event triggering order to determine the candidate correspondence between each process feature vector and each log segment.

[0074] For example, in the embodiments of this application, feature extraction operations can be performed on each modality of data separately. For example, visual semantic features can be extracted from image data, speech transcription results and semantic features can be extracted from audio data, action timing features and scene change features can be extracted from video data, text semantic features can be extracted from text data, and state change features and operation event features can be extracted from execution data. Then, corresponding timestamp information can be attached to different modality features, and different modality features can be converted into multiple process feature vectors through a unified semantic space mapping.

[0075] Furthermore, the log generation method according to embodiments of this application can also match each log segment with at least one of the time position, duration, and event triggering order of each process feature vector. For example, if the time interval of a process feature vector overlaps with the time range corresponding to a log segment, or if the event order reflected by a process feature vector is consistent with the contextual order of a log segment, then a candidate correspondence can be established between the two. This approach helps improve the temporal alignment capability between multimodal features and log segments.

[0076] S104. Based on the business semantic distribution corresponding to each log segment and the candidate correspondence between each process feature vector and each log segment, calculate the correlation between each process feature vector and the first business content, and determine the target process feature vector with a correlation greater than the preset correlation threshold.

[0077] In step S104, based on the business semantic distribution corresponding to each log segment and the candidate correspondence between each process feature vector and each log segment, the correlation between each process feature vector and the first business content can be calculated, and the target process feature vector with a correlation greater than a preset correlation threshold can be determined. In this embodiment, correlation can be used to characterize whether a certain process feature vector has a sufficiently strong correlation with the first business content, so as to filter out the target process feature vector that is truly related to the current business content from multiple process feature vectors. The electronic device can comprehensively utilize the business semantic distribution corresponding to the log segment and the candidate correspondence to perform correlation calculation on each process feature vector separately.

[0078] For example, if there is a candidate correspondence between a process feature vector and a log segment, and the log segment has a high business semantic weight in the first business content, then the correlation between the process feature vector and the first business content can be improved accordingly. Conversely, if a process feature vector is close to a log segment in time, but the business semantic distribution of the log segment deviates significantly from the first business content, then the correlation between the process feature vector and the log segment can be reduced.

[0079] After obtaining the correlation between each process feature vector and the first business content, in step S104, process feature vectors with a correlation greater than a preset correlation threshold can be identified as target process feature vectors for further correlation calculation and log completion. This filtering process can reduce the probability of irrelevant process features entering subsequent processing flows.

[0080] For example, in this embodiment, relevance can be represented as a continuous numerical value or as a discrete level. In the continuous numerical form, relevance can take values ​​between 0 and 1; in the discrete level form, relevance can be divided into three or more levels: high, medium, and low. The electronic device can normalize the calculated semantic matching degree, temporal matching degree, and business rule matching degree, and fuse them according to preset weights to obtain the final relevance.

[0081] For example, in a specific business scenario, the semantic matching degree can be determined by the vector similarity between the process feature vector and the first business content; the temporal matching degree can be determined by the overlap rate of the target time interval; and the business rule matching degree can be determined by whether the target process features meet the preset process constraints. Electronic devices can fuse these three dimensions with weights of, for example, 0.5, 0.3, and 0.2 to obtain the correlation between the process feature vector and the first business content. This application does not limit the specific weight values.

[0082] Furthermore, in this embodiment of the application, calculating the correlation between each process feature vector and the first business content may include: based on the business semantic distribution corresponding to each log segment and the candidate correspondence between each process feature vector and each log segment, calculating the degree of matching between each process feature vector and the first business content in at least one of the semantic dimension, time dimension and business rule dimension; and determining the correlation between each process feature vector and the first business content based on the calculated degree of matching.

[0083] For example, in step S104, the degree of matching can be calculated from multiple dimensions. The semantic dimension can reflect the semantic similarity between the process information corresponding to the process feature vector and the first business content; the time dimension can reflect the degree of matching between the occurrence time or duration of the process feature vector and the time position of the log segment; the business rule dimension can reflect whether the business behavior corresponding to the process feature vector conforms to the process rules, node dependencies, or operational constraints that the target business should have.

[0084] In step S104, the matching degrees of one or more of the above dimensions can be weighted and fused to obtain the correlation between the process feature vector and the first business content. For example, the semantic dimension weight can be set higher than the time dimension weight, or the weights of each dimension can be dynamically adjusted according to the specific business scenario. Through this multi-dimensional correlation calculation method, the accuracy and robustness of process feature selection can be improved.

[0085] S105, based on the candidate correspondence between each target process feature vector and each log segment, calculate the correlation between each target process feature vector and the first business content, and determine the parts to be completed in the target log and the target process feature vectors corresponding to each part to be completed according to the correlation between each target process feature vector and the first business content.

[0086] In this embodiment, after the target process feature vectors are selected, the correlation between these target process feature vectors and the first business content can be further calculated. Compared with the correlation in S104, the correlation can focus more on characterizing the specific support of a target process feature vector for the target log completion task, such as the ability to complete missing nodes, missing attributes, or missing time-series information.

[0087] In step S105, the candidate correspondence between the target process feature vector and each log segment can be combined to analyze which segments in the target log may have missing content, incomplete information, or time sequence anomalies. For example, when a log segment lacks a record corresponding to the expected process of the target business, but there is a highly relevant target process feature vector within the corresponding time range, the part corresponding to that log segment can be identified as the part to be completed.

[0088] After determining the parts to be completed, a mapping relationship can be established between each part to be completed and the corresponding target process feature vector, so that the corresponding parts to be completed can be updated in a targeted manner according to the corresponding process feature vector, instead of indiscriminately completing the entire target log.

[0089] For example, relevance can be used to characterize the completion value of a target process feature vector for a specific part to be completed. Relevance considers not only the overall semantic consistency between the target process feature vector and the first business content, but also the targeted completion capability of the target process feature vector for missing content in the target log segment. For instance, if a target process feature vector can directly reflect the occurrence time, executing entity, and execution result of the missing node, then the relevance of that target process feature vector to the corresponding part to be completed can be high.

[0090] In step S104, the log completeness of the target log segment can be quantitatively evaluated first, and then the part to be completed can be determined by combining the correlation between the target process feature vector and the first business content. The quantitative result of log completeness can be event completeness rate, attribute completeness rate, temporal continuity rate, semantic closure rate, or comprehensive score. When the comprehensive score is lower than the preset completeness threshold, the missing, abnormal, or incomplete positions in the corresponding log segment can be marked as the part to be completed. This method can avoid misjudgment caused by completing and locating solely based on the target process feature vector.

[0091] Furthermore, in this embodiment of the application, determining the parts to be completed in the target log and the target process feature vectors corresponding to each part to be completed based on the correlation between each target process feature vector and the first business content may further include: determining the target log segment corresponding to each target process feature vector based on the candidate correspondence between each target process feature vector and each log segment; determining the parts to be completed based on the correlation between each target process feature vector and the first business content and the log completeness of each target log segment; and determining the corresponding target process feature vector based on the determined parts to be completed.

[0092] For example, in step S105, each target process feature vector can be mapped to a corresponding target log segment based on the candidate correspondence. A target log segment can be a single log fragment or a local interval composed of multiple consecutive log fragments. Afterward, the electronic device can evaluate the log completeness of each target log segment. Log completeness can be determined based on indicators such as whether key events are missing, whether attribute fields are missing, whether there are time breakpoints, and whether there are business context breaks.

[0093] When the log integrity of a target log segment is lower than the preset integrity requirement, and the corresponding target process feature vector has a high correlation with the first business content, the corresponding part in the target log segment can be identified as the part to be completed, and the corresponding target process feature vector can be used as the data basis for completing this part. This method can improve the precision of identifying the part to be completed.

[0094] S106. Based on the target process feature vectors corresponding to each part to be completed, the correlation between each target process feature vector and the first business content, and the target log, generate the updated log.

[0095] In step S106, after determining the part to be completed and the corresponding target process feature vector, the target log can be completed, corrected, or updated based on the process information provided by these target process feature vectors to generate an updated log. The newly added log content in the updated log may include new event records, supplementary attribute information, corrected time sequence, supplementary node status, supplementary operation instructions, etc.

[0096] During the generation of updated logs, not only can the process information reflected by the target process feature vectors be utilized, but the credibility, priority, or update weight of different completion information can also be determined by combining the correlation between each target process feature vector and the first business content. For example, a target process feature vector with higher relevance can be used first to update the corresponding part to be completed, or when multiple candidate completion contents conflict, the content corresponding to the process feature vector with higher relevance can be used first.

[0097] Furthermore, when generating updated logs, it can be ensured that the newly added log content meets preset timing constraints and business logic constraints compared to the original log content. Timing constraints ensure that the order of events is reasonable, while business logic constraints ensure that the newly added content conforms to the process dependencies and rule requirements of the target business. Through these constraints, it is possible to avoid generating updated logs that do not conform to the actual business execution logic.

[0098] In this embodiment, the update position may include at least one of the following: insertion position, replacement position, correction position, and append position. The insertion position is used to add new log content between existing log content; the replacement position is used to replace existing erroneous content with the new log content; the correction position is used to modify time fields, status fields, or attribute fields in the existing log content; and the append position is used to add supplementary explanations or attribute information after the existing log content. The electronic device can select the appropriate update position based on the relevance of the target process feature vector and the type of the part to be completed.

[0099] For example, timing constraint checks and business logic constraint checks can be performed before or after generating the updated log. If the newly added log content does not meet the timing constraint or business logic constraint, the newly added log content can be discarded, its write position adjusted, its contribution weight reduced, or it can be reserved as candidate completion content instead of being directly written to the updated log. In this way, the reliability of the updated log and business consistency can be improved.

[0100] Furthermore, in this embodiment of the application, generating an updated log based on the target process feature vector corresponding to each part to be completed, the correlation between each target process feature vector and the first business content, and the target log may include: converting the target log into a log vector; generating a completed log vector based on the target process feature vector corresponding to each part to be completed; determining the update position of each completed log vector in the target log based on the correlation between each target process feature vector and the first business content; and updating the log vector to generate the updated log.

[0101] For example, the target log can first be vectorized, encoding the text content, field content, time information, and node relationships in the log into log vectors. Then, based on the target process feature vectors corresponding to each part to be completed, a completed log vector representing the completed content can be generated. The completed log vector can reflect the semantics, attributes, temporal position, or node type of the newly added log content.

[0102] Furthermore, the update position of each completed log vector in the target log can be determined based on the correlation between the target process feature vector and the first business content. For example, the update position could be an insertion position, a replacement position, a correction position, or a supplementary position. Finally, the electronic device can update the log vectors and restore the updated log vectors to text log or structured log format to generate the updated log. This method improves the structured processing capability of the log update process.

[0103] Furthermore, in this embodiment, the preset timing constraint is used to limit at least one of the following relationships among the new log content in the updated log and the original log content in the target log: the order relationship, the time interval relationship, and the node connection relationship.

[0104] In this embodiment, the electronic device can verify and constrain the temporal relationship between newly added log content and existing log content. For example, if a newly added log content represents approval, the time position of this newly added log content should not be earlier than the application submission event; if there is a minimum or maximum time interval requirement between two nodes, it can be restricted by preset timing constraints; if a newly added node should form a continuous connection with the preceding and following nodes, it can also be restricted by node connection relationships. Through these constraints, the rationality of the updated log in the time dimension can be improved.

[0105] Furthermore, in this embodiment of the application, the business logic constraint is used to limit at least one of the following relationships among the process dependency, node triggering relationship, and business rule consistency relationship between the newly added log content in the updated log and the baseline business content corresponding to the business identifier.

[0106] In this embodiment, business logic constraints can be constructed based on baseline business content in a preset business database. For example, certain business nodes can only be triggered after the preceding node is completed, certain nodes have mutual exclusion relationships, certain attribute fields must remain consistent with the state of the previous node, and certain actions must be performed by a specific entity. When generating the updated log, it can be determined whether the newly added log content meets these logical constraints, and only if the constraints are met can it be written into the updated log. This can further improve the consistency between the updated log and the actual business logic.

[0107] S107, use the first machine learning model to perform semantic analysis on the updated log to obtain the second business content, and based on the difference between the second business content and the baseline business content, determine the contribution of each newly added log content in the updated log to the integrity of the target business.

[0108] In step S107, after generating the updated log, the first machine learning model can be used again to perform semantic analysis on the updated log to obtain the second business content. The second business content can be understood as the business execution content reflected in the updated log at the semantic level. Compared with the first business content obtained in S102, the second business content is usually closer to the true and complete content of the target business because the updated log has incorporated the supplementary parts from the multimodal process data.

[0109] Furthermore, in this embodiment, the second business content can be compared with the baseline business content to determine the differences between them. These differences can manifest as a reduction in missing nodes, more complete business semantic coverage, a more reasonable process sequence, or higher rule consistency. Based on these differences, the electronic device can assess the contribution of each newly added log entry in the updated log to the completeness of the target business. The contribution can be in numerical, scoring, weighted, or probabilistic form.

[0110] For example, if a newly added log entry fills in a critical missing node in the target business, its contribution can be high; if a newly added log entry only fills in a minor attribute field, its contribution can be low. By calculating the contribution of each newly added log entry to the completeness of the target business, a basis can be provided for subsequent output judgments.

[0111] In this embodiment, the electronic device can calculate the contribution of each newly added log entry. The contribution calculation can comprehensively consider at least one of the following indicators: whether the newly added log entry fills a key missing business link; whether the newly added log entry improves the log sequence continuity; whether the newly added log entry improves the consistency of business logic; the increase in similarity between the second business content and the baseline business content after the addition of the newly added log entry; and the importance of the newly added log entry in the target business process. For newly added log entries corresponding to key business nodes, a higher node weight can be assigned, thereby making its contribution higher than that of newly added log entries corresponding to ordinary auxiliary nodes.

[0112] For example, when a newly added log entry completes a missing approval result confirmation node in the original target log, and that node is a key closed-loop node in the target business, the electronic device can assign a higher contribution to the newly added log entry; conversely, when a newly added log entry only supplements additional attribute information of an existing node, its contribution can be relatively lower. This approach more accurately reflects the actual improvement that different newly added log entries make to the completeness of the target business.

[0113] S108. When the similarity between the second business content and the baseline business content is greater than the preset similarity threshold and the contribution of each newly added log content to the integrity of the target business is greater than the preset contribution threshold, the updated log is output.

[0114] In this embodiment, the updated log can be judged to meet the output conditions based on the similarity between the second service content and the baseline service content, and the contribution of the newly added log content to the integrity of the target service. Similarity measures the overall closeness of the updated log to the target service standard content, while contribution measures whether the newly added log content truly improves the integrity of the target service. Only when both conditions are met will the electronic device output the updated log as the final result.

[0115] This dual-judgment mechanism can avoid the following situations: although the updated logs are semantically similar to the baseline business content, the actual improvement of the target business integrity by the added log content is limited; or although the added log content has a high contribution, the overall updated logs still deviate significantly from the baseline business content. By introducing both similarity thresholds and contribution thresholds, the reliability of the final output results can be improved.

[0116] After outputting the updated logs, the updated logs can be provided to subsequent business audit systems, process analysis systems, anomaly detection systems, process prediction systems, or risk control systems, or they can be written to log libraries, business databases, or reporting systems.

[0117] In this embodiment, the preset similarity threshold can be determined based on the similarity statistics between historical complete business log samples and corresponding baseline business content, and the preset contribution threshold can be determined based on the contribution statistics of valid newly added log content in historical supplementary samples. For high-risk businesses, businesses with high compliance requirements, or critical process businesses, the preset similarity threshold and preset contribution threshold can be set higher; for scenarios with high fault tolerance requirements or high business flexibility, the thresholds can be appropriately reduced.

[0118] For example, if the updated logs do not meet the output conditions, an iterative update process can begin. Besides reaching a preset threshold number of iterations, iteration can be terminated based on at least one of the following conditions: the similarity increase corresponding to the logs after two consecutive updates is less than a preset change threshold; the change in contribution corresponding to the newly added log content after two consecutive updates is less than a preset change threshold; no new parts to be completed are identified after reanalysis; or the set of newly added log content remains unchanged compared to the previous round. Adding these termination conditions can prevent invalid iterations.

[0119] Furthermore, in this embodiment, when the similarity between the second business content and the baseline business content is less than a preset similarity threshold and / or the contribution of each newly added log content to the target business integrity is less than a preset contribution threshold, semantic analysis is performed again based on the updated log to obtain new first business content; the candidate correspondence between each process feature vector and each log segment, the correlation between each process feature vector and the new first business content, and the parts to be completed are re-determined based on the new first business content; the updated log is updated again based on the re-determined parts to be completed, until the similarity between the second business content and the baseline business content is greater than a preset similarity threshold and the contribution of each newly added log content to the target business integrity is greater than a preset contribution threshold, or the number of iterations reaches a preset number threshold.

[0120] For example, if the updated log does not yet meet the output conditions, the electronic device can treat the current updated log as a new analysis object, perform semantic analysis and subsequent update processing again, thus forming an iterative update mechanism. In other words, when the result of the previous round of completion is insufficient to achieve the goal, the first business content can be further modified based on the current result, and a more suitable process feature vector can be selected accordingly to redetermine the part to be completed and update again.

[0121] This iterative approach gradually narrows the gap between the second set of business content and the baseline business content, improving the effectiveness of newly added log content. When a preset threshold is met, the final updated log can be output; when the iteration count threshold is reached, processing can be terminated to control computational resource consumption and processing latency. This method helps to further improve the accuracy and stability of log generation results.

[0122] This application provides a log generation method based on multimodal process data. It obtains process data, target logs, and corresponding baseline business content during the execution of a target business based on the business identifier of the target business. First, a first machine learning model is used to perform semantic analysis on the target logs to obtain the first business content representing the business execution content reflected in the current logs and the business semantic distribution corresponding to each log segment. Then, a second machine learning model is used to extract features from the process data to obtain multiple process feature vectors. Based on the candidate correspondence between the process feature vectors and each log segment, and combined with the business semantic distribution corresponding to each log segment, the method calculates the relationship between each process feature vector and the first business content. The correlation and relevance between the log entries are analyzed to determine the parts to be completed in the target log and the target process feature vectors corresponding to each part. Based on the target process feature vectors, relevance, and target log, an updated log that meets preset time constraints and business logic constraints is generated. Then, semantic analysis is performed on the updated log to obtain the second business content. The contribution of the new log content to the target business integrity is determined based on the difference between the second business content and the baseline business content. When the similarity between the second business content and the baseline business content is greater than a preset similarity threshold and the contribution of each new log content to the target business integrity is greater than a preset contribution threshold, the updated log is output. Compared to existing technologies that rely solely on single text logs for log completion or process analysis, which struggle to effectively utilize multimodal process information such as images, audio, video, text, and execution data during business execution—leading to problems like difficulty in accurately locating missing log portions, inconsistencies between completed content and actual business scenarios, lack of temporal rationality and business logic consistency in completed results, difficulty in verifying update results, and difficulty in accurately reconstructing the true business execution trajectory—this application addresses these issues by associating and collaboratively utilizing the semantic analysis results of the target log with the feature extraction results of multimodal process data. Furthermore, it combines this with baseline business content to perform dual verification of similarity and contribution of the updated log. This effectively improves the accuracy of identifying the parts to be completed, enhances the matching degree between the newly added log content and the actual execution process of the target business, strengthens the rationality of the updated log in terms of temporal sequence and business logic, and improves the ability of the log generation results to represent the integrity of the target business. This, in turn, enhances the accuracy, reliability, and usability of log completion and updates, providing a more reliable data foundation for subsequent process analysis, business auditing, anomaly identification, and risk management.

[0123] Third Embodiment Figure 2 This is a schematic diagram of the structure of an embodiment of the log generation apparatus based on multimodal process data provided in this application. Figure 2As shown in the illustration, the log generation device based on multimodal process data provided in this application embodiment may include: a data acquisition module 21, a first analysis module 22, a feature extraction module 23, a correlation determination module 24, a completion determination module 25, a log generation module 26, a second analysis module 27, and an output module 28. The above modules can be implemented through software, hardware, or a combination of both, and can be integrated into the same processing platform or distributed across multiple computing nodes.

[0124] The data acquisition module 21 can be used to acquire process data during the execution of the target business and the target log corresponding to the business identifier based on the business identifier of the target business, and to acquire the baseline business content corresponding to the target business from the preset business database based on the business identifier.

[0125] In this embodiment, the data acquisition module 21 can establish a communication connection with a business system, database, object storage, log platform, monitoring platform, or external device to obtain log data and process data corresponding to the target business. The data acquisition module 21 can also access a preset business database based on a business identifier to read baseline business content.

[0126] The data acquired by the data acquisition module 21 can be used uniformly by subsequent modules, thereby ensuring that the entire device performs log analysis and log generation around the same target business. The data acquisition module 21 can also perform data preprocessing operations, such as format standardization, field mapping, abnormal data filtering, and time unification processing, to improve the efficiency of subsequent processing.

[0127] Furthermore, in the embodiments of this application, the process data includes at least two of the following: image data, audio data, video data, text data, and execution data.

[0128] For example, the data acquisition module 21 can collect or receive data from at least two modalities from different sources and provide this data to subsequent modules for multimodal feature analysis. This enhances the device's ability to perceive the actual business execution process.

[0129] The first analysis module 22 can be used to perform semantic analysis on the target log using a first machine learning model to obtain the first business content, and determine the business semantic distribution corresponding to each log fragment in the target log based on the first business content. The first business content is used to characterize the target business execution content inferred from the target log.

[0130] In this embodiment, the first analysis module 22 can invoke a pre-trained first machine learning model to perform semantic understanding, event parsing, business classification, and content inference on the target log. The first analysis module 22 can output the first business content and the business semantic distribution at the log fragment level.

[0131] The output of the first analysis module 22 not only reflects the business content corresponding to the current log record, but also provides a semantic basis for subsequent multimodal process feature matching. By decomposing the target log to the log fragment level and generating a business semantic distribution, the accuracy of subsequent candidate correspondence filtering and correlation calculation can be improved.

[0132] Furthermore, in this embodiment of the application, using a first machine learning model to perform semantic analysis on the target log to obtain first business content, and determining the business semantic distribution corresponding to each log segment in the target log based on the first business content may include: semantically encoding multiple text units in the target log to obtain multiple semantic representation vectors; dividing the target log into segments based on the multiple semantic representation vectors to obtain multiple log segments; performing business semantic recognition on each log segment to obtain the business semantic distribution corresponding to each log segment; and determining the first business content corresponding to the target log based on the business semantic distribution corresponding to each log segment.

[0133] For example, the first analysis module 22 may further include an encoding unit, a fragment segmentation unit, and a business semantic recognition unit. The encoding unit is used to convert text units into semantic representation vectors, the fragment segmentation unit is used to generate multiple log fragments, the business semantic recognition unit is used to output the business semantic distribution corresponding to each log fragment, and finally the content determination unit generates the first business content.

[0134] The feature extraction module 23 can be used to extract features from process data using a second machine learning model, obtain multiple process feature vectors, and determine the candidate correspondence between each process feature vector and each log segment based on the distribution relationship of each process feature vector in the time dimension.

[0135] In this embodiment, the feature extraction module 23 can perform feature extraction, encoding, and fusion processing on data of different modalities to generate process feature vectors with a unified representation. The feature extraction module 23 can also utilize time information to establish candidate correspondences between the process feature vectors and log fragments.

[0136] Candidate correspondences can serve as important inputs for subsequent correlation determination module 24 and completion determination module 25, enabling the device to prioritize process features that have a temporal correspondence with log fragments when processing multimodal data, thereby improving overall processing efficiency.

[0137] Furthermore, in this embodiment of the application, using a second machine learning model to extract features from the process data to obtain multiple process feature vectors, and determining the candidate correspondence between each process feature vector and each log segment based on the distribution relationship of each process feature vector in the time dimension may include: extracting features from data of different modalities in the process data to obtain multiple modal features; performing timestamp calculation and semantic mapping on the multiple modal features to obtain multiple process feature vectors; and matching each log segment with each process feature vector based on at least one of the time position, duration, and event triggering order to determine the candidate correspondence between each process feature vector and each log segment.

[0138] For example, the feature extraction module 23 may further include a modality feature extraction unit, a timestamp processing unit, a semantic mapping unit, and a matching unit. The matching unit can perform matching operations based on time position, duration, and event order to establish candidate correspondences.

[0139] The correlation determination module 24 can be used to calculate the correlation between each process feature vector and the first business content based on the business semantic distribution corresponding to each log segment and the candidate correspondence between each process feature vector and each log segment, and determine the target process feature vector with a correlation greater than a preset correlation threshold.

[0140] In this embodiment, the correlation determination module 24 can score the correlation of each process feature vector and filter target process feature vectors according to a preset correlation threshold. The correlation determination module 24 can interact with the first analysis module 22 and the feature extraction module 23 to simultaneously utilize business semantic distribution and candidate correspondence.

[0141] By using correlation filtering, the device can identify target process feature vectors that are truly relevant to the current business content from a large number of process feature vectors, reducing irrelevant interference in subsequent completion determination and log generation, and improving completion efficiency and accuracy.

[0142] Furthermore, in this embodiment of the application, calculating the correlation between each process feature vector and the first business content may include: based on the business semantic distribution corresponding to each log segment and the candidate correspondence between each process feature vector and each log segment, calculating the degree of matching between each process feature vector and the first business content in at least one of the semantic dimension, time dimension and business rule dimension; and determining the correlation between each process feature vector and the first business content based on the calculated degree of matching.

[0143] For example, the relevance determination module 24 may further include a multi-dimensional matching unit and a fusion calculation unit. The multi-dimensional matching unit is used to perform matching on the semantic dimension, the time dimension, and the business rule dimension respectively, and the fusion calculation unit is used to fuse the results of each dimension to obtain the final relevance.

[0144] The module 25 for determining the parts to be completed can be used to calculate the correlation between each target process feature vector and the first business content based on the candidate correspondence between each target process feature vector and each log segment, and to determine the parts to be completed in the target log and the target process feature vectors corresponding to each part to be completed based on the correlation between each target process feature vector and the first business content.

[0145] In this embodiment, the completion determination module 25 can further perform correlation analysis on the target process feature vector based on the correlation screening results, and locate the part to be completed by combining the completeness of the target log segment. The output of the completion determination module 25 can clearly indicate which positions in the target log need to be completed, and which target process feature vector(s) should support each part to be completed.

[0146] In this way, the module 25 that needs to be completed can avoid performing coarse-grained completion on the entire log, and instead achieve fine-grained completion and positioning based on local log segments, thereby improving the relevance of log generation.

[0147] Furthermore, in this embodiment of the application, determining the parts to be completed in the target log and the target process feature vectors corresponding to each part to be completed based on the correlation between each target process feature vector and the first business content may include: determining the target log segment corresponding to each target process feature vector based on the candidate correspondence between each target process feature vector and each log segment; determining the parts to be completed based on the correlation between each target process feature vector and the first business content and the log completeness of each target log segment; and determining the corresponding target process feature vector based on the determined parts to be completed.

[0148] For example, the module 25 for determining the log section to be completed may further include a log section mapping unit, a completeness assessment unit, and a correspondence determination unit. The completeness assessment unit can judge the target log section from aspects such as missing events, missing attributes, time breakpoints, and logical breaks.

[0149] Log generation module 26 can be used to generate updated logs based on the target process feature vectors corresponding to each part to be completed, the correlation between each target process feature vector and the first business content, and the target log. The newly added log content in the updated logs and the original log content in the target logs satisfy the preset timing constraints and the business logic constraints corresponding to the business identifier.

[0150] In this embodiment, the log generation module 26 can perform insertive updates, replacement updates, or modification updates on the parts to be completed, and generate the final updated log. During the update process, the log generation module 26 can refer to correlation results to determine the priority, reliability, or writing order of different completed content.

[0151] Log generation module 26 can also perform time-series constraint checks and business logic constraint checks on newly added log content to ensure that the time and business relationships between the new content and the original logs are reasonable, thereby improving the quality of log generation results.

[0152] Furthermore, in this embodiment of the application, generating an updated log based on the target process feature vector corresponding to each part to be completed, the correlation between each target process feature vector and the first business content, and the target log may include: converting the target log into a log vector; generating a completed log vector based on the target process feature vector corresponding to each part to be completed; determining the update position of each completed log vector in the target log based on the correlation between each target process feature vector and the first business content; and updating the log vector to generate the updated log.

[0153] For example, the log generation module 26 may further include a log vector conversion unit, a log vector completion generation unit, an update position determination unit, and a log update unit. These units work together to achieve vectorized log updates.

[0154] Furthermore, in this embodiment, the preset timing constraint is used to limit at least one of the following relationships among the new log content in the updated log and the original log content in the target log: the order relationship, the time interval relationship, and the node connection relationship.

[0155] For example, the log generation module 26 may include a timing constraint verification unit, which is used to check the event sequence, time interval and node connection before or after writing new log content.

[0156] Furthermore, in this embodiment of the application, business logic constraints can be used to limit at least one of the following relationships among the process dependency, node triggering relationship, and business rule consistency relationship between the newly added log content in the updated log and the baseline business content corresponding to the business identifier:

[0157] For example, the log generation module 26 may include a logical constraint verification unit, which is used to determine whether the newly added log content meets the requirements of process dependencies, node triggering conditions and business rules by combining the baseline business content.

[0158] The second analysis module 27 can be used to perform semantic analysis on the updated logs using the first machine learning model to obtain the second business content, and based on the difference between the second business content and the baseline business content, determine the contribution of each newly added log content in the updated logs to the integrity of the target business.

[0159] In this embodiment, the second analysis module 27 can perform semantic understanding on the updated log again to obtain the second business content. The second analysis module 27 can also compare the second business content with the baseline business content to evaluate the effectiveness of the newly added log content.

[0160] The second analysis module 27 can output the contribution of each newly added log entry to the integrity of the target business, which is then used by the output module 28 for threshold judgment. Through this design, the device can not only generate logs but also perform content-level quality assessments of the generated results.

[0161] The output module 28 can be used to output the updated log when the similarity between the second business content and the baseline business content is greater than a preset similarity threshold and the contribution of each newly added log content to the integrity of the target business is greater than a preset contribution threshold.

[0162] In this embodiment, the output module 28 can receive the second business content and contribution information output by the second analysis module 27, and further calculate the similarity between the second business content and the baseline business content, or receive similarity results output by other modules. The output module 28 can determine whether the updated log meets the output conditions based on a preset similarity threshold and a preset contribution threshold.

[0163] When the output conditions are met, the output module 28 can output the updated log to the log storage system, audit system, process analysis system, risk control system, or user interface. When the output conditions are not met, it can trigger an iterative update process or return intermediate results.

[0164] Furthermore, in this embodiment, when the similarity between the second business content and the baseline business content is less than a preset similarity threshold and / or the contribution of each newly added log content to the target business integrity is less than a preset contribution threshold, semantic analysis is performed again based on the updated log to obtain new first business content; the candidate correspondence between each process feature vector and each log segment, the correlation between each process feature vector and the new first business content, and the parts to be completed are re-determined based on the new first business content; the updated log is updated again based on the re-determined parts to be completed, until the similarity between the second business content and the baseline business content is greater than a preset similarity threshold and the contribution of each newly added log content to the target business integrity is greater than a preset contribution threshold, or the number of iterations reaches a preset number threshold.

[0165] For example, the device may also include an iterative update module. This iterative update module can work in conjunction with the first analysis module 22, the feature extraction module 23, the correlation determination module 24, the completion determination module 25, the log generation module 26, and the second analysis module 27 to re-execute one or more rounds of update processing when threshold conditions are not met. Through this iterative mechanism, the quality of the final log generation result can be gradually improved.

[0166] In the embodiments of this application, Figure 2 The modules shown can be implemented as software modules, hardware circuits, firmware units, or a combination of software and hardware. The modules can interact with each other via bus, shared memory, message mechanisms, interface call mechanisms, or network communication mechanisms. Figure 2 The module divisions described herein are for illustrative purposes only and do not limit the actual physical deployment method. In practice, two or more modules can be integrated into the same processing unit, or they can be further divided into multiple sub-modules.

[0167] In this embodiment, the correlation result output by the correlation determination module 24 is mainly used for filtering the target process feature vector, and the correlation result output by the completion determination module 25 is mainly used to determine the part to be completed and the update position and update weight in the log generation module 26. The contribution result output by the second analysis module 27 can be used together with the similarity result between the second business content and the baseline business content calculated or received by the output module 28 as the final output judgment basis.

[0168] Fourth embodiment The above describes the internal functions and structure of a log generation device based on multimodal process data, which can be implemented as an electronic device. Figure 3 A schematic diagram illustrating the structure of an embodiment of the electronic device provided in this application. (See attached diagram.) Figure 3 As shown, the electronic device includes a memory 31 and a processor 32.

[0169] Memory 31 is used to store programs. In addition to the programs described above, memory 31 can also be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, contact data, phonebook data, messages, pictures, videos, etc.

[0170] The memory 31 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0171] Processor 32 is not limited to a central processing unit (CPU), but may also be a graphics processing unit (GPU), a field-programmable gate array (FPGA), an embedded neural network processor (NPU), or an artificial intelligence (AI) chip. Processor 32 is coupled to memory 31 and executes the program stored in memory 31. When the program runs, it executes the log generation method based on multimodal process data in Embodiment 2 described above.

[0172] Furthermore, such as Figure 3 As shown, the electronic device may also include other components such as a communication component 33, a power supply component 34, an audio component 35, and a display 36. Figure 3 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 3 The components shown.

[0173] Communication component 33 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic devices can access wireless networks based on communication standards, such as WiFi, 3G, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 33 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 33 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0174] Power supply component 34 provides power to various components of the electronic device. Power supply component 34 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0175] Audio component 35 is configured to output and / or input audio signals. For example, audio component 35 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 31 or transmitted via communication component 33. In some embodiments, audio component 35 also includes a speaker for outputting audio signals.

[0176] Display 36 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can detect not only the boundaries of touch or swipe actions, but also the duration and pressure associated with the touch or swipe operation.

[0177] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A log generation method based on multimodal process data, characterized in that, include: Based on the business identifier of the target business, obtain the process data during the execution of the target business and the target log corresponding to the business identifier, and obtain the baseline business content corresponding to the target business from the preset business database based on the business identifier; The first machine learning model is used to perform semantic analysis on the target log to obtain the first business content, and the business semantic distribution corresponding to each log fragment in the target log is determined based on the first business content. The first business content is used to characterize the target business execution content inferred from the target log. The second machine learning model is used to extract features from the process data to obtain multiple process feature vectors. Based on the distribution relationship of each process feature vector in the time dimension, the candidate correspondence between each process feature vector and each log segment is determined. Based on the business semantic distribution corresponding to each log segment and the candidate correspondence between each process feature vector and each log segment, the correlation between each process feature vector and the first business content is calculated, and the target process feature vector with a correlation greater than the preset correlation threshold is determined. Based on the candidate correspondence between each target process feature vector and each log segment, the correlation between each target process feature vector and the first business content is calculated. Based on the correlation between each target process feature vector and the first business content, the parts to be completed in the target log and the target process feature vectors corresponding to each part to be completed are determined. Based on the target process feature vectors corresponding to each part to be completed, the correlation between each target process feature vector and the first business content, and the target log, an updated log is generated. The newly added log content in the updated log satisfies the preset time sequence constraints and the business logic constraints corresponding to the business identifier with the original log content in the target log. The first machine learning model is used to perform semantic analysis on the updated logs to obtain the second business content. Based on the difference between the second business content and the baseline business content, the contribution of each newly added log content in the updated logs to the integrity of the target business is determined. When the similarity between the second business content and the baseline business content is greater than the preset similarity threshold, and the contribution of each newly added log content to the integrity of the target business is greater than the preset contribution threshold, the updated log is output.

2. The log generation method based on multimodal process data according to claim 1, characterized in that, The step of using a first machine learning model to perform semantic analysis on the target log to obtain first business content, and determining the business semantic distribution corresponding to each log segment in the target log based on the first business content, includes: Semantic encoding is performed on multiple text units in the target log to obtain multiple semantic representation vectors; The target log is divided into segments based on multiple semantic representation vectors to obtain multiple log segments; Perform business semantic recognition on each log segment to obtain the business semantic distribution corresponding to each log segment; Based on the business semantic distribution corresponding to each log segment, determine the first business content corresponding to the target log.

3. The log generation method based on multimodal process data according to claim 1, characterized in that, The process involves using a second machine learning model to extract features from the process data, obtaining multiple process feature vectors, and determining candidate correspondences between each process feature vector and each log segment based on the distribution relationship of each process feature vector in the time dimension. This includes: Feature extraction was performed on data from different modalities in the process data to obtain multiple modal features; Multiple process feature vectors are obtained by performing timestamp calculation and semantic mapping on multiple modal features; Based on at least one of the time position, duration, and event triggering order of each process feature vector, it is matched with each log segment to determine the candidate correspondence between each process feature vector and each log segment.

4. The log generation method based on multimodal process data according to claim 1, characterized in that, The calculation of the correlation between the feature vectors of each process and the first business content includes: Based on the business semantic distribution corresponding to each log segment and the candidate correspondence between each process feature vector and each log segment, calculate the degree of matching between each process feature vector and the first business content in at least one of the semantic dimension, time dimension and business rule dimension. Based on the calculated matching degree, the correlation between the feature vectors of each process and the first business content is determined.

5. The log generation method based on multimodal process data according to claim 1, characterized in that, The step of determining the parts to be completed in the target log and the target process feature vectors corresponding to each part to be completed based on the correlation between each target process feature vector and the first business content includes: Based on the candidate correspondence between the feature vectors of each target process and each log segment, the target log segment corresponding to each feature vector of the target process is determined. Based on the correlation between the feature vectors of each target process and the first business content, as well as the log completeness of each target log segment, the parts to be supplemented are determined. Based on the identified parts to be completed, determine the corresponding target process feature vector.

6. The log generation method based on multimodal process data according to claim 1, characterized in that, The updated log is generated based on the target process feature vectors corresponding to each part to be completed, the correlation between each target process feature vector and the first business content, and the target log, including: Convert the target log into a log vector; Generate a completion log vector based on the target process feature vector corresponding to each part to be completed; Based on the correlation between the feature vectors of each target process and the first business content, the update position of each log completion vector in the target log is determined. Update the log vector to generate the updated log.

7. The log generation method based on multimodal process data according to claim 1, characterized in that, Also includes: When the similarity between the second business content and the baseline business content is less than the preset similarity threshold and / or the contribution of each newly added log content to the integrity of the target business is less than the preset contribution threshold, semantic analysis is performed again based on the updated logs to obtain new first business content. Based on the new first business content, the candidate correspondence between the feature vectors of each process and each log segment, the correlation between the feature vectors of each process and the new first business content, and the parts to be completed are re-determined; The updated logs are updated again based on the newly determined parts to be completed, until the similarity between the second business content and the baseline business content is greater than the preset similarity threshold and the contribution of each newly added log content to the integrity of the target business is greater than the preset contribution threshold, or the number of iterations reaches the preset number threshold.

8. The log generation method based on multimodal process data according to claim 1, characterized in that, The process data includes at least two of the following: image data, audio data, video data, text data, and execution data.

9. A log generation device based on multimodal process data, characterized in that, include: The data acquisition module is used to acquire process data during the execution of the target business and the target log corresponding to the business identifier based on the business identifier of the target business, and to acquire the baseline business content corresponding to the target business from the preset business database based on the business identifier. The first analysis module is used to perform semantic analysis on the target log using a first machine learning model to obtain the first business content, and to determine the business semantic distribution corresponding to each log fragment in the target log based on the first business content. The first business content is used to characterize the target business execution content inferred from the target log. The feature extraction module is used to extract features from the process data using a second machine learning model, obtain multiple process feature vectors, and determine the candidate correspondence between each process feature vector and each log segment based on the distribution relationship of each process feature vector in the time dimension. The correlation determination module is used to calculate the correlation between each process feature vector and the first business content based on the business semantic distribution corresponding to each log segment and the candidate correspondence between each process feature vector and each log segment, and to determine the target process feature vector with a correlation greater than a preset correlation threshold. The module for determining the parts to be completed is used to calculate the correlation between the feature vectors of each target process and the first business content based on the candidate correspondence between the feature vectors of each target process and each log segment, and to determine the parts to be completed in the target log and the target process feature vectors corresponding to each part to be completed based on the correlation between the feature vectors of each target process and the first business content. The log generation module is used to generate an updated log based on the target process feature vector corresponding to each part to be completed, the correlation between each target process feature vector and the first business content, and the target log. The newly added log content in the updated log satisfies the preset time sequence constraints and the business logic constraints corresponding to the business identifier in the original log content in the target log. The second analysis module is used to perform semantic analysis on the updated logs using the first machine learning model to obtain the second business content, and based on the difference between the second business content and the baseline business content, to determine the contribution of each newly added log content in the updated logs to the integrity of the target business. The output module is used to output updated logs when the similarity between the second business content and the baseline business content is greater than a preset similarity threshold and the contribution of each newly added log content to the integrity of the target business is greater than a preset contribution threshold.

10. An electronic device, characterized in that, include: Memory, used to store programs; A processor is configured to run the program stored in the memory to perform the log generation method based on multimodal process data as described in any one of claims 1 to 8.

Images