A method and system for generating a multi-modal business process graph and a computer device
By integrating audio recordings, screen recordings, and event logs, a multimodal business process diagram is generated, which solves the problem of insufficient data integration in existing technologies and enables comprehensive monitoring and optimization of business processes.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI YISAIQI SOFTWARE CO LTD
- Filing Date
- 2022-11-30
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies struggle to effectively integrate multimodal data such as audio recordings, screen recordings, and event logs, making it impossible to generate complete business process diagrams and hindering enterprises from fully understanding and optimizing their business processes.
By acquiring audio recordings, screen recordings, and event logs containing user IDs and timestamps, the data is segmented and merged based on predefined rules. Audio clips and screenshots are extracted, and speech recognition and OCR recognition are performed. Keywords are merged, and a process mining algorithm is applied to generate a multimodal business process diagram.
It integrates multimodal data from audio recordings, screen recordings, and event logs to generate multimodal business process diagrams. Users can monitor and optimize business processes from both global and local perspectives, and fully reconstruct the business execution process.
Smart Images
Figure CN116011945B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method, system, and computer equipment for generating multimodal business process diagrams, belonging to the field of process mining technology. Background Technology
[0002] Process mining is a technique for discovering, monitoring, and improving actual business processes from event logs of information systems. For example, business process data can be discovered from workflow logs in an ERP system, and corresponding business process diagrams can be reconstructed, helping enterprises better understand and improve their business processes. Task mining is the application of process mining algorithms at the user operation level. It can reconstruct actual business operation processes from user operation logs and screenshots, rebuild business execution flowcharts, and help enterprises identify bottlenecks in business operations and discover opportunities for business automation.
[0003] Process mining typically only handles text-based data, while task mining can process data containing both text and images. However, in some scenarios, such as customer service, voice data also needs to be processed. Only by combining and analyzing all three types of data can the complete business operation process be reconstructed, thus providing an opportunity to improve and optimize the process. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, system, and computer device for generating multimodal business process diagrams, capable of simultaneously processing multimodal data such as audio recording, screen recording, and event logs to generate multimodal business process diagrams. To achieve the above objective, this invention employs the following technical solution:
[0005] In a first aspect, the present invention provides a method for generating a multimodal business process diagram, comprising:
[0006] Retrieve audio recordings, screen recordings, and event logs containing user ID and timestamp attributes;
[0007] The event logs are segmented and merged based on predefined rules. Each segment of the event log corresponds to a business step in the business process. The earliest timestamp in each segment is the start time of the business step, and the latest timestamp is the end time of the business step.
[0008] Based on the start time, end time, and user ID of each business step, extract audio segments from the audio recording data and extract screenshots from the screen recording data;
[0009] Speech recognition is performed on the audio clips to obtain the audio text, and keywords are extracted from the audio text; OCR recognition is performed on the screenshots to obtain the screen text, and screen keywords are extracted from the screen text.
[0010] Merge all event logs for the same business step, extract repeated operation sequences to obtain operation sequences representing business semantics; merge all recording keywords for the same business step to obtain a recording word cloud; merge all screen keywords for the same business step to obtain a screen word cloud.
[0011] Based on the segmented and merged event logs, a process mining algorithm is applied to obtain a business process diagram.
[0012] Add corresponding operation sequences, audio word clouds, and screen word clouds to each business step in the business process diagram to obtain a multimodal business process diagram.
[0013] In conjunction with the first aspect, further, the acquisition of audio recording data, screen recording data, and event logs containing user ID and timestamp attributes includes:
[0014] In response to the fulfillment of the triggering conditions, the system records the call to obtain audio data, records the screen operations to obtain screen recording data, and saves the logs of the business system to obtain event logs.
[0015] In conjunction with the first aspect, the segmentation and merging of event logs based on predefined rules further includes:
[0016] Event logs are grouped according to business number and user number. Each group of events is sorted by timestamp. An event attribute is selected as the segment attribute. Consecutive events with the same attribute value are considered as a segment. Each log segment corresponds to a business step in the business process. The earliest timestamp in the log segment is taken as the start time of the business step, and the latest timestamp in the log segment is taken as the end time of the business step. The business number remains unchanged.
[0017] In conjunction with the first aspect, the grouping of event logs based on service number and user number further includes:
[0018] The event log includes a business number attribute; events with the same business number represent events in the same business process.
[0019] The event log also includes a user ID attribute; events with the same user ID represent events from the same user.
[0020] In conjunction with the first aspect, further, the extraction of screen keywords from screen text includes:
[0021] OCR recognition is performed on the screenshot to obtain all the text in the image. The text that changes between the two screenshots is retained. All the changed text is merged and the keyword extraction algorithm is applied to obtain the screen keywords.
[0022] In conjunction with the first aspect, further, the extraction of recording keywords from the recording text includes:
[0023] Speech recognition algorithms are used to convert speech into text, and keyword extraction algorithms are applied to extract keywords from the recordings.
[0024] If no audio segment exists, the audio keywords will be empty.
[0025] Secondly, the present invention provides a system for generating multimodal business process diagrams, comprising:
[0026] Acquisition module: Used to acquire audio recording data, screen recording data, and event logs containing user ID and timestamp attributes;
[0027] Segmentation module: Used to segment and merge event logs based on predefined rules. Each segment of event log corresponds to a business step in the business process. The earliest timestamp in each segment is the start time of the business step, and the latest timestamp is the end time of the business step.
[0028] The first extraction module is used to extract audio segments from audio recording data and screenshots from screen recording data based on the start time, end time, and user ID of each business step.
[0029] The second extraction module is used to perform speech recognition on the audio clips to obtain the audio text, and extract the audio keywords from the audio text; it also performs OCR recognition on the screenshots to obtain the screen text, and extracts the screen keywords from the screen text.
[0030] The merging module is used to merge all event logs of the same business step, extract repeated operation sequences to obtain operation sequences representing business semantics; merge all recording keywords of the same business step to obtain a recording word cloud; and merge all screen keywords of the same business step to obtain a screen word cloud.
[0031] Process mining module: Used to obtain business process diagrams by applying process mining algorithms based on the segmented and merged event logs;
[0032] Output module: Used to add corresponding operation sequences, audio word clouds, and screen word clouds to each business step in the business process diagram to obtain a multimodal business process diagram.
[0033] Thirdly, the present invention provides a computer device, including a processor and a storage medium;
[0034] The storage medium is used to store instructions;
[0035] The processor is configured to operate according to the instructions to perform the steps of the method described in the first aspect.
[0036] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.
[0037] Compared with the prior art, the beneficial effects achieved by the multimodal business process diagram generation method, system, and computer equipment provided in this embodiment of the invention include:
[0038] This invention obtains audio recording data, screen recording data, and event logs containing user ID and timestamp attributes. Based on predefined rules, it segments and merges the event logs to obtain the start and end times of a business step. By extracting the event logs, audio recording data, and screen recording data generated by each user within that time period, aligned multimodal data can be obtained.
[0039] This invention extracts audio segments from audio recordings and screenshots from screen recordings based on the start time, end time, and user ID of each business step; it performs speech recognition on the audio segments to obtain audio text and extracts audio keywords from the audio text; it performs OCR on the screenshots to obtain screen text and extracts screen keywords from the screen text; it merges all event logs of the same business step, extracts repeated operation sequences to obtain operation sequences representing business semantics; it merges all audio keywords of the same business step to obtain an audio word cloud; and it merges all screen keywords of the same business step to obtain a screen word cloud. This invention uses a repeated operation sequence discovery algorithm, speech-to-text conversion, and OCR technology to obtain the simplest semantic representation of each business step from multimodal data.
[0040] This invention uses a process mining algorithm based on segmented and merged event logs to obtain a business process diagram; it adds corresponding operation sequences, audio word clouds, and screen word clouds to each business step in the business process diagram to obtain a multimodal business process diagram; this invention applies a process mining algorithm to segmented event logs to obtain a business process diagram, and then adds multimodal data to the business process diagram to obtain a multimodal business process diagram.
[0041] With multimodal business process diagrams, users can continue to use existing business process analysis functions to perform various business process analysis and optimization operations. They can also quickly understand the main semantics of the corresponding business steps through typical operation sequences, audio word clouds, and screen word clouds. Furthermore, they can fully reconstruct the user's actual business execution process through the details of individual business processes, making it convenient for users to monitor, analyze, and optimize business processes from both global and local perspectives. Attached Figure Description
[0042] Figure 1 This is a flowchart of a method for generating a multimodal business process provided in Embodiment 1 of the present invention;
[0043] Figure 2 The multimodal business process diagram is obtained by a multimodal business process generation method provided in Embodiment 1 of the present invention. Detailed Implementation
[0044] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.
[0045] Example 1:
[0046] like Figure 1 This invention provides a method for generating a multimodal business process diagram, comprising:
[0047] Retrieve audio recordings, screen recordings, and event logs containing user ID and timestamp attributes;
[0048] The event logs are segmented and merged based on predefined rules. Each segment of the event log corresponds to a business step in the business process. The earliest timestamp in each segment is the start time of the business step, and the latest timestamp is the end time of the business step.
[0049] Based on the start time, end time, and user ID of each business step, extract audio segments from the audio recording data and extract screenshots from the screen recording data;
[0050] Speech recognition is performed on the audio clips to obtain the audio text, and keywords are extracted from the audio text; OCR recognition is performed on the screenshots to obtain the screen text, and screen keywords are extracted from the screen text.
[0051] Merge all event logs for the same business step, extract repeated operation sequences to obtain operation sequences representing business semantics; merge all recording keywords for the same business step to obtain a recording word cloud; merge all screen keywords for the same business step to obtain a screen word cloud.
[0052] Based on the segmented and merged event logs, a process mining algorithm is applied to obtain a business process diagram.
[0053] Add corresponding operation sequences, audio word clouds, and screen word clouds to each business step in the business process diagram to obtain a multimodal business process diagram.
[0054] The specific steps include:
[0055] Step 1: Obtain audio recording data, screen recording data, and event logs containing user ID and timestamp attributes.
[0056] In response to the fulfillment of the triggering conditions, the system records the call to obtain audio data, records the screen operations to obtain screen recording data, and saves the logs of the business system to obtain event logs.
[0057] The recording data is a file saved during a user's call, and the file format can be WAV or MP3. The recording data also includes the recording start time and recording end time. Taking a customer service scenario as an example, when a call comes in, the call is automatically recorded, resulting in recording data containing the user's ID and timestamp attributes.
[0058] Screen recording data consists of multiple screenshots captured while a user is interacting with the screen. Captures can be taken at each keyboard and mouse action, or at fixed time intervals. Screenshots can be saved as multiple files or merged into a single video. Each screenshot includes a timestamp representing the execution time of the action. For example, in a customer service scenario, when a call comes in, the on-screen actions are recorded, resulting in screen recording data containing the user's ID and a timestamp.
[0059] Event logs are log files or database records generated by the business system backend. They consist of multiple records, each containing multiple attributes representing the time of an event, the user performing the operation, the department to which it belongs, the content of the operation, the business type, etc. Taking a customer service scenario as an example, when a call comes in, the event logs corresponding to all screen operations are saved in the backend, resulting in an event log containing user ID and timestamp attributes.
[0060] The timestamps in audio recordings, screen recordings, and event logs are all taken from the system time. If audio recordings, screen recordings, and event logs are obtained from multiple computers, system time synchronization is required to ensure timestamp consistency.
[0061] In this embodiment, the audio recording data is a WAV file, and the filename contains the user ID, recording start time and duration. The screen recording data consists of multiple images, and the filename of each image contains the user ID and timestamp. The event log is a CSV file, and the example data is shown in Table 1.
[0062] Table 1 Event Logs with User ID and Timestamp Attributes
[0063] Timestamp user application operate Business Number Attribute 1 Attribute 2 t1 user1 Excel Edit A Xx Xx t2 user2 Browser Click B Xx Xx t3 user1 Excel Edit A Xx Xx t4 user1 Browser Click A Xx Xx t5 user3 Excel Edit A Xx Xx t6 user1 Excel Edit A Xx Xx
[0064] Step 2: Segment and merge the event logs based on predefined rules. Each segment of the event log corresponds to a business step in the business process. The earliest timestamp in each segment is the start time of the business step, and the latest timestamp is the end time of the business step.
[0065] The event log contains the execution processes of multiple business operations and users, and can be divided into business steps based on business content and analysis needs. A business step contains multiple event log entries, representing the detailed execution process of that business step.
[0066] There are several ways to group event logs. In this embodiment, event logs are grouped according to business number, with events having the same business number representing events within the same business process; and according to user ID attribute, event logs are grouped with the same user ID, with events having the same user ID representing events for the same user. By splitting the event logs into multiple groups based on business type and user ID, and sorting each group by timestamp, multiple groups of event log records for the same business and the same user, arranged chronologically, are obtained, representing the complete execution process of a user performing a business.
[0067] Select an event attribute as the segmentation attribute, and segment each group of event logs according to the segmentation attribute. Each event log segment corresponds to a business step in the business process, representing a user's operation process within an application. The duration of each business step is generally no less than half a minute and no more than one hour. Taking the above data as an example, using the "application" attribute as the segmentation attribute, the event log can be divided into multiple log segments based on the different business, users, and applications. Each log segment represents all the operations of a certain user on a certain application within a certain business.
[0068] After segmenting the event log, multiple log records are merged into a single record, including start time, end time, business ID, and activity name, to meet the needs of subsequent process analysis. The start time is the earliest time in each log segment, and the end time is the latest time in each log segment. If there is only one record, the start time is the time of the previous operation record of the same user, and the end time is the time of the next operation record of the same user. The business ID is the original business ID in the log. The activity name is the concatenation of attributes representing the activity content from multiple consecutive logs, or the activity content of the first record in each log segment. In this embodiment, the content of the first log is used as the representative.
[0069] Taking the sample data in Table 1 as an example, there are multiple logs with business number "A", which are sorted by time as follows:<t1,user1, excel, edit> ,<t3, user1, excel, edit> ,<t4, user1, browser, click> ,<t5, user3, excel, edit> ,<t6, user1, excel, edit> After filtering for user "user1" by user ID, the event logs are obtained:<t1, user1, excel, edit> ,<t3, user1, excel, edit> ,<t4,user1, browser, click> ,<t5, user3, excel, edit> ,<t6, user1, excel, edit> Select the "Apply" attribute as the segmentation attribute to segment this group of event logs, and select the "Apply" attribute of the first entry in each segment as the activity name for that segment. The results are as follows:<t1, t3, A, excel> ,<t3, t5, A,browser> ,<t5, t6, A, excel> .
[0070] Step 3: Extract audio segments from the audio recording data and extract screenshots from the screen recording data based on the start time, end time, and user ID of each business step.
[0071] Based on the start and end times and user ID of each event log segment, retrieve the corresponding user's audio recording data and screenshot data within the specified time period.
[0072] Step 4: Perform speech recognition on the audio clip to obtain the audio text, and extract the audio keywords from the audio text; perform OCR recognition on the screenshot to obtain the screen text, and extract the screen keywords from the screen text.
[0073] If the corresponding user's voice data exists within the specified time period, a speech recognition algorithm is used to translate it into text, and then keywords are extracted. In this embodiment, the Whisper model is used to convert speech into text, and the TF-IDF algorithm is used to extract keywords. If no recording segment exists, the corresponding keywords are left blank.
[0074] If a screenshot of the corresponding user exists within the corresponding time period, OCR technology is used to recognize the text in each image, and the differences between the text in two images are compared, saving only the different text. Then, a keyword extraction algorithm is applied to extract keywords. In this embodiment, a CRNN model is used for text recognition, and the TF-IDF algorithm is used to extract keywords. The recorded keywords and screen keywords are added to the segmented log.
[0075] Step 5: Merge all event logs for the same business step, extract repeated operation sequences to obtain operation sequences representing business semantics; merge all recording keywords for the same business step to obtain a recording word cloud representing the recording content; merge all screen keywords for the same business step to obtain a screen word cloud representing the screen operation content.
[0076] Retrieve log records corresponding to segmented logs with the same activity name, and calculate typical operation sequences that represent these different operation sequences. In this embodiment, the operation sequence with the most occurrences is selected. Using the example data above as an illustration, segmented logs...<t1, t3, user1, excel> Includes two log records<excel, edit> ,<excel, edit> The activity is named "excel". Count the number of times a completely identical operation sequence appears in all segmented logs named "excel". If it appears the most frequently, then it is the typical operation sequence.
[0077] Obtain keywords from segmented logs with the same activity name, simply sum them up to obtain multiple keywords and their frequency, and generate a word cloud. The frequency of occurrence is the weight, which is reflected in the font size; the more frequently a keyword appears, the greater its weight, and the larger the font size of the corresponding keyword in the word cloud, indicating that the keyword is more important. Merge the audio recording keywords and screen keywords separately to generate word clouds representing the audio recording content and screen operation content, respectively.
[0078] Step 6: Based on the segmented and merged event logs, apply a process mining algorithm to obtain the business process diagram.
[0079] Based on the attributes in the segmented event logs—start time, end time, business number, and activity name—a process mining algorithm is applied to generate a business process diagram. In this embodiment, an inductive process mining algorithm is used, and the generated business process diagram is as follows: Figure 2 As shown on the left.
[0080] Step 7: Add corresponding operation sequences, audio word clouds, and screen word clouds to each business step in the business process diagram to obtain a multimodal business process diagram.
[0081] When displaying a business process diagram, the system queries the corresponding typical operation sequence, audio word cloud, and screen word cloud based on the activity name. These can be displayed when the user needs them, allowing for simultaneous viewing of multimodal data. In this embodiment, when the user moves the mouse over a node in the business process diagram, a node details page is automatically displayed, such as... Figure 2As shown on the right, this area contains a list of operation sequences, audio word clouds, and screen word clouds. When a user views a specific business process, a complete segmented event log is displayed, containing multiple business steps. Each business step includes the corresponding operation sequence, audio recordings, and screenshots.
[0082] With multimodal business process diagrams, users can continue to use existing business process analysis functions to perform various analysis and optimization operations. They can also quickly obtain the main semantics of the corresponding business steps through typical operation sequences, audio word clouds, and screen word clouds. Furthermore, they can fully reconstruct the user's actual business execution process through the details of a single business process, making it convenient for users to monitor and analyze business processes from both global and local perspectives.
[0083] Example 2:
[0084] This invention provides a system for generating multimodal business process diagrams, comprising:
[0085] Acquisition module: Used to acquire audio recording data, screen recording data, and event logs containing user ID and timestamp attributes;
[0086] Segmentation module: Used to segment and merge event logs based on predefined rules. Each segment of event log corresponds to a business step in the business process. The earliest timestamp in each segment is the start time of the business step, and the latest timestamp is the end time of the business step.
[0087] The first extraction module is used to extract audio segments from audio recording data and screenshots from screen recording data based on the start time, end time, and user ID of each business step.
[0088] The second extraction module is used to perform speech recognition on the audio clips to obtain the audio text, and extract the audio keywords from the audio text; it also performs OCR recognition on the screenshots to obtain the screen text, and extracts the screen keywords from the screen text.
[0089] The merging module is used to merge all event logs of the same business step, extract repeated operation sequences to obtain operation sequences representing business semantics; merge all recording keywords of the same business step to obtain a recording word cloud; and merge all screen keywords of the same business step to obtain a screen word cloud.
[0090] Process mining module: Used to obtain business process diagrams by applying process mining algorithms based on the segmented and merged event logs;
[0091] Output module: Used to add corresponding operation sequences, audio word clouds, and screen word clouds to each business step in the business process diagram to obtain a multimodal business process diagram.
[0092] Example 3:
[0093] This invention provides a computer device, including a processor and a storage medium;
[0094] The storage medium is used to store instructions;
[0095] The processor is configured to operate according to the instructions to execute the steps of the method described in Embodiment 1.
[0096] Example 4:
[0097] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in Embodiment 1.
[0098] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0099] This application is described with reference to business process diagrams and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each process and / or block in the business process diagram and / or block diagram, and combinations of processes and / or blocks in the business process diagram and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the business process... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0100] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a business process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0101] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing business processes. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0102] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for generating a multimodal business process diagram, characterized in that, Retrieve audio recordings, screen recordings, and event logs containing user ID and timestamp attributes; Event logs are segmented and merged based on predefined rules. Each segment corresponds to a business step in the business process. The earliest timestamp in each segment is the start time of the business step, and the latest timestamp is the end time of the business step. The segmentation and merging of event logs based on predefined rules includes: The event logs are grouped according to the business number and user number. Each group of events is sorted by timestamp. An event attribute is selected as the segment attribute. Consecutive events with the same attribute value are grouped into a segment. Each segment of the log corresponds to a business step in the business process. The business number remains unchanged. Based on the start time, end time, and user ID of each business step, extract audio segments from the audio recording data and extract screenshots from the screen recording data; The process involves: performing speech recognition on an audio clip to obtain audio text, and extracting keywords from the audio text; performing OCR on a screenshot to obtain screen text, and extracting screen keywords from the screen text; wherein, the extraction of screen keywords from the screen text includes: After obtaining all the text in the screenshot by OCR recognition, the text that changes between the two screenshots is retained, all the changed text is merged, and the keyword extraction algorithm is applied to obtain the screen keywords. Merge all event logs for the same business step, extract repeated operation sequences, and obtain operation sequences representing business semantics; merge all audio keywords for the same business step to obtain an audio word cloud; merge all screen keywords for the same business step to obtain a screen word cloud; wherein, obtaining the operation sequence representing business semantics includes: obtaining log records corresponding to segmented logs with the same activity name, calculating typical operation sequences that can represent these different operation sequences, and selecting the operation sequence with the most occurrences as the operation sequence representing business semantics; wherein, obtaining the word cloud includes: obtaining keywords from segmented logs with the same activity name, simply summing the keywords to obtain multiple keywords and their occurrence counts, using the occurrence counts as weights to generate a word cloud, and merging audio keywords and screen keywords respectively to generate audio word clouds and screen word clouds, representing audio content and screen operation content; Based on the segmented and merged event logs, a process mining algorithm is applied to obtain a business process diagram. Add corresponding operation sequences, audio word clouds, and screen word clouds to each business step in the business process diagram to obtain a multimodal business process diagram.
2. The method for generating a multimodal business process diagram according to claim 1, characterized in that, The acquisition of audio recording data, screen recording data, and event logs containing user ID and timestamp attributes includes: In response to the fulfillment of the triggering conditions, the system records the call to obtain audio data, records the screen operations to obtain screen recording data, and saves the logs of the business system to obtain event logs.
3. The method for generating a multimodal business process diagram according to claim 1, characterized in that, The grouping of event logs based on service number and user number includes: The event log includes a business number attribute; events with the same business number represent events in the same business process. The event log also includes a user ID attribute; events with the same user ID represent events from the same user.
4. The method for generating a multimodal business process diagram according to claim 1, characterized in that, The extraction of audio keywords from the audio transcript includes: After converting the audio clips into text using a speech recognition algorithm, a keyword extraction algorithm is applied to extract the keywords from the audio clips. If no audio segment exists, the audio keywords will be empty.
5. A system for generating multimodal business process diagrams, characterized in that, include: Acquisition module: Used to acquire audio recording data, screen recording data, and event logs containing user ID and timestamp attributes; Segmentation Module: Used to segment and merge event logs based on predefined rules. Each segment of the event log corresponds to a business step in the business process. The earliest timestamp in each segment is the start time of the business step, and the latest timestamp is the end time of the business step. The segmentation and merging of event logs based on predefined rules includes: The event logs are grouped according to the business number and user number. Each group of events is sorted by timestamp. An event attribute is selected as the segment attribute. Consecutive events with the same attribute value are grouped into a segment. Each segment of the log corresponds to a business step in the business process. The business number remains unchanged. The first extraction module is used to extract audio segments from audio recording data and screenshots from screen recording data based on the start time, end time, and user ID of each business step. The second extraction module is used to perform speech recognition on the audio clips to obtain audio text, and extract audio keywords from the audio text; to perform OCR recognition on the screenshots to obtain screen text, and extract screen keywords from the screen text; wherein, the extraction of screen keywords from the screen text includes: After obtaining all the text in the screenshot by OCR recognition, the text that changes between the two screenshots is retained, all the changed text is merged, and the keyword extraction algorithm is applied to obtain the screen keywords. The merging module is used to merge all event logs of the same business step, extract repeated operation sequences, and obtain operation sequences representing business semantics; merge all recording keywords of the same business step to obtain a recording word cloud; and merge all screen keywords of the same business step to obtain a screen word cloud. Obtaining the operation sequence representing business semantics includes: acquiring log records corresponding to segmented logs with the same activity name, calculating typical operation sequences that can represent these different operation sequences, and selecting the operation sequence with the most occurrences as the operation sequence representing business semantics. Obtaining the word cloud includes: acquiring keywords from segmented logs with the same activity name, simply summing the keywords to obtain multiple keywords and their occurrence counts, using the occurrence counts as weights to generate a word cloud, and merging recording keywords and screen keywords respectively to generate recording word clouds and screen word clouds, representing recording content and screen operation content. Process mining module: Used to obtain business process diagrams by applying process mining algorithms based on the segmented and merged event logs; Output module: Used to add corresponding operation sequences, audio word clouds, and screen word clouds to each business step in the business process diagram to obtain a multimodal business process diagram.
6. A computer device, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 4.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 4.