Employee database construction and state analysis method and system
By building an employee database and adopting multimodal fusion technology, the problems of single data and high computing resources in existing employee status analysis have been solved, realizing efficient and accurate employee status analysis, improving enterprise management efficiency and privacy protection.
Patent Information
- Application Number
- CN202510985732.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-17
- Publication Date
- 2025-10-31
AI Technical Summary
Existing employee status analysis methods rely on single-modal data, making it difficult to capture non-verbal behavioral features. They also suffer from low feature fusion efficiency, high computational resource requirements, and lack privacy protection, making it difficult to meet the needs of modern enterprise management.
An employee database is built to manage text, audio, and video data in a unified manner. A multimodal fusion framework is adopted, which uses Transformer networks and self-attention mechanisms to fuse data. Combined with keyframe extraction and speaker separation, the data volume is reduced and privacy is protected.
It enables the capture of multi-dimensional employee status characteristics, improves analysis accuracy and computational efficiency, ensures privacy and security, and provides an efficient and intelligent employee status analysis solution.
Smart Images
Figure CN120873064A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer technology and automated office management technology, and in particular to a method and system for constructing an employee database and analyzing its status. Background Technology
[0002] With the development of the times, enterprises have higher requirements for the scientific management of employees. Currently, they often understand and manage employees' work by checking monitoring or workload. Such management models are relatively complex and cumbersome and cannot meet the management needs of modern enterprises.
[0003] Existing methods for statistically analyzing employee work status rely on videos or images, making it difficult to determine whether employees are meeting performance standards. This requires manual review of individual images or videos, which limits the scope of companies with large workforces to random checks and hinders effective scientific management.
[0004] With the rapid development of artificial intelligence technology, multimodal data analysis has demonstrated enormous application potential across various industries. In the field of enterprise management, scientifically managing employees is of great significance for ensuring production efficiency, improving safety management, and optimizing human resource allocation. Analyzing employee work status is the foundation of scientific employee management.
[0005] Currently, enterprises mainly rely on single text or attendance data for employee status analysis, such as employee self-assessment, daily attendance, and work hour records. These methods are simple and easily affected by subjective factors. In some high-end scenarios, video surveillance is combined for behavior recognition or physiological sensors are used to obtain physiological signals such as heart rate and skin conductance for health warnings. However, these are often deployed in a scattered manner and lack a unified multimodal fusion framework.
[0006] Existing technologies still have the following main shortcomings in employee status analysis:
[0007] 1) The data modality is singular. Traditional methods usually rely on single modalities such as attendance records, questionnaires or office logs, which make it difficult to capture the non-verbal behavioral characteristics of employees during the work process, such as facial expressions, posture changes, and tone of voice, resulting in a one-sided state assessment.
[0008] 2) Low feature fusion efficiency. Existing multimodal fusion methods mostly adopt simple fusion strategies such as serialization or weighted averaging, ignoring the temporal, spatial and semantic relationships between different modal data, resulting in insufficient utilization of cross-modal information and limited analysis accuracy;
[0009] 3) High-frequency video acquisition generates massive amounts of data, which places high demands on the computing resources for multimodal fusion and can easily lead to system latency.
[0010] In recent years, deep learning-based multimodal fusion technology has made groundbreaking progress. Methods such as Transformer, cross-modal attention, and graph neural networks can achieve joint modeling of heterogeneous data such as images, text, audio, and video. However, existing multimodal fusion technologies are difficult to directly apply to employee status analysis, mainly facing a series of challenges such as data heterogeneity, diverse scenarios, real-time performance, and privacy protection. Summary of the Invention
[0011] To overcome the problems of heterogeneous data, diverse scenarios, poor real-time performance, and poor privacy when applying multimodal fusion technology to employee status analysis, this invention proposes an employee database construction method. It adopts a unified multimodal data access and preprocessing framework to achieve standardized management of heterogeneous data such as text, images, audio, and video, laying the foundation for multimodal data fusion.
[0012] The present invention proposes an employee database construction method, which first establishes a work status database, and generates a structured data table with associated identity for each employee in the work status database;
[0013] Collect employee text monitoring data, audio reporting data, and video surveillance data, and store them in the corresponding structured data tables;
[0014] The sources of text-based monitoring data include attendance records, office logs, and project reports; the sources of audio reporting data include recordings of work reports and excerpts from video conference recordings in which employees participate; and the sources of video surveillance data are surveillance videos from public areas.
[0015] Preferably, the video surveillance data is acquired by: collecting surveillance video, identifying employees using a face database and extracting video clips of designated employees; after adaptively adjusting the image quality and cropping blank segments of the video clips, extracting key frames using adaptive keyframe extraction technology to form the video surveillance data of the designated employees.
[0016] Preferably, the text monitoring data collection method is as follows: first, collect the text data of employees' attendance records, office logs and project reports; extract and clean the text content of each piece of text data, and then output it in Markdown format and store it in the structured data table corresponding to the work status database.
[0017] Preferably, the method for collecting video conference recording excerpts is as follows: first, retrieve the video conference recordings in which the employee participated; separate the speakers, identify and extract the voice segments of the designated employees; perform voice noise reduction and enhancement on the voice segments and trim blank segments to form the video conference recording excerpts corresponding to the designated employees.
[0018] Preferably, the identity information associated with the structured data table includes: employee ID, job type, and department attribute.
[0019] This invention proposes an employee status analysis method based on multimodal data. First, a report generation model and a work status database are constructed. The report generation model obtains text monitoring data, audio reporting data, and video monitoring data of each target employee within the monitoring period from the work status database, and generates work reports for the target employees. The report generation model is trained on a known dataset {text monitoring data, audio reporting data, and video monitoring data of target employees within the monitoring period; employee analysis reports} until the parameters are fixed.
[0020] The work status database is updated in real time. At each monitoring cycle node, the text supervision data, audio reporting data and video monitoring data of each target employee in the most recent monitoring cycle are input into the report generation model to obtain the target employee's analysis report.
[0021] Preferably, the report generation model includes a sequentially connected encoding layer, a temporal layer, multiple Transformer networks, and an integration module;
[0022] The encoding layer includes a text encoder, an audio encoder, and a video encoder. The text encoder extracts and encodes the tokens from the text monitoring data, and then performs linear dimensionality reduction on the token encodings to obtain text vectors. The audio encoder encodes the audio reporting data to obtain audio vectors. The video encoder encodes each keyframe of the video monitoring data, and then performs linear dimensionality reduction on the keyframe vectors to obtain video vectors. The semantic spaces of the text vectors, audio vectors, and video vectors are the same.
[0023] The temporal layer includes a temporal embedding module and a self-attention mechanism. The text vectors, audio vectors, and video vectors output from the encoding layer are embedded with timestamps using a learnable time-aware encoding technique through the temporal embedding module, and then encoded into attention feature data through the self-attention mechanism. The attention feature data is then processed by the Transformer network for encoding and decoding to obtain key information encoding, which includes keywords and images. The integration module refers to the known text encoding-lexicon lookup table and image encoding-RGB lookup table to convert the text encoding and image encoding in the key information encoding into keyword lexicon and key image, respectively, and then generates and outputs an analysis report.
[0024] Preferably, when embedding timestamps into a vector using learnable time-aware coding techniques, the time embedding module adds periodic coding to each dimension using the following formula:
[0025]
[0026] Where, learnable_bias is a learnable parameter; PE 2i (t) is the periodic code of the 2i-th column of the vector, PE 2i+1 (t) represents the periodic code of the (2i+1)th column of the vector, where i is a positive integer. This indicates rounding down from d / 2; d is the vector dimension, and t represents the timestamp information.
[0027] Preferably, whenever an analysis report is generated, the text monitoring data, audio reporting data, and video monitoring data for the corresponding monitoring period are deleted from the work status database, with the monitoring period set to one week.
[0028] The present invention proposes an employee status analysis system, comprising: a human-computer interaction module, a work status database, a data acquisition and processing module, and a report generation module;
[0029] The human-computer interaction module is used to add, delete, and edit structured data tables in the work status database;
[0030] The data acquisition and processing module is used to collect data from data sources, parse the text monitoring data, audio reporting data and video surveillance data of each employee, and store them in the structured data table corresponding to the work status database.
[0031] The report generation module has a built-in report generation model. The report generation module uses the report generation model to periodically process the text supervision data, audio reporting data and video monitoring data stored in the structured data table of a specified employee in the work status database, and obtains and outputs the corresponding analysis report for the specified employee.
[0032] The advantages of this invention are:
[0033] (1) The employee database construction method proposed in this invention supports simultaneous access to multiple data sources such as text, audio, and video, and can capture multi-dimensional state characteristics of employees during work, such as language, behavior, facial expressions, and tone of voice, to achieve a comprehensive portrayal of employee status. This invention realizes structured and unified data management, improves data processing efficiency, and achieves efficient and standardized data management processes by establishing a private structured data table for each employee and integrating automatic cleaning, format conversion, and unified time alignment mechanisms.
[0034] (2) This invention significantly reduces the amount of video data processing through keyframe adaptive extraction technology; speaker separation and blank segment trimming compress audio data. This invention employs a lightweight multimodal data processing strategy, effectively solving the problem of computationally intensive multimodal data processing while maintaining computational efficiency. All raw data in the work status database is periodically deleted, retaining only analysis reports, effectively reducing the risk of long-term storage of sensitive data, protecting employee privacy, and strengthening privacy protection.
[0035] (3) The present invention proposes an employee status analysis method based on multimodal data, which constructs a cross-modal association modeling mechanism, realizes information complementarity between different modalities by fusing attention networks, and introduces a temporal network layer to strengthen the temporal change information of employee status, improve the accuracy of employee status identification, and enhance the comprehensiveness of assessment.
[0036] (4) The report generation model in this invention adopts the Decoder-Only autoregressive generation model, and the setting of the temporal layer improves the temporal sensitivity of the model. The semantic space of the text vector, audio vector and video vector output by the encoding layer in the report generation model is the same, which facilitates subsequent feature fusion and ensures the full fusion of multimodal data.
[0037] (5) In this invention, the temporal layer can effectively capture the dynamic change trend of employee status. Through learnable positional encoding and periodic modeling, it can identify temporal characteristics such as employee work patterns and fatigue cycles, thereby improving the accuracy of status change trend analysis.
[0038] (6) The work status database, multimodal data analysis, and analysis report generation process constructed in this invention can achieve in-depth understanding and intelligent analysis of multiple modal data such as text, audio, and video, significantly improving the efficiency of timely feedback to enterprises on employee work results and changes in personal status. This invention constructs an efficient and accurate method for analyzing employee work status through deep fusion and intelligent processing of multimodal data, providing a new technological paradigm for the field of smart office management.
[0039] (7) The present invention proposes an intelligent report generation system for employee status analysis, which improves readability and decision support value. The report generation model is based on multimodal content to realize content summarization and visualization of text and graphics. The output results are clear and well-structured, helping managers to quickly understand employee status and work performance. The present invention provides an intelligent, scalable, secure and efficient employee status analysis solution, providing a new generation of technical support for intelligent enterprise management systems. Attached Figure Description
[0040] Figure 1 This is a flowchart of an employee database construction method proposed in this invention;
[0041] Figure 2 This is a flowchart of an employee status analysis method based on multimodal data proposed in this invention;
[0042] Figure 3 Generate a model structure diagram for the report;
[0043] Figure 4 Example of an analysis report generated for an implementation example. Detailed Implementation
[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0045] Reference Figure 1 This implementation proposes a method for constructing an employee database to establish a unique work status database for each employee. For each employee, a structured data table containing employee ID, job type, and department attribute is generated in the work status database, and the employee's text monitoring data, audio reporting data, and video surveillance data are aggregated.
[0046] Text-based monitoring data includes: employees' daily attendance records, office logs, project reports, etc.
[0047] The text monitoring data is collected as follows: First, text data such as attendance records, office logs, email records, and project reports are collected through attendance systems and OA systems; text content is extracted and cleaned for each piece of text data, and then output in Markdown format.
[0048] Text content extraction and cleaning operations include text content recognition and text layout analysis. In practice, a natural language processing module can be used to recognize content, extract keywords such as "project progress," "customer communication," and "technical challenges," and then convert them into Markdown format for storage.
[0049] Audio reporting data includes recordings of work reports and excerpts from video conference recordings in which employees participate.
[0050] The method for collecting video conference recording excerpts is as follows: First, the video conference recordings in which employees participated are retrieved based on information such as meeting minutes; the speakers are separated, the voice segments of designated employees are identified and retained, and the voices of other participants are filtered out; the voice segments are enhanced by voice denoising, background noise is eliminated, and blank segments are automatically trimmed to form the video conference recording excerpt corresponding to the designated employee.
[0051] Work report recordings are directly linked to employee information, so we only need to perform noise reduction and enhancement to link them to employees and store them in the work status database.
[0052] The video surveillance data comes from surveillance videos in public areas. Employees are automatically identified based on the company's facial recognition database, and only video clips featuring specific employees are retained. After adaptive quality adjustment and blank segment cropping of the video clips, keyframes are extracted using adaptive keyframe extraction technology to construct the video surveillance data. In specific implementation, it can be set to mark a keyframe when the RGB value of the image exceeds a set threshold.
[0053] The adaptive image quality adjustment function can adjust the video resolution to 256×256.
[0054] In this embodiment, the location of the target employee is automatically detected based on the company's facial database, enabling target person recognition. This ensures that only the target employee's activity segments are retained in the video, effectively preventing interference from other individuals. The adaptive keyframe extraction operation automatically detects frames where the RGB value variation exceeds a set threshold, significantly reducing redundant information in the video and improving video data processing speed.
[0055] In this embodiment, the collected text monitoring data, audio reporting data, and video surveillance data are all accompanied by precise timestamps for subsequent time-series analysis.
[0056] In practical implementation, considering the database storage load and employee analysis cycles, the original multimodal data can be automatically cleaned up periodically, retaining only the analysis reports to ensure employee information security and data privacy. The analysis cycle can be set to "weekly".
[0057] Reference Figure 2 , Figure 3 This embodiment proposes an employee status analysis method based on multimodal data, which includes the following steps:
[0058] S1. Construct a report generation model. The report generation model obtains text monitoring data, audio reporting data and video monitoring data of each target person within the monitoring period from the work status database, and generates work reports for the target person.
[0059] The report generation model includes a sequentially connected encoding layer, a temporal layer, multiple Transformer networks, and an integration module; the encoding layer includes a text encoder, an audio encoder, and a video encoder.
[0060] The text encoder is used to encode each piece of text monitoring data of the target personnel within the monitoring period, and obtain the d-dimensional text vector of each piece of text monitoring data. When the text encoder encodes the text monitoring data, it first extracts words through natural language processing technology, encodes each word into a d-dimensional vector, and obtains an m×d-dimensional vector of text monitoring data. After concatenating them into an (1,m×d) encoded vector, it reduces the dimension through a linear layer to obtain a d-dimensional text vector.
[0061] The audio encoder is used to encode each audio report data of the target personnel within the monitoring period, and obtain the d-dimensional audio vector of each audio report data;
[0062] The video encoder is used to encode each video surveillance data point of the target personnel within the monitoring period, obtaining a d-dimensional video vector for each video surveillance data point. Since each video surveillance data point contains n keyframes, the video encoder first encodes each keyframe, encoding each keyframe into a d-dimensional vector, thus obtaining an n×d-dimensional vector corresponding to the video surveillance data. After concatenating these into an (1, n×d) encoded vector, a linear layer is used for dimensionality reduction to obtain the d-dimensional video vector.
[0063] The temporal layer includes a temporal embedding module and a self-attention mechanism. d-dimensional text vectors, d-dimensional audio vectors, and d-dimensional video vectors are input to the temporal embedding module, where timestamps are embedded using a learnable time-aware encoding technique, and then encoded into attention feature data through the self-attention mechanism.
[0064] d-dimensional text vectors, d-dimensional audio vectors, and d-dimensional video vectors are collectively referred to as d-dimensional vectors. The method for embedding timestamps into each d-dimensional vector using learnable time-aware coding techniques is as follows:
[0065] Periodic encoding is added to each dimension, and the periodic encoding expression is as follows:
[0066]
[0067] Wherein, learnable_bias is a learnable parameter used to optimize the numerical value of positional encoding; PE 2i (t) is the periodic code of the 2i-th column of the d-dimensional vector, PE 2i+1 (t) is the periodic code of the (2i+1)th column of the d-dimensional vector, where i is a positive integer. This indicates rounding down from d / 2; t represents timestamp information; These are the sine and cosine functions, respectively.
[0068] The temporal layer processes the input d-dimensional text vector, d-dimensional audio vector, and d-dimensional video vector as follows:
[0069] TemporalLayer(x)=Self-Attention(x+PE(t))
[0070] Where x represents the set of input text vectors, audio vectors, and video vectors, PE(t) is periodic encoding, Self-Attention(·) is the self-attention mechanism, and TemporalLayer(x) is the output of the temporal layer, i.e., attention feature data.
[0071] Attention feature data is sequentially processed by Transformer network encoding and decoding to obtain key information encoding, which includes keywords and images. The integration module refers to the known text encoding-lexical lookup table and image encoding-RGB lookup table to convert the text encoding and image encoding in the key information encoding into keyword words and key images, respectively, and then generates and outputs an analysis report.
[0072] S2. The report generation model is trained on a known dataset {text monitoring data, audio reporting data, and video surveillance data of target employees during the monitoring period; employee analysis reports} until the model is fixed.
[0073] Specifically, the loss function in the training process of the report generation model is the cross-entropy loss between the text and image encodings output by the last Transformer network in the model and the corresponding encodings in the real report.
[0074] In the real report, the lexical reference text encoding-lexical reference table is converted into the real lexical encoding, and the images in the real report are converted into the real image encoding by referring to the image encoding-RGB reference table.
[0075] S3. Update the work status database in real time. At each monitoring cycle node, input the text supervision data, audio reporting data and video monitoring data of each target employee within the monitoring cycle into the report generation model to obtain the analysis report of the target employee.
[0076] S4. Whenever an analysis report is generated, delete the text monitoring data, audio reporting data, and video monitoring data for the corresponding monitoring period from the work status database.
[0077] In this embodiment, an employee status analysis system is also proposed, including a human-computer interaction module, a work status database, a data acquisition and processing module, and a report generation module.
[0078] The human-computer interaction module is used to add, delete, and edit structured data tables in the work status database.
[0079] The data acquisition and processing module is used to collect data from data sources and parse the text monitoring data, audio reporting data and video surveillance data of each employee, and store them in the structured data table corresponding to the work status database.
[0080] Specifically, the data acquisition and processing module obtains text data such as employees' daily attendance records, office logs, and performance results, audio data such as work report recordings, and surveillance video data from public areas. It also performs preprocessing operations such as data cleaning on the data and stores the employees' text monitoring data, audio report data, and video surveillance data into the employee's structured data table in the work status database.
[0081] The report generation module has a built-in report generation model. The report generation module uses the report generation model to periodically process the text supervision data, audio reporting data and video monitoring data stored in the structured data table of a specified employee in the work status database, and obtains and outputs the corresponding analysis report for the specified employee.
[0082] The report generation module supports both text and image output formats, providing rich and intuitive analysis results. Furthermore, to ensure employee information security and data privacy, the work status database periodically and automatically cleans up the original multimodal data, retaining only the analysis reports.
[0083] The following describes the above-mentioned employee status analysis method based on multimodal data in conjunction with specific embodiments.
[0084] In this embodiment, the structured data table stores employee text monitoring data, audio reporting data, and video surveillance data on a daily basis. Special characters containing time information and modality type special characters are added to the input data for each day to enhance the model's perception of time series and the differentiation of different modalities, such as... <day1> <text> a< / text> <audio> b< / audio> <video> c< / video> < / day1> When generating the report, prompts can be entered to guide the report generation model, such as "Please generate an employee work status analysis report based on employee A's work data from June 9th to June 15th, 2025." The model's output includes... <text> ,< / text> , Special characters such as commas are used to ensure the model can output both text and image data, and to distinguish between the text and image portions of the output content. The output image data is converted using a pre-defined image encoding-RGB lookup table and integrated into a complete image with a size controlled at 256×256. This allows the image data to be modified according to the semantics of the generated analysis report, such as adding bounding boxes.
[0085] In this embodiment, to ensure the clarity of the output image, a super-resolution model can also be used to perform image enhancement operations to obtain high-definition images of 512×512 or 1024×1024.
[0086] In a specific embodiment, after the system starts, it first creates a unique employee work status database, generating a structured data table containing employee ID, job type, and department attributes. A data lifecycle strategy is set: the raw multimodal data storage period is ≤7 days, analysis reports are permanently stored, and the system performs an automatic cleanup task every morning. An enterprise face database index table is constructed to establish a mapping relationship between employee IDs and biometric features for target recognition in video data.
[0087] The data acquisition and analysis module uploads and analyzes multimodal data such as text, audio, and video, and stores it in the work status database.
[0088] Taking "Employee 1" as an example, from June 9th to June 15th, 2025, the system automatically collected multimodal work data for Employee 1, including: work logs, email records, attendance records, project reports, and other text data obtained from the enterprise OA system; audio recordings of video conferences and work reports participated in by Employee 1; and work videos obtained from office area surveillance cameras. All collected and uploaded data were accompanied by precise timestamps for subsequent time-series analysis.
[0089] The data acquisition and analysis module organizes the collected data into text monitoring data, audio reporting data, and video monitoring data, and then stores them in the structured data table corresponding to employee 1 in the work status database; the text monitoring data is stored in Markdown format.
[0090] Organize the data collected within a week in a structured data table according to time sequence and label it with modality tags, such as... <day1> <text> a< / text> <audio> b< / audio> <video> c< / video> < / day1> The encoding layer of the report generation model maps data from various modalities to a unified semantic space to ensure representation consistency; and introduces a learnable positional encoding temporal layer to model periodicity and temporal dependency features, forming a temporal embedding.
[0091] The report generation model employs a Decoder-Only autoregressive generative model, generating the analysis report from the fused temporal embedding vectors. The input prompt, "Please generate the corresponding employee work status analysis report based on employee 1's work data from June 9th to June 15th, 2025," guides the model in report generation. The model output supports a text + image structure, and the image data visualization processing uses tagging and automatic content block segmentation to select and annotate employee behaviors in video keyframes. The analysis report is output in Markdown format, with a clear structure, facilitating reading and archiving.
[0092] The generated reports are automatically pushed to the enterprise management system interface for managers to view. All original text and audio / video are periodically deleted, retaining only the report content to protect employee privacy.
[0093] In this embodiment, d = 8192 is set, and the number of Transformer networks in the report generation model is 12. To enhance the readability of the analysis report, all models are generated in Markdown format, including employee work overview, attendance records, work logs, key achievements and value, work report summary, and work emotion recognition summary.
[0094] Output report text content such as Figure 4 As shown.
[0095] Of course, those skilled in the art will recognize that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
[0096] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
[0097] The technologies, shapes, and structures not described in detail in this invention are all known technologies.
Claims
1. A method for constructing an employee database, characterized in that, First, a work status database is established, and a structured data table with associated identity is generated for each employee in the work status database; Collect employee text monitoring data, audio reporting data, and video surveillance data, and store them in the corresponding structured data tables; The sources of text-based monitoring data include attendance records, office logs, and project reports; the sources of audio reporting data include recordings of work reports and excerpts from video conference recordings in which employees participate; and the sources of video surveillance data are surveillance videos from public areas.
2. The employee database construction method as described in claim 1, characterized in that, The video surveillance data is acquired as follows: surveillance video is collected, employees are identified by combining the face database, and video clips of designated employees are extracted; after adaptive quality adjustment and blank segment cropping of the video clips, key frames are extracted using adaptive keyframe extraction technology to form the video surveillance data of the designated employees.
3. The employee database construction method as described in claim 1, characterized in that, The text monitoring data is collected as follows: First, the text data of employee attendance records, office logs, and project reports are collected; the text content of each piece of text data is extracted and cleaned, and then output in Markdown format and stored in the structured data table corresponding to the work status database.
4. The employee database construction method as described in claim 1, characterized in that, The method for collecting video conference recording excerpts is as follows: First, retrieve the video conference recordings in which employees participate; separate the speakers, identify and extract the voice segments of designated employees; perform voice noise reduction and enhancement on the voice segments and trim blank segments to form the video conference recording excerpts corresponding to the designated employees.
5. The employee database construction method according to any one of claims 1-4, characterized in that, The identity information associated with the structured data table includes: employee ID, job type, and department attribute.
6. A method for employee status analysis based on multimodal data, employing the employee database construction method as described in any one of claims 1-5, characterized in that, First, a report generation model and a work status database are constructed. The report generation model obtains text monitoring data, audio reporting data, and video monitoring data of each target employee during the monitoring period from the work status database, and generates work reports for the target employees. The report generation model is trained on a known dataset {text monitoring data, audio reporting data, and video monitoring data of target employees during the monitoring period; employee analysis reports} until the parameters are fixed. The work status database is updated in real time. At each monitoring cycle node, the text supervision data, audio reporting data and video monitoring data of each target employee in the most recent monitoring cycle are input into the report generation model to obtain the target employee's analysis report.
7. The employee status analysis method based on multimodal data as described in claim 6, characterized in that, The report generation model includes a sequentially connected encoding layer, a temporal layer, multiple Transformer networks, and an integration module; The encoding layer includes a text encoder, an audio encoder, and a video encoder. The text encoder extracts and encodes the tokens from the text monitoring data, and then performs linear dimensionality reduction on the token encodings to obtain text vectors. The audio encoder encodes the audio reporting data to obtain audio vectors. The video encoder encodes each keyframe of the video monitoring data, and then performs linear dimensionality reduction on the keyframe vectors to obtain video vectors. The semantic spaces of the text vectors, audio vectors, and video vectors are the same. The temporal layer includes a temporal embedding module and a self-attention mechanism. The text vectors, audio vectors, and video vectors output by the encoding layer are embedded with timestamps through the temporal embedding module using learnable time-aware coding techniques, and then encoded into attention feature data through the self-attention mechanism. Attention feature data is sequentially processed by Transformer network encoding and decoding to obtain key information encoding, which includes keywords and images. The integration module refers to the known text encoding-lexical lookup table and image encoding-RGB lookup table to convert the text encoding and image encoding in the key information encoding into keyword words and key images, respectively, and then generates and outputs an analysis report.
8. The employee status analysis method based on multimodal data as described in claim 7, characterized in that, When embedding timestamps into vectors using learnable time-aware coding techniques, the time embedding module adds periodic coding to each dimension using the following formula: Where, learnable_bias is a learnable parameter; PE 2i (t) is the periodic code of the 2i-th column of the vector, PE 2i+1 (t) represents the periodic code of the (2i+1)th column of the vector, where i is a positive integer. This indicates rounding down from d / 2; d is the vector dimension, and t represents the timestamp information.
9. The employee status analysis method based on multimodal data as described in claim 6, characterized in that, Whenever an analysis report is generated, the text monitoring data, audio reporting data, and video monitoring data for the corresponding monitoring period are deleted from the work status database. The monitoring period is set to one week.
10. A system for performing the employee status analysis method based on multimodal data as described in any one of claims 6-9, characterized in that, include: Human-computer interaction module, working status database, data acquisition and processing module, and report generation module; The human-computer interaction module is used to add, delete, and edit structured data tables in the work status database; The data acquisition and processing module is used to collect data from data sources, parse the text monitoring data, audio reporting data and video surveillance data of each employee, and store them in the structured data table corresponding to the work status database. The report generation module has a built-in report generation model. The report generation module uses the report generation model to periodically process the text supervision data, audio reporting data and video monitoring data stored in the structured data table of a specified employee in the work status database, and obtains and outputs the corresponding analysis report for the specified employee.