Intelligent conference summary generation method and system based on multi-modal time sequence characteristics

Abstract

The invention relates to the technical field of multimedia data processing, in particular to a conference summary intelligent generation method and system based on multi-modal time sequence characteristics. The method comprises the following steps: acquiring a real-time video stream and a real-time audio stream of a conference site by using an intelligent docking station, and preprocessing; spatial dispersion is calculated by using geometric distribution of inter-frame difference pixels, global page turning and local interference are distinguished based on the spatial dispersion, and a visual reference timestamp is locked; constructing a causal search window by taking the timestamp as a starting point, detecting a first mutation point of an audio energy gradient in the window, and determining actual effective time; semantic segmentation is carried out by using the actual effective time, and a structured summary is generated in combination with the visual reference frame. According to the method, a mechanism of combining spatial dispersion filtering and causal time sequence calibration is constructed, so that the problem of summary affiliation dislocation caused by audio-visual time difference of a speaker is effectively solved, and the robustness of the finally generated conference summary is improved.

Description

Technical Field

[0001] This invention relates to the field of multimedia data processing technology, specifically to a method and system for intelligent generation of meeting minutes based on multimodal temporal features. Background Technology

[0002] With the popularization of remote work and digital meetings, automatic meeting minutes generation technology has become a key tool for improving office efficiency. Existing technical solutions usually adopt a hard synchronization strategy of audio and video, that is, mechanically detect the page turning time of the slide in the video screen, and directly use this time as the dividing point to classify the speech-to-text before and after the time into the content of the two slides respectively.

[0003] However, in practical applications, this hard synchronization scheme suffers from serious pseudo-synchronization defects, leading to misattribution. Behavioral studies show that human speech has a natural audiovisual time difference. Speakers usually turn slides first, followed by a 1 to 3 second pause to organize their thoughts or observe the audience's reaction before starting their audio explanation of the new page. Traditional methods often incorrectly assign these few seconds of pause or the closing remarks of the previous page to the new page, or omit the opening remarks of the current page from the previous page, resulting in a chaotic and semantically incoherent summary structure.

[0004] Furthermore, existing technologies often use external cameras to photograph the screen, which is highly susceptible to factors such as ambient light reflection, shooting angle deviation, and moiré interference, resulting in a large amount of noise in the acquired image data and significant shortcomings in environmental adaptability. In addition, existing technologies often use fixed threshold methods, such as determining page turning as a pixel change rate >50%. However, for common operations during conference presentations, such as large mouse movements, IM software message pop-ups, and partial animations within slides, although the pixel change may be large, it does not necessarily indicate a change in topic. Existing technologies lack the spatial discrimination capability between global page turning and local interference, easily fragmenting a complete topic into invalid fragments, seriously affecting the readability of the meeting minutes. Summary of the Invention

[0005] To address the problem that existing technologies cannot intelligently generate accurate meeting minutes based on changes in the scenario, this invention provides a method and system for intelligently generating meeting minutes based on multimodal temporal features.

[0006] In a first aspect, the present invention provides a method for intelligent generation of meeting minutes based on multimodal temporal features, employing the following technical solution: The system acquires and preprocesses real-time video and audio streams from the meeting. It performs inter-frame difference operations on the video stream to extract a set of differing pixels. Based on the geometric distribution characteristics of the coordinates of these differing pixels on the image plane, it calculates a spatial dispersion index characterizing the degree of dispersion of image changes. The spatial dispersion index is compared with a preset global change threshold. If it exceeds the threshold, local interference is excluded, and a global topic switch is determined to have occurred at the current moment, which is then marked as a visual reference timestamp. Starting from the visual reference timestamp, a backward-extending causal search window is constructed in the real-time audio stream. Within this window, the audio energy gradient is calculated, and an audio abrupt change exceeding a preset gradient threshold is detected. If such a change exists, the time corresponding to the abrupt change is marked as the actual effective time. The actual effective time is used to semantically segment the real-time audio stream. The segmented audio content is then associated with the video frame image corresponding to the visual reference timestamp to generate a structured meeting minutes.

[0007] This invention effectively solves the problem of misalignment of text and image attribution caused by speaker cognitive delay in traditional audiovisual synchronization methods by calculating the spatial dispersion index that characterizes the degree of dispersion of image changes and combining it with causal search window for temporal calibration, thereby improving the accuracy and semantic coherence of meeting minutes in complex interference scenarios.

[0008] Furthermore, the calculation of the spatial dispersion index characterizing the degree of dispersion of image changes specifically includes: obtaining the horizontal and vertical resolutions of the real-time video stream; obtaining the abscissa and ordinate of each pixel in the set of differing pixels; calculating the variance of the abscissa and the variance of the ordinate respectively; normalizing and weighting the variances of the abscissa and the ordinate based on the horizontal and vertical resolutions; calculating the ratio of the total number of pixels in the set of differing pixels to the product of the horizontal and vertical resolutions to obtain the pixel density factor; and multiplying the result of the normalization and weighted summation with the pixel density factor to obtain the spatial dispersion index.

[0009] Furthermore, the global change threshold is calibrated based on the classification optimization experiment of the interference samples.

[0010] This invention introduces a global change threshold based on interference sample classification optimization experiments, and constructs an adversarial sample library for real meeting scenarios to automatically optimize parameters. This ensures a low false trigger rate when faced with large mouse movements, system pop-ups, and local animation demonstrations, and achieves intelligent filtering of visual interference sources.

[0011] Furthermore, the causal search window is set based on the cognitive delay characteristics between human visual perception and language expression.

[0012] This invention constructs a causal search window based on the characteristics of human cognitive delay, establishing a dynamic monitoring time domain after the visual reference timestamp. This restores the speaker's behavioral pattern from operating the document to organizing thoughts and then starting to explain, automatically compensating for the natural time difference between visual operation and verbal expression, and preventing the conclusion of the previous topic from being mistakenly attributed to the current topic.

[0013] Furthermore, the intelligent generation method for meeting minutes based on multimodal temporal features further includes: dividing the audio data within the causal search window into continuous short frames, calculating the energy value of each short frame, calculating the difference between the energy value of the current short frame and the energy value of the previous short frame to obtain the audio energy gradient, comparing the audio energy gradient with the gradient threshold, and determining that the current short frame is the audio abrupt change point if the audio energy gradient exceeds the gradient threshold.

[0014] Furthermore, the intelligent generation method for meeting minutes based on multimodal temporal features also includes: if no audio mutation point is detected within the causal search window, the event corresponding to the visual reference timestamp is determined to be a silent operation demonstration, and semantic segmentation is not performed.

[0015] This invention effectively solves the problem of invalid segmentation that occurs when speakers quickly skip slides or make purely visual presentations, avoids generating empty minutes with only titles and no body content, and ensures the logical continuity and compactness of the meeting minutes text flow.

[0016] Furthermore, the preprocessing includes: denoising the real-time video stream and the real-time audio stream, and calibrating and aligning the timelines of the real-time video stream and the real-time audio stream.

[0017] Furthermore, the generation of structured meeting minutes includes: performing optical character recognition on the video frame image corresponding to the visual reference timestamp to extract the title text as an index title; performing speech recognition on the data in the real-time audio stream located after the actual effective time to convert it into body text; and encapsulating and storing the index title, the body text, and the video frame image according to their time relationship.

[0018] Furthermore, the length of the causal search window is set based on the cognitive delay characteristics and in combination with the speaker's cumulative speaking rate.

[0019] This invention establishes a correlation adjustment mechanism between the length of the causal search window and the speaker's cumulative speaking rate. It adaptively adjusts the length of the causal search window according to the speaker's speaking rhythm, enabling a rapid response in fast-paced brainstorming sessions and providing sufficient buffer time in slow-paced academic reports, thus improving adaptability to different speaking styles.

[0020] Secondly, this invention provides an intelligent meeting minutes generation system based on multimodal temporal features, employing the following technical solution: The intelligent meeting minutes generation system based on multimodal time-series features includes: a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the above-mentioned intelligent meeting minutes generation method based on multimodal time-series features is implemented. By adopting the above technical solution, the intelligent generation method for meeting minutes based on multimodal time-series characteristics is generated into a computer program and stored in a container, which is then loaded and executed by the processor. This allows for the creation of a terminal device based on the memory and processor, making it convenient to use.

[0021] The present invention has the following technical effects: This invention breaks through the limitations of existing technologies that rely excessively on pixel changes for event monitoring. From the perspective of human cognitive behavior, it establishes a causal search window mechanism that includes cognitive delay compensation. By monitoring audio mutation points that exceed a preset gradient threshold after the visual reference timestamp, it accurately locates the actual effective time of the voice explanation, achieving a leap from physical time alignment to semantic logic alignment, and effectively suppressing the misalignment of text and image due to the speaker's audiovisual time difference. The discreteness analysis model based on spatial geometric distribution constructed in this invention provides a low-computing-power, high-precision discrimination method for distinguishing between effective global topic switching and invalid local interference by utilizing the spatial discreteness index. Without introducing a deep learning model with high computational cost, it effectively improves the filtering ability of mouse movement, pop-up interference, and local animation. This invention possesses intelligent archiving capabilities for silent presentation scenarios and adaptive adjustment capabilities for the speaker's speaking speed. This results in structured meeting minutes that are not only clear and well-organized with accurate text-image correspondence, but also adaptable to various complex scenarios such as fast-paced discussions and slow-paced academic reports, demonstrating significant economic benefits and application prospects. Attached Figure Description

[0022] Figure 1 This is a flowchart of the intelligent generation method for meeting minutes based on multimodal temporal features provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the timing alignment principle provided in an embodiment of the present invention. Detailed Implementation

[0023] This invention provides a method for intelligent generation of meeting minutes based on multimodal temporal features, referring to... Figure 1 This includes steps S1-S4: S1: Data Acquisition and Preprocessing.

[0024] Specifically, the system uses a smart docking station as the core sensing node to acquire real-time video and audio streams from the conference at the data transmission link layer. It then uses a unified hardware clock inside the docking station for timing alignment, performs grayscale conversion and filtering on the real-time video stream to remove visual noise, and performs background noise suppression on the real-time audio stream to improve the clarity of human voices. The result is multimodal data with high signal-to-noise ratio and timing alignment.

[0025] First, we set up the hardware acquisition environment: To ensure accurate capture of meeting content, this embodiment uses an HB985 smart docking station with integrated signal processing unit as the core acquisition device.

[0026] The expansion dock is connected in series between the conference host and the display terminal. Utilizing the video processing chip integrated inside the expansion dock, a clean digital video stream is mirrored through the internal bus while transmitting the video signal. Considering the differences in resolution and aspect ratio of different conference terminals, such as 4K, 1080P, 9:16 portrait mobile phones, or 21:9 ultrawide screens, this embodiment adopts a proportional downsampling strategy based on total pixel limit in order to reduce the computing load while maintaining the geometric features of the image. The standard total pixel threshold is set at 2.07 million pixels, corresponding to standard 1080P. When the total pixel count of the digital video stream exceeds this threshold, the video frames are proportionally scaled while maintaining the original aspect ratio, so that the total pixel count is close to the threshold. At the same time, the frame rate is normalized to 30fps to obtain a real-time video stream. The omnidirectional digital microphone array integrated on the docking station is used as an audio acquisition device to collect real-time audio streams. The audio is then converted from analog to digital by the internal SoC main control chip. The audio sampling rate is set to 44.1kHz and the quantization bit depth is 16bit to ensure accurate reproduction of the speaker's voice information and coverage of the full frequency range of human voice.

[0027] Next, multimodal timing alignment is performed: Traditional methods often use cameras and microphones to collect video and audio data separately. However, since cameras and microphones are usually heterogeneous hardware and transmit data through different network buses, there is a millisecond-level startup time difference. In order to ensure strict alignment of visual images and auditory sounds on the timeline, this embodiment adopts a board-level hardware timing mechanism. The main control chip of the expansion dock immediately adds a unified 64-bit high-precision Unix timestamp the moment it reads a frame of video data and a frame of audio data, controlling the physical time alignment error between video frames and audio frames to within 1ms, and providing a reliable data foundation for subsequent operations.

[0028] Perform real-time video stream preprocessing: For the acquired real-time video stream, this embodiment adopts a preprocessing strategy that combines grayscale conversion and Gaussian filtering, as detailed below; Grayscale processing: Since the core of slide flip detection lies in the sudden change of brightness information rather than color information, in order to reduce the computational load of subsequent algorithms, the acquired three-channel color image needs to be converted into a single-channel grayscale image. Considering the difference in sensitivity of the human eye to different wavelengths of light, the grayscale image obtained by directly taking the average value will lose the brightness levels perceived by the human eye. This embodiment adopts a psychological grayscale conversion model based on the ITU-R BT.061 standard. Therefore, red is taken as 0.299, green as 0.587, and blue as 0.114 as weights to perform weighted dimensionality reduction on the acquired color image. This method can effectively preserve the edge information of low-contrast text in the slide and prevent texture loss during the grayscale process. Gaussian smoothing filter: Considering that compression artifacts or jagged fonts may exist during video transmission, this embodiment uses a Gaussian smoothing filter with a size of 5×5 and a standard deviation of 1.5 to perform spatial convolution processing on the acquired real-time video stream, smoothing image texture and suppressing high-frequency noise.

[0029] Finally, a grayscale image sequence is obtained.

[0030] Perform real-time audio stream preprocessing: For the acquired real-time audio stream, considering that the conference room environment is often accompanied by steady-state background noise such as air conditioning vents and projector cooling fans, these persistent low-frequency noises will raise the audio energy benchmark and affect the sensitivity of subsequent gradient detection of audio abrupt changes. Therefore, this embodiment uses spectral subtraction based on minimum statistics to denoise the acquired real-time audio stream, as follows: Background noise reference spectrum calibration: Ambient sound is collected within 2 seconds after the audio acquisition device is started and recorded as raw audio data. The raw audio data is divided into continuous audio frames according to the standard frame length of 20ms and the frame shift step length of 10ms. A short-time Fourier transform is performed on each frame of raw audio data to convert it from a time domain signal to a frequency domain signal. The average amplitude of all audio frames at each frequency point is calculated to obtain the background noise reference spectrum representing the current acoustic environment characteristics. During the conference, for each new frame of the noisy signal in the real-time audio stream, a short-time Fourier transform is first used to convert it into a frequency domain amplitude spectrum. Then, an amplitude subtraction operation is performed, that is, at each frequency point, the corresponding amplitude in the background noise reference spectrum is subtracted from the current frame's spectral amplitude to obtain the clean spectral amplitude. At the same time, half-wave rectification is performed, that is, when the result of the subtraction at a certain frequency point is negative, its amplitude is set to zero to ensure the accuracy of the physical meaning. The pure spectral amplitude is extracted and then subjected to a short-time Fourier transform (SFT) with the original phase information retained during the original audio data. This transforms the frequency domain signal back into a time domain signal, resulting in independent denoised audio frames. The processor then overlays and stitches these audio frames on the time axis to reconstruct a continuous, noise-free real-time audio stream.

[0031] It should be noted that the above description of hardware selection and preprocessing algorithms is only one preferred embodiment of the present invention; those skilled in the art should understand that the data acquisition source is not limited to smart docking stations, but can also be screen recording data or offline screen recording files from smartphones, AR glasses, cloud conferencing software, etc.; the timing alignment method is not limited to board-level hardware timing, but can also use audio-visual board synchronization, audio fingerprint matching, software post-synchronization based on key frames, etc.; the preprocessing methods can also be replaced by median filtering, Wiener filtering, wavelet transform, and end-to-end noise reduction networks based on deep learning; any technical solution that acquires multimodal conference data and performs basic cleaning and spatiotemporal alignment should be included within the protection scope of the present invention.

[0032] S2: Visual reference timestamp extraction.

[0033] Specifically, inter-frame difference operations are performed on the real-time video stream to obtain the set of differing pixels that have undergone pixel changes. Based on the geometric distribution characteristics of the differing pixel set on the image plane, a spatial dispersion index that characterizes the global nature of the image changes is calculated, thereby obtaining a visual reference timestamp.

[0034] Perform inter-frame difference calculation on the grayscale image sequence obtained from S1, and assume... For a moment Image frames in coordinates grayscale value at that location For a moment Image frames in coordinates The grayscale value at the specified location is calculated using the absolute difference method in this embodiment to determine the difference image. The relation is: ; To eliminate minor noise caused by digital signal transmission jitter or video compression encoding, this embodiment sets a pixel change threshold of 20. When determining coordinates The pixels that are different are identified by the difference; the entire difference image is traversed to extract the coordinates of all different pixels, thus constructing a set of different pixels. .

[0035] Since the pixels changed by mouse movement or pop-ups are usually clustered in space, while the pixels changed by slide flipping are usually distributed in various areas of the screen, this embodiment introduces a spatial dispersion index to distinguish between global slide flipping and local interference. For the set of differences ,in For a moment The number of differing pixels in an image frame. For the first Calculate the horizontal and vertical coordinates of each difference pixel on the image plane; Mean values ​​in the horizontal and vertical directions: , ,in, for The mean in the horizontal direction, for In the vertical direction, by calculating the mean, we can find the geometric center of the area where the pixel changes, that is, the centroid. If it is mouse movement, the centroid is in the middle of the mouse trajectory. If it is a pop-up window, the centroid is in the center of the pop-up window. If it is a slideshow, the centroid is usually near the center of the entire screen. Traditional methods only determine the number of pixels that changed while ignoring their distribution. Therefore, this paper calculates the variance of the distribution of differing pixels in the horizontal and vertical directions. , ,in, Let V be the variance of the distribution of differing pixels in the horizontal direction. The variance of the distribution of difference pixels in the vertical direction is used to measure the extent of the spread of the changed area. The larger the calculated variance, the farther the difference pixels are from the centroid. This corresponds to when the slide changes, the title in the upper left corner changes, the page number in the lower right corner changes, the body text in the middle changes, etc. The pixels corresponding to these changes are all far from the centroid.

[0036] According to the book "Digital Image Processing," the second-order central moment is used to describe the physical dispersion of an image region. The second-order central moment is physically equivalent to the moment of inertia in rigid body dynamics, reflecting the physical discreteness or aggregation of pixel distribution relative to its centroid. Based on this, to adapt to different conference terminals, such as landscape projectors, portrait mobile phone screen mirroring, and ultra-widescreen displays with varying display ratios, this embodiment employs an adaptive weighting strategy based on video frame resolution to construct a spatial dispersion index. The specific relationship is as follows:

[0037] in, For a moment Spatial dispersion index; , These are the horizontal and vertical pixel resolutions of the real-time video stream, respectively. , The adaptive direction weights are as follows: , This weight is related to the aspect ratio of the video frame at the current moment. For example, in a 16:9 screen, the weight in the horizontal direction is... The weight in the vertical direction is 0.64. The value is 0.36, which can mathematically enhance the algorithm's sensitivity to changes in the main visual axis. For pixel density factor, This is used to prevent misjudgments caused by the discrete distribution of a very small number of pixels.

[0038] When a speaker rapidly draws circles on the screen or moves the mouse drastically from left to right, the rapid movement creates a comet-tail effect, resulting in a large number of changing pixels. This can easily exceed the threshold set by traditional methods, leading to misinterpretations as page turning. However, this invention analyzes that while the mouse trajectory is long, it exhibits a linear distribution, meaning it is very narrow in the direction perpendicular to the trajectory. or There must be a very small one that is ultimately calculated. If the value is lowered, it is considered interference. When pop-ups appear in WeChat, system update notifications, or antivirus software floating windows, some of which are large or have animations and many pixel variations, this invention analyzes that a pop-up is a rectangular block, which is highly aggregated, and its variance is much smaller than that of a slideshow. The final calculated... It will also be very low; If pixelation or random errors occur during the decoding process of a digital video stream, discrete noise often appears across the entire screen with a large variance, but the total number of pixels is very small. Introducing a pixel density factor can reduce noise in such cases. The value should be set to avoid interference.

[0039] The result The comparison is performed in real time with the global change threshold. If... If the change is less than or equal to the global change threshold, the current image is considered to have local interference, and the change is ignored; if... If the change exceeds the global change threshold, it is determined that a global topic switch has occurred on the current screen, i.e., a slide flip. The time corresponding to this screen is marked as the visual baseline timestamp, which serves as the starting point for subsequent analysis.

[0040] It should be noted that the global change threshold was calibrated based on the classification optimization experiment using interfering samples. The specific steps are as follows: First, in a standard conference room environment, 50 hours of original conference video were recorded using the same acquisition equipment as the S1. After manual editing and annotation, a sample library containing 2000 independent events was constructed, including: 1000 positive samples: marked as valid slide transitions, including 500 sets of instantaneous transitions of plain text slides with large pixel changes and abrupt changes in texture structure; 300 sets of gradual transitions of slides with mixed text and images, including fade-in, fade-out, push-pull and other transition animations; and 200 sets of slide transitions with only partial text changes, such as displaying lists line by line. 1000 negative samples: marked as invalid interference, including 400 sets of rapid and large-scale mouse cursor movements with large pixel changes but low dispersion; 300 sets of desktop pop-up interference, including WeChat and DingTalk message notifications, and pop-ups in the lower right corner of Windows systems, characterized by rectangular block distribution; and 300 sets of embedded video playback or local animation, including local areas with continuously changing pixels, such as video playback in small windows inside slides.

[0041] Next, the global change threshold is preset to a range of [0,1]. A linear scan is performed with a step size of 0.001. For each scan, all samples are traversed, and the spatial dispersion index is calculated. If the spatial dispersion index is greater than the global change threshold of the current scan, it is marked as a predicted positive sample; otherwise, it is marked as a predicted negative sample. Then, it is compared with the manually labeled real labels to calculate the precision, i.e., how many of the predicted positive samples are valid page turns; the recall is calculated, i.e., how many of the valid page turns are predicted positive samples; based on the precision and recall, the F1 score is calculated.

[0042] A performance trend curve was plotted with the global change threshold on the x-axis and the F1 score on the y-axis. The curve shows that when the global change threshold is less than 0.2, although the recall is close to 100%, a large number of mouse movements are misjudged as slide turning, resulting in extremely low precision. When the global change threshold is greater than 0.6, the precision is close to 100%, but a large number of minor slide turning actions, such as displaying lists line by line, are missed, resulting in extremely low recall. When the global change threshold is 0.35, the performance trend curve reaches its peak, at which point the F1 score reaches its maximum value of 0.988.

[0043] Based on the above experimental results, this embodiment sets the global change threshold to 0.35, which represents the mathematically optimal solution for balancing anti-interference capability and detection sensitivity under the current hardware configuration and algorithm logic.

[0044] As can be seen, the core of this step lies in constructing a set of spatial morphology filters. By calculating the spatial dispersion index of the difference pixel set, the traditional pixel quantity detection is upgraded to geometric topology analysis, which gives the algorithm the ability to perceive spatial changes in the screen. It can effectively distinguish between global topic switching with high dispersion characteristics, such as slide turning, and local visual interference with high aggregation characteristics, such as fast mouse movement and pop-up prompts. Without increasing the computing power cost, it effectively solves the misjudgment caused by high frequency local interference and ensures the purity and accuracy of the visual reference timestamp.

[0045] It should be noted that the use of weighted variance to characterize spatial dispersion in this embodiment is only a preferred mathematical expression. Those skilled in the art should understand that any algorithm that uses the geometric distribution characteristics of differing pixels on the image plane to distinguish between global and local changes falls within the protection scope of this invention. For example, the calculation of spatial dispersion index can also be replaced by statistical indicators such as information entropy, convex hull area ratio, average nearest neighbor distance, and spatial autocorrelation coefficient. The judgment logic of the global change threshold can also be replaced by a dynamic adaptive threshold based on a sliding window or a lightweight support vector machine classifier. Any technical solution that segments the scene based on image changes rather than simply the number of pixels should be covered within the protection scope of this invention.

[0046] S3: Audio start-edge locking based on energy gradient.

[0047] This step aims to establish a causal relationship between visual perception and auditory expression, reconstructing the speaker's behavioral pattern of operating the presentation document → organizing their thought process → beginning the presentation, thereby pinpointing the semantic starting point of the new topic. Specifically, starting with the visual baseline timestamp, a causal search window extending backward is constructed in the real-time audio stream. Within the causal search window, the audio energy gradient is calculated, and it is detected whether there are audio abrupt changes exceeding a preset gradient threshold. If so, the time corresponding to the audio abrupt change is marked as the actual effective time.

[0048] Starting with a visual reference timestamp, a causal search window is constructed extending towards future time, with a length denoted as . This indicates the maximum waiting time from the change in visuals to the speaker beginning to explain the new visuals; based on behavioral psychology statistics, The baseline value is set to 3 seconds; To accommodate the speaking speed habits of different speakers, this embodiment adopts an adaptive adjustment strategy based on historical data; a circular buffer with a time span of 5 minutes is maintained to store historical speaking speed data, and the following segmented calculations are performed: Cold start period: When the meeting lasts less than 5 minutes, the cumulative average method is used to count the total number of recognized syllables from the start of the meeting to the current time and calculate the cumulative speech rate. For the initial time, the cumulative speech rate is set to the standard speech rate of 4 syllables / second. Stable period: When the meeting duration is greater than or equal to 5 minutes, switch to the sliding window averaging method for the specified time period. Only calculate The cumulative speaking rate over a time period ensures that the calculated cumulative speaking rate reflects the speaker's current state, such as fatigue causing a slower speaking rate.

[0049] As pointed out in *Psycholinguistics: Experiments in Spontaneous Speech*, the apparent speaking speed of humans during spontaneous speech is not determined by the speed of mouth muscle movement, but rather by the cognitive pause time generated by the brain during sentence planning in the lexical retrieval domain. That is, there is an inverse physical correlation between perceived speaking speed and physical pause time; as the physical pause time increases, the speaking speed decreases. Based on this, this embodiment uses cumulative speaking speed to calculate the length of the causal search window: ,in, For a moment The length of the causal search window; This serves as a baseline value for the length of the causal search window; Standard speaking speed; For a moment The cumulative speech rate.

[0050] Within the causal search window, the acquired real-time audio stream is divided into consecutive short frames, each with a frame length of 20ms and a frame shift step of 10ms. Assume there are several short frames within the causal search window, and each short frame contains... Each discrete amplitude data point represents a quantized value obtained after analog-to-digital conversion and encoding modulation of the audio signal. Each discrete amplitude data point represents the instantaneous amplitude intensity of the audio signal at that sampling moment. According to *Machine Learning for Audio, Image and Video Analysis*, sound intensity is proportional to the square of the pressure change. Furthermore, when processing speech data, calculating the short-time energy with a window function is the fundamental basis for distinguishing speech segments from silent segments and detecting abrupt changes. Based on this, for the... Energy value of a short frame The specific relationship is as follows:

[0051] in: For the first The first short frame Discrete amplitude data; For the Hamming window function at the th Weight coefficients at each position; To prevent logarithmic aberrations, this embodiment uses a very small positive number, 10. -8 .

[0052] For the Calculate the audio energy gradient in a short time frame. ,in, For the first The energy value of a short frame will Compare with the preset gradient threshold; if If it is greater than the gradient threshold, then determine the first... Each short frame is an audio abrupt change point.

[0053] It should be noted that, in order to ensure stable detection in different acoustic environments such as quiet small meeting rooms or noisy large lecture halls, the gradient threshold is adaptively calibrated based on the ambient noise floor characteristics, as detailed below: Extract the time-domain energy data corresponding to the background noise benchmark spectrum constructed in S1, calculate the standard deviation, and follow the 3σ principle in statistics to set the gradient threshold to three times the standard deviation to ensure the confidence of the detection results.

[0054] Extract the sequence number of the first audio mutation point within the causal search window. Since the starting point of the causal search window corresponds to the moment of visual change, and the time displacement of each short frame relative to the previous short frame is determined by the frame shift step size, the speaker's thinking time is equal to the sequence number multiplied by the frame shift step size, thus obtaining the relative time offset. Add the relative time offset to the visual reference timestamp to obtain the actual effective time.

[0055] If no audio abrupt change is detected after traversing all short frames within the causal search window, the event corresponding to the visual reference timestamp is determined to be a silent operation demonstration. In this case, no new actual effective time is generated, no semantic segmentation operation is performed, and the video frame at that moment is directly archived as an image attachment at the end of the previous meeting minutes.

[0056] S4: Time alignment and summary generation based on audiovisual causality.

[0057] Specifically, the real-time audio stream is divided into several independent semantic segments using the actual effective time, and combined with the video frame images corresponding to the visual reference timestamps to generate structured meeting minutes containing text-image correspondence.

[0058] Based on the actual effective time obtained from S3, the collected real-time audio stream is logically segmented to distinguish the boundary between the previous topic and the current topic. The specific segmentation logic is as follows: All audio data on the timeline between the actual effective time of the previous topic and the actual effective time of the current time are classified as the audio content of the previous topic. It should be noted that the time difference between the visual reference timestamp and the actual effective time usually includes the closing particle or a brief silence of the previous topic. In order to maintain semantic coherence, this embodiment archives it to the end of the previous topic. The actual effective time is marked as the semantic zero point of the new topic, and the real-time audio stream is input from that moment until the next actual effective time is detected.

[0059] After segmentation, the optical character recognition engine and automatic speech recognition engine are invoked to write content from the audiovisual data, as follows: Extract the video frame corresponding to the visual reference timestamp, i.e. the first frame after the slide flipping stabilizes. Perform optical character recognition on this frame to obtain all text information in the frame. Calculate the font height and screen position coordinates of each text block. Select the text line with the largest font and the highest position and mark it as the index title. Input the audio segment captured from the actual effective time into the automatic speech recognition engine and convert it into a text stream as the main text.

[0060] The time information, visual information, and text information obtained above are associated and mapped, encapsulated into standardized data objects, and stored. In this embodiment, JSON format is used for encapsulation. The generated structured data includes title sequence number, visual reference timestamp, actual effective time, video frame corresponding to the visual reference timestamp, minutes title, body text, and storage path.

[0061] Through the above steps, a smart meeting minutes with strict correspondence between images, text, and audio is finally generated. After clicking on the minutes title, users can directly jump to the audio at the moment when the speaker begins to explain the image, realizing a leap from physical time playback to semantic logic playback.

[0062] Figure 2 This is a schematic diagram of the timing alignment principle provided by an embodiment of the present invention. This diagram illustrates the timing alignment principle based on audiovisual causality. The upper waveform represents the drastic change in the video image, with the peak corresponding to a visual abrupt change point, such as the instant a PPT page is turned. The lower waveform represents the audio energy intensity, with the step rise position corresponding to the audio start edge, such as when a speaker begins to speak. The present invention utilizes the causal timing difference between the two, i.e., the pause in thinking, to construct a dynamic causal window, locking the first significant audio start edge after the visual abrupt change point as the final segmentation point of the system, ensuring that the title and content of the meeting minutes are accurately aligned in semantic timing.

[0063] It should be noted that the specific implementations of the JSON format, optical character recognition engine, and automatic speech recognition engine mentioned in this embodiment are only examples. Those skilled in the art should understand that the encapsulation format of structured data can also be XML, HTML, Markdown, or a proprietary binary protocol. Content recognition technology can run on local edge computing nodes or be transmitted to a cloud server for processing via a network. For the extraction logic of index titles, natural language processing algorithms can be combined to extract summaries of full-screen text, rather than relying on font size. Any technical solution that uses the actual effective time proposed in this invention to associate audiovisual content and generate documents should be included within the protection scope of this invention.

Claims

1. A method for intelligent generation of meeting minutes based on multimodal temporal features, characterized in that, include: Acquire and preprocess real-time video and audio streams from the meeting venue; The real-time video stream is subjected to inter-frame difference operation to extract the set of difference pixels. Based on the geometric distribution characteristics of the coordinates of the set of difference pixels on the image plane, a spatial dispersion index characterizing the degree of dispersion of image changes is calculated. The spatial dispersion index is compared with a preset global change threshold. If it is greater than the global change threshold, local interference is excluded, it is determined that a global topic switch has occurred at the current moment, and the current moment is marked as a visual reference timestamp. Starting from the visual reference timestamp, a causal search window extending backward is constructed in the real-time audio stream. The audio energy gradient is calculated within the causal search window, and it is detected whether there are audio abrupt changes exceeding a preset gradient threshold. If there are, the time corresponding to the audio abrupt change is marked as the actual effective time. The real-time audio stream is semantically segmented using the actual effective time, and the segmented audio content is associated with the video frame image corresponding to the visual reference timestamp to generate structured meeting minutes.

2. The intelligent meeting minutes generation method based on multimodal temporal features according to claim 1, characterized in that, The calculation of the spatial dispersion index, which characterizes the degree of dispersion of image changes, specifically includes: obtaining the horizontal and vertical resolutions of the real-time video stream; obtaining the x-coordinate and y-coordinate of each pixel in the set of differing pixels; calculating the variance of the x-coordinate and the variance of the y-coordinate respectively; normalizing and weighting the variances of the x-coordinate and the y-coordinate based on the horizontal and vertical resolutions; calculating the ratio of the total number of pixels in the set of differing pixels to the product of the horizontal and vertical resolutions to obtain the pixel density factor; and multiplying the result of the normalization and weighted summation by the pixel density factor to obtain the spatial dispersion index.

3. The intelligent meeting minutes generation method based on multimodal temporal features according to claim 1, characterized in that, The global change threshold is calibrated based on the classification optimization experiment of interference samples.

4. The intelligent generation method for meeting minutes based on multimodal temporal features according to claim 1, characterized in that, The causal search window is designed based on the cognitive delay characteristics between human visual perception and language expression.

5. The intelligent generation method for meeting minutes based on multimodal temporal features according to claim 1, characterized in that, The method further includes: dividing the audio data within the causal search window into consecutive short frames, calculating the energy value of each short frame, calculating the difference between the energy value of the current short frame and the energy value of the previous short frame to obtain the audio energy gradient, comparing the audio energy gradient with the gradient threshold, and determining that the current short frame is the audio mutation point if the audio energy gradient exceeds the gradient threshold.

6. The intelligent meeting minutes generation method based on multimodal temporal features according to claim 1, characterized in that, The method further includes: if no audio mutation point is detected in the causal search window, the event corresponding to the visual reference timestamp is determined to be a silent operation demonstration, and the semantic segmentation operation is not performed.

7. The intelligent generation method for meeting minutes based on multimodal temporal features according to claim 1, characterized in that, The preprocessing includes: denoising the real-time video stream and the real-time audio stream, and calibrating and aligning the timelines of the real-time video stream and the real-time audio stream.

8. The intelligent meeting minutes generation method based on multimodal temporal features according to claim 1, characterized in that, The generation of structured meeting minutes includes: performing optical character recognition on the video frame image corresponding to the visual reference timestamp to extract the title text as the index title; performing speech recognition on the data in the real-time audio stream that is after the actual effective time and converting it into body text; and encapsulating and storing the index title, the body text, and the video frame image according to their time relationship.

9. The intelligent generation method for meeting minutes based on multimodal temporal features according to claim 4, characterized in that, The length of the causal search window is set based on the cognitive delay characteristics and in combination with the speaker's cumulative speaking rate.

10. A meeting minutes intelligent generation system based on multimodal temporal features, characterized in that, include: The processor and memory, wherein the memory stores computer program instructions, which, when executed by the processor, implement the intelligent generation method for meeting minutes based on multimodal timing features according to any one of claims 1-9.

Images