A multi-modal data recording method and apparatus for a conferencing system

Abstract

The application provides a multi-modal data recording method and device of a conference system, and relates to the field of multi-modal data processing, and comprises the following steps: acquiring resource data, a video stream and an audio stream of a conference system, and determining a current processing mode according to attribute information of the resource data; in the case that the current processing mode is a real-time processing mode, constructing a face ID time sequence according to the video stream, constructing a text time sequence according to the audio stream, performing space-time alignment on the face ID time sequence and the text time sequence, and obtaining first recording data; in the case that the current processing mode is an offline processing mode, buffering the video stream and the audio stream; after detecting that a conference ends, constructing second recording data according to the buffered video stream and audio stream, wherein the construction process of the second recording data is the same as that of the first recording data; and splicing the first recording data and the second recording data to obtain final recording data.

Description

Technical Field

[0001] This invention relates to the field of multimodal data processing, and more particularly to a method and apparatus for recording multimodal data in a conference system. Background Technology

[0002] A web conferencing system is a software platform or service based on internet technology that allows multiple participants in different geographical locations to conduct real-time audio and video communication, data sharing, and collaborative work through terminal devices such as computers and mobile devices.

[0003] Existing web conferencing systems, limited by hardware and network speed, primarily focus on single-modal processing. To ensure data transmission quality, they only perform facial recognition or speech-to-text transcription, outputting only text or video recognition results. They lack timestamped audio-video synchronization and speaker ID association, failing to comprehensively record multimodal data from web conferences. Furthermore, existing web conferencing systems rely on high computing power and stable networks for data processing. When special circumstances occur, such as network fluctuations, data loss can occur, resulting in poor data quality stored by the web conferencing system. Summary of the Invention

[0004] To address the above problems, this invention provides a multimodal data recording method for a conference system, comprising:

[0005] Acquire resource data, video streams, and audio streams from the conference system, and determine the current processing mode based on the attribute information of the resource data;

[0006] In the current processing mode of real-time processing, a face ID time series is constructed based on the video stream, and a text time series is constructed based on the audio stream. The face ID time series and the text time series are spatiotemporally aligned to obtain the first record data.

[0007] When the current processing mode is offline processing mode, the video stream and audio stream are cached; after the meeting ends, the second record data is constructed based on the cached video stream and audio stream, and the construction process of the second record data is the same as that of the first record data.

[0008] The first and second record data are concatenated to obtain the final record data.

[0009] Optionally, determining the current processing mode based on the attribute information of the resource data includes:

[0010] The attribute information of the resource data includes device computing power data and network speed. The device computing power data and network speed are input into a preset decision maker built based on the XGBoost model for evaluation. If the device computing power data is greater than the minimum device computing power threshold and the network speed is greater than the minimum network speed threshold, the current processing mode is set to real-time processing mode; otherwise, the current processing mode is set to offline processing mode.

[0011] Optionally, the step of constructing a face ID time series based on the video stream and a text time series based on the audio stream includes:

[0012] The video stream is segmented into frames to obtain video images at each time step. The RetinaFace model generates prediction boxes based on the motion vector field at each time step, and the face region of the video image is cropped using the prediction boxes to obtain face images. The ArcFace model extracts features from the face images to obtain face feature vectors. The comprehensive similarity between face images at adjacent time steps is calculated based on the motion vector field, prediction boxes, and face feature vectors. Face IDs are labeled on the video images based on face feature vectors and comprehensive similarity. The labeled face IDs are stored in chronological order to obtain a face ID time series.

[0013] The audio stream is divided into multiple audio segments according to a preset time interval. The audio segments are converted into text using an ASR model. The text of each audio segment is stored in chronological order to obtain a text time series.

[0014] Optionally, the RetinaFace model generates prediction boxes based on the motion vector field at each time step, including:

[0015] The RetinaFace model calculates the motion vector field for the next moment based on the face image from the previous moment and the face image from the current moment. It then adjusts the predicted bounding box for the current moment based on the motion vector field for the next moment to generate the predicted bounding box for the next moment.

[0016] Optionally, the step of calculating the comprehensive similarity of face images between adjacent time points based on the motion vector field, prediction box, and face feature vector includes:

[0017] The cosine similarity between facial feature vectors at adjacent time points is used as the first matching factor, the overlap between predicted bounding boxes at adjacent time points is used as the second matching factor, and the vector distance between motion vector fields at adjacent time points is used as the third matching factor. The first matching factor, the second matching factor, and the third matching factor are weighted and summed to obtain the comprehensive similarity of facial images at adjacent time points.

[0018] Optionally, the step of labeling face IDs on video images based on face feature vectors and comprehensive similarity includes:

[0019] All facial images contained in the video image at the current moment are obtained based on the facial feature vector, and the corresponding facial IDs are labeled for different facial images. If the comprehensive similarity between facial images at adjacent moments is greater than a preset value, the facial image inherits the facial ID of the current moment in the next moment; otherwise, the facial ID is re-labeled for the facial image based on the facial feature vector.

[0020] Optionally, the step of spatiotemporally aligning the face ID time series and the text time series to obtain the first record data includes:

[0021] S11: Obtain the i-th text in the text time series. The time range of the i-th text is [t1, t2]. Extend the time range [t1, t2] forward and backward by a preset time δ to obtain the extended time range [t1-δ, t2+δ].

[0022] S12: In the face ID time series, obtain the face IDs at each time point within the extended time range, and take the face ID that appears most frequently as the speaker; take the speaker, the time range [t1,t2], and the i-th text as the structured data corresponding to the i-th text;

[0023] S13: Repeat steps S11-S12 to obtain the structured data corresponding to all texts, store each structured data in chronological order, and obtain the first record data.

[0024] The present invention also provides a multimodal data recording device for a conference system, used to implement the multimodal data recording method of the conference system, the device comprising:

[0025] The processing mode switching module is used to acquire resource data, video streams, and audio streams from the conference system, and determine the current processing mode based on the attribute information of the resource data.

[0026] The first record data acquisition module is used to construct a face ID time series based on the video stream and a text time series based on the audio stream when the current processing mode is real-time processing mode, and to align the face ID time series and the text time series in time and space to obtain the first record data.

[0027] The second recording data acquisition module is used to cache the video stream and audio stream when the current processing mode is offline processing mode; after the meeting ends, it constructs the second recording data based on the cached video stream and audio stream, and the construction process of the second recording data is the same as that of the first recording data.

[0028] The final record data acquisition module is used to concatenate the first record data and the second record data to obtain the final record data.

[0029] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal data recording method of the conference system.

[0030] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multimodal data recording method of the conference system.

[0031] The present invention has the following beneficial effects:

[0032] 1. Based on the attribute information of the resource data, the current processing mode can be flexibly switched between real-time processing mode and offline processing mode. When the device's computing power and network speed meet the requirements, the video and audio streams are processed in real time, ensuring the real-time data processing of the conference system when the device's computing power and network speed are good. When the device's computing power and network speed do not meet the requirements, the video and audio streams are cached first, and then processed after the meeting ends. This avoids potential data loss when the device's computing power and network speed are poor, ensuring the integrity of the data processed by the conference system and taking into account both the needs of real-time annotation during the meeting and batch processing after the meeting.

[0033] 2. The RetinaFace model is used to calculate the motion vector field of the next moment based on the face image of the previous moment and the face image of the current moment. The prediction box of the current moment is adjusted according to the motion vector field of the next moment to generate the prediction box of the next moment. This makes the position of the prediction box of the next frame more consistent with the actual motion trajectory of the object on the image plane, which greatly reduces the jitter of the detection box in the time series and makes the face recognition of subsequent processing smoother and more stable.

[0034] 3. Mark face IDs on video images based on face feature vectors and comprehensive similarity. If the comprehensive similarity between adjacent time points of a face image is greater than a preset value, the face image will inherit the face ID of the current time point in the next time point. Even if the face fluctuates briefly due to changes in angle or other reasons, the face ID can still be tracked and marked based on comprehensive similarity, reducing tracking interruptions and face ID jumps.

[0035] 4. By spatiotemporally aligning the face ID time series and text time series to obtain recorded data, and by extending the time range of the text before and after, mismatch problems caused by video frame acquisition delays or text transcription time deviations are effectively avoided; the face ID that appears most frequently within the extended time range is identified as the speaker, which can filter out momentary interfering faces and ensure that the truly continuous speaker is identified; by treating the speaker, time range, and text as structured data, each text segment forms a three-dimensional association with its corresponding speaker and speaking time, transforming the original separate video stream and transcribed text into a traceable spatiotemporal event chain; the structured data stored in chronological order supports fast time interval queries and conditional filtering, improving the efficiency of finding meeting content. Attached Figure Description

[0036] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0037] Figure 2 This is a structural diagram of the device according to an embodiment of the present invention;

[0038] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0039] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0040] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0041] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0042] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0043] Reference Figure 1 This invention provides a multimodal data recording method for a conference system, comprising:

[0044] Acquire resource data, video streams, and audio streams from the conference system, and determine the current processing mode based on the attribute information of the resource data;

[0045] In some embodiments, a multi-channel synchronous camera (PTP or hardware-triggered synchronization) is used to acquire the original video stream. Bilateral filtering or median filtering is applied to each frame of the original video stream to suppress noise. Histogram equalization or adaptive histogram equalization is used to improve the visual quality of each frame. The processed video stream is then used for subsequent processing steps.

[0046] The raw audio stream is acquired synchronously using a multi-channel array microphone. A bandpass filter (300–3400 Hz) is used to remove low-frequency wind noise and high-frequency noise from the raw audio stream. Speech activity is detected based on energy thresholds or short-time energy statistics, and the effective audio stream is extracted for subsequent processing steps.

[0047] In some embodiments, determining the current processing mode based on the attribute information of the resource data includes:

[0048] The attribute information of the resource data includes device computing power data and network speed. The device computing power data and network speed are input into a preset decision maker built based on the XGBoost model for evaluation. If the device computing power data is greater than the minimum device computing power threshold and the network speed is greater than the minimum network speed threshold, the current processing mode is set to real-time processing mode; otherwise, the current processing mode is set to offline processing mode.

[0049] In some embodiments, the system calls the operating system interface to collect device computing power data and network speed. Device computing power data includes resource indicators such as CPU parameters, GPU parameters, and memory usage. The preset decision-maker is built based on the XGBoost model. XGBoost (eXtreme Gradient Boosting) is a high-performance gradient boosting decision tree (GBDT) algorithm containing multiple decision trees. The overall decision accuracy of the preset decision-maker is improved through iterative training of each decision tree. The prediction results of all decision trees are weighted and summed to obtain the final prediction result. The system learns the correspondence between "computing power-network speed" and processing mode from historical data, automatically assessing the minimum device computing power threshold and minimum network speed threshold. If the current processing mode is real-time processing mode, it indicates that the system's computing power and network speed meet the requirements, and real-time processing of video and audio streams is possible, with the video and audio streams being constructed as recorded data. If the current processing mode is offline processing mode, it indicates that the system configuration is poor or the network speed is too slow, and the system cannot process video and audio streams in real time. The video and audio streams need to be cached first and processed after the meeting ends.

[0050] In the current processing mode of real-time processing, a face ID time series is constructed based on the video stream, and a text time series is constructed based on the audio stream. The face ID time series and the text time series are spatiotemporally aligned to obtain the first record data.

[0051] In some embodiments, constructing a face ID time series based on a video stream and a text time series based on an audio stream includes:

[0052] The video stream is segmented into frames to obtain video images at each time step. The RetinaFace model generates prediction boxes based on the motion vector field at each time step, and the face region of the video image is cropped using the prediction boxes to obtain face images. The ArcFace model extracts features from the face images to obtain face feature vectors. The comprehensive similarity between face images at adjacent time steps is calculated based on the motion vector field, prediction boxes, and face feature vectors. Face IDs are labeled on the video images based on face feature vectors and comprehensive similarity. The labeled face IDs are stored in chronological order to obtain a face ID time series.

[0053] In some embodiments, the RetinaFace model is a deep learning-based face detection model that combines a Feature Pyramid Network (FPN) and multi-task learning techniques to simultaneously detect face location, key points (such as eyes, nose, and mouth), and attribute information (such as gender and age). The RetinaFace model's backbone network uses a ResNet network for image feature extraction. The Feature Pyramid Network generates multi-level feature maps, accommodating both large-scale and small-scale face detection. The detection head includes a classification and regression head, a key point prediction head, etc., and its performance is optimized through multi-task training. The ArcFace model is an improved deep learning face recognition model. Its core is the introduction of an Angular Margin loss function to enhance feature discriminability, thereby improving recognition accuracy and robustness. The ArcFace model is based on a convolutional neural network, containing structures such as convolutional layers, pooling layers, and residual blocks to extract face feature vectors and utilizes an Additive Angular Margin Loss function to optimize model parameters. This loss function improves classification performance by introducing angular intervals into the feature space, maximizing the angular similarity of feature vectors within the same class, while increasing the distance between different classes.

[0054] In some embodiments, the RetinaFace model generates prediction boxes based on the motion vector field at each time step, including:

[0055] The RetinaFace model calculates the motion vector field for the next moment based on the face image from the previous moment and the face image from the current moment. It then adjusts the predicted bounding box for the current moment based on the motion vector field for the next moment to generate the predicted bounding box for the next moment.

[0056] In some embodiments, the face image at time t is obtained. Face images at time t+1 The Farneback dense optical flow algorithm is used to calculate the motion vector field between time t and time t+1. According to the motion vector field Calculate the prediction box at time t median displacement of internal sampling points According to the median displacement and prediction boxes Calculate and obtain the initial prediction box at time t+1. , the initial prediction box The 120% range is used as the prediction box at time t+1. The essence of motion vector fields is to predict the dense displacement of pixels between frames. This displacement information is used to adjust the bounding boxes in the current frame, making the predicted bounding box position in the next frame more consistent with the actual motion trajectory of the object on the image plane. This allows the model to better handle rapid motion and pose changes, significantly reducing the false negative rate and improving the accuracy of bounding box localization, especially in motion-blurred scenes. Furthermore, by utilizing image information from two consecutive frames to infer the motion of the next frame, the model can perceive the movement trend and continuity of the face, making predictions that are more consistent with the laws of physical motion.

[0057] In some embodiments, calculating the comprehensive similarity of face images between adjacent time points based on the motion vector field, prediction box, and face feature vector includes:

[0058] The cosine similarity between facial feature vectors at adjacent time points is used as the first matching factor, the overlap between predicted bounding boxes at adjacent time points is used as the second matching factor, and the vector distance between motion vector fields at adjacent time points is used as the third matching factor. The first matching factor, the second matching factor, and the third matching factor are weighted and summed to obtain the comprehensive similarity of facial images at adjacent time points.

[0059] In some embodiments, the motion vector field of face image i at time t is: The prediction box is The facial feature vector is The motion vector field of face image j at time t+1 is: The prediction box is The facial feature vector is .

[0060] Facial feature vector With the facial feature vector The first matching factor between The calculation formula is:

[0061]

[0062] Where || is the norm symbol, used to measure the length of a vector in a vector space;

[0063] Predicted box With prediction box Overlapping area and prediction frame The proportion is used as the degree of overlap;

[0064] Based on the motion vector field Calculate and obtain the prediction box median displacement of internal sampling points According to the motion vector field Calculate and obtain the prediction box median displacement of internal sampling points According to the median displacement and median displacement The vector distance d between the motion vector fields is calculated using the following formula:

[0065]

[0066] The weights of the first matching factor, the second matching factor, and the third matching factor are set to 0.6, 0.3, and 0.1, respectively.

[0067] Both the bounding box overlap and the vector distance of the motion vector field utilize the strong prior knowledge of the spatiotemporal continuity of the target's position and trajectory between adjacent frames. Even if facial features fluctuate briefly due to changes in angle, the continuity of position and motion helps maintain correct association, reducing tracking interruptions and identity jumps.

[0068] Combining the first, second, and third matching factors allows them to compensate for each other's shortcomings. When one factor performs poorly, the others can still provide reliable matching clues. In complex scenarios involving similar-looking faces, rapid movement, camera movement, and brief occlusion, the overall similarity score is more resistant to interference and maintains stable tracking correlation.

[0069] If face image i and face image j do not belong to the same person, the value of the first matching factor will be extremely small, and the overall similarity will certainly not exceed the preset value. For example, when a new face suddenly appears on the motion path of a face in the previous frame, relying solely on motion vectors or position may result in a mismatch. However, the feature vectors of the old and new faces are usually very different, and the weighted overall similarity will score this match low to prevent mismatches. Only when the facial features are similar, the positional overlap is high, and the motion trend is consistent will the overall similarity be high. This is equivalent to a multi-dimensional cross-validation mechanism, which greatly reduces the probability of mismatches. Therefore, the overall similarity can accurately determine whether two face images correspond to the same person.

[0070] In some embodiments, the step of annotating the face ID on the video image based on the face feature vector and comprehensive similarity includes:

[0071] All facial images contained in the video image at the current moment are obtained based on the facial feature vector, and the corresponding facial IDs are labeled for different facial images. If the comprehensive similarity between facial images at adjacent moments is greater than a preset value, the facial image inherits the facial ID of the current moment in the next moment; otherwise, the facial ID is re-labeled for the facial image based on the facial feature vector.

[0072] In some embodiments, face images are labeled with face IDs according to trajectory management. In the field of video analysis, "trajectory" specifically refers to the spatiotemporal motion path and state evolution of the same target object in consecutive video frames. First, the confidence level of each face ID is obtained based on the face feature vector. The face ID with the highest confidence level is used as the unique identifier of the face image. When the comprehensive similarity between adjacent moments of the face image is greater than a preset value, it indicates that the face ID matches the face image correctly, and the face in the new frame directly "inherits" the ID of the corresponding face in the previous frame. Even if the face features of the current frame are not very similar to the best match in the database due to occlusion or pose changes, as long as its comprehensive similarity with the tracked target in the previous frame is high enough, the correct ID labeling can be maintained. This ensures that when the same individual appears consecutively in a video sequence, its labeled ID remains unchanged (e.g., ID_001 remains ID_001 throughout the tracking process). This effectively solves the problem of frequent and meaningless ID jumps between adjacent frames that may occur when identification is based solely on a single frame (e.g., the same face is ID_001 in frame 1, becomes ID_007 in frame 2, and then reverts to ID_001 in frame 3). Furthermore, as long as the overall similarity is greater than a preset value, there is no need to repeatedly calculate the confidence level of the face ID; only the face image needs to inherit the face ID, greatly reducing the computational load of labeling face IDs on face images and saving computational resources.

[0073] The audio stream is divided into multiple audio segments according to a preset time interval. The audio segments are converted into text using an ASR model. The text of each audio segment is stored in chronological order to obtain a text time series.

[0074] In some embodiments, an Automatic Speech Recognition (ASR) model is a model that converts human speech into text. An ASR model transforms audio segments into readable and editable text. Its two core components are an acoustic model and a language model. The acoustic model is responsible for mapping speech signals to phonemes or words in the ASR model. It uses a deep neural network algorithm at its core, learning and training on various acoustic phenomena, and then selecting the learning unit with the highest probability corresponding to that phenomenon as its output. The language model is responsible for predicting the probability of word sequences in the ASR model. It is based on linguistic knowledge and describes the statistical characteristics of word sequences. The main task of the language model is to evaluate the plausibility of a given word sequence and provide the ASR system with the prior probability of the word sequence. The audio segments, after being segmented from the audio stream, are fed into the ASR model, transcribed into text in real time, and the start and end timestamps are retained.

[0075] When the current processing mode is offline processing mode, the video stream and audio stream are cached; after the meeting ends, the second record data is constructed based on the cached video stream and audio stream, and the construction process of the second record data is the same as that of the first record data.

[0076] In some embodiments, the step of spatiotemporally aligning the face ID time series and the text time series to obtain the first record data includes:

[0077] S11: Obtain the i-th text in the text time series. The time range of the i-th text is [t1, t2]. Extend the time range [t1, t2] forward and backward by a preset time δ to obtain the extended time range [t1-δ, t2+δ].

[0078] In some embodiments, by extending the time window before and after the video frame capture, mismatch problems caused by video frame capture delays or text transcription time deviations are effectively avoided. For example, when the speaker pauses briefly or the camera briefly drops frames, the extended time range can still capture the correct face ID.

[0079] S12: In the face ID time series, obtain the face IDs at each time point within the extended time range, and take the face ID that appears most frequently as the speaker; take the speaker, the time range [t1,t2], and the i-th text as the structured data corresponding to the i-th text;

[0080] In some embodiments, the text in the text time series is defined as a TextEvent, and the face ID in the face ID time series is defined as a FaceEvent. The disappearance time Fe.t_end and the appearance time Fe.t_start of each FaceEvent in the face ID time series are obtained. All FaceEvents that satisfy Fe.t_end > T1 and Fe.t_start < T2 are selected from the face ID time series to form a candidate set C. If C is empty, the speaker_id of the TextEvent is temporarily set to "unknown". The appearance duration D of each FaceEvent in the candidate set C is calculated, D = min(Fe.t_end, T2) – max(Fe.t_start, T1). The number of appearances of each FaceEvent is obtained based on the appearance duration D. The FaceEvent with the most appearances is used as the speaker. If there are multiple FaceEvents with equal appearance durations, the FaceEvent with the highest confidence corresponding to the face feature vector of the FaceEvent is used as the speaker. If there are no candidate faces in N (such as 3) consecutive texts, it is marked as "speaker missing", and the user is prompted to supplement. If the appearance duration of all face IDs is less than D_min (such as 50 ms), it is marked as "speaker uncertain", and "speaker + time range + text" is used as the structured data.

[0081] S13: Repeat steps S11 - S12 to obtain the structured data corresponding to all texts, and store the structured data in chronological order to obtain the first recorded data.

[0082] The first recorded data and the second recorded data are spliced to obtain the final recorded data.

[0083] In some embodiments, the structured data in the first recorded data and the second recorded data are sorted by time, the overlapping parts at both ends of the time range of the structured data are deleted, the texts that repeatedly appear in the structured data are merged, and if other face IDs with a number of appearances less than a preset number appear between consecutive appearances of the same face ID, the other face IDs are deleted to maintain the consistency of the speaker. Finally, the structured data after removing duplicates and fragmentation are spliced by time into the final recorded data, and the template engine (such as Pandoc) is called to fill "speaker + time range + text" in the final recorded data into DOCX, PDF, and TXT templates for storage according to the time axis.

[0084] Refer to Figure 2 , the present invention also provides a multimodal data recording device 20 for a conference system to implement the multimodal data recording method of the conference system. The device includes:

[0085] The processing mode switching module 21 is used to acquire resource data, video stream and audio stream of the conference system, and determine the current processing mode based on the attribute information of the resource data.

[0086] The first record data acquisition module 22 is used to construct a face ID time series based on the video stream and a text time series based on the audio stream when the current processing mode is real-time processing mode, and to align the face ID time series and the text time series in time and space to obtain the first record data.

[0087] The second recording data acquisition module 23 is used to cache the video stream and audio stream when the current processing mode is offline processing mode; after the meeting ends, it constructs the second recording data based on the cached video stream and audio stream, and the construction process of the second recording data is the same as that of the first recording data.

[0088] The final record data acquisition module 24 is used to concatenate the first record data and the second record data to obtain the final record data.

[0089] This application provides an electronic device, including a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements a multimodal data recording method for a conferencing system according to any of the above-described schemes.

[0090] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0091] Memory can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and also random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0092] This application also provides a computer-readable medium storing a computer program thereon, which, when executed by a processor, implements the multimodal data recording method of the conferencing system according to any of the above-described schemes. The computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the method as described in the embodiments of this application.

[0093] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0094] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments, but should be defined not only by the appended claims, but also by their equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method of multi-modal data recording of a meeting system, characterized in that, The method comprises the following steps: acquiring resource data, a video stream and an audio stream of a conference system, and determining a current processing mode according to attribute information of the resource data; in the case where the current processing mode is a real-time processing mode, constructing a face ID time sequence according to the video stream, constructing a text time sequence according to the audio stream, and performing spatio-temporal alignment on the face ID time sequence and the text time sequence to obtain first record data; in the case where the current processing mode is an offline processing mode, buffering the video stream and the audio stream; after detecting the end of the conference, constructing second record data according to the buffered video stream and audio stream, the construction process of the second record data being the same as that of the first record data; splicing the first record data and the second record data to obtain final record data; the method further comprises the following steps: the attribute information of the resource data comprises device computing power data and network speed, the device computing power data and the network speed are input into a preset decision maker based on an XGBoost model for evaluation, if the device computing power data is greater than a minimum device computing power threshold and the network speed is greater than a minimum network speed threshold, the current processing mode is set to the real-time processing mode, otherwise, the current processing mode is set to the offline processing mode; the method further comprises the following steps: frame operation is performed on the video stream to obtain video images at each time; a RetinaFace model generates a prediction box according to a motion vector field at each time, the face region of the video image is cropped through the prediction box to obtain a face image; a feature vector of the face image is obtained through feature extraction of the face image by an ArcFace model; the comprehensive similarity between the face images at adjacent times is calculated according to the motion vector field, the prediction box and the feature vector of the face image, the face ID is labeled on the video image according to the feature vector of the face image and the comprehensive similarity, the labeled face ID is stored in time sequence to obtain the face ID time sequence; the audio stream is divided into a plurality of audio segments according to a preset time interval, the audio segments are converted into texts through an ASR model, and the texts of the audio segments are stored in time sequence to obtain the text time sequence; the method further comprises the following steps: S11: obtaining an ith text in the text time sequence, the time range of the ith text being [t1, t2], the time range [t1, t2] is extended by a preset time δ forward and backward to obtain an extended time range [t1-δ, t2+δ]; S12: in the face ID time sequence, obtaining the face ID at each time within the extended time range, taking the face ID with the maximum continuous appearance frequency as a speaker; taking the speaker, the time range [t1, t2] and the ith text as the structured data corresponding to the ith text; S13: repeating steps S11-S12 to obtain the structured data corresponding to all texts, and storing the structured data in time sequence to obtain the first record data.

2. The multi-modal data recording method of a conference system according to claim 1, wherein, The RetinaFace model generates prediction boxes based on the motion vector field at each time step, including: The RetinaFace model calculates the motion vector field for the next moment based on the face image from the previous moment and the face image from the current moment. It then adjusts the predicted bounding box for the current moment based on the motion vector field for the next moment to generate the predicted bounding box for the next moment.

3. The method of claim 1, wherein, The step of calculating the comprehensive similarity of face images between adjacent time points based on the motion vector field, prediction box, and face feature vector includes: The cosine similarity between facial feature vectors at adjacent time points is used as the first matching factor, the overlap between predicted bounding boxes at adjacent time points is used as the second matching factor, and the vector distance between motion vector fields at adjacent time points is used as the third matching factor. The first matching factor, the second matching factor, and the third matching factor are weighted and summed to obtain the comprehensive similarity of facial images at adjacent time points.

4. The method of claim 1, wherein, The step of labeling face IDs on video images based on face feature vectors and comprehensive similarity includes: All facial images contained in the video image at the current moment are obtained based on the facial feature vector, and the corresponding facial IDs are labeled for different facial images. If the comprehensive similarity between facial images at adjacent moments is greater than a preset value, the facial image inherits the facial ID of the current moment in the next moment; otherwise, the facial ID is re-labeled for the facial image based on the facial feature vector.

5. A multi-modal data recording apparatus of a conference system for implementing the multi-modal data recording method of the conference system according to any one of claims 1 to 4, characterized in that, The device includes: The processing mode switching module is used to acquire resource data, video streams, and audio streams from the conference system, and determine the current processing mode based on the attribute information of the resource data. The first record data acquisition module is used to construct a face ID time series based on the video stream and a text time series based on the audio stream when the current processing mode is real-time processing mode, and to align the face ID time series and the text time series in time and space to obtain the first record data. The second recording data acquisition module is used to cache the video stream and audio stream when the current processing mode is offline processing mode; after the meeting ends, it constructs the second recording data based on the cached video stream and audio stream, and the construction process of the second recording data is the same as that of the first recording data. The final record data acquisition module is used to concatenate the first record data and the second record data to obtain the final record data.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multimodal data recording method of the conference system as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal data recording method of the conference system as described in any one of claims 1 to 4.

Images