Method, device, equipment, storage medium and program product for generating conference record

Abstract

The present disclosure provides a conference record generation method, device, equipment, storage medium and program product, comprising: acquiring audio and video of a conference; extracting a voiceprint feature and a face feature of the audio and video at a current time, and fusing to obtain a speaker fusion feature at the current time; determining whether a sample speaker fusion feature matched with the speaker fusion feature at the current time exists in a face voiceprint fusion feature library constructed in real time based on the conference; if not, determining a speaker at the current time and a face image corresponding to the speaker fusion feature, and storing them in the face voiceprint fusion feature library; if so, determining a speaker at the current time and a face image corresponding to the audio and video at the current time based on the sample speaker fusion feature; converting audio in the audio and video at the current time into text, and generating a conference record of the speaker at the current time based on the text, the speaker at the current time and the face image. The present disclosure constructs a feature library in real time, without the need to construct a personnel feature library in advance, thereby improving the conference record generation efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of meeting minutes technology, and in particular to a method, apparatus, device, storage medium, and program product for generating meeting minutes. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] The automated generation of meeting minutes refers to the process of automatically converting the audio content of a meeting into text using technologies such as speech recognition and natural language processing, and then further analyzing, summarizing, and organizing the text content to ultimately generate a structured meeting record.

[0004] However, in the meeting record generation solutions provided by related technologies, a feature database of participants needs to be built in advance before the meeting records are generated. In this solution, when new participants join the meeting, the feature database needs to be rebuilt. Summary of the Invention

[0005] In view of this, the purpose of this disclosure is to provide a method, apparatus, device, storage medium and program product for generating meeting minutes, which at least partially solves one of the technical problems in the related art.

[0006] To achieve the above objectives, a first aspect of the exemplary embodiments of this disclosure provides a method for generating meeting minutes, comprising:

[0007] Acquire the audio and video of the meeting, wherein the audio and video include audio and video;

[0008] Extract the voiceprint and facial features of the audio and video at the current moment, and fuse the voiceprint and facial features to obtain the speaker's fused features at the current moment;

[0009] Determine whether there exists a sample speaker fusion feature in the face and voiceprint fusion feature library constructed in real time based on the meeting that matches the speaker fusion feature at the current moment;

[0010] In response to determining that there is no sample speaker fusion feature in the face-voiceprint fusion feature library that matches the speaker fusion feature at the current moment, the speaker at the current moment and the face image corresponding to the speaker fusion feature are determined, and the speaker fusion feature, the speaker at the current moment and the face image are stored in the face-voiceprint fusion feature library accordingly.

[0011] In response to determining that there exists a sample speaker fusion feature in the face and voiceprint fusion feature library that matches the speaker fusion feature at the current moment, the speaker at the current moment and the face image corresponding to the audio and video at the current moment are determined based on the sample speaker fusion feature;

[0012] The audio in the current moment's audio and video is converted into text. Based on the text, the speaker at the current moment, and the facial image, the meeting minutes of the speaker at the current moment are generated.

[0013] In some exemplary embodiments, the extraction of voiceprint features and facial features of the audio and video at the current moment includes:

[0014] The audio and video are segmented to obtain at least two audio and video segments;

[0015] Each of the aforementioned audio and video segments is processed as the audio and video at the current moment;

[0016] Then, the step of converting the audio in the current moment's audio and video into text, and generating the meeting minutes of the current moment's speaker based on the text, the current moment's speaker, and the facial image, includes:

[0017] Determine whether the current audio / video segment is the first of the at least two audio / video segments;

[0018] In response to determining that the current audio / video segment is not the first audio / video segment among the at least two audio / video segments, it is determined whether the speaker fusion feature at the current moment matches the speaker fusion feature of the previous audio / video segment;

[0019] In response to determining that the speaker fusion feature at the current moment matches the speaker fusion feature of the previous audio-visual segment, the current audio-visual segment is aggregated with the previous audio-visual segment to obtain an aggregated audio-visual segment;

[0020] The audio in the aggregated audio and video segments is converted into text, and based on the text, the speaker at the current moment, and the facial image, a meeting record of the speaker at the current moment is generated.

[0021] In some exemplary embodiments, extracting the voiceprint features of the audio and video at the current moment includes:

[0022] The audio in the audio and video at the current moment is preprocessed to obtain preprocessed audio;

[0023] Determine the spectral parameters of the preprocessed audio;

[0024] The spectral parameters are filtered to obtain the output;

[0025] Perform a logarithmic transformation on the output to obtain the logarithmic energy;

[0026] The logarithmic energy is subjected to discrete cosine transform to obtain cepstral coefficients and the first-order difference coefficients of the cepstral coefficients. The cepstral coefficients and the first-order difference coefficients are used as the voiceprint features.

[0027] In some exemplary embodiments, the step of preprocessing the audio in the audio and video at the current moment to obtain preprocessed audio includes:

[0028] The audio is pre-emphasized to improve the signal-to-noise ratio of the high-frequency components.

[0029] The audio is divided into multiple audio frames by frame segmentation and windowing.

[0030] Endpoint detection is performed on the audio to remove non-speech signals from the audio.

[0031] In some exemplary embodiments, the method for determining the face image includes:

[0032] Obtain a sequence of facial images from the video in the audio and video at the current moment;

[0033] Select candidate face images from the face image sequence whose face size is larger than a preset size;

[0034] The face image is determined from the candidate face images based on preset rules.

[0035] In some exemplary embodiments, determining the face image from the candidate face images based on preset rules includes:

[0036] The face image is determined from the candidate face images based on the face blur evaluation rules, face pose evaluation rules, face image quality evaluation model, and smile expression matching degree.

[0037] Based on the same inventive concept, a second aspect of the exemplary embodiments of this disclosure provides an apparatus for generating meeting minutes, comprising:

[0038] The audio and video acquisition module is configured to acquire the audio and video of the meeting, wherein the audio and video include audio and video.

[0039] The speaker fusion feature extraction module is configured to extract the voiceprint features and facial features of the audio and video at the current moment, and fuse the voiceprint features and facial features to obtain the speaker fusion features at the current moment;

[0040] The speaker fusion feature matching module is configured to determine whether there is a sample speaker fusion feature in the face and voiceprint fusion feature library built in real time based on the meeting that matches the speaker fusion feature at the current moment;

[0041] The speaker and face image determination module is configured to, in response to determining that there is no sample speaker fusion feature in the face and voiceprint fusion feature library that matches the speaker fusion feature at the current moment, determine the speaker and face image at the current moment corresponding to the speaker fusion feature, and store the speaker fusion feature, the speaker at the current moment, and the face image into the face and voiceprint fusion feature library accordingly; and in response to determining that there is a sample speaker fusion feature in the face and voiceprint fusion feature library that matches the speaker fusion feature at the current moment, determine the speaker and face image at the current moment corresponding to the audio and video at the current moment based on the sample speaker fusion feature.

[0042] The meeting record generation module is configured to convert the audio in the current moment's audio and video into text, and generate the meeting record of the speaker at the current moment based on the text, the speaker at the current moment, and the facial image.

[0043] Based on the same inventive concept, a third aspect of the exemplary embodiments of this disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in the first aspect.

[0044] Based on the same inventive concept, a fourth aspect of the exemplary embodiments of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method as described in the first aspect.

[0045] Based on the same inventive concept, a fifth aspect of the exemplary embodiments of this disclosure provides a computer program product including computer program instructions that, when run on a computer, cause the computer to perform the method as described in the first aspect.

[0046] As can be seen from the above description, the meeting record generation method, apparatus, device, storage medium, and program product provided in this disclosure include: acquiring audio and video of a meeting, wherein the audio and video include audio and video; extracting voiceprint features and facial features from the audio and video at the current moment, fusing the voiceprint features and facial features to obtain a speaker fusion feature at the current moment; determining whether there is a sample speaker fusion feature matching the speaker fusion feature at the current moment in a facial voiceprint fusion feature library constructed in real time based on the meeting; and responding to determining that there is no sample speaker fusion feature matching the speaker fusion feature at the current moment in the facial voiceprint fusion feature library. The present invention constructs a face-voiceprint fusion feature library by combining the speaker's fusion features to determine the speaker and face image at the current moment corresponding to the speaker's fusion features. The speaker's fusion features, the speaker at the current moment, and the face image are then stored in the face-voiceprint fusion feature library. In response to the determination that a sample speaker fusion feature exists in the face-voiceprint fusion feature library that matches the speaker's fusion features at the current moment, the speaker and face image at the current moment are determined based on the sample speaker fusion features. The audio in the current moment audio-visual recording is converted into text, and the meeting minutes of the speaker at the current moment are generated based on the text, the speaker at the current moment, and the face image. This invention constructs a face-voiceprint fusion feature library in real time, without the need to pre-construct a feature library for attendees. When new attendees join the meeting, there is no need to rebuild the feature library, thus improving the efficiency of meeting minutes generation. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this disclosure or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A schematic diagram illustrating an application scenario of the meeting record generation method provided as an exemplary embodiment of this disclosure;

[0049] Figure 2 A flowchart illustrating a method for generating meeting minutes provided as an exemplary embodiment of this disclosure;

[0050] Figure 3 A schematic diagram of an audio feature extraction process provided for an exemplary embodiment of this disclosure;

[0051] Figure 4 A schematic diagram illustrating an audio feature extraction and facial feature extraction process provided for exemplary embodiments of this disclosure;

[0052] Figure 5Another flowchart illustrating a method for generating meeting minutes provided as an exemplary embodiment of this disclosure;

[0053] Figure 6 A schematic diagram of a face and voiceprint fusion feature library and a face queue provided as an exemplary embodiment of this disclosure;

[0054] Figure 7 A schematic diagram illustrating an aggregation method provided for an exemplary embodiment of this disclosure;

[0055] Figure 8 A schematic diagram of the segmented recognition result provided for an exemplary embodiment of this disclosure;

[0056] Figure 9 A schematic diagram of a meeting record generation apparatus provided for an exemplary embodiment of this disclosure;

[0057] Figure 10 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0058] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0059] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this application's technical solution, based on the prompt message.

[0060] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0061] It is understood that the above notification and user authorization process is merely illustrative and does not limit the implementation of this application. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this application.

[0062] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0063] To make the objectives, technical solutions, and advantages of this disclosure clearer, the principles and spirit of this disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0064] In this article, it is important to understand that any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0065] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar words used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly. The article "a" or "an" preceding an element does not exclude the existence of multiple such elements.

[0066] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments.

[0067] As described in the background section, the automated generation of meeting minutes refers to the process of automatically converting the audio content of a meeting into text through technologies such as speech recognition and natural language processing, and then further analyzing, summarizing, and organizing the text content to ultimately generate a structured meeting record.

[0068] However, the inventors of this disclosure have discovered that in the meeting record generation schemes provided by related technologies, a feature database of the participants needs to be built in advance before the meeting records are generated. In this scheme, when new participants join the meeting, the feature database needs to be rebuilt.

[0069] To address the aforementioned issues, this disclosure provides a scheme for generating meeting minutes, specifically including:

[0070] The process involves: acquiring the audio and video of a meeting, including both audio and video; extracting the voiceprint and facial features of the audio and video at the current moment, fusing the voiceprint and facial features to obtain the speaker's fusion feature at the current moment; determining whether a sample speaker fusion feature matching the speaker's fusion feature exists in the face and voiceprint fusion feature library constructed in real-time based on the meeting; in response to determining that no sample speaker fusion feature matching the speaker's fusion feature exists in the face and voiceprint fusion feature library, determining the speaker and facial image at the current moment corresponding to the speaker fusion feature, and storing the speaker fusion feature, the speaker at the current moment, and the facial image in the face and voiceprint fusion feature library; in response to determining that a sample speaker fusion feature matching the speaker's fusion feature exists in the face and voiceprint fusion feature library, determining the speaker and facial image at the current moment corresponding to the audio and video at the current moment based on the sample speaker fusion feature; converting the audio in the audio and video at the current moment into text, and generating the meeting record of the speaker at the current moment based on the text, the speaker at the current moment, and the facial image.

[0071] This invention enables real-time construction of a facial and voiceprint fusion feature library, eliminating the need to pre-build a feature library for attendees. When new attendees join the meeting, there is no need to rebuild the feature library, thus improving the efficiency of meeting record generation.

[0072] After introducing the basic principles of this disclosure, various non-limiting embodiments of this disclosure will be described in detail below.

[0073] refer to Figure 1 This is a schematic diagram illustrating an application scenario of the meeting record generation method provided in an exemplary embodiment of this disclosure.

[0074] This application scenario includes a terminal device 101, a server 102, and a data storage system 103. The terminal device 101, server 102, and data storage system 103 can all be connected via wired or wireless communication networks to achieve data interaction.

[0075] Terminal device 101 may be an electronic device located close to the user side, possessing data transmission and multimedia input / output functions, including but not limited to desktop computers, mobile phones, portable computers, tablet computers, media players, smart wearable devices, personal digital assistants (PDAs), or other electronic devices capable of performing the aforementioned functions. This electronic device may include a processor and a display screen with touch input functionality. The display screen is used to present a graphical user interface (GUI), which can display an application interface. The processor is used to process application data, generate the GUI, and control the display of the GUI on the screen.

[0076] Both server 102 and data storage system 103 can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0077] In some exemplary embodiments, the method for generating meeting minutes may run on terminal device 101 or server 102.

[0078] When the meeting record generation method runs on terminal device 101, terminal device 101 provides meeting record generation services to users of terminal device 101. A client is installed on terminal device 101. The client acquires the audio and video of the meeting, which includes audio and video. It extracts the voiceprint and facial features of the audio and video at the current moment, fuses the voiceprint and facial features to obtain the speaker's fused features at the current moment, determines whether there is a sample speaker fused feature matching the speaker's fused features at the current moment in the facial and voiceprint fused feature library constructed in real-time based on the meeting, and responds to the determination that there is no fused feature matching the speaker's fused features at the current moment in the facial and voiceprint fused feature library. The system uses a sample speaker fusion feature for feature matching to determine the current speaker and facial image corresponding to the fusion feature. The speaker fusion feature, the current speaker, and the facial image are then stored in the facial voiceprint fusion feature database. In response to the determination that a sample speaker fusion feature matching the current speaker fusion feature exists in the database, the system determines the current speaker and facial image corresponding to the current audio / video based on the sample speaker fusion feature. The audio in the current audio / video is converted to text, and based on the text, the current speaker, and the facial image, a meeting record of the current speaker is generated. The client can then display the meeting record to the user.

[0079] In the above exemplary embodiments, the method for generating meeting minutes is described as running on terminal device 101. However, this disclosure is not limited thereto. In some exemplary embodiments, the method for generating meeting minutes may also run on server 102.

[0080] When the meeting record generation method runs on server 102, server 102 provides meeting record generation services to users of terminal device 101. Terminal device 101 has a client installed that communicates with server 102. Users can use this client to send the audio and video of the meeting captured by terminal device 101 to server 102. Server 102 obtains the audio and video of the meeting, which includes audio and video. It extracts the voiceprint and facial features of the audio and video at the current moment, fuses the voiceprint and facial features to obtain the speaker's fusion features at the current moment, determines whether there is a sample speaker fusion feature matching the speaker's fusion features at the current moment in the facial and voiceprint fusion feature library constructed in real-time based on the meeting, and responds to the determination of the face... If no sample speaker fusion feature matches the speaker fusion feature at the current moment in the voiceprint fusion feature library, determine the speaker and face image at the current moment corresponding to the speaker fusion feature, and store the speaker fusion feature, the speaker at the current moment, and the face image in the face voiceprint fusion feature library. In response to determining that a sample speaker fusion feature matches the speaker fusion feature at the current moment in the face voiceprint fusion feature library, determine the speaker and face image at the current moment corresponding to the audio / video at the current moment based on the sample speaker fusion feature. Convert the audio in the audio / video at the current moment into text, and generate the meeting minutes of the speaker at the current moment based on the text, the speaker at the current moment, and the face image. Server 102 can send the meeting minutes to the client, and the client can display the meeting minutes to the user.

[0081] The data storage system 103 is used to assist the terminal device 101 and the server 102 in storing data, such as storing a facial and voiceprint fusion feature library.

[0082] The following is combined Figure 1 The above application scenarios are used to describe the method for generating meeting minutes according to exemplary embodiments of this disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this disclosure, and the embodiments of this disclosure are not limited in any way. Rather, the embodiments of this disclosure can be applied to any applicable scenario.

[0083] refer to Figure 2 This is a flowchart illustrating a method for generating meeting minutes provided in an exemplary embodiment of this disclosure.

[0084] The method for generating meeting minutes includes the following steps:

[0085] Step S210: Obtain the audio and video of the meeting, wherein the audio and video include audio and video.

[0086] In some exemplary embodiments, video and audio during a meeting are captured using a terminal device with a camera, or a device with audio and video capture and transmission capabilities, such as a vertical camera or a pan-tilt camera.

[0087] As an example, it could be a smart 360° panoramic camera or a smart PTZ camera, a device with speaker tracking, audio and video capture and transmission capabilities. The camera can identify the speaker and capture a close-up shot of the speaker by locating the sound source and detecting and comparing the visual activities (lip movements) of multiple participants in the video information.

[0088] Step S220: Extract the voiceprint features and facial features of the audio and video at the current moment, and fuse the voiceprint features and facial features to obtain the speaker's fused features at the current moment.

[0089] In some exemplary embodiments, the extraction of voiceprint features and facial features of the audio and video at the current moment includes:

[0090] The audio and video are segmented to obtain at least two audio and video segments;

[0091] Each of the aforementioned audio and video segments is processed as the audio and video at the current moment.

[0092] In some exemplary embodiments, extracting the voiceprint features of the audio and video at the current moment includes:

[0093] The audio in the audio and video at the current moment is preprocessed to obtain preprocessed audio;

[0094] Determine the spectral parameters of the preprocessed audio;

[0095] The spectral parameters are filtered to obtain the output;

[0096] Perform a logarithmic transformation on the output to obtain the logarithmic energy;

[0097] The logarithmic energy is subjected to discrete cosine transform to obtain cepstral coefficients and the first-order difference coefficients of the cepstral coefficients. The cepstral coefficients and the first-order difference coefficients are used as the voiceprint features.

[0098] In some exemplary embodiments, the step of preprocessing the audio in the audio and video at the current moment to obtain preprocessed audio includes:

[0099] The audio is pre-emphasized to improve the signal-to-noise ratio of the high-frequency components.

[0100] The audio is divided into multiple audio frames by performing frame segmentation and windowing.

[0101] Endpoint detection is performed on the audio to remove non-speech signals from the audio.

[0102] refer to Figure 3 As an example, the audio input is first preprocessed, including pre-emphasis, frame windowing, and endpoint detection.

[0103] Among them, pre-emphasis: improves the signal-to-noise ratio of the high-frequency part of the speech signal;

[0104] Frame-based windowing: This makes the speech signal to be processed relatively stable by dividing the acquired speech data into multiple small speech frames, typically 10ms to 30ms per frame.

[0105] Endpoint detection: After voice data acquisition, some non-speech signal segments in the obtained data are not conducive to representing the speaker's voiceprint characteristics, such as silence. The acquired data is processed by retaining the useful speech signals and removing non-speech signals that are detrimental to voiceprint recognition.

[0106] After audio preprocessing, a Fast Fourier Transform (FFT) is performed to calculate the spectral parameters of each frame. These parameters are then convolved through a Mel frequency filter consisting of N (typically 20-40) triangular bandpass filters. The logarithm of the output for each frequency band is then taken to calculate the logarithmic energy S(m), where m = 1, 2, 3, ..., N. Finally, a Discrete Cosine Transform (DCT) is performed on these N parameters to obtain the Mel cepstral coefficients (MFCC) and the first-order difference coefficients of the MFCC (ΔMFCC), which serve as the voiceprint features.

[0107] Step S230: Determine whether there is a sample speaker fusion feature in the face and voiceprint fusion feature library constructed in real time based on the meeting that matches the speaker fusion feature at the current moment.

[0108] In practice, this disclosure collects video data (including facial image data) and audio data in real time during the meeting, and constructs a facial voiceprint fusion feature library frame by frame starting from the first frame. For the specific construction method, refer to step S240.

[0109] Step S240: In response to determining that there is no sample speaker fusion feature in the face-voiceprint fusion feature library that matches the speaker fusion feature at the current moment, determine the speaker at the current moment and the face image corresponding to the speaker fusion feature, and store the speaker fusion feature, the speaker at the current moment and the face image into the face-voiceprint fusion feature library.

[0110] refer to Figure 4 As an example, video input is preprocessed, including grayscale conversion, noise reduction, and equalization.

[0111] Among them, grayscale conversion involves converting facial data to grayscale, transforming the grayscale image into a two-dimensional matrix for processing.

[0112] Noise reduction: Face image filtering, median filtering removes noise from the acquired image while retaining information useful for image processing;

[0113] Histogram equalization of face images improves image contrast, increases the grayscale difference between pixels in the image, and makes the features in the image more obvious.

[0114] Then, feature extraction is performed to generate a facial image feature sequence.

[0115] The voiceprint features and facial image features are fused at the feature level to obtain facial voiceprint features, and a facial voiceprint feature database is established.

[0116] Finally, feature recognition is performed on the segmented audio and video information to determine the corresponding speaker.

[0117] In some exemplary embodiments, the method for determining the face image includes:

[0118] Obtain a sequence of facial images from the video in the audio and video at the current moment;

[0119] Select candidate face images from the face image sequence whose face size is larger than a preset size;

[0120] The face image is determined from the candidate face images based on preset rules.

[0121] In some exemplary embodiments, determining the face image from the candidate face images based on preset rules includes:

[0122] The face image is determined from the candidate face images based on the face blur evaluation rules, face pose evaluation rules, face image quality evaluation model, and smile expression matching degree.

[0123] In some exemplary embodiments, the selection of the speaker's head image for optimal facial representation involves the following steps: obtaining a sequence of facial images from a target facial video; selecting facial images from the sequence whose face size is larger than a preset size; and conducting a comprehensive evaluation based on four dimensions: facial blurriness evaluation rules, facial pose evaluation rules, a facial image quality evaluation model, and smile expression matching degree, to determine the best image as the target facial image. To standardize the speaker's head image, a custom background color can be used to make the image cleaner.

[0124] Step S250: In response to determining that there is a sample speaker fusion feature in the face and voiceprint fusion feature library that matches the speaker fusion feature at the current moment, determine the speaker at the current moment and the face image corresponding to the audio and video at the current moment based on the sample speaker fusion feature.

[0125] In specific implementation, if there is a sample speaker fusion feature in the face and voiceprint fusion feature library that matches the speaker fusion feature at the current moment, then the speaker and face image at the current moment corresponding to the speaker fusion feature at the current moment are the same as the speaker and face image at the current moment corresponding to the sample speaker fusion feature.

[0126] Step S260: Convert the audio in the current moment's audio and video into text, and generate the meeting minutes of the current moment's speaker based on the text, the current moment's speaker, and the face image.

[0127] In some exemplary embodiments, the step of converting the audio in the current moment's audio-visual recording into text, and generating the meeting minutes of the current moment's speaker based on the text, the current moment's speaker, and the facial image, includes:

[0128] Determine whether the current audio / video segment is the first of the at least two audio / video segments;

[0129] In response to determining that the current audio / video segment is not the first audio / video segment among the at least two audio / video segments, it is determined whether the speaker fusion feature at the current moment matches the speaker fusion feature of the previous audio / video segment;

[0130] In response to determining that the speaker fusion feature at the current moment matches the speaker fusion feature of the previous audio-visual segment, the current audio-visual segment is aggregated with the previous audio-visual segment to obtain an aggregated audio-visual segment;

[0131] The audio in the aggregated audio and video segments is converted into text, and based on the text, the speaker at the current moment, and the facial image, a meeting record of the speaker at the current moment is generated.

[0132] As an example, in practical implementation:

[0133] The audio and video are segmented to obtain at least two audio and video segments;

[0134] For each audio / video segment, the voiceprint features and facial features are extracted, and the voiceprint features and facial features are fused to obtain the speaker's fused features;

[0135] Determine whether the audio / video segment is the first audio / video segment among the at least two audio / video segments;

[0136] In response to determining that the audio / video segment is the first audio / video segment among the at least two audio / video segments, the speaker corresponding to the speaker fusion feature is determined, the speaker fusion feature is stored as a sample speaker fusion feature and the speaker is stored in the face voiceprint fusion feature library, the face image corresponding to the audio / video segment is determined, the face image is stored in the face image queue, and the speaker and the face image are mapped to the audio / video segment;

[0137] In response to determining that the audio / video segment is not the first audio / video segment among the at least two audio / video segments, it is determined whether the speaker fusion feature matches the speaker fusion feature of the previous audio / video segment;

[0138] In response to determining that the speaker fusion feature matches the speaker fusion feature of the previous audio / video segment, the audio / video segment is aggregated with the previous audio / video segment to obtain an aggregated audio / video segment;

[0139] In response to determining that the speaker fusion feature does not match the speaker fusion feature of the previous audio / video segment, the previous audio / video segment is stored in the segmented recognition result, and it is determined whether there is a sample speaker fusion feature in the face and voiceprint fusion feature library that matches the speaker fusion feature;

[0140] In response to determining that there exists a sample speaker fusion feature in the face and voiceprint fusion feature library that matches the speaker fusion feature, the speaker and face image corresponding to the sample speaker fusion feature are mapped to the audio and video segment;

[0141] In response to determining that there is no sample speaker fusion feature in the face-voiceprint fusion feature library that matches the speaker fusion feature, the speaker corresponding to the speaker fusion feature is determined, the speaker fusion feature and the speaker are stored in the face-voiceprint fusion feature library, the face image corresponding to the audio-video segment is determined, the face image is stored in the face image queue, and the speaker and the face image are mapped to the audio-video segment.

[0142] refer to Figure 5As an example, the entire process involves real-time acquisition of facial image and audio data during a meeting, and then segmenting the data frame by frame to build a facial-voiceprint fusion feature library, starting from the first frame. First, the target audio and video are acquired through the audio / video acquisition unit and then segmented. Video frame rates are 24fps, 30fps, 60fps, etc., while audio frame segmentation and windowing typically use 10ms-30ms per frame. The segmentation time is a multiple of both, taking 100ms as an example. The segmented audio and video are then arranged sequentially and processed segment by segment. The processing involves preprocessing the audio and video, extracting features, and then fusing the audio and image features. When the target audio / video segment is the first segment, the target audio / video features are fused and stored in the facial-voiceprint fusion feature library, while the image is stored in the face queue. (Refer to...) Figure 6 This audio / video segment is mapped to the corresponding speaker and face. If the target audio / video segment is not the first segment, it is compared with the facial and voiceprint fusion feature database to determine if it belongs to the same speaker. If so, it is aggregated with the previous audio / video segment, referencing... Figure 7 If not, the previous aggregation result is stored in the segmented recognition result to obtain the recognition result. Simultaneously, the facial and voiceprint fusion feature library is traversed to determine if a matching speaker exists. If a matching speaker exists, the audio / video segment is mapped to the corresponding speaker and face. If no matching result exists, the target audio / video feature-level fusion is stored in the facial and voiceprint fusion feature library, and the main image frame is stored in the face queue. The audio / video segment is then mapped to the corresponding speaker and face. Recognition using multimodal facial and voiceprint fusion features can reduce the impact of environmental factors on the recognition process and improve the recognition error tolerance. Finally, the audio is processed for speech-to-text conversion, and the result is stored in the segmented recognition result for reference. Figure 8 .

[0143] In some exemplary embodiments, after converting the audio in the audio / video recording into text, and generating the meeting minutes of the speaker at the current moment based on the text, the speaker at the current moment, and the facial image, the method further includes:

[0144] In response to determining that each audio and video segment has been processed, a meeting record of the meeting is generated based on the segmentation and recognition results.

[0145] In some exemplary embodiments, the segmented recognition results are presented through the client to generate the original speech text with the speaker's avatar, and a meeting summary, speaker summary and to-do list are generated through the meeting minutes generation model.

[0146] As can be seen from the above, the meeting record generation method provided in this disclosure includes: acquiring the audio and video of the meeting, wherein the audio and video include audio and video; extracting the voiceprint features and facial features of the audio and video at the current moment, fusing the voiceprint features and the facial features to obtain the speaker fusion feature at the current moment; determining whether there is a sample speaker fusion feature matching the speaker fusion feature at the current moment in the facial voiceprint fusion feature library constructed in real time based on the meeting; and, in response to determining that there is no sample speaker fusion feature matching the speaker fusion feature at the current moment in the facial voiceprint fusion feature library, determining the speaker fusion feature. The speaker and facial image at the current moment are correspondingly stored in the facial voiceprint fusion feature library. In response to determining that a sample speaker fusion feature matching the current moment speaker fusion feature exists in the facial voiceprint fusion feature library, the speaker and facial image at the current moment corresponding to the current moment audio / video are determined based on the sample speaker fusion feature. The audio in the current moment audio / video is converted to text, and the meeting minutes of the current moment speaker are generated based on the text, the current moment speaker, and the facial image.

[0147] This invention enables real-time construction of a facial and voiceprint fusion feature library, eliminating the need to pre-build a feature library for attendees. When new attendees join the meeting, there is no need to rebuild the feature library, thus improving the efficiency of meeting record generation.

[0148] Furthermore, based on multimodal fusion features, the fusion features are used to accurately identify and distinguish speakers in the meeting, reducing the false recognition rate caused by single features.

[0149] Furthermore, a method for selecting the best facial image in complex environments is proposed to ensure that the speaker's image is of high quality and standardized.

[0150] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0151] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0152] Based on the same inventive concept, corresponding to any of the above embodiments, this disclosure also provides a meeting minutes generation apparatus.

[0153] refer to Figure 9 This is a schematic diagram of a meeting record generation apparatus provided in an exemplary embodiment of the present disclosure.

[0154] The meeting minutes generation device includes the following modules:

[0155] The audio and video acquisition module 910 is configured to acquire the audio and video of the meeting, wherein the audio and video include audio and video.

[0156] The speaker fusion feature extraction module 920 is configured to extract the voiceprint features and facial features of the audio and video at the current moment, and fuse the voiceprint features and facial features to obtain the speaker fusion features at the current moment;

[0157] The speaker fusion feature matching module 930 is configured to determine whether there is a sample speaker fusion feature in the face and voiceprint fusion feature library constructed in real time based on the meeting that matches the speaker fusion feature at the current moment;

[0158] The speaker and face image determination module 940, in response to determining that there is no sample speaker fusion feature in the face and voiceprint fusion feature library that matches the speaker fusion feature at the current moment, determines the speaker and face image at the current moment corresponding to the speaker fusion feature, and stores the speaker fusion feature, the speaker at the current moment, and the face image into the face and voiceprint fusion feature library accordingly; in response to determining that there is a sample speaker fusion feature in the face and voiceprint fusion feature library that matches the speaker fusion feature at the current moment, determines the speaker and face image at the current moment corresponding to the audio and video at the current moment based on the sample speaker fusion feature;

[0159] The meeting record generation module 950 is configured to convert the audio in the current moment's audio and video into text, and generate the meeting record of the current moment's speaker based on the text, the current moment's speaker, and the facial image.

[0160] In some exemplary embodiments, the audio / video acquisition module 910 is specifically configured as follows:

[0161] The audio and video are segmented to obtain at least two audio and video segments;

[0162] Each of the aforementioned audio and video segments is processed as the audio and video at the current moment;

[0163] Therefore, the meeting minutes generation module 950 is specifically configured as follows:

[0164] Determine whether the current audio / video segment is the first of the at least two audio / video segments;

[0165] In response to determining that the current audio / video segment is not the first audio / video segment among the at least two audio / video segments, it is determined whether the speaker fusion feature at the current moment matches the speaker fusion feature of the previous audio / video segment;

[0166] In response to determining that the speaker fusion feature at the current moment matches the speaker fusion feature of the previous audio-visual segment, the current audio-visual segment is aggregated with the previous audio-visual segment to obtain an aggregated audio-visual segment;

[0167] The audio in the aggregated audio and video segments is converted into text, and based on the text, the speaker at the current moment, and the facial image, a meeting record of the speaker at the current moment is generated.

[0168] In some exemplary embodiments, the speaker fusion feature extraction module 920 is specifically configured as follows:

[0169] The audio in the audio and video at the current moment is preprocessed to obtain preprocessed audio;

[0170] Determine the spectral parameters of the preprocessed audio;

[0171] The spectral parameters are filtered to obtain the output;

[0172] Perform a logarithmic transformation on the output to obtain the logarithmic energy;

[0173] The logarithmic energy is subjected to discrete cosine transform to obtain cepstral coefficients and the first-order difference coefficients of the cepstral coefficients. The cepstral coefficients and the first-order difference coefficients are used as the voiceprint features.

[0174] In some exemplary embodiments, the speaker fusion feature extraction module 920 is specifically configured as follows:

[0175] The audio is pre-emphasized to improve the signal-to-noise ratio of the high-frequency components.

[0176] The audio is divided into multiple audio frames by performing frame segmentation and windowing.

[0177] Endpoint detection is performed on the audio to remove non-speech signals from the audio.

[0178] In some exemplary embodiments, the speaker and face image determination module 940 are specifically configured as follows:

[0179] Obtain a sequence of facial images from the video in the audio and video at the current moment;

[0180] Select candidate face images from the face image sequence whose face size is larger than a preset size;

[0181] The face image is determined from the candidate face images based on preset rules.

[0182] In some exemplary embodiments, the speaker and face image determination module 940 are specifically configured as follows:

[0183] The face image is determined from the candidate face images based on the face blur evaluation rules, face pose evaluation rules, face image quality evaluation model, and smile expression matching degree.

[0184] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0185] The apparatus described above is used to implement the corresponding meeting record generation method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0186] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the meeting record generation method described in any of the above embodiments.

[0187] Figure 10 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0188] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0189] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0190] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0191] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0192] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0193] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0194] The electronic devices described above are used to implement the corresponding meeting record generation method in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0195] The memory 1020 stores machine-readable instructions executable by the processor 1010. When the electronic device is running, the processor 1010 communicates with the memory 1020 via the bus 1030, causing the processor 1010 to execute the following instructions during operation:

[0196] Acquire the audio and video of the meeting, wherein the audio and video include audio and video;

[0197] Extract the voiceprint and facial features of the audio and video at the current moment, and fuse the voiceprint and facial features to obtain the speaker's fused features at the current moment;

[0198] Determine whether there exists a sample speaker fusion feature in the face and voiceprint fusion feature library constructed in real time based on the meeting that matches the speaker fusion feature at the current moment;

[0199] In response to determining that there is no sample speaker fusion feature in the face-voiceprint fusion feature library that matches the speaker fusion feature at the current moment, the speaker at the current moment and the face image corresponding to the speaker fusion feature are determined, and the speaker fusion feature, the speaker at the current moment and the face image are stored in the face-voiceprint fusion feature library accordingly.

[0200] In response to determining that there exists a sample speaker fusion feature in the face and voiceprint fusion feature library that matches the speaker fusion feature at the current moment, the speaker at the current moment and the face image corresponding to the audio and video at the current moment are determined based on the sample speaker fusion feature;

[0201] The audio in the current moment's audio and video is converted into text. Based on the text, the speaker at the current moment, and the facial image, the meeting minutes of the speaker at the current moment are generated.

[0202] In one possible implementation, the instructions executed by the processor 1010, namely, extracting the voiceprint features and facial features of the audio and video at the current moment, include:

[0203] The audio and video are segmented to obtain at least two audio and video segments;

[0204] Each of the aforementioned audio and video segments is processed as the audio and video at the current moment;

[0205] Then, the step of converting the audio in the current moment's audio and video into text, and generating the meeting minutes of the speaker at the current moment based on the text, the speaker at the current moment, and the facial image, includes:

[0206] Determine whether the current audio / video segment is the first of the at least two audio / video segments;

[0207] In response to determining that the current audio / video segment is not the first audio / video segment among the at least two audio / video segments, it is determined whether the speaker fusion feature at the current moment matches the speaker fusion feature of the previous audio / video segment;

[0208] In response to determining that the speaker fusion feature at the current moment matches the speaker fusion feature of the previous audio-visual segment, the current audio-visual segment is aggregated with the previous audio-visual segment to obtain an aggregated audio-visual segment;

[0209] The audio in the aggregated audio and video segments is converted into text, and based on the text, the speaker at the current moment, and the facial image, a meeting record of the speaker at the current moment is generated.

[0210] In one possible implementation, the instructions executed by the processor 1010, including the step of extracting the voiceprint features of the audio and video at the current moment, include:

[0211] The audio in the audio and video at the current moment is preprocessed to obtain preprocessed audio;

[0212] Determine the spectral parameters of the preprocessed audio;

[0213] The spectral parameters are filtered to obtain the output;

[0214] Perform a logarithmic transformation on the output to obtain the logarithmic energy;

[0215] The logarithmic energy is subjected to discrete cosine transform to obtain cepstral coefficients and the first-order difference coefficients of the cepstral coefficients. The cepstral coefficients and the first-order difference coefficients are used as the voiceprint features.

[0216] In one possible implementation, the instruction executed by processor 1010 to preprocess the audio in the audio and video at the current moment to obtain preprocessed audio includes:

[0217] The audio is pre-emphasized to improve the signal-to-noise ratio of the high-frequency components.

[0218] The audio is divided into multiple audio frames by performing frame segmentation and windowing.

[0219] Endpoint detection is performed on the audio to remove non-speech signals from the audio.

[0220] In one possible implementation, the method for determining the face image in the instructions executed by the processor 1010 includes:

[0221] Obtain a sequence of facial images from the video in the audio and video at the current moment;

[0222] Select candidate face images from the face image sequence whose face size is larger than a preset size;

[0223] The face image is determined from the candidate face images based on preset rules.

[0224] In one possible implementation, the instructions executed by processor 1010, wherein determining the face image from the candidate face images based on preset rules, includes:

[0225] The face image is determined from the candidate face images based on the face blur evaluation rules, face pose evaluation rules, face image quality evaluation model, and smile expression matching degree.

[0226] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the meeting record generation method as described in any of the above embodiments.

[0227] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0228] The aforementioned non-transitory computer-readable storage media can be any available medium or data storage device that a computer can access, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0229] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the meeting record generation method as described in any of the embodiments in the exemplary method section above, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0230] Based on the same inventive concept, corresponding to the meeting record generation method described in any of the above embodiments, this disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processor to perform the meeting record generation method. Corresponding to the execution entity for each step in each embodiment of the meeting record generation method, the processor executing the corresponding step may belong to the corresponding execution entity.

[0231] The computer program product of the above embodiments is used to cause the computer and / or the processor to execute the meeting record generation method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0232] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, method, or computer program product. Therefore, this disclosure can be implemented as entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this disclosure can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0233] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (not exhaustive) of a computer-readable storage medium may include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0234] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0235] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0236] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0237] It should be understood that each block of a flowchart and / or block diagram, as well as combinations of blocks in a flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine that, when executed by a computer or other programmable data processing device, creates means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0238] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce a product comprising an instruction apparatus that implements the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0239] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable apparatus can provide a process for implementing the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0240] Furthermore, although the operations of the methods of this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be executed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0241] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0242] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0243] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0244] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0245] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0246] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

[0247] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be interpreted in the broadest sense, thereby encompassing all such modifications and equivalent structures and functions.

Claims

1. A method for generating meeting minutes, characterized in that, include: Acquire the audio and video of the meeting, wherein the audio and video include audio and video; Extract the voiceprint and facial features of the audio and video at the current moment, and fuse the voiceprint and facial features to obtain the speaker's fused features at the current moment; Determine whether there exists a sample speaker fusion feature in the face and voiceprint fusion feature library constructed in real time based on the meeting that matches the speaker fusion feature at the current moment; In response to determining that there is no sample speaker fusion feature in the face-voiceprint fusion feature library that matches the speaker fusion feature at the current moment, the speaker at the current moment and the face image corresponding to the speaker fusion feature are determined, and the speaker fusion feature, the speaker at the current moment and the face image are stored in the face-voiceprint fusion feature library accordingly. In response to determining that there exists a sample speaker fusion feature in the face and voiceprint fusion feature library that matches the speaker fusion feature at the current moment, the speaker at the current moment and the face image corresponding to the audio and video at the current moment are determined based on the sample speaker fusion feature; The audio in the current moment's audio and video is converted into text. Based on the text, the speaker at the current moment, and the facial image, the meeting minutes of the speaker at the current moment are generated.

2. The method according to claim 1, characterized in that, The extraction of voiceprint and facial features from the audio and video at the current moment includes: The audio and video are segmented to obtain at least two audio and video segments; Each of the aforementioned audio and video segments is processed as the audio and video at the current moment; Then, the step of converting the audio in the current moment's audio and video into text, and generating the meeting minutes of the current moment's speaker based on the text, the current moment's speaker, and the facial image, includes: Determine whether the current audio / video segment is the first of the at least two audio / video segments; In response to determining that the current audio / video segment is not the first audio / video segment among the at least two audio / video segments, it is determined whether the speaker fusion feature at the current moment matches the speaker fusion feature of the previous audio / video segment; In response to determining that the speaker fusion feature at the current moment matches the speaker fusion feature of the previous audio-visual segment, the current audio-visual segment is aggregated with the previous audio-visual segment to obtain an aggregated audio-visual segment; The audio in the aggregated audio and video segments is converted into text, and based on the text, the speaker at the current moment, and the facial image, a meeting record of the speaker at the current moment is generated.

3. The method according to claim 1, characterized in that, The extraction of the voiceprint features of the audio and video at the current moment includes: The audio in the audio and video at the current moment is preprocessed to obtain preprocessed audio; Determine the spectral parameters of the preprocessed audio; The spectral parameters are filtered to obtain the output; Perform a logarithmic transformation on the output to obtain the logarithmic energy; The logarithmic energy is subjected to discrete cosine transform to obtain cepstral coefficients and the first-order difference coefficients of the cepstral coefficients. The cepstral coefficients and the first-order difference coefficients are used as the voiceprint features.

4. The method according to claim 3, characterized in that, The step of preprocessing the audio in the audio and video at the current moment to obtain preprocessed audio includes: The audio is pre-emphasized to improve the signal-to-noise ratio of the high-frequency components. The audio is divided into multiple audio frames by frame segmentation and windowing. Endpoint detection is performed on the audio to remove non-speech signals from the audio.

5. The method according to claim 1, characterized in that, The method for determining the facial image includes: Obtain a sequence of facial images from the video in the audio and video at the current moment; Select candidate face images from the face image sequence whose face size is larger than a preset size; The face image is determined from the candidate face images based on preset rules.

6. The method according to claim 5, characterized in that, The step of determining the face image from the candidate face images based on preset rules includes: The face image is determined from the candidate face images based on the face blur evaluation rules, face pose evaluation rules, face image quality evaluation model, and smile expression matching degree.

7. A device for generating meeting minutes, characterized in that, include: The audio and video acquisition module is configured to acquire the audio and video of the meeting, wherein the audio and video include audio and video. The speaker fusion feature extraction module is configured to extract the voiceprint features and facial features of the audio and video at the current moment, and fuse the voiceprint features and facial features to obtain the speaker fusion features at the current moment; The speaker fusion feature matching module is configured to determine whether there is a sample speaker fusion feature in the face and voiceprint fusion feature library built in real time based on the meeting that matches the speaker fusion feature at the current moment; The speaker and face image determination module is configured to, in response to determining that there is no sample speaker fusion feature in the face and voiceprint fusion feature library that matches the speaker fusion feature at the current moment, determine the speaker and face image at the current moment corresponding to the speaker fusion feature, and store the speaker fusion feature, the speaker at the current moment, and the face image into the face and voiceprint fusion feature library accordingly; and in response to determining that there is a sample speaker fusion feature in the face and voiceprint fusion feature library that matches the speaker fusion feature at the current moment, determine the speaker and face image at the current moment corresponding to the audio and video at the current moment based on the sample speaker fusion feature. The meeting record generation module is configured to convert the audio in the current moment's audio and video into text, and generate the meeting record of the speaker at the current moment based on the text, the speaker at the current moment, and the facial image.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes computer program instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1 to 6.

Images