Intelligent conference system based on multi-modal data

Abstract

The invention discloses an intelligent conference system based on multi-modal data, and belongs to the field of computer science and digital signal processing. The system comprises a data acquisition module, a data transmission module, a participant identification module, a content visualization module, a conference voting module, a knowledge recommendation module, a voice transcription module, an archiving processing module and an auxiliary decision-making module. According to the invention, the information technology means and the conference three elements are organically fused, interaction and fusion of multi-modal data are enhanced, the adaptive capacity of the system is improved by applying the artificial intelligence technology, a solution after the conference is provided, the reliability, compatibility and service capability of the intelligent conference system are enhanced, and the intelligent conference system is more intelligent. And a better condition is created for the practical application of an intelligent and intelligent scene.

Description

Technical Field

[0001] The present application relates to fields such as computer science and digital signal processing, and in particular to an intelligent conference system based on multimodal data. Background Art

[0002] A meeting is an organized, led, and purposeful deliberative activity. It is conducted at a limited time and place according to a certain procedure. There are meetings in almost every organized place. The main functions of meetings include decision-making, control, coordination, and education. In recent years, with the rapid development of information technology, the form of meetings has gradually evolved from traditional meeting modes to meetings in various terminal forms. Offline meetings, online meetings, and online and offline combined meetings are becoming more and more common. Meetings generally include three elements: discussion, decision, and action. Therefore, it is necessary to have discussions during meetings, decisions during discussions, and actions after decisions.

[0003] Traditional conference systems serve as auxiliary media for holding meetings and provide basic convenience for meetings. Although existing conference systems have initially realized the function of voice tracking, they still have problems such as poor intelligence, many limitations on application scenarios, and few application functions, which have restricted the actual application and development of information technology in conference systems to varying degrees.

[0004] Therefore, making full use of the various information data of the conference system, effectively realizing basic functions such as identity recognition and data collection, using information technology to broaden application scenarios, and developing more efficient, scientific, and intelligent functions, and building an intelligent conference system based on big data analysis has become an engineering problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides an intelligent conference system based on multimodal data to enhance the reliability, compatibility and business capabilities of the intelligent conference system.

[0006] The technical solution adopted in this application is:

[0007] An intelligent conference system based on multimodal data, comprising: a data acquisition module, a data transmission module, a participant identification module, a content visualization module, a conference voting module, a knowledge recommendation module, a voice transcription module, an archiving processing module and a decision support module;

[0008] The data collection module collects data from the target area and / or target object based on the received data collection instruction or the input user operation instruction, and encapsulates the collected data to obtain a collection data packet, and then uploads it to the data transmission module, wherein the collection data packet includes a service identifier of the collected data for the data transmission module to distribute the collected data to the corresponding module of the intelligent conference system;

[0009] The data transmission module performs data packet parsing processing on the uploaded collected data packets, transmits the collected data to the corresponding module based on the service identifier, and stores the collected data in the corresponding storage device of the data transmission module;

[0010] A participant identification module is used to identify and confirm the participants based on the collected data;

[0011] The content visualization module is used to visualize the conference content configured by the user based on the output terminals deployed at the conference venue, and to visualize the conference process related information;

[0012] The meeting voting module provides voting functions including various voting forms, collects voting information and counts voting results to form meeting voting results;

[0013] The knowledge recommendation module generates recommended knowledge information of the current meeting's associated auxiliary knowledge based on the deployed knowledge recommendation model according to the current meeting's theme and content, and pushes the recommended knowledge information to designated meeting participants;

[0014] A speech transcription module, which performs speech transcription processing on the speech data collected by the data collection module based on the deployed machine learning model to obtain transcribed text information of the meeting process;

[0015] The archiving processing module automatically generates meeting minutes based on the meeting-related information based on the deployed large language model; and classifies and organizes the meeting-related materials to form and save the meeting archive materials;

[0016] The decision support module, based on the deployed decision support model, extracts the specified information from the conference archives as its model input, and generates text information of specific measures of the conference through the decision support model.

[0017] Furthermore, the intelligent conference system based on multimodal data of the present application also includes a conference initiation notification module, which initiates conference notification information to designated notified objects based on the conference notification information configured by the user.

[0018] Furthermore, the conference initiation notification module generates a conference notification and a confirmed list of participants based on the conference information fed back by the notified party.

[0019] Furthermore, the service identifier is set to be an identifier of a module that issues a data collection instruction and / or an identifier of a module corresponding to a user operation instruction.

[0020] Furthermore, the acquisition device of the data acquisition module includes but is not limited to: an audio acquisition device, a video acquisition device, an image acquisition device and a text acquisition device.

[0021] Furthermore, the data transmission module regularly cleans and deletes the collected data stored in the storage device according to the configured cleanup cycle.

[0022] Furthermore, the participant identification module is specifically:

[0023] Collect basic information of the person to be identified based on the data collection module;

[0024] The collected basic information of the person to be identified is matched with the information of the participants recorded in the preset identification database. If the current person to be identified is in the database, it is confirmed as a participant of this meeting; if not in the database, the basic information of the current person to be identified is added to the identification database, and the current person to be identified is regarded as a participant of this meeting;

[0025] Based on the set participant counting time or the participant counting operation initiated by the conference host, the participant identification module counts the participants of the current meeting and matches them with the participant list determined when the meeting was initiated. If they are consistent, the meeting starts; if they are inconsistent, a confirmation message is sent to the conference host to confirm whether the meeting has started;

[0026] If the meeting host agrees to start the meeting, the meeting will be started; if the host disagrees to start the meeting, the participants of this meeting will be re-determined based on the participant information fed back by the host.

[0027] Furthermore, the identification data of the participant identification module includes but is not limited to: face, fingerprint and voiceprint.

[0028] Furthermore, the knowledge recommendation module is specifically as follows:

[0029] Extract conference contents of multiple modes and perform feature extraction on them respectively to obtain feature vectors of conference contents of each mode. Usually, the modes include video, image, and text. Feature vectors of different modes can be extracted based on a large language model to obtain feature vectors of conference contents of each mode.

[0030] Extract the feature vector of the text of the conference theme to obtain the feature vector of the conference theme;

[0031] In each modality, the feature vector of the conference content and the feature vector of the conference theme are concatenated as the input of the causal transfer module based on the attention mechanism. The fusion features in each modality are obtained based on the output of the causal transfer module.

[0032] The fusion features under all modes are spliced ​​as the input of the preset generation prompt network model, and the text prompt of the meeting-related auxiliary knowledge of the current meeting is obtained based on the output of the generation prompt network model;

[0033] The text prompt of the conference-related auxiliary knowledge is then input into the large language model, and the conference-related auxiliary knowledge of the current conference is obtained based on its output.

[0034] Furthermore, the conference voting module includes the following forms of conference voting: oral, action, text and other forms.

[0035] Furthermore, meeting-related materials include: information about participants, basic meeting materials, meeting-related auxiliary knowledge, text information about the meeting process, meeting voting results, and meeting minutes.

[0036] Furthermore, the meeting-related materials extracted by the archiving processing module also include the multimodal collected data saved by the data transmission module; and after completing the meeting archiving materials of the corresponding meeting, the archiving processing module notifies the data transmission module to clean up and delete the collected data corresponding to the meeting.

[0037] Furthermore, the archiving processing module generates corresponding conference news based on the deployed conference news generation template through a large language model and pushes the corresponding conference news to the configured conference news notification users.

[0038] The technical solution provided by this application brings at least the following beneficial effects:

[0039] The intelligent conference system based on multimodal data provided in this application is based on the organic integration of information technology means with the three elements of the meeting (discussion, decision, and action), which strengthens the interaction and fusion of multimodal data, uses artificial intelligence technology to improve the adaptability of the system, provides post-meeting solutions, and enhances the reliability and compatibility of the intelligent conference system, creating better conditions for the practical application of intelligent and smart scenes.

[0040] In order to better utilize causal information at the feature level, the knowledge recommendation module of this application adopts a causal relationship enhancement method, which can extract features from multimodal information such as images, text, and language, and then connect them with the conference topic feature information respectively, and then input them into the causal migration module respectively. In the cover module, the fusion features are calculated through the attention mechanism to guide the large model to generate related knowledge information, so as to realize the use of large language model related technologies, push knowledge blind spots and reference materials in the meeting process in real time, and provide knowledge references to meeting decision makers.

[0041] The archiving processing module of this application provides users with a text generation function. It uses big data technology to perform machine learning for text writing based on the meeting situation, and can automatically generate meeting minutes, meeting news, etc. It can also develop corresponding text generation functions as needed.

[0042] The auxiliary decision-making module of this application provides users with auxiliary problem-solving functions. It uses large language model-related technologies to automatically generate notifications, plans, methods, etc. for the next step in solving the problem based on the situation of the meeting, providing support for the next step in solving the problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0044] Figure 1 A schematic diagram of the structure of an intelligent conference system based on multimodal data provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of a hierarchical architecture of an intelligent conference system based on multimodal data provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of a participant identification process in an embodiment of the present application;

[0047] Figure 4 This is a schematic diagram of the processing process of the knowledge recommendation module of an embodiment of the present application;

[0048] Figure 5 This is a schematic diagram of the processing process of the conference voting module of an embodiment of the present application.

[0049] Figure 6 This is a workflow diagram of an intelligent conference system based on multimodal data according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions of the embodiments of the present application will be described in detail and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the embodiments described with reference to the drawings are exemplary and are intended to be used to explain the present application, and cannot be understood as limiting the present application.

[0051] In order to build an intelligent platform based on the field of information technology for meetings, by organically integrating information technology means with the three elements of meetings (discussion, decision, and action), strengthening the interaction and integration of multimodal data, using artificial intelligence technology to improve the adaptability of the system, providing post-meeting solutions, and enhancing the reliability and compatibility of intelligent conference systems, the embodiment of the present application provides an intelligent conference system based on multimodal data.

[0052] like Figure 1As shown, an intelligent conference system based on multimodal data provided by an embodiment of the present application includes: a data acquisition module, a data transmission module, a participant identification module, a content visualization module, a conference voting module, a knowledge recommendation module, a voice transcription module, an archiving processing module and a decision support module;

[0053] Among them, the data acquisition module collects data on the target area and / or target object based on the data acquisition instructions issued by other modules of the intelligent conference system or the input user operation instructions, and encapsulates the collected data to obtain a collection data packet, and then uploads it to the data transmission module, wherein the collection data packet includes a task / business identifier of the collected data (such as an identifier of the module issuing the data acquisition instruction, an identifier of the working module corresponding to the user operation instruction, etc.), so that the data transmission module distributes the collected data to the corresponding functional module for corresponding processing; wherein the data modality of the collected data includes but is not limited to voice data, image / video data, text data, etc. Specifically, the data acquisition module of this embodiment can be specifically set to: an audio acquisition device, a video acquisition device, an image acquisition device and a text acquisition device.

[0054] The data transmission module performs data packet parsing processing on the uploaded collected data packets, and then transmits the collected data to the corresponding module; for example, if the current collected data is the data to be identified of the participants (such as face data to be identified, fingerprint data to be identified, sound wave data to be identified, etc.), it will be transmitted to the participant identification module; for another example, if the current collected data is data related to conference voting, it will be transmitted to the conference voting module; if it is data related to voice transcription, it will be transmitted to the voice transcription module; if it is data related to the conference process, it will be transmitted to the archiving processing module and the knowledge recommendation module, etc. In addition, the corresponding storage space can be set for the data transmission module to facilitate the storage of the parsed collected data (with the conference number as the storage data index). In order to improve the utilization rate of the storage space, the collected data corresponding to the conference archived materials or the conference number of the completed conference archived materials issued by the archiving processing module can also be regularly cleared / deleted.

[0055] The participant identification module is used to identify and confirm participants based on collected data (such as face, fingerprint, voiceprint, etc.).

[0056] The content visualization module is used to visualize the conference content configured by the user based on the output terminals deployed at the conference venue; that is, the module usually configures a user interaction interface for the user to configure the conference content to be output and displayed; at the same time, it can also realize personalized display for different conference participants, such as the output display of basic conference content for participants, and the separate presentation of information content such as supplementary materials that can be collected by the conference host / conference speaker. In addition, the content visualization module is also used to realize the output display of the voting process and conference voting results of the conference voting module.

[0057] The meeting voting module is used to provide voting functions in various forms (such as oral, action, text and other forms), collect voting information and count the voting situation to form the meeting voting results; in addition, the meeting voting results can be output and displayed to designated personnel through the content visualization module.

[0058] The knowledge recommendation module, based on the deployed knowledge recommendation model (which can be implemented based on the large language model), generates recommended knowledge information of the current meeting's related auxiliary knowledge according to the theme and content of the current meeting, and pushes the recommended knowledge information to designated meeting personnel to provide basic support for the scientific judgment of meeting decision makers.

[0059] The speech transcription module performs speech transcription on the speech data collected by the data acquisition module based on the deployed machine learning model to obtain the transcribed text information of the meeting process for subsequent processing by the archiving processing module, auxiliary decision-making module, etc.

[0060] The archiving processing module automatically generates meeting minutes based on the deployed large language model and meeting-related information; and classifies and organizes meeting-related materials such as participant information, basic meeting materials, meeting-related auxiliary knowledge, meeting process text information, meeting voting results and meeting minutes to form and save meeting archive materials. In addition, when forming meeting archive materials, the archiving processing module can also extract the multimodal collected data saved by the data transmission module. After completing the meeting archive materials of the corresponding meeting, the data transmission module can be notified to delete the collected data corresponding to the meeting. Furthermore, the archiving processing module can also generate corresponding meeting news based on the deployed meeting news generation template through the large language model and push it to the configured meeting news notification user. The meeting news can also be stored as an item in the meeting archive materials.

[0061] The decision support module, based on the deployed decision support model (which can be implemented based on the large language model), extracts specified information from the meeting archive materials (such as meeting content and meeting process text information, etc.) as its model input, and generates text information of specific measures of the meeting through the decision support model. That is, artificial intelligence technology based on the large model automatically forms implementation notices, plans, methods, measures, etc. according to the discussion of the meeting, to assist decision makers in carrying out actual work more efficiently.

[0062] In one embodiment, the intelligent conference system based on multimodal data of the present application also includes a conference initiation notification module, which initiates conference notification information to designated notified objects based on the conference notification information configured by the user (such as time, location, host, topic, etc.). Furthermore, the conference notification and the confirmed list of participants can be formed based on the meeting information fed back by the notified objects.

[0063] Furthermore, in actual application, the architecture of the intelligent conference system based on multimodal data provided by the embodiment of the present application can be divided into five layers, such as Figure 2 As shown, it includes the physical layer, data layer, control layer, application layer and extension layer, each layer is specifically:

[0064] The physical layer is mainly composed of data information collection devices. This layer is the lowest level of the system, and a large number of sensors, devices and equipment are deployed. The equipment includes but is not limited to mobile phones, computers, microphones, speakers, cameras, image capture devices, smart conference equipment, etc., which are used to collect and transmit audio, images, videos, texts and other basic materials required for meetings, and provide basic guarantees for the convening of meetings.

[0065] The data layer is mainly for data recording and storage. This layer stores data of different categories, different times, different devices, and different objects. It marks the data during the data recording process and fully considers the efficient storage and access of different types of data.

[0066] The control layer is mainly responsible for data processing and interaction. In the process of multimodal data collection, data synchronization and interaction are involved. Synchronization or interaction may occur in one modality at different times or in different modalities at the same time. Some data-level markings need to be done through the control layer. At the same time, the data may contain redundant information and data noise, so data control and supervision need to be strengthened. Digital signal processing technology can be used to construct data feature extraction and application.

[0067] The application layer is mainly used to develop basic applications. Combined with the conference scenario, the reserved database is used to develop applications for actual scenarios. The participant positioning function identifies the participants according to the pre-booked conference plan, collects the facial image data of the participants, compares the existing personnel information database, performs facial recognition, and locks the participants; the voice transcription function uses natural language processing methods to transcribe voice data in real time; the meeting voting function records and counts the voting results according to the actual situation through trigger voting, raising hands, oral voting, etc.

[0068] The extension layer mainly provides interfaces for data interactive applications. The text generation function uses big data technology to perform machine learning for text writing according to the situation of the meeting, automatically generates meeting minutes, meeting news, etc., and can also develop corresponding text generation functions as needed; the knowledge recommendation function uses large language model related technologies to push knowledge blind spots and reference materials in the meeting process in real time, providing knowledge references for meeting decision makers; the auxiliary solution function uses large language model related technologies to automatically generate notifications, plans, and methods for the next step of solving problems according to the situation of the meeting, providing support for the next step of solving problems.

[0069] In one embodiment, Figure 3 As shown, the specific processing process of the participant identification module of this application is: when the meeting is initiated, the scope of participants of this meeting has been clarified. Personnel identification is carried out before the meeting. If the person is in a known database (which can be a database configured by an intelligent conference system, such as a participant identification database, which can be set based on the participant confirmation list) and is confirmed as a participant of this meeting, the person is confirmed to attend the meeting. If the person is not in the known database, it is necessary to add the person's basic information (face, fingerprint, voiceprint, etc.) and update the database, and finally determine whether the person will attend the meeting. After all the expected participants have arrived, the host will review and agree to hold the meeting.

[0070] In the embodiment of the present application, the basic information of the person to be identified can be first collected based on the data collection module (which can be a basic information collection device for personnel deployed in a designated area such as the entrance of the venue, such as collecting face, fingerprint, voiceprint, etc.), and transmitted to the participant identification module through the data transmission module; the module performs personnel matching (such as face matching, fingerprint matching, voiceprint matching, etc.) based on the basic information of the personnel and the participant information recorded in the pre-configured identification database. If the current person to be identified is in the database, it is confirmed as a participant of this meeting; if not in the database, the basic information of the current person to be identified is added to the identification database, and the identification is realized. The identification database is now updated, and the current person to be identified is regarded as a participant of this meeting; based on the set participant counting time (which can be set based on the meeting start time) or the participant counting operation initiated by the meeting host, the participant identification module counts the participants of this meeting and matches them with the list of participants determined when the meeting was initiated. If they are consistent, the meeting starts; if they are inconsistent, a confirmation message on whether the meeting has started is sent to the meeting host, and the meeting is started based on the feedback of the meeting host; if the host disagrees, the participants of this meeting are re-determined based on the processing information of their feedback.

[0071] The theme of a conference often guides the discussion direction of the entire conference. The content of the conference is generally presented in the form of video, image, text, voice and other visual information. The multimodal information in the conference theme and content contains causal relationships. In order to better utilize causal information at the feature level, the knowledge recommendation module of this application adopts a causal enhancement method. Taking the three modal information of image, text and language as an example, the three modal information is extracted and connected with the conference theme feature information respectively, and then respectively input into the causal migration module (denoted as module T), in which the fusion features are calculated through the attention mechanism:

[0072]

[0073]

[0074]

[0075] in, and Represent the input features and output fusion features / data respectively, represents the output of the causal transfer module, represents the softmax function, represents the attention score, which is expressed as:

[0076]

[0077] in, , , Represent three fully connected layers, namely , , They are the outputs of the query linear layer, the key linear layer, and the value linear layer, respectively. In the embodiment of the present application, they are all set to be fully connected layers. Represents a preset constant. The specific value can be set based on the actual application scenario.

[0078] In one embodiment, the specific processing process of the knowledge recommendation module of the present application based on the causal migration module is as follows: Figure 4 As shown: The feature vectors of three modal information (image data, text data, and voice data) (respectively recorded as image features , text features , voice features ) and the feature vector corresponding to the conference theme information (referred to as theme feature ) as the input of the causal migration module, corresponding to three causal migration modules respectively , , , that is, each modality corresponds to one; the process of obtaining the corresponding fusion feature based on the corresponding causal migration module can be expressed as:

[0079]

[0080]

[0081]

[0082]

[0083] in, , , They are the fusion features of image and topic, the fusion features of text and topic, and the fusion features of speech and topic. Indicates the connection operation. Represents the generated text prompt, Represents the generation of a prompt network model. As the input of the big model, it guides the big model to generate related knowledge information, namely, conference-related auxiliary knowledge.

[0084] In one embodiment, a pre-trained machine learning model (also referred to as a speech transcription model) can be directly used as the initial structure, and after transfer learning training is performed on it in combination with the manually verified text, it is deployed in the speech transcription module. Among them, the specific process of transfer learning training includes: based on the selected initial structure, the speech data to be transcribed is pre-processed, such as segmentation, so that the pre-processed speech data matches the input of the machine learning model. Then the pre-processed speech data is input into the model, and the corresponding text information is obtained based on its output, and it is matched with the manually verified text corresponding to the input. If it is inconsistent, the model parameters of the machine learning model are optimized based on the set loss function (such as the cross entropy loss of the text information output by the model and the manually verified text, etc.) to complete the model correction and deep learning; if it is consistent, the transfer learning training is stopped, so as to output the final transcribed text information based on the current model.

[0085] In addition, after the transfer training is completed, the machine learning model can be regularly updated based on the processed voice data and manually proofread text as training data to further improve the transcription accuracy of the model.

[0086] In one embodiment, Figure 5 As shown, the conference voting module of the present application can support voting forms including: oral, action, text and other forms. Among them, oral voting corresponds to voice recognition, action voting (such as raising hands) corresponds to image recognition, and text voting corresponds to text recognition. Oral voting can be set to: agree, pass, approve, etc., which can all express the inclination towards the topic or matter. Action voting: raising hands, nodding, etc. can all express approval. Text voting is more direct, and the text record will be more direct and clear through ticking, agreeing, etc. There are other voting methods, such as clicking on a specific mark on the screen, pressing a specific button, etc., which is also a form of voting. Based on the voting form selected by the meeting host, the corresponding data acquisition device is started, and the statistical results of the meeting voting are obtained based on the recognition processing of the data acquisition device, that is, after the vote, the voting situation will be counted according to the collected and identified voting information, and finally the meeting voting results will be formed.

[0087] Figure 6 The workflow diagram of an intelligent conference system based on multimodal data provided in the embodiment of the present application is described in the figure according to the links of the meeting, which is mainly divided into three parts, namely before the meeting, during the meeting and after the meeting. The specific process is as follows:

[0088] Before the meeting is officially held, the host will initiate the meeting arrangement based on the meeting initiation notification module of this system, including the basic information of the meeting: time, place, host, participants, agenda, etc. After the meeting is initiated, the preparatory work for the meeting will be carried out: preparation of meeting materials, preparation of hardware facilities, preparation of software facilities. It should be noted that the intelligent conference system of this application has added multimedia system management on the basis of traditional conferences, and at the same time has higher requirements for the update of databases, knowledge bases, etc., and conducts machine learning and knowledge reserves according to the theme of the meeting.

[0089] During the meeting, participants are first confirmed based on the participant identification module. The confirmation method is selected according to the different conference media: face recognition, fingerprint recognition, sound wave recognition and other recognition methods.

[0090] After the participants are confirmed, the meeting is formally held. The meeting is conducted based on an intelligent conference system, whose content visualization module is connected to multimedia devices to facilitate the transmission and reception of multimedia information, including: video, images, text, voice and other visual information.

[0091] It is worth mentioning that this application introduces an auxiliary decision-making system during the meeting, that is, the knowledge recommendation module of this application can use artificial intelligence technology based on the large language model (Large Language Model, referred to as the large model) to recommend related knowledge information according to the theme and content of the meeting, providing basic support for the scientific judgment of decision makers. At the same time, the text information of the meeting process can also be obtained based on the voice transcription module of this application, and its content includes but is not limited to: the speech information of the speaker of the meeting, the discussion information of the participants, etc.

[0092] The end of a meeting usually requires judgment and decision-making, the most common of which is approval or rejection. At this time, a variety of voting methods are introduced: oral voting, raising hands, text voting and other voting methods. Based on the selected voting method, the corresponding meeting voting results are obtained through the meeting voting module of this application.

[0093] After the meeting, it is necessary to organize the situation of the meeting and arrange the next steps. Traditional meeting archives only classify and store the materials during the meeting. The information in the archiving processing module provided by the intelligent conference system of this application is classified and organized, and after processing by methods such as noise reduction and cleaning, it is numbered and stored. At the same time, according to the actual situation of the meeting, artificial intelligence means are used to automatically generate meeting minutes as an archiving material. As one of the results of the meeting, specific implementation measures are indispensable. Artificial intelligence technology based on large models automatically forms implementation notices, plans, methods, measures, etc. according to the discussion of the meeting, assisting decision makers to carry out actual work more efficiently.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0095] The above are only some embodiments of the present application. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. An intelligent conference system based on multimodal data, characterized in that: include: Data collection module, data transmission module, participant identification module, content visualization module, meeting voting module, knowledge recommendation module, voice transcription module, archiving processing module and decision support module; The data collection module collects data from the target area and / or target object based on the received data collection instruction or the input user operation instruction, and encapsulates the collected data to obtain a collection data packet, and then uploads it to the data transmission module, wherein the collection data packet includes a service identifier of the collected data for the data transmission module to distribute the collected data to the corresponding module of the intelligent conference system; The data transmission module performs data packet parsing processing on the uploaded collected data packets, transmits the collected data to the corresponding module based on the service identifier, and stores the collected data in the corresponding storage device of the data transmission module; A participant identification module is used to identify and confirm the participants based on the collected data; The content visualization module is used to visualize the conference content configured by the user based on the output terminals deployed at the conference venue, and to visualize the conference process related information; The meeting voting module provides voting functions including various voting forms, collects voting information and counts voting results to form meeting voting results; The knowledge recommendation module generates recommended knowledge information of the current meeting's associated auxiliary knowledge based on the deployed knowledge recommendation model according to the current meeting's theme and content, and pushes the recommended knowledge information to designated meeting participants; A speech transcription module, which performs speech transcription processing on the speech data collected by the data collection module based on the deployed machine learning model to obtain transcribed text information of the meeting process; The archiving processing module automatically generates meeting minutes based on the meeting-related information based on the deployed large language model; and classifies and organizes the meeting-related materials to form and save the meeting archive materials; The decision support module, based on the deployed decision support model, extracts the specified information from the conference archives as its model input, and generates text information of specific measures of the conference through the decision support model.

2. The intelligent conference system based on multimodal data according to claim 1, characterized in that: It also includes a conference initiation notification module, which initiates conference notification information to designated notified objects based on the conference notification information configured by the user.

3. The intelligent conference system based on multimodal data according to claim 1, characterized in that: The service identifier is set to the identifier of the module that issues the data collection instruction and / or the identifier of the module corresponding to the user operation instruction.

4. The intelligent conference system based on multimodal data according to claim 1, characterized in that: The acquisition device of the data acquisition module includes an audio acquisition device, a video acquisition device, an image acquisition device and a text acquisition device.

5. The intelligent conference system based on multimodal data according to claim 1, characterized in that: The data transmission module regularly cleans and deletes the collected data stored in the storage device according to the configured cleanup cycle.

6. The intelligent conference system based on multimodal data according to claim 1, characterized in that: The specific module for identifying participants is as follows: Collect basic information of the person to be identified based on the data collection module; The collected basic information of the person to be identified is matched with the information of the participants recorded in the preset identification database. If the current person to be identified is in the database, it is confirmed as a participant of this meeting; if not in the database, the basic information of the current person to be identified is added to the identification database, and the current person to be identified is regarded as a participant of this meeting; Based on the set participant counting time or the participant counting operation initiated by the conference host, the participant identification module counts the participants of the current meeting and matches them with the participant list determined when the meeting was initiated. If they are consistent, the meeting starts; if they are inconsistent, a confirmation message is sent to the conference host to confirm whether the meeting has started; If the meeting host agrees to start the meeting, the meeting will be started; if the host disagrees to start the meeting, the participants of this meeting will be re-determined based on the participant information fed back by the host.

7. The intelligent conference system based on multimodal data according to claim 1, characterized in that: The knowledge recommendation module is as follows: Extract conference contents of multiple modes, and perform feature extraction on them respectively to obtain feature vectors of conference contents of each mode; Extract the feature vector of the text of the conference theme to obtain the feature vector of the conference theme; In each modality, the feature vector of the conference content and the feature vector of the conference theme are concatenated as the input of the causal transfer module based on the attention mechanism. The fusion features in each modality are obtained based on the output of the causal transfer module. The fusion features under all modes are spliced ​​as the input of the preset generation prompt network model, and the text prompt of the meeting-related auxiliary knowledge of the current meeting is obtained based on the output of the generation prompt network model; The text prompt of the conference-related auxiliary knowledge is then input into the large language model, and the conference-related auxiliary knowledge of the current conference is obtained based on its output.

8. The intelligent conference system based on multimodal data according to claim 1, characterized in that: Meeting-related materials include: information about participants, basic meeting materials, meeting-related auxiliary knowledge, text information about the meeting process, meeting voting results and meeting minutes.

9. The intelligent conference system based on multimodal data according to claim 1, characterized in that: The meeting-related data extracted by the archiving processing module also includes the multimodal collected data saved by the data transmission module; and after completing the meeting archiving data of the corresponding meeting, the archiving processing module notifies the data transmission module to clean up and delete the collected data corresponding to the meeting.

10. The intelligent conference system based on multimodal data according to claim 1, characterized in that: The archiving processing module generates corresponding conference news based on the deployed conference news generation template through a large language model and pushes it to the configured conference news notification users.

Images