Conference information output method and device, terminal equipment and storage medium

Abstract

The invention provides a conference information output method and device, terminal equipment and a storage medium, and the method comprises the steps: carrying out the intention analysis of conference site information under the condition that the conference site information is received, and determining a user request type; inputting the conference site information into a preset recognition model corresponding to the user request type, and outputting a semantic-free recognition model output result; according to the user request type and a preset recognition model, performing matching from a preset dynamic prompt word library to obtain a target prompt word; and inputting the recognition model output result and the target cue word into a preset multi-modal large model, and outputting structured target conference information. Therefore, the proper preset recognition model is selected through intention analysis, then the prompt word bank is combined, and the multi-modal large model outputs the structured information, so that the precision problem when the multi-modal large model is directly used for processing the vertical field task can be avoided, meanwhile, the multi-modal large model does not need to be finely adjusted, the cost is saved, and the efficiency is improved. And the conference information output effect is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent conference, and particularly relates to a conference information output method and device, a terminal device and a storage medium. BACKGROUND

[0002] In an intelligent conference scenario, an ideal conference assistant not only needs to process voice and text, but also needs to "see" the conference site (such as conference participants and presentation content) and "understand" the conference context, so as to complete complex tasks such as meeting management, content summary, real-time question and answer, etc.

[0003] In the related art, a multimodal large model is generally directly used to process all information in a conference, expecting that the multimodal large model can complete a series of operations such as target detection, speech recognition, semantic understanding and content generation in an end-to-end manner. However, a general multimodal large model has a high missing detection and false detection rate, and cannot provide accurate and reliable structured data. If the performance of the multimodal large model in a specific vertical field is to be improved, a high cost needs to be paid.

[0004] Therefore, how to improve the effect of conference information output has become a technical problem to be solved. SUMMARY

[0005] Embodiments of the application provide a conference information output method and device, a terminal device and a storage medium, which can improve the effect of conference information output.

[0006] A first aspect of embodiments of the application provides a conference information output method, comprising: in the case of receiving conference site information, performing intent analysis on the conference site information to determine a user request type; inputting the conference site information into a preset recognition model corresponding to the user request type, the preset recognition model outputting a recognition model output result without semantics; according to the user request type and the preset recognition model, matching a target prompt word from a preset dynamic prompt word library; inputting the recognition model output result and the target prompt word into a preset multimodal large model, the multimodal large model outputting structured target conference information.

[0007] In the technical solution of the application, a suitable preset recognition model is selected through intent analysis, and then a multimodal large model is used to output structured information in combination with a prompt word library, so that the precision problem of directly using the multimodal large model to process a vertical field task can be avoided, and the multimodal large model does not need to be finely adjusted, thereby saving cost and improving the effect of conference information output.

[0008] Optionally, in a possible implementation manner of the first aspect, the conference site information comprises a conference site image, the preset recognition model comprises a face detection model and a face recognition model, and the inputting the conference site information into the preset recognition model corresponding to the user request type and the outputting the semantic-free recognition model output result by the preset recognition model comprise: inputting the conference site image into the face detection model, and outputting position information of each participant in the conference site image by the face detection model; inputting the conference site image and the position information of each participant into the face recognition model, and outputting identity information of each participant by the face recognition model. In this way, the conference site image is automatically processed, and the identity of each participant is accurately positioned and recognized.

[0009] Optionally, in another possible implementation manner of the first aspect, the target conference information comprises participant information and current speaker information, and the inputting the recognition model output result and the target prompt word into the preset multi-modal large model and the outputting the structured target conference information by the multi-modal large model comprise: inputting the position information of each participant, the identity information of each participant and the target prompt word into the multi-modal large model, and outputting the participant information and the current speaker information by the multi-modal large model. In this way, the inference ability of the multi-modal large model is used to convert the original identity information and position information into conference state information (for example, who is speaking) with semantics.

[0010] Optionally, in another possible implementation manner of the first aspect, the conference site information comprises a conference script image and conference audio data, the preset recognition model comprises an optical character recognition (OCR) model and a speech recognition model, and the inputting the conference site information into the preset recognition model corresponding to the user request type and the outputting the semantic-free recognition model output result by the preset recognition model comprise: inputting the conference script image into the optical character recognition model, and outputting conference script text by the optical character recognition model; inputting the conference audio data into the speech recognition model, and outputting conference audio text by the speech recognition model. Through the above scheme, the conference script image and the conference audio data in the conference are converted into data that can be further processed by the multi-modal large model.

[0011] Optionally, in a further possible implementation manner of the first aspect, the target conference information includes conference speech records, conference speech summaries, and conference discussion opinions, and the inputting the recognition model output result and the target prompt word into the preset multi-modal large model and the outputting the structured target conference information by the multi-modal large model include: inputting the conference audio data, the conference script text, the conference audio text, and the target prompt word into the multi-modal large model, and outputting the conference speech records, the conference speech summaries, and the conference discussion opinions by the multi-modal large model. In this way, the multi-modal large model is used for deep processing of conference content, and valuable conference records, summaries, and opinions are extracted from multi-modal information.

[0012] Optionally, in a further possible implementation manner of the first aspect, the conference site information includes conference site images or conference audio data, the preset recognition model includes a face recognition model in a case where the conference site information includes the conference site images, and the preset recognition model includes a voiceprint recognition model in a case where the conference site information includes the conference audio data, and the inputting the conference site information into the preset recognition model corresponding to the user request type and the outputting the semantic-free recognition model output result by the preset recognition model include: inputting the conference site images into the face recognition model, and outputting identity information of each conference participant by the face recognition model; or inputting the conference audio data into the voiceprint recognition model, and outputting the identity information of each conference participant by the voiceprint recognition model. The above technical solution provides a scheme of identity verification through two different modalities of face or voiceprint.

[0013] Optionally, in a further possible implementation manner of the first aspect, the identity information of each conference participant includes identity information of a communication initiator, and the target conference information includes a target call request, and the inputting the recognition model output result and the target prompt word into the preset multi-modal large model and the outputting the structured target conference information by the multi-modal large model include: inputting the conference site images, the conference audio data, the identity information of the communication initiator, and the target prompt word into the multi-modal large model, and outputting the target call request by the multi-modal large model, where the target call request includes the identity information of the communication initiator and identity information of a communication receiver. In this way, the intelligent call function is implemented by combining face recognition, voiceprint recognition, and a multi-modal large model, and the generated target call request can identify who initiates the call, and the called person can know who initiates the call.

[0014] The second aspect of the embodiment of the present application provides a conference information output apparatus, including:

[0015] an intent analysis module configured to, in a case where the conference site information is received, perform intent analysis on the conference site information, and determine a user request type;

[0016] The first model calling module is configured to input the conference live information into a preset recognition model corresponding to the user request type, and the preset recognition model outputs a recognition model output result without semantics.

[0017] The prompt word matching module is configured to match a target prompt word from a preset dynamic prompt word library according to the user request type and the preset recognition model.

[0018] The second model calling module is configured to input the recognition model output result and the target prompt word into a preset multi-modal large model, and the multi-modal large model outputs structured target conference information.

[0019] The third aspect of the embodiment of the present application provides a terminal device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the conference information output method of the first aspect when executing the computer program.

[0020] The fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the conference information output method of the first aspect.

[0021] The fifth aspect of the embodiment of the present application provides a computer program product, which, when executed on a terminal device, causes the terminal device to execute the conference information output method of the first aspect.

[0022] It can be understood that the beneficial effects of the second aspect to the fifth aspect can be referred to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0024] Figure 1 is a flowchart of a conference information output method provided by the embodiment of the present application;

[0025] Figure 2 is a conference scene flowchart provided by the embodiment of the present application;

[0026] Figure 3 is another conference scene flowchart provided by the embodiment of the present application;

[0027] Figure 4is another conference scenario flowchart provided by an embodiment of the present application;

[0028] Figure 5 is another conference scenario flowchart provided by an embodiment of the present application;

[0029] Figure 6 is a structural diagram of a conference information output device provided by an embodiment of the present application;

[0030] Figure 7 is a structural diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0031] In the following description, for the purposes of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the application. However, it will be apparent to those skilled in the art that the application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the application with unnecessary detail.

[0032] It is to be understood that the terminology "includes", "has", "holds", "contains" and / or "comprising", when used in this specification and in the following claims, indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0033] It is also to be understood that the terminology "and / or" when used in this specification and in the following claims, refers to at least one of the items, or any combination of one or more of the items, and includes any possible combination of the items.

[0034] As used in this specification and in the claims, the term "if" can be interpreted as meaning "when", or "once", or "in response to a determination", or "in response to detecting", as appropriate, depending on the context. Similarly, the phrase "if determined", or "if detected [the described condition or event]" can be interpreted as meaning "once determined", or "in response to a determination", or "once detected [the described condition or event]", or "in response to detecting [the described condition or event]", as appropriate, depending on the context.

[0035] In addition, in the description of the application and in the following claims, the terms "first", "second", "third", etc. are only used to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0036] Reference within the specification of this application to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in additional embodiments," and so on, in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily referring to some, but not all, embodiments, unless otherwise indicated by the context. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.

[0037] It should be understood that the magnitude of the serial number of each step in the embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the application.

[0038] In related technologies, in the intelligent meeting scenario, a multi-modal large model is usually directly used to process all information in the meeting, expecting to complete a series of operations such as target detection, speech recognition, semantic understanding and content generation end-to-end.

[0039] However, when the general multi-modal large model is directly applied to such vertical fields with high accuracy requirements, obvious defects and deficiencies are exposed: first, when performing specific visual tasks such as target detection and character recognition (OCR), the positioning and recognition accuracy of the multi-modal large model is often difficult to compare with that of a dedicated and optimized domain small model (such as a dedicated face recognition model or an OCR model), and there is a high rate of missed detection and false detection, which cannot provide accurate and reliable structured data (such as "who is present", "who is speaking", and "what is the specific text on the screen"). Second, if these huge multi-modal models are fine-tuned in order to improve their performance in specific vertical fields, they will face high costs, including the need to collect a large amount of high-quality domain annotation data, consume huge computing resources for training, and require higher computing power for deployment devices, which is not economical and difficult to implement for many application scenarios that have already deployed existing edge computing devices. Therefore, how to improve the effect of meeting information output has become a technical problem to be solved.

[0040] To this end, the embodiment of the present application provides a conference information output method and device, terminal equipment and storage medium, wherein the method first analyzes the intention of the conference live information to determine the user request type when the conference live information is received; then inputs the conference live information into a preset recognition model corresponding to the user request type, and the preset recognition model outputs a semantic-free recognition model output result; then matches the target prompt word from the preset dynamic prompt word library according to the user request type and the preset recognition model; finally, inputs the recognition model output result and the target prompt word into the preset multi-modal large model, and the multi-modal large model outputs the structured target conference information. Thus, by selecting the appropriate preset recognition model through intention analysis, and combining the prompt word library to let the multi-modal large model output structured information, the precision problem when directly using the multi-modal large model to process the vertical field task can be avoided, and the multi-modal large model does not need to be finely adjusted, saving the cost, and thus the effect of conference information output is improved.

[0041] The application scenario of the conference information output method provided by the embodiment of the present application is exemplified below. The present application can be applied to digital human intelligent conference assistants, conference computers, and conference assistants with microphones and cameras. Through the embodiment of the present application, the conference assistant can not only make pre-conference arrangements, in-conference questions and answers, and post-conference summaries, but also has "eyes" to see the conference scene and the conference large screen, and "ears" to hear the speeches of on-site and remote participants. In addition, the conference assistant is endowed with "wisdom" to know who attends the conference and what is said by combining the end-side recognition model and the multi-modal large model, and to participate in the conference discussion by combining the private knowledge base.

[0042] To illustrate the technical solutions of the present application, specific embodiments are described below.

[0043] Referring to Figure 1 , a flowchart of a conference information output method provided by an embodiment of the present application is shown. The conference information output method can be executed by a conference assistant. As Figure 1 indicated, the conference information output method can include the following steps:

[0044] Step 101, when receiving conference live information, analyzing the intention of the conference live information to determine the user request type.

[0045] The conference live information can include conference audio data, conference live images, conference document images (PPT or other document screenshots used in the conference, document screenshots being discussed, etc.).

[0046] In the embodiments of the present application, the intention recognition is performed according to the conference site information input by the user, which can identify which aspect of information the user is concerned about, i.e., the user request type. Specifically, the core task is to understand the raw, multi-modal conference site information (such as a user's voice instruction "Xiaowang, who is speaking now?", a conference site screenshot, or a conference audio) received by a microphone, a camera, or the like, and determine the user's intention.

[0047] In one embodiment, the intention analysis can classify a user's natural language query or instruction into a predefined category with clear meaning, thereby determining the user's intention and invoking the corresponding service or process. The intention analysis can be implemented by classical intention recognition methods, Large Language Model (LLM) based intention recognition methods, etc. For example, a series of keywords and rule patterns (such as regular expressions) can be manually defined, and if the user query matches a certain rule, it is classified into the corresponding intention. For example, the user says "Help me find Xiaowang", the intention is "call", then a rule [help | find | contact] + [name] can be defined in advance to match the "call" intention, and in actual application, when the user says "Help me contact Xiaowang", it will be successfully triggered.

[0048] In one embodiment, the conference assistant has a sleep function. When the conference starts, if the conference assistant is in a sleep state, the wake-up function (including audio wake-up, face recognition wake-up, infrared perception wake-up, etc.) on the conference assistant can be used to wake up the conference assistant.

[0049] Step 102, input the conference site information into the preset recognition model corresponding to the user request type, and the preset recognition model outputs a semantic-free recognition model output result.

[0050] In the embodiments of the present application, the aspect of information that the user is concerned about can call different models, and different models output different results. Specifically, according to the user request type determined in the foregoing steps, one or more pre-deployed preset recognition models specialized in specific tasks can be called. These recognition models are more accurate than general large models in specific tasks (such as face recognition and OCR). In addition, the output of these recognition models is raw, accurate but lacks context understanding data.

[0051] It should be understood that the preset recognition model can include a face detection model, a face recognition model, an OCR model, a speech recognition model, a voiceprint recognition model, etc., which can be determined in combination with actual application scenarios and needs, and the embodiments of the present application do not limit this.

[0052] Step 103, according to the user request type and the preset recognition model, the target prompt word is matched from the preset dynamic prompt word library.

[0053] In the embodiments of the present application, a dynamic prompt word library is pre-set, which stores templates for different scenarios (intents) and different model outputs. According to the results of the foregoing steps (user request type and which model is specifically called), the most suitable target prompt word template is intelligently selected from the prompt word library, so that the output results of different recognition models are combined with the corresponding prompt words in the prompt word library.

[0054] In one embodiment, at the beginning of the meeting, the face recognition algorithm is used to identify which people attend the meeting, and the prompt words recorded in the prompt word library about the meeting scene can be as follows: "The following people (the names of the participants identified by the face recognition algorithm) attend the meeting, who is speaking in the meeting, how is the atmosphere of the meeting...", and the content of the meeting scene is described in the prompt word.

[0055] In another embodiment, the prompt word library can include prompt words for summarizing the content of the meeting. When a person finishes speaking, the content of his speech is summarized. For example, "xxx has finished speaking, please summarize his speech content, including which questions are raised, which solutions are provided for the questions raised in the previous speech, and the summary of the speech content...". Then the text recognized by the speech recognition (all the content spoken by the speaker at that time) + prompt word 2 + historical meeting records are sent to the multi-modal large model to summarize the content of the speech.

[0056] In yet another embodiment, the prompt word library can include content recording prompt words for communication with the other party after the call: "xxx called xxx in the meeting, please summarize the communication content, including but not limited to: the caller, the called person, the problem discussed by the two parties, the result of the discussion, etc.

[0057] Step 104, input the recognition model output result and the target prompt word into the preset multi-modal large model, and the multi-modal large model outputs the structured target meeting information.

[0058] In the embodiments of the present application, the semantic-free recognition model output result and the target prompt word are combined, and the meeting information can also be combined and input to the multi-modal large model. The multi-modal large model has strong natural language understanding and generation capabilities, and can digest these information and output structured, easy-to-understand and use target meeting information.

[0059] In one embodiment, a post-processing module can also be set up to adjust the output results of the multi-modal large model according to the actual application scenario, such as outputting formatted text or pictures with highlighted markers.

[0060] The conference information output method disclosed in the above embodiments of the application first performs intent analysis on the conference live information in the case of receiving the conference live information, determines the user request type, then inputs the conference live information into a preset recognition model corresponding to the user request type, and the preset recognition model outputs a recognition model output result without semantics, then according to the user request type and the preset recognition model, a target prompt word is matched from a preset dynamic prompt word library, and finally the recognition model output result and the target prompt word are input into a preset multi-modal large model, and the multi-modal large model outputs structured target conference information. Thus, by selecting a suitable preset recognition model through intent analysis, and combining the prompt word library to let the multi-modal large model output structured information, the precision problem when directly using the multi-modal large model to process vertical field tasks can be avoided, and the multi-modal large model does not need to be finely adjusted, saving costs, and thus improving the effect of conference information output.

[0061] In order to facilitate understanding of the above embodiments, Figure 2 A conference scene flow schematic diagram provided by an embodiment of the application is shown. As shown in Figure 2 According to the intent recognition of the conference live text information, which information the user is interested in is recognized. Then different recognition models are called, different recognition models output different results, the output results of different recognition models are combined with the corresponding prompt words in the prompt word library and the prompt word application skills (use methods of various existing prompt words), and then the conference live text information and the conference live image information are input into the multi-modal large model. The multi-modal large model combines the conference live image information and the accurate output results of each recognition model, outputs the information the user is interested in, and after post-processing, the target conference information is obtained, improving the user interaction ability.

[0062] The following introduces several specific examples of conference information output methods.

[0063] In one embodiment, referring to the conference scene flow schematic diagram as shown in Figure 3 As shown in Figure 3 The conference live image is input into a face detection model, the face detection model outputs the position information of each participant in the conference live image, and the conference live image and the position information of each participant are input into a face recognition model, and the face recognition model outputs the identity information of each participant. Thus, the automatic processing of the conference live image is realized, and the identity of each participant is accurately positioned and recognized.

[0064] Furthermore, the location information, identity information, and target cue words of each participant can be input into the multimodal big data model. The multimodal big data model outputs participant information and the current speaker information. Thus, by utilizing the reasoning capabilities of the multimodal big data model, the original identity and location information can be transformed into semantic meeting state information (e.g., who is speaking).

[0065] Specifically, a face detection model can be used to detect the coordinates of faces in the meeting room, and then the corresponding face images can be transmitted to a face recognition model to identify the attendees' names, employee numbers, and other identity information. The recognition results are then combined with prompts and input into a multimodal large model, which can output attendee information, whether someone has spoken, the current speaker's information, and the situation in the meeting.

[0066] In another embodiment, see as follows Figure 4 The diagram illustrates the meeting scenario flow. Figure 4 As shown, meeting information is input into a preset recognition model corresponding to the user request type. The preset recognition model outputs a semantically neutral result, including: inputting meeting document images into an optical character recognition model, which outputs the meeting document text; and inputting meeting audio data into a speech recognition model, which outputs the meeting audio text. This approach transforms meeting document images and audio data into data that can be further processed by a multimodal large-scale model.

[0067] Furthermore, the meeting audio data, meeting transcripts, and target cue words are input into a multimodal big data model. The model then outputs meeting minutes, a summary of the minutes, and discussion points. Thus, the multimodal big data model is used to deeply process the meeting content, extracting valuable meeting minutes, summaries, and viewpoints from the multimodal information.

[0068] Specifically, it can combine OCR-related algorithms to summarize the meeting transcripts used in the meeting (such as the content on a PPT); and combine speech recognition algorithms to identify the speakers' speech content and output a meeting summary, discussion points, etc. for the current time period.

[0069] In yet another embodiment, see as follows: Figure 5 The diagram illustrates the meeting scenario flow. Figure 5 As shown, the meeting scene images are input into a face recognition model, which outputs the identity information of each participant; alternatively, the meeting audio data is input into a voiceprint recognition model, which outputs the identity information of each participant. The above technical solution provides a method for identity verification using two different modalities: face or voiceprint.

[0070] Further, taking the intelligent call function as an example, the conference assistant not only can play the role of "conference secretary", but also can discuss and contact outside the meeting, communicate across departments in the meeting, and improve the efficiency of multi-department cooperation. That is, the identity information of each participant includes the identity information of the communication initiator, the conference scene image, the conference audio data, the identity information of the communication initiator, and the target prompt word are input into the multi-modal large model, and the multi-modal large model outputs the target call request, wherein the target call request includes the identity information of the communication initiator and the identity information of the communication receiver. Therefore, the intelligent call function is realized by combining face recognition, voiceprint recognition and multi-modal large model, and the generated target call request can identify who initiates the call and the called person can know who initiates the call.

[0071] Specifically, the call function can be realized by combining a voice recognition algorithm, a face recognition algorithm, a multi-modal large model, and a preset address book, etc., to determine who calls who and realize two-way identity confirmation. Of course, a voiceprint recognition algorithm can also be used to confirm the identity. In the conference scenario, when it is necessary to contact personnel outside the meeting, the conference assistant can be told the contact person, such as "help me find Wang XX". The conference assistant will first identify who initiates the call and use the meeting address book to contact the person to be called, and the called person can know who initiates the call. In addition, the conference assistant can participate in the discussion of conference topics by combining a private knowledge base, such as: "Xiaowang" is the name of the conference assistant, "Xiaowang, we discuss the product function of the company, which products does the company have, and please explain the comparison." "Xiaowang, to realize the scheme discussed just now, which technologies does the company reserve, and which departments need to be connected?" "Xiaowang, help me contact the technical responsible person of xxx product, I want to consult the feasibility of the technical scheme." and the like.

[0072] Referring to Figure 6 , a structure schematic diagram of a conference information output device provided by an embodiment of the present application is shown, only the parts related to the embodiments of the present application are shown for ease of description.

[0073] The conference information output device can specifically include the following modules:

[0074] The intent analysis module 601 is configured to, in the case of receiving the conference scene information, perform intent analysis on the conference scene information to determine the user request type.

[0075] The first model calling module 602 is configured to input the conference scene information into a preset recognition model corresponding to the user request type, and the preset recognition model outputs a semantic-free recognition model output result.

[0076] The prompt word matching module 603 is configured to match a target prompt word from a preset dynamic prompt word library according to the user request type and the preset recognition model.

[0077] The second model calling module 604 is configured to input the recognition model output result and the target prompt word into a preset multi-modal large model, and the multi-modal large model outputs structured target conference information.

[0078] The conference information output device disclosed in the above embodiments of the present application first performs intent analysis on the conference site information to determine the user request type when the conference site information is received. Then, the conference site information is input into a preset recognition model corresponding to the user request type, and the preset recognition model outputs a recognition model output result without semantics. Next, the user request type and the preset recognition model are used to match a target prompt word from a preset dynamic prompt word library. Finally, the recognition model output result and the target prompt word are input into a preset multi-modal large model, and the multi-modal large model outputs structured target conference information. In this way, by selecting a suitable preset recognition model through intent analysis and combining the prompt word library, the multi-modal large model outputs structured information, which can avoid the precision problem when directly using the multi-modal large model to process vertical field tasks, and does not need to finely adjust the multi-modal large model, thereby saving costs and improving the conference information output effect.

[0079] Further, in a possible implementation manner of the embodiments of the present application, the conference site information includes conference site images, and the preset recognition model includes a face detection model and a face recognition model. The first model calling module 602 can specifically include the following units:

[0080] The first model calling unit is configured to input the conference site images into the face detection model, and the face detection model outputs position information of each participant in the conference site images.

[0081] The second model calling unit is configured to input the conference site images and the position information of each participant into the face recognition model, and the face recognition model outputs identity information of each participant.

[0082] In this way, the conference site images are automatically processed, and the identity of each participant is accurately positioned and recognized.

[0083] Further, in another possible implementation manner of the embodiments of the present application, the target conference information includes participant information and current speaker information. The second model calling module 604 can specifically include the following units:

[0084] The third model calling unit is configured to input the position information of each participant, the identity information of each participant, and the target prompt word into the multi-modal large model, and the multi-modal large model outputs the participant information and the current speaker information.

[0085] Thus, the original identity information and location information are converted into conference state information (e.g., who is speaking) with semantics by using the inference capability of the multi-modal large model.

[0086] Further, in another possible implementation manner of the embodiment of the present application, the conference site information includes conference script images and conference audio data, the preset recognition model includes an OCR model and a speech recognition model, and the first model calling module 602 can specifically include the following units:

[0087] A fourth model calling unit is configured to input the conference script images into an optical character recognition model, and the optical character recognition model outputs conference script texts.

[0088] A fifth model calling unit is configured to input the conference audio data into a speech recognition model, and the speech recognition model outputs conference audio texts.

[0089] Through the above scheme, the conference script images and the conference audio data in the conference are converted into data that can be further processed by the multi-modal large model.

[0090] Further, in another possible implementation manner of the embodiment of the present application, the target conference information includes conference speech records, conference speech summaries, and conference discussion viewpoints, and the second model calling module 604 can specifically include the following units:

[0091] A sixth model calling unit is configured to input the conference audio data, the conference script texts, the conference audio texts, and the target prompt word into the multi-modal large model, and the multi-modal large model outputs the conference speech records, the conference speech summaries, and the conference discussion viewpoints.

[0092] Thus, the conference content is deeply processed by using the multi-modal large model, and valuable conference records, summaries, and viewpoints are extracted from the multi-modal information.

[0093] Further, in another possible implementation manner of the embodiment of the present application, the conference site information includes conference site images or conference audio data, in the case where the conference site information includes the conference site images, the preset recognition model includes a face recognition model, and in the case where the conference site information includes the conference audio data, the preset recognition model includes a voiceprint recognition model, and the first model calling module 602 can specifically include the following units:

[0094] A seventh model calling unit is configured to input the conference site images into the face recognition model, and the face recognition model outputs identity information of each participant; or input the conference audio data into the voiceprint recognition model, and the voiceprint recognition model outputs the identity information of each participant.

[0095] The above technical solution provides a scheme of identity verification through two different modalities of face or voiceprint.

[0096] Further, in another possible implementation of the embodiment of the application, the identity information of each participant includes identity information of the communication initiator, and the target conference information includes a target call request. The second model calling module 604 can specifically include the following units:

[0097] An eighth model calling unit is configured to input the conference live image, the conference audio data, the identity information of the communication initiator, and the target prompt word into a multi-modal large model, and output a target call request from the multi-modal large model, where the target call request includes the identity information of the communication initiator and the identity information of the communication receiver.

[0098] Therefore, the intelligent call function is realized by combining face recognition, voiceprint recognition, and a multi-modal large model, and the generated target call request can identify who initiates the call, so that the called person can know who initiates the call.

[0099] The conference information output apparatus provided in the embodiment of the application can be applied in the foregoing method embodiment, and details are referred to the description of the method embodiment, which will not be repeated here.

[0100] Figure 7 is a structural schematic diagram of a terminal device provided in the embodiment of the application. As shown in the Figure 7 embodiment, the terminal device 700 includes at least one processor 710 (only one processor is shown in the Figure 7 embodiment), a memory 720, and a computer program 721 stored in the memory 720 and executable on the at least one processor 710. The processor 710 implements the steps in the conference information output method embodiments described above when executing the computer program 721.

[0101] The terminal device 700 can be a desktop computer, a notebook computer, a palm computer, a cloud server, and other computing devices. The terminal device can include, but is not limited to, the processor 710 and the memory 720. Those skilled in the art can understand that Figure 7 The terminal device 700 is only an example and does not limit the terminal device 700, which can include more or fewer components than shown, or combine certain components, or include different components, for example, can also include an input / output device, a network access device, and the like.

[0102] The processor 710 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0103] The memory 720 can be an internal storage unit of the terminal device 700 in some embodiments, for example, a hard disk or a memory of the terminal device 700. The memory 720 can also be an external storage device of the terminal device 700 in other embodiments, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 720 can include both an internal storage unit and an external storage device of the terminal device 700. The memory 720 is used to store an operating system, application programs, a boot loader, data, and other programs, for example, program codes of the computer program, etc. The memory 720 can also be used to temporarily store data that has been output or will be output.

[0104] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the above-described functions. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0105] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in a certain embodiment can be referred to the relevant description of other embodiments.

[0106] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0107] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are only schematic, and the division of the modules or units is only a logical function division, and there can be another division in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or in other forms.

[0108] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place, or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0109] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0110] The integrated module / unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can implement the steps of each method embodiment described above when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer readable medium can include any entity or device, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. that can carry the computer program code. It should be noted that the contents included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0111] The above-mentioned embodiment methods can also be completed by a computer program product, which, when running on a terminal device, causes the terminal device to execute the steps in the above-mentioned various method embodiments.

[0112] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, rather than limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A conference information output method characterized by comprising: The method comprises the following steps: In the case of receiving conference site information, performing intent analysis on the conference site information to determine the user request type; Input the conference site information into a preset recognition model corresponding to the user request type, and the preset recognition model outputs a semantic-free recognition model output result; According to the user request type and the preset recognition model, a target prompt word is matched from a preset dynamic prompt word library; Input the recognition model output result and the target prompt word into a preset multi-modal large model, and the multi-modal large model outputs structured target conference information.

2. The method of claim 1, wherein, The conference site information includes conference site images, and the preset recognition model includes a face detection model and a face recognition model. The conference site information is input into the preset recognition model corresponding to the user request type, and the preset recognition model outputs a semantic-free recognition model output result, which comprises: Input the conference site image into the face detection model, and the face detection model outputs the position information of each participant in the conference site image; Input the conference site image and the position information of each participant into the face recognition model, and the face recognition model outputs the identity information of each participant.

3. The method of claim 2, wherein, The target conference information includes participant information and current speaker information. The recognition model output result and the target prompt word are input into a preset multi-modal large model, and the multi-modal large model outputs structured target conference information, which comprises: Input the position information of each participant, the identity information of each participant, and the target prompt word into the multi-modal large model, and the multi-modal large model outputs participant information and current speaker information.

4. The method of claim 1, wherein, The conference site information includes conference document images and conference audio data, and the preset recognition model includes an optical character recognition model and a speech recognition model. The conference site information is input into the preset recognition model corresponding to the user request type, and the preset recognition model outputs a semantic-free recognition model output result, which comprises: Input the conference document image into the optical character recognition model, and the optical character recognition model outputs conference document text; Input the conference audio data into the speech recognition model, and the speech recognition model outputs conference audio text.

5. The method of claim 4, wherein, The target conference information includes conference speech records, conference speech summaries, and conference discussion points. The recognition model output result and the target prompt word are input into a preset multi-modal large model, and the multi-modal large model outputs structured target conference information, which comprises: Input the conference audio data, the conference document text, the conference audio text, and the target prompt word into the multi-modal large model, and the multi-modal large model outputs the conference speech records, conference speech summaries, and conference discussion points.

6. The method of claim 1, wherein, The conference site information includes conference site images or conference audio data, in the case of the conference site information including the conference site images, the preset recognition model includes a face recognition model, in the case of the conference site information including the conference audio data, the preset recognition model includes a voiceprint recognition model, the conference site information is input into the preset recognition model corresponding to the user request type, the preset recognition model outputs a semantic-free recognition model output result, including: The conference site images are input into the face recognition model, and the face recognition model outputs identity information of each participant; or, The conference audio data is input into the voiceprint recognition model, and the voiceprint recognition model outputs the identity information of each participant.

7. The method of claim 6, wherein, The identity information of each participant includes identity information of a communication initiator, the target conference information includes a target call request, the recognition model output result and the target prompt word are input into a preset multi-modal large model, and the multi-modal large model outputs structured target conference information, including: The conference site images, the conference audio data, the identity information of the communication initiator and the target prompt word are input into the multi-modal large model, and the multi-modal large model outputs a target call request, wherein the target call request includes the identity information of the communication initiator and identity information of a communication receiver.

8. A conference information output apparatus characterized by comprising: Including: An intent analysis module is configured to, in the case of receiving conference site information, perform intent analysis on the conference site information to determine a user request type; A first model calling module is configured to input the conference site information into a preset recognition model corresponding to the user request type, and the preset recognition model outputs a semantic-free recognition model output result; A prompt word matching module is configured to, according to the user request type and the preset recognition model, match a target prompt word from a preset dynamic prompt word library; A second model calling module is configured to input the recognition model output result and the target prompt word into a preset multi-modal large model, and the multi-modal large model outputs structured target conference information.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the method of any one of claims 1 to 7.

Images