Voice and character display method and device for video conference

Abstract

The embodiment of the invention discloses a voice and character display method and device for a video conference. The specific implementation mode of the method comprises the steps that a hash value set is generated according to personal audio data of participants, and one audio frame in the personal audio data generates one hash value; voice recognition is carried out on the personal audio data of the participants, voice recognition results of the participants are obtained, and the voice recognition results comprise the starting time and the ending time of recognized statements; the starting time and the ending time of the audio frames corresponding to the hash values in the hash value set are inquired in a hash time table, and the hash time table is generated according to the multi-person mixed audio frames in the video conference data; calibrating the starting time and the ending time of a corresponding statement in the voice recognition result according to the starting time and the ending time of the queried audio frame to obtain a conference record; and combining and displaying the video conference data and the conference record. According to the embodiment, the problem that a voiceprint system needs to train and recognize participant user information is solved, and the time of characters transferred by a voice-to-character system is consistent with the time in a mixed stream video.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method and device for displaying voice and text in a video conference. Background Art

[0002] In the existing services, the voice information recorded by participants in the video conferencing system during the meeting is presented in text form after the meeting by extracting the voice file wav from the recorded mixed video file. The meeting minutes service calls the AI ​​speech-to-text system function for recognition, and the AI ​​needs to turn on the role separation function to identify multiple speakers in the voice file. If the identity of each speaker needs to be accurately identified, the voiceprint recognition system must be trained in advance.

[0003] If the text display of conference records is served based on the existing technical framework, the role separation function of the AI ​​system used in the conference record service will be very demanding, and the effect is not expected. If there are too many participants, in medium or large video conferences, there are often dozens or hundreds of participants. The more speakers there are, the more roles need to be separated, and the higher the requirements for the subsequent AI speech-to-text system. Usually, the role separation capacity of a single audio of the AI ​​speech-to-text system is dozens, which can easily trigger its upper limit.

[0004] In addition, all possible participants need to undergo voiceprint recognition training before they can be separated and identified. For example, training is required before speaker A can be identified as an employee through voiceprint recognition after the roles are separated. The accuracy of this solution cannot reach 100% and its feasibility is relatively poor. Summary of the invention

[0005] The embodiments of the present disclosure provide a method and device for displaying voice and text in a video conference.

[0006] In a first aspect, an embodiment of the present disclosure provides a method for displaying voice and text in a video conference, comprising: generating a hash value set based on personal audio data of a participant, wherein a hash value is generated for an audio frame in the personal audio data; performing voice recognition on the personal audio data of the participant to obtain a voice recognition result of the participant, wherein the voice recognition result includes a start time and an end time of a recognized sentence; querying a hash timetable for the start time and the end time of an audio frame corresponding to a hash value in the hash value set, wherein the hash timetable is generated based on audio frames mixed by multiple people in the video conference data; calibrating the start time and the end time of the corresponding sentence in the voice recognition result according to the start time and the end time of the queried audio frame to obtain a meeting record; and displaying the video conference data together with the meeting record.

[0007] In some embodiments, generating a hash value set based on the personal audio data of the participant includes: generating hash values ​​for audio frames in the personal audio data of the participant using an MD5 algorithm to obtain a hash value set.

[0008] In some embodiments, generating a hash value set based on the personal audio data of the attendees includes: in response to detecting the personal audio data of the attendees from multiple clients, generating hash values ​​for the audio frames in the personal audio data of the attendees using the MD5 algorithm, and obtaining a hash value set of the attendees corresponding to the client identifiers.

[0009] In some embodiments, the method also includes: generating a hash value based on a pulse code modulation audio frame in the video conference data; performing multi-channel audio mixing on the pulse code modulation audio frame to generate an advanced audio coding audio frame; and generating a hash timetable based on a mapping relationship between the advanced audio coding audio frame and the hash value, and the start time and end time of the advanced audio coding audio frame.

[0010] In some embodiments, the method further includes: in response to detecting that the video conference data comes from multiple clients, adding client identifications to the hash timetable.

[0011] In some embodiments, the displaying of the video conference data and the conference record together includes: obtaining the participant's attendance information and adding it to the conference record; and outputting the conference record while playing the video conference data.

[0012] In a second aspect, an embodiment of the present disclosure provides a voice and text display device for a video conference, comprising: a generation unit, configured to generate a hash value set based on personal audio data of a participant, wherein one audio frame in the personal audio data generates a hash value; a recognition unit, configured to perform voice recognition on the personal audio data of the participant to obtain a voice recognition result of the participant, wherein the voice recognition result includes the start time and end time of the recognized sentence; a query unit, configured to query the start time and end time of the audio frame corresponding to the hash value in the hash value set in a hash time table, wherein the hash time table is generated based on audio frames mixed by multiple people in the video conference data; a calibration unit, configured to calibrate the start time and end time of the corresponding sentence in the voice recognition result according to the start time and end time of the queried audio frame to obtain a meeting record; and a display unit, configured to display the video conference data and the meeting record together.

[0013] In some embodiments, the generating unit is further configured to: generate hash values ​​for audio frames in the personal audio data of the participant by using the MD5 algorithm to obtain a hash value set.

[0014] In some embodiments, the generation unit is further configured to: in response to detecting personal audio data of participants from multiple clients, generate hash values ​​for audio frames in the personal audio data of the participants using the MD5 algorithm to obtain a set of hash values ​​of the participants corresponding to the client identifiers.

[0015] In some embodiments, the generation unit is further configured to: generate a hash value based on a pulse code modulated audio frame in the video conferencing data; perform multi-channel audio mixing on the pulse code modulated audio frame to generate an advanced audio coding audio frame; and generate a hash timetable based on a mapping relationship between the advanced audio coding audio frame and the hash value, and the start time and end time of the advanced audio coding audio frame.

[0016] In some embodiments, the generating unit is further configured to: in response to detecting that the video conference data comes from multiple clients, add a client identification to the hash timetable.

[0017] In some embodiments, the display unit is further configured to: obtain the attendance information of the participants and add it to the meeting record; and output the meeting record while playing the video conference data.

[0018] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device on which one or more computer programs are stored, and when the one or more computer programs are executed by the one or more processors, the one or more processors implement a method as described in any one of the first aspects.

[0019] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of the first aspects.

[0020] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which implements the method as described in any one of the first aspects when executed by a processor.

[0021] The embodiment of the present disclosure provides a method and device for displaying voice and text in a video conference. By recording each participant's audio stream separately, the audio stream in the video conference is received and recorded separately according to the participant's peerId (client identifier). First, the limiting bottleneck of role separation in the voice-to-text system is solved (the audio stream pulled from the RTC worker (real-time communication thread) carries peerId information, which can be used to distinguish participants); the separately recorded participant's peerId is used to query the participant's user information registered in the existing conference control service to output the participant's identity, solving the problem of requiring voiceprint system training and identification of participant user information. Since the mixed-stream video recording service and the meeting minutes service are two separate services, the audio data recorded by the two are inconsistent. When displaying, the text generated by the meeting minutes service needs to correspond one-to-one with the time in the mixed-stream video service, and timestamp calibration is performed to achieve the consistency between the text time transcribed by the voice-to-text system (asr) and the time in the mixed-stream video.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0024] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0025] Figure 2 is a flow chart of an embodiment of a method for displaying voice and text in a video conference according to the present disclosure;

[0026] Figure 3 is a schematic diagram of an application scenario of the method for displaying speech and text in a video conference according to the present disclosure;

[0027] Figure 4 is a flow chart of another embodiment of the method for displaying voice and text in a video conference according to the present disclosure;

[0028] Figure 5 is a structural schematic diagram of an embodiment of a voice and text display device for a video conference according to the present disclosure;

[0029] Figure 6 It is a structural diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0031] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0032] Figure 1 An exemplary system architecture of an embodiment of a method for displaying speech and text in a video conference or a device for displaying speech and text in a video conference disclosed in the present invention is shown.

[0033] like Figure 1 As shown, the system architecture may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0034] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as video conferencing applications, 3D video players, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0035] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices that support video conferencing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) players, laptop computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0036] Server 105 can be a server that provides video conferencing services. First, establish a UDP connection with RTC and pull the audio stream of the video conference room. Record the audio frame information received for each peerId, generate the pcm file and hash file of the peerId, and upload them. Identify the pcm file of each peerId to obtain the original peer.asr file. Calibrate the audio frame hash time information of each peer.asr, peer.hash and the meet.mp4 generated by the mixed stream to obtain the calibrated peer.asr file. Merge the peer.asr files of all participants to obtain the meet.asr file of the entire meeting. Obtain the user participation information corresponding to each peerId from the user service, such as name, department, etc., and update it to the meet.asr file. Upload the meet.asr file to the object storage service for comprehensive display at the front desk.

[0037] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, multiple software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0038] It should be noted that the method for displaying voice and text in a video conference provided in the embodiments of the present disclosure is generally executed by the server 105 , and accordingly, the device for displaying voice and text in a video conference is generally disposed in the server 105 .

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0040] Continue to refer Figure 2 , shows a process 200 of an embodiment of a method for displaying voice and text in a video conference according to the present disclosure. The method for displaying voice and text in a video conference includes the following steps:

[0041] Step 201, generating a hash value set based on the personal audio data of the participant.

[0042] In this embodiment, the execution subject of the method for displaying voice and text in a video conference (for example Figure 1The server shown in the figure can receive personal audio data sent by the participant through the client through a wired connection or a wireless connection. Each participant has a peerId to identify his / her identity. The audio data can be converted into a hash value through an audio fingerprint algorithm. For example, a section of audio is first preprocessed, then framed and converted, and then features are extracted, such as the most classic peak feature. It is then hashed and queried through a hash table. Among them, an audio frame in the personal audio data generates a hash value.

[0043] Step 202: Perform speech recognition on the participant's personal audio data to obtain the participant's speech recognition result.

[0044] In this embodiment, the speech is converted into text by using existing speech recognition technology, wherein the speech recognition result includes the start time and end time of the recognized sentence.

[0045] Step 203: Search the hash time table for the start time and end time of the audio frame corresponding to the hash value in the hash value set.

[0046] In this embodiment, the hash timetable is generated based on the audio frames of multiple people mixed in the video conference data. The method of generating hash values ​​is the same as that of personal audio data. Therefore, the audio data of the same person will generate the same hash value. The hash timetable includes not only the hash value of the audio, but also the start time and end time of the audio and the client identifier. The hash value of the audio frame in the mixed video conference data can be detected by a custom detection tool.

[0047] The hash value generated by the personal audio data is in the hash table generated by the audio frames mixed by multiple people in the video conference data. By looking up the table, the start time and end time of the personal audio frame in the video conference data can be found.

[0048] Step 204, calibrate the start time and end time of the corresponding sentence in the speech recognition result according to the start time and end time of the queried audio frame to obtain the meeting record.

[0049] In this embodiment, since comfort noise is added during the mixing process, the start time and end time of the audio frame in the video conference data are different from those in the personal audio data, so the start time and end time in the speech recognition result need to be calibrated so that it can match the video and audio data of the video conference. The calibrated speech recognition result can be used as the meeting record.

[0050] Step 205: Combine the video conference data and the conference record for display.

[0051] In this embodiment, the meeting record can be displayed while the video is being played. The user can jump to the video by clicking on the content of the meeting record.

[0052] The method provided by the above-mentioned embodiment of the present disclosure upgrades the meeting minutes service to separately record the audio stream information of each participant, generate a separate audio file and an audio frame hash file, and send the audio files recorded by the audio stream information of each participant to the AI ​​speech-to-text system to obtain the text information of each participant, wait for the mixed stream service to generate the video file of the meeting and extract the hash timetable of all mixed audios therefrom, for the text information of each participant, combined with the audio frame hash file, obtain the time point corresponding to each sentence in the text information in the mixed stream video file from the hash timetable, finally merge all the calibrated text information files and obtain the user information of the participants from the user service, generate the text information file of the entire meeting, and display it with the video file of the meeting to achieve the expected effect.

[0053] In some optional implementations of this embodiment, generating a hash value set based on the personal audio data of the participant includes: generating a hash value for the audio frame in the personal audio data of the participant using the MD5 algorithm to obtain a hash value set. Generating a hash value by encrypting the MD5 algorithm can generate a hash value conveniently and quickly.

[0054] In some optional implementations of this embodiment, generating a hash value set based on the personal audio data of the participant includes: in response to detecting the personal audio data of the participant from multiple clients, generating a hash value for the audio frame in the personal audio data of the participant using the MD5 algorithm to obtain a hash value set of the participant corresponding to the client identifier. If there are multiple participants, a corresponding hash value is generated for each participant and distinguished by the client identifier, which facilitates subsequent query retrieval.

[0055] In some optional implementations of this embodiment, the method further includes: generating a hash value based on a pulse code modulation audio frame in the video conference data; performing multi-channel audio mixing on the pulse code modulation audio frame to generate an advanced audio coding audio frame; generating a hash timetable based on a mapping relationship between the advanced audio coding audio frame and the hash value, and the start time and end time of the advanced audio coding audio frame. The original audio frame is in PCM (pulse code modulation) format, but after mixing, it is compressed into AAC (advanced audio coding) format. Therefore, it is necessary to map the hash value generated in PCM format to AAC format. This ensures that the hash timetable reflects the correspondence between the time of the AAC audio frame and the hash value.

[0056] In some optional implementations of this embodiment, the method further includes: in response to detecting that the video conference data comes from multiple clients, adding a client identifier to the hash timetable. If there are participants, a corresponding hash value is generated for each participant and distinguished by the client identifier, which facilitates subsequent query retrieval.

[0057] Continue to see Figure 3 , Figure 3 This is a schematic diagram of an application scenario of the method for displaying voice and text in a video conference according to this embodiment. The specific process is as follows:

[0058] 1. In the mixed streaming service, the audio frame data (with comfort noise) and hash value pulled from the RTC worker are written into the aac audio frame when mixing multiple audio channels. Then, each frame of audio data in the meet.mp4 file of the video conference data finally recorded and generated will contain one or more hash values.

[0059] 2. In the meeting minutes service, the audio frame data and hash value pulled from the RTC worker are dumped locally to generate peer.hash and peer.pcm files. The pcm file is sent to AI to generate the speech recognition result peer.asr file, which records the start and end time points of each recognized sentence (relative to the peer.pcm audio file).

[0060] 3. Use a custom detection tool to detect the hash values ​​of all audio frames in the meet.mp4 file recorded by the mixing service and generate audio frame information audio.frameInfo.

[0061] 4. Find the corresponding hash value from peer.hash for the starting time point of each sentence in peer.asr, find the time information of the meet.mp4 file corresponding to the hash value from audio.frameInfo obtained in step 3, and update it to the peer.asr file, and finally get the calibrated peer.asr file.

[0062] 5. Merge the calibrated peer.asr files of all participants to generate the final meet.asr file of the entire meeting for front-end display.

[0063] Further references Figure 4 , which shows a process 400 of another embodiment of a method for displaying voice and text in a video conference. The process 400 of the method for displaying voice and text in a video conference includes the following steps:

[0064] Step 401, generating a hash value set based on the personal audio data of the participant, wherein a hash value is generated for one audio frame in the personal audio data.

[0065] Step 402: Perform speech recognition on the participant's personal audio data to obtain the participant's speech recognition result.

[0066] Step 403: Search the hash time table for the start time and end time of the audio frame corresponding to the hash value in the hash value set.

[0067] Step 404, calibrate the start time and end time of the corresponding sentence in the speech recognition result according to the start time and end time of the queried audio frame to obtain the meeting record.

[0068] Steps 401-404 are substantially the same as steps 201-204, and thus are not described in detail herein.

[0069] Step 405: Obtain the meeting information of the participants and add it to the meeting record.

[0070] In this embodiment, user conference information corresponding to each client identifier, such as name, department, etc., is obtained from the user service and updated into the conference record.

[0071] Step 406: Output the conference record while playing the video conference data.

[0072] In this embodiment, the conference record including the conference information of the participants is outputted while the video conference data is played, so that it is possible to clearly see who is speaking and to select to jump to the corresponding video clip according to the identity of the participant.

[0073] The meeting minutes service of this application directly pulls the audio stream of each participant from the RTC worker, and no longer relies on the synthesized audio stream recording file in the mixing service, which improves the accuracy of identifying participants and reduces the requirements for the AI ​​system.

[0074] Since the audio stream pulled by the meeting minutes service cannot be completely consistent with the audio stream pulled by the mixed stream service, the audio stream pulled by the mixed stream must have good sound quality, and the audio data contains comfortable noise data passed through Neteq; and the audio data in the audio stream pulled by the meeting minutes service must be complete original data in order not to affect the accuracy of AI recognition. Therefore, the solution of this application can not only accurately identify the information of participants, but also calibrate the mixed stream audio and the meeting minutes asr content, so that the content of the asr file can correspond one to one when the mixed stream video is played.

[0075] Further references Figure 5As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a voice and text display device for a video conference. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0076] like Figure 5 As shown, the speech and text display device 500 for video conference of this embodiment includes: a generating unit 501, a recognizing unit 502, a querying unit 503, a calibrating unit 504 and a displaying unit 505. The generating unit 501 is configured to generate a hash value set according to the personal audio data of the participant, wherein one audio frame in the personal audio data generates a hash value; the recognizing unit 502 is configured to perform speech recognition on the personal audio data of the participant to obtain the speech recognition result of the participant, wherein the speech recognition result includes the start time and end time of the recognized sentence; the querying unit 503 is configured to query the start time and end time of the audio frame corresponding to the hash value in the hash value set in the hash time table, wherein the hash time table is generated according to the mixed audio frames of multiple people in the video conference data; the calibrating unit 504 is configured to calibrate the start time and end time of the corresponding sentence in the speech recognition result according to the start time and end time of the queried audio frame to obtain the conference record; the displaying unit 505 is configured to display the video conference data and the conference record together.

[0077] In this embodiment, the specific processing of the generation unit 501, the recognition unit 502, the query unit 503, the calibration unit 504 and the display unit 505 of the speech and text display device 500 for video conferencing can be referred to in Figure 2 Corresponding to step 201, step 202, step 203, step 204 and step 205 in the embodiment.

[0078] In some optional implementations of this embodiment, the generating unit 501 is further configured to: generate hash values ​​for audio frames in the personal audio data of the participant by using the MD5 algorithm to obtain a hash value set.

[0079] In some optional implementations of this embodiment, the generation unit 501 is further configured to: in response to detecting personal audio data of participants from multiple clients, generate hash values ​​for audio frames in the personal audio data of the participants using the MD5 algorithm, and obtain a set of hash values ​​of the participants corresponding to the client identifiers.

[0080] In some optional implementations of this embodiment, the generation unit 501 is further configured to: generate a hash value based on a pulse code modulation audio frame in the video conference data; perform multi-channel audio mixing on the pulse code modulation audio frame to generate an advanced audio coding audio frame; generate a hash timetable based on the mapping relationship between the advanced audio coding audio frame and the hash value, and the start time and end time of the advanced audio coding audio frame.

[0081] In some optional implementations of this embodiment, the generating unit 501 is further configured to: in response to detecting that the video conference data comes from multiple clients, add a client identifier to the hash timetable.

[0082] In some optional implementations of this embodiment, the display unit 505 is further configured to: obtain the participant's attendance information and add it to the conference record; and output the conference record while playing the video conference data.

[0083] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0084] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0085] An electronic device comprises: one or more processors; a storage device on which one or more computer programs are stored, and when the one or more computer programs are executed by the one or more processors, the one or more processors implement the method described in process 200 or 400.

[0086] A computer-readable medium stores a computer program, wherein the computer program implements the method described in process 200 or 400 when executed by a processor.

[0087] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0088] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0089] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0090] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the road area planning method. For example, in some embodiments, the road area planning method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the road area planning method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the road area planning method in any other appropriate manner (e.g., by means of firmware).

[0091] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0092] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0093] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0094] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0095] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0096] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a server of a distributed system, or a server combined with a blockchain. The server may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The server may be a server of a distributed system, or a server combined with a blockchain. The server may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0097] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0098] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for displaying voice and text in a video conference, comprising: Generate a set of hash values ​​based on the personal audio data of the participants, where Generating a hash value for an audio frame in the personal audio data; Performing speech recognition on the personal audio data of the participant to obtain a speech recognition result of the participant, wherein the speech recognition result includes a start time and an end time of the recognized sentence; Querying the start time and end time of the audio frame corresponding to the hash value in the hash value set in the hash time table, wherein the hash time table is generated according to the audio frames mixed by multiple people in the video conference data; Calibrate the start time and end time of the corresponding sentence in the speech recognition result according to the start time and end time of the queried audio frame to obtain the meeting record; The video conference data is combined with the conference record for display.

2. The method according to claim 1, wherein: The generating of a hash value set according to the personal audio data of the participant includes: A hash value is generated for the audio frame in the personal audio data of the participant using the MD5 algorithm to obtain a hash value set.

3. The method according to claim 1, wherein: The generating of a hash value set according to the personal audio data of the participant includes: In response to detecting personal audio data of participants from multiple clients, hash values ​​are generated for audio frames in the personal audio data of the participants using an MD5 algorithm to obtain a set of hash values ​​of the participants corresponding to the client identifiers.

4. The method according to claim 1, wherein: The method further comprises: generating a hash value based on a pulse code modulated audio frame in the video conference data; Performing multi-channel audio mixing on the pulse code modulation audio frame to generate an advanced audio coding audio frame; A hash time table is generated according to a mapping relationship between an advanced audio coding audio frame and a hash value, and a start time and an end time of the advanced audio coding audio frame.

5. The method according to claim 4, wherein: The method further comprises: In response to detecting that the video conference data comes from multiple clients, client identifications are added to the hash timetable.

6. The method according to claim 1, wherein: The combining and displaying the video conference data and the conference record includes: Obtaining the meeting information of the participants and adding it to the meeting record; The conference record is outputted while playing the video conference data.

7. A voice and text display device for video conferencing, comprising: A generating unit is configured to generate a hash value set according to the personal audio data of the participant, wherein: Generating a hash value for an audio frame in the personal audio data; A recognition unit configured to perform speech recognition on the personal audio data of the participant to obtain a speech recognition result of the participant, wherein the speech recognition result includes a start time and an end time of the recognized sentence; A query unit configured to query the start time and end time of the audio frame corresponding to the hash value in the hash value set in the hash time table, wherein the hash time table is generated according to the audio frames mixed by multiple people in the video conference data; A calibration unit configured to calibrate the start time and end time of the corresponding sentence in the speech recognition result according to the start time and end time of the queried audio frame to obtain a meeting record; A display unit is configured to display the video conference data and the conference record in combination.

8. An electronic device, comprising: one or more processors; a storage device having one or more computer programs stored thereon, When the one or more computer programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images