Manuscript explanation method, device and equipment and storage medium

Abstract

The invention discloses a manuscript explanation method and device, equipment and a storage medium, and the method comprises the steps: generating an explanation script based on a presentation manuscript, and enabling the explanation script to comprise an explanation text corresponding to the presentation manuscript; video data corresponding to the explanation script are synthesized and played, and audios in the video data are generated based on the explanation text; receiving an explanation adjustment instruction triggered by the user in the process of playing the video data, and determining a target explanation position in the explanation script based on the explanation adjustment instruction; and continuing to play the video data corresponding to the explanation script from the target explanation position. In this way, the flexibility of manuscript explanation and the user experience can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a manuscript explanation method, device, equipment and storage medium. Background Art

[0002] Traditional manuscript explanations usually rely on manual explanations or pre-recorded videos. Although these methods meet the needs of information transmission to a certain extent, they have many shortcomings in terms of flexibility and user experience.

[0003] Among them, although the pre-recorded videos can be played repeatedly, they can only be played in a fixed order and speed, and cannot be adjusted according to user needs. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a method, device, equipment and storage medium for document explanation, which can improve the flexibility of document explanation and user experience.

[0005] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a manuscript explanation method, the method comprising: generating an explanation script based on a presentation manuscript, wherein the explanation script includes an explanation text corresponding to the presentation manuscript; synthesizing and playing video data corresponding to the explanation script, wherein the audio in the video data is generated based on the explanation text; receiving an explanation adjustment instruction triggered by the user during the playback of the video data, and determining a target explanation position in the explanation script based on the explanation adjustment instruction; and continuing to play the video data corresponding to the explanation script from the target explanation position.

[0006] In order to solve the above technical problems, another technical solution adopted by the present application is: to provide a manuscript explanation device, which includes: a script generation module, a first playback module, an explanation position determination module and a second playback module. The script generation module is used to generate an explanation script based on the presentation manuscript, wherein the explanation script includes the explanation text corresponding to the presentation manuscript; the first playback module is used to synthesize and play the video data corresponding to the explanation script, wherein the audio in the video data is generated based on the explanation text; the explanation position determination module is used to receive the explanation adjustment instruction triggered by the user during the playback of the video data, and determine the target explanation position in the explanation script based on the explanation adjustment instruction; the second playback module is used to continue playing the video data corresponding to the explanation script from the target explanation position.

[0007] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising a memory and a processor coupled to each other, the memory storing program instructions; the processor is used to execute the program instructions stored in the memory to implement the above method.

[0008] In order to solve the above technical problem, another technical solution adopted by the present application is: providing a computer-readable storage medium for storing program instructions, which can be executed to implement the above method.

[0009] The above scheme, after generating the explanation script based on the presentation, merges and plays the video data corresponding to the explanation script, and when receiving the explanation adjustment instruction triggered by the user during the video data playback process, determines the target explanation position in the explanation script based on the explanation adjustment instruction; and continues to play the video data corresponding to the explanation script from the target explanation position. It can be seen that in the process of playing video data, the present application can adjust the target explanation position according to demand. Compared with the explanation method of fixing the explanation order according to the explanation script, the present application adjusts the target explanation position based on the user's explanation adjustment instruction, and can adjust the playback according to user needs, greatly improving the flexibility of the explanation and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of an embodiment of a manuscript explanation method provided by the present application;

[0011] Figure 2 yes Figure 1 The flowchart of step S13 is shown as an embodiment;

[0012] Figure 3 It is a schematic diagram of an embodiment of determining a target explanation position based on an interruption position provided by the present application;

[0013] Figure 4 It is a schematic diagram of the framework of an embodiment of a document explanation device provided by the present application;

[0014] Figure 5 It is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application;

[0015] Figure 6 It is a schematic diagram of the framework of the computer-readable storage medium provided by this application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail below with reference to the accompanying drawings and examples.

[0017] In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0018] See also Figure 1 , Figure 11 is a flow chart of an embodiment of the manuscript explanation method provided by the present application. It should be noted that if there are substantially the same results, this embodiment is not used. Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:

[0019] S11: Generate an explanation script based on the presentation, wherein the explanation script includes an explanation text corresponding to the presentation.

[0020] This embodiment is used to adjust the playback of video data corresponding to the explanation script based on the explanation adjustment instruction triggered by the user during the playback of video data, wherein the video data is the video data corresponding to the explanation script generated based on the presentation manuscript.

[0021] Presentations are multimedia files used to present information, ideas, data or teaching content to users. These files are organized in the form of document units, and each document unit may contain rich multimedia elements, such as text, pictures, charts, hyperlinks or animations.

[0022] Exemplarily, the presentation may include, but is not limited to, a PPT file, and may also be a PDF file, etc.

[0023] Step S11 is used to generate a script corresponding to the existing presentation based on the content of the presentation, which is used to explain the content of the presentation. The generated explanation script includes the explanation text corresponding to the presentation. The explanation text is the text content to be explained to the user by voice.

[0024] In some embodiments, the presentation document can be divided into multiple document units, wherein the document unit can be, but is not limited to, a page, or a paragraph, etc. The generated explanation script includes explanation information of each document unit in the presentation document, wherein the explanation information of the document unit includes the explanation text corresponding to the document unit.

[0025] In one embodiment, the explanation information of the manuscript unit includes not only the explanation text corresponding to the manuscript unit, but also at least one of the serial number of the manuscript unit, the total playback time, and the acquisition address of the manuscript image corresponding to the manuscript unit.

[0026] Taking the document unit as a page as an example, the serial number of the document unit is used to indicate which page of the presentation document each document unit belongs to.

[0027] The total playback time is used to characterize the total playback time corresponding to the start of playback of the manuscript unit to the completion of playback of the manuscript unit. In one embodiment, the total playback time corresponding to the manuscript unit can be estimated based on the length of the explanation text corresponding to the manuscript unit and the playback speed. In another embodiment, considering that the explanation texts corresponding to different manuscript units have different contents, punctuation and sentence breaks, and the corresponding syllable pronunciation duration and pause duration are different, the total playback time estimated based on the length of the explanation text corresponding to the manuscript unit and the playback speed may have errors, so the total playback time of the explanation text corresponding to the manuscript unit can be calculated by speech pre-synthesis.

[0028] The acquisition address of the document image corresponding to the document unit is the storage address corresponding to the document image, and the document image corresponding to the document unit can be acquired based on the acquisition address of the document image.

[0029] It should be noted that a presentation generally includes explanation texts and manuscript images of each manuscript unit. The explanation texts and manuscript images of each manuscript unit can be stored separately, but in order to ensure that the subsequent explanation and manuscript image presentation are carried out according to the explanation texts of each manuscript unit, the acquisition address of the manuscript image in each manuscript unit and the explanation text can be stored in association, so that the manuscript image corresponding to the manuscript unit can be obtained according to the acquisition address of the manuscript image.

[0030] In a specific embodiment, the presentation is a PPT document, and each slide in the PPT document contains notes and images. The image and notes in each slide can be extracted using relevant technology, and the image and notes are stored in the form of image + text. The notes correspond to the explanation text in each slide.

[0031] S12: synthesize and play the video data corresponding to the explanation script, wherein the audio in the video data is generated based on the explanation text.

[0032] In one embodiment, the video data corresponding to each manuscript unit can be synthesized and played based on the explanation information of each manuscript unit. The video data corresponding to each manuscript unit can be synthesized and played based on the explanation text corresponding to each manuscript unit; in addition to the explanation text corresponding to each manuscript unit, the video data corresponding to each manuscript unit can also be synthesized and played based on at least one of the sequence number of each manuscript unit, the total playback time, and the acquisition address of the manuscript image corresponding to the manuscript unit.

[0033] It should be noted that the video data is speech data expressed by combining audio and images. The video data can be played through a web page sent to the user terminal, that is, the user can open the corresponding web page through the terminal to watch the video data. Of course, it can also be achieved in other ways, such as playing the video data through a related application.

[0034] In some embodiments, the audio of the video data corresponding to the manuscript unit corresponds to the explanation text corresponding to the manuscript unit, that is, the audio in the video data is the audio about the explanation text, that is, the explainer explains the explanation text.

[0035] The image frame of the video data corresponding to the manuscript unit includes at least one of the following: the manuscript image corresponding to the manuscript unit, and the speaker for giving a speech on the explanation text corresponding to the manuscript unit. That is, the image frame of the video data corresponding to the manuscript unit may only include the manuscript image corresponding to the manuscript unit, and the user can only see the manuscript image corresponding to the manuscript unit but not the speaker, or only include the speaker of the speech explanation text, and the user can only see the speaker but not the manuscript image corresponding to the manuscript unit; of course, in order to allow the user to see both the manuscript image corresponding to the manuscript unit and the speaker, the image frame of the video data corresponding to the manuscript unit may include both the manuscript image corresponding to the manuscript unit and the speaker of the speech explanation text.

[0036] In a specific embodiment, the image frame in the video data includes the speaker of the audio in the speech video data, and the video data corresponding to the manuscript unit is the speaker video data, or a combination of the speaker video data and the manuscript image corresponding to the manuscript unit. In this embodiment, based on the explanation text corresponding to the manuscript unit, the speaker video data about the speaker explaining the manuscript unit can be synthesized. Among them, the speaker video data about the speaker explaining the manuscript unit can be synthesized based on the explanation text corresponding to the manuscript unit by voice synthesis; in addition, the speaker can be a digital virtual person with interactive capabilities that is comprehensively generated by multiple technologies such as computer technology, graphics, artificial intelligence, speech synthesis and recognition or natural language processing. The virtual person can simulate human behavior, expression, voice and even emotion to provide a more vivid, personalized and intelligent interactive experience.

[0037] Furthermore, the explanation information also includes an acquisition address of a manuscript image corresponding to the manuscript unit. After the video data corresponding to the manuscript unit is synthesized, the video data corresponding to the manuscript unit can be played in the following manner: first, the explanation video data corresponding to the manuscript unit and the acquisition address of the manuscript image are sent to the user terminal, so that the user terminal decodes and renders the explanation video data; then, the manuscript image corresponding to the manuscript unit is acquired using the acquisition address, and the manuscript image is superimposed on the explanation video data.

[0038] In a specific embodiment, a full-duplex streaming interactive channel can be pre-established, and common streaming media protocols such as rtmp and webrtc can be used to obtain the narrator video data stream in h264 encoding format for efficient video decoding and rendering.

[0039] In one embodiment, the synthesis and playback of the video data corresponding to each manuscript unit are performed in sequence. Specifically, the manuscript units are first synthesized in sequence to obtain the video data corresponding to each manuscript unit, and then the video data corresponding to each manuscript unit is played in sequence; or the video data corresponding to each manuscript unit is synthesized and played in sequence. The video data of the latter manuscript unit is played when the video data of the latter manuscript unit has been synthesized and the video data of the previous manuscript unit has been played.

[0040] Among them, the video data of each manuscript unit can be synthesized and played according to the serial number of each manuscript unit in the presentation manuscript.

[0041] It should be noted that, considering that the target explanation position needs to be determined in the explanation script based on the explanation adjustment instructions triggered by the user during the playback of the video data, and the target explanation position may be located in the currently playing manuscript unit, or may be located in the manuscript unit before or after the currently playing manuscript unit. Therefore, in order to better adjust the playback according to the needs of the user, each manuscript unit can be synthesized and played in real time, so that the currently synthesized and played manuscript unit is the manuscript unit expected by the user.

[0042] Furthermore, in the process of playing the video data corresponding to each manuscript unit in sequence, the video data corresponding to each manuscript unit can be played based on the explanation end signal indicating that the explainer has finished explaining the manuscript unit. Specifically, if the explanation end signal of the explainer in the video data corresponding to the previous manuscript unit is received, and it is determined that the video data corresponding to the next manuscript unit has been synthesized, the video data corresponding to the next manuscript unit is played. That is, this embodiment controls the playback of the video data corresponding to the next manuscript unit based on the explanation end signal of the explainer in the video data corresponding to the previous manuscript unit, so as to achieve continuous playback of each manuscript unit.

[0043] S13: receiving an explanation adjustment instruction triggered by the user during the video data playback process, and determining a target explanation position in the explanation script based on the explanation adjustment instruction.

[0044] S14: Continue playing the video data corresponding to the explanation script from the target explanation position.

[0045] The explanation adjustment instruction in this embodiment can be a voice instruction issued by the user. The voice instruction issued by the user can be recognized to obtain a recognized text, and the corresponding explanation adjustment instruction can be determined based on the recognized text. It can also be a plurality of explanation adjustment instruction options pre-set for user selection. During the process of playing video data, the user's explanation adjustment instruction can be recognized in response to the user's selection operation, wherein the plurality of explanation adjustment instruction options can be determined based on the statistics of the user's adjustment intentions during the explanation process.

[0046] Among them, the explanation adjustment instruction can be a jump instruction based on the document unit, such as "return to the previous page", "next page" or "jump to page n", etc.; the explanation adjustment instruction can also be an explanation interruption instruction that represents the interruption of the current playback. Among them, the explanation interruption instruction is, for example, a pause instruction for pausing the video data being played, and can also be a question instruction when the user wants to interact with the explainer in a question-and-answer session. Of course, the explanation adjustment instruction can also be an explanation resumption instruction that is triggered after the explanation interruption instruction and is used to guide the explainer to resume the explanation.

[0047] In some embodiments, the explanation information also includes the serial number of the manuscript unit, for example, each page of the manuscript is the first page of the explanation script. When playing the video data, the serial number of the manuscript unit currently being played can be recorded as the current playing serial number.

[0048] In one embodiment, when the received explanation adjustment instruction is a jump instruction in the form of a document unit, the current playback sequence number can be obtained first, and then the target jump sequence number is determined based on the jump content indicated by the jump instruction and the current playback sequence number, and used as the target explanation position. Among them, the document unit corresponding to the target jump sequence number is the target document unit, that is, the intended document unit of the user. After the target document unit is determined, the video data of the target document unit and the document units after it are played in sequence.

[0049] Exemplarily, when the user's explanation adjustment instruction is "return to the previous page", the current playback sequence number is first obtained, for example, the fifth page; then based on the jump content indicated by the jump instruction (return to the previous page) and the current playback sequence number (fifth page), the target jump number is determined to be the fourth page, and the fourth page is used as the target explanation position, and then the video data of the fourth page document unit and the subsequent document units are played sequentially.

[0050] In another embodiment, if the jump instruction is "jump to page n", the jump unit indicated by the jump instruction can be directly determined as the target jump sequence number, and the document unit corresponding to the target jump sequence number can be determined as the target document unit, and the video data of the target document unit and the document units after it can be played in sequence.

[0051] In one embodiment, see Figure 2 , Figure 2 yes Figure 1 The flowchart of step S13 of an embodiment is shown. In this embodiment, based on the explanation adjustment instruction, the target explanation position is determined in the explanation script, including:

[0052] S21: In response to the explanation adjustment instruction being an explanation interruption instruction, the total playback time of the current manuscript unit is obtained from the explanation information corresponding to the current manuscript unit, and the recorded playback time of the current manuscript unit is obtained. The current manuscript unit is the manuscript unit played when the user triggers the explanation adjustment instruction.

[0053] This embodiment is for some corresponding processing when the explanation adjustment instruction is an explanation interruption instruction. The explanation information in this embodiment also includes the total playback time of the manuscript unit. Among them, the method for obtaining the total playback time of the manuscript unit can refer to the above description, which will not be repeated here.

[0054] S22: Based on the played time and the total played time of the current manuscript unit, calculate the interruption position in the explanation text corresponding to the current manuscript unit when the user triggers the explanation interruption instruction.

[0055] In this embodiment, the ratio of the product of the text length of the explanation text corresponding to the current manuscript unit and the played time to the total played time can be directly used as the interruption position.

[0056] S23: Determine the target explanation position based on the interruption position.

[0057] In this embodiment, the interruption position can be directly used as the target explanation position. The first punctuation mark before the interruption position can also be found in the explanation text, and the position of the first punctuation mark can be used as the target explanation position; of course, the position of the first character after the first punctuation mark can also be used as the target explanation position. The target explanation position is determined in order to play the video data corresponding to the current manuscript unit and the video data corresponding to the manuscript unit after the current manuscript unit from the target explanation position.

[0058] It should be noted that, considering that the interruption position may be somewhere in the middle of a sentence, if the interruption position is directly used as the target explanation position, and the video data corresponding to the current manuscript unit and the video data corresponding to the manuscript unit after the current manuscript unit are played from the interruption position, the explanation content will appear stiff and unnatural due to the lack of contextual semantics. Therefore, it is preferred to search the explanation text for the first punctuation mark before the interruption position, and use the position of the first punctuation mark or the position of the first character after the first punctuation mark as the target explanation position.

[0059] For example, see Figure 3 , Figure 3 1 is a schematic diagram of an embodiment of determining a target explanation position based on an interruption position provided by the present application. Figure 3As shown, the explanation text of the current document unit includes "the positioning information of the real-time synthetic intelligent interaction platform for virtual humans, oriented to business scenarios such as voice-controlled large screens and virtual human intelligent question and answer, and the integration of virtual humans, voices, and semantics", among which the determined interruption position is the position of "business" in the "business scenario", and the finally determined target explanation position is the first punctuation mark before the interruption position, that is, the position of "," in "the positioning information of the real-time synthetic intelligent interaction platform for virtual humans,"

[0060] In one implementation scenario, the image frame in the video data includes the speaker of the audio in the speech video data, and the explanation interruption instruction triggered by the user is a question instruction, that is, the user triggers the explanation interruption instruction in order to interact with the speaker through questions and answers. In this implementation scenario, after receiving the user's question instruction, the question instruction is first semantically understood to obtain the corresponding question text, and the answer text corresponding to the question text is generated using the preset question and answer module, and the answer audio corresponding to the answer text is obtained, and the corresponding answer audio is fed back to the user using the speaker in the video data. Exemplarily, the speaker is controlled to voice broadcast the answer audio, and the speaker's lips are controlled to change to match the lip shape of the answer audio. Further, some body movements can be used to make the speaker give a more vivid explanation. Among them, the preset question and answer module can be, but is not limited to, a preset knowledge base.

[0061] In one implementation scenario, after the user triggers the explanation interruption instruction, the explanation resumption instruction is triggered. In this case, when the explanation resumption instruction of the user is received, based on the determined target explanation position, the step of playing the video data corresponding to the current manuscript unit and the video data corresponding to the manuscript unit after the current manuscript unit from the target explanation position is executed. The explanation interruption instruction can be a pause instruction or a question instruction.

[0062] In another implementation scenario, the explanation interruption instruction triggered by the user is a question instruction. In this case, the step of playing the video data corresponding to the current manuscript unit and the video data corresponding to the manuscript unit after the current manuscript unit starting from the target explanation position is performed upon receiving the user's explanation resumption instruction or determining that the conversation corresponding to the question instruction has ended.

[0063] That is, in addition to executing the step of playing the video data corresponding to the current manuscript unit and the video data corresponding to the manuscript unit after the current manuscript unit from the target explanation position when receiving the user's explanation resumption instruction, the step of playing the video data corresponding to the current manuscript unit and the video data corresponding to the manuscript unit after the current manuscript unit from the target explanation position can also be executed when it is determined that the dialogue corresponding to the question instruction has ended.

[0064] Exemplarily, if no further questioning instructions are received from the user within a preset time, the dialogue corresponding to the questioning instruction is determined to be over, and the step of playing the video data corresponding to the current manuscript unit and the video data corresponding to the manuscript unit after the current manuscript unit starting from the target explanation position is executed.

[0065] The above scheme, after generating the explanation script based on the presentation, merges and plays the video data corresponding to the explanation script, and when receiving the explanation adjustment instruction triggered by the user during the video data playback process, determines the target explanation position in the explanation script based on the explanation adjustment instruction; and continues to play the video data corresponding to the explanation script from the target explanation position. It can be seen that in the process of playing video data, the present application can adjust the target explanation position according to demand. Compared with the explanation method of fixing the explanation order according to the explanation script, the present application adjusts the target explanation position based on the user's explanation adjustment instruction, and can adjust the playback according to user needs, greatly improving the flexibility of the explanation and user experience.

[0066] See also Figure 4 , Figure 4 It is a schematic diagram of the framework of an embodiment of a document explanation device provided by the present application. In this embodiment, the document explanation device 40 includes a script generation module 41, a first playback module 42, an explanation position determination module 43 and a second playback module 44. The script generation module 41 is used to generate an explanation script based on a presentation, wherein the explanation script includes an explanation text corresponding to the presentation; the first playback module 42 is used to synthesize and play video data corresponding to the explanation script, wherein the audio in the video data is generated based on the explanation text; the explanation position determination module 43 is used to receive an explanation adjustment instruction triggered by a user during the playback of video data, and determine a target explanation position in the explanation script based on the explanation adjustment instruction; the second playback module 44 is used to continue playing the video data corresponding to the explanation script from the target explanation position.

[0067] In some embodiments, the presentation is divided into multiple document units, and the explanation script generated by the script generation module 41 includes explanation information of each document unit, and the explanation information includes the explanation text corresponding to the document unit; the first playback module 42 synthesizes and plays the video data corresponding to the explanation script, including: based on the explanation information of each document unit, synthesizing and playing the video data corresponding to each document unit.

[0068] In some embodiments, the explanation information also includes at least one of the serial number of the manuscript unit, the total playback time, and the acquisition address of the manuscript image corresponding to the manuscript unit; the audio of the video data corresponding to the manuscript unit corresponds to the explanation text corresponding to the manuscript unit, and the image frame of the video data corresponding to the manuscript unit includes at least one of the following: the manuscript image corresponding to the manuscript unit, and the explainer used to give a speech on the explanation text corresponding to the manuscript unit.

[0069] In some embodiments, the explanation information also includes the serial number of the manuscript unit; the method also includes: when playing the video data, recording the serial number of the manuscript unit currently being played as the current playback serial number; the explanation position determination module 43 determines the target explanation position in the explanation script based on the explanation adjustment instruction, including: in response to the explanation adjustment instruction being a jump instruction in units of manuscript units, obtaining the current playback serial number; based on the jump content indicated by the jump instruction and the current playback serial number, determining the target jump serial number as the target explanation position, wherein the manuscript unit corresponding to the target jump serial number is the target manuscript unit; the second playback module 44 continues to play the video data corresponding to the explanation script from the target explanation position, including: playing the video data of the target manuscript unit and the manuscript units thereafter in sequence.

[0070] In some embodiments, the explanation information also includes the total playback time of the manuscript unit; the method also includes: when playing the video data, recording the playback time of the currently played manuscript unit; the explanation position determination module 43 determines the target explanation position in the explanation script based on the explanation adjustment instruction, including: in response to the explanation adjustment instruction being an explanation interruption instruction, obtaining the total playback time of the current manuscript unit from the explanation information corresponding to the current manuscript unit, and obtaining the recorded playback time of the current manuscript unit, the current manuscript unit being the manuscript unit played when the user triggers the explanation adjustment instruction; based on the playback time and total playback time of the current manuscript unit, calculating the interruption position in the explanation text corresponding to the current manuscript unit when the user triggers the explanation interruption instruction; determining the target explanation position based on the interruption position; the second playback module 44 continues to play the video data corresponding to the explanation script from the target explanation position, including: playing the video data corresponding to the current manuscript unit and the video data corresponding to the manuscript unit after the current manuscript unit from the target explanation position.

[0071] In some embodiments, the total playback time is estimated based on the length of the explanation text corresponding to the manuscript unit and the playback speed; and / or, based on the playback time and total playback time of the current manuscript unit, the interruption position in the explanation text corresponding to the current manuscript unit when the user triggers the explanation interruption command is calculated, including: taking the ratio of the product of the text length of the explanation text corresponding to the current manuscript unit and the playback time to the total playback time as the interruption position; and / or, determining the target explanation position based on the interruption position, including: taking the interruption position as the target explanation position; or, searching the explanation text for the first punctuation mark before the interruption position, and taking the position of the first punctuation mark as the target explanation position.

[0072] In some embodiments, the explanation interruption instruction is a pause instruction, and the step of playing the video data corresponding to the current manuscript unit and the video data corresponding to the manuscript unit after the current manuscript unit from the target explanation position is performed when the user's explanation resumption instruction is received; or, the explanation interruption instruction is a question instruction, and the image frame in the video data includes the explainer of the audio in the speech video data; the method also includes: semantically understanding the question instruction to obtain the question text, and using a preset question and answer module to generate an answer text corresponding to the question text, obtaining the answer audio corresponding to the answer text, and using the explainer in the video data to feed back the answer audio to the user; the step of playing the video data corresponding to the current manuscript unit and the video data corresponding to the manuscript unit after the current manuscript unit from the target explanation position is performed when the user's explanation resumption instruction is received, or it is determined that the dialogue corresponding to the question instruction is over.

[0073] In some embodiments, the first playback module 42 synthesizes and plays the video data corresponding to each document unit, including: synthesizing each document unit in sequence to obtain the video data corresponding to each document unit; playing the video data corresponding to each document unit in sequence, wherein the video data of the latter document unit is played when the video data of the latter document unit has been synthesized and the video data of the previous document unit has been played.

[0074] In some embodiments, the image frame in the video data includes a speaker for the audio in the speech video data; the video data corresponding to each manuscript unit is played in sequence, including: in response to receiving a lecture end signal from the speaker in the video data corresponding to the previous manuscript unit, and determining that the video data corresponding to the next manuscript unit has been synthesized, playing the video data corresponding to the next manuscript unit.

[0075] In some embodiments, the image frame in the video data includes the speaker of the audio in the speech video data; the first playback module 42 synthesizes the video data corresponding to the manuscript unit, including: based on the explanation text corresponding to the manuscript unit, synthesizing the speaker video data about the speaker explaining the manuscript unit, wherein the video data corresponding to the manuscript unit is the speaker video data, or is a combination of the speaker video data and the manuscript image corresponding to the manuscript unit.

[0076] In some embodiments, the explanation information also includes an acquisition address of a document image corresponding to the document unit; playing the video data corresponding to the document unit includes: sending the explainer video data and the acquisition address corresponding to the document unit to the user terminal, so that the user terminal decodes and renders the explainer video data, and uses the acquisition address to obtain the document image corresponding to the document unit, and superimposes the document image on the explainer video data.

[0077] In some embodiments, the document unit is a page.

[0078] In some embodiments, the explanation adjustment instruction is a voice instruction; and / or, the presentation includes PPT; and / or, the playback of the video data is achieved through a web page sent to the user terminal.

[0079] See also Figure 5 , Figure 5 1 is a schematic diagram of a framework of an electronic device according to an embodiment of the present application. In this embodiment, the electronic device 50 includes a memory 51 and a processor 52 coupled to each other.

[0080] The memory 51 stores program instructions, and the processor 52 is used to execute the program instructions stored in the memory 51 to implement the steps of any of the above method implementations. In a specific implementation scenario, the electronic device 50 may include, but is not limited to: a microcomputer, a server, and in addition, the electronic device 50 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0081] Specifically, the processor 52 is used to control itself and the memory 51 to implement the steps of any of the above-mentioned embodiments. The processor 52 can also be called a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 52 can be implemented by an integrated circuit chip.

[0082] See also Figure 6 , Figure 6It is a schematic diagram of the framework of the computer-readable storage medium provided by the present application. The computer-readable storage medium 60 of the embodiment of the present application stores a program instruction 61, and when the program instruction 61 is executed, the method provided by any embodiment of the above method and any non-conflicting combination is implemented. Among them, the program instruction 61 can form a program file and be stored in the above-mentioned computer-readable storage medium 60 in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of each implementation method of the present application. The aforementioned computer-readable storage medium 60 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0083] The above scheme, after generating the explanation script based on the presentation, merges and plays the video data corresponding to the explanation script, and when receiving the explanation adjustment instruction triggered by the user during the video data playback process, determines the target explanation position in the explanation script based on the explanation adjustment instruction; and continues to play the video data corresponding to the explanation script from the target explanation position. It can be seen that in the process of playing video data, the present application can adjust the target explanation position according to demand. Compared with the explanation method of fixing the explanation order according to the explanation script, the present application adjusts the target explanation position based on the user's explanation adjustment instruction, and can adjust the playback according to user needs, greatly improving the flexibility of the explanation and user experience.

[0084] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0085] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0086] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0087] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0088] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0089] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0090] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A manuscript explanation method, characterized in that: The method comprises: Generate an explanation script based on the presentation, wherein the explanation script includes an explanation text corresponding to the presentation; Synthesizing and playing video data corresponding to the explanation script, wherein the audio in the video data is generated based on the explanation text; receiving an explanation adjustment instruction triggered by a user during the playing of the video data, and determining a target explanation position in the explanation script based on the explanation adjustment instruction; Continue playing the video data corresponding to the explanation script from the target explanation position.

2. The method according to claim 1, characterized in that The presentation manuscript is divided into a plurality of manuscript units, the explanation script includes explanation information of each of the manuscript units, and the explanation information includes explanation texts corresponding to the manuscript units; The synthesizing and playing the video data corresponding to the explanation script includes: Based on the explanation information of each of the manuscript units, the video data corresponding to each of the manuscript units are synthesized and played.

3. The method according to claim 2, characterized in that The explanation information also includes at least one of the serial number of the manuscript unit, the total playing time, and the acquisition address of the manuscript image corresponding to the manuscript unit; The audio of the video data corresponding to the manuscript unit corresponds to the explanation text corresponding to the manuscript unit, and the image frame of the video data corresponding to the manuscript unit includes at least one of the following: the manuscript image corresponding to the manuscript unit and the commentator for delivering the explanation text corresponding to the manuscript unit.

4. The method according to claim 2, characterized in that: The explanation information also includes the serial number of the manuscript unit; The method further comprises: When playing the video data, recording the serial number of the document unit currently being played as the current playing serial number; The step of determining a target explanation position in the explanation script based on the explanation adjustment instruction includes: In response to the explanation adjustment instruction being a jump instruction in manuscript units, obtaining the current play sequence number; Based on the jump content indicated by the jump instruction and the current play sequence number, a target jump sequence number is determined as the target explanation position, wherein the document unit corresponding to the target jump sequence number is the target document unit; The step of continuing to play the video data corresponding to the explanation script from the target explanation position includes: The video data of the target document unit and subsequent document units are played in sequence.

5. The method according to claim 2, characterized in that: The explanation information also includes the total playing time of the manuscript unit; The method further comprises: When playing the video data, recording the playing time of the currently played document unit; The step of determining a target explanation position in the explanation script based on the explanation adjustment instruction includes: In response to the explanation adjustment instruction being an explanation interruption instruction, obtaining the total playing time of the current manuscript unit from the explanation information corresponding to the current manuscript unit, and obtaining the recorded playing time of the current manuscript unit, wherein the current manuscript unit is the manuscript unit played when the user triggers the explanation adjustment instruction; Based on the played time of the current manuscript unit and the total played time, calculating the interruption position in the explanation text corresponding to the current manuscript unit when the user triggers the explanation interruption instruction; Determining the target explanation position based on the interruption position; The step of continuing to play the video data corresponding to the explanation script from the target explanation position includes: The video data corresponding to the current manuscript unit and the video data corresponding to the manuscript unit after the current manuscript unit are played starting from the target explanation position.

6. The method according to claim 5, characterized in that The total playback time is estimated based on the length of the explanation text corresponding to the manuscript unit and the playback speed; And / or, the calculating, based on the played time of the current manuscript unit and the total played time, the interruption position in the explanation text corresponding to the current manuscript unit when the user triggers the explanation interruption instruction comprises: The ratio of the product of the text length of the explanation text corresponding to the current manuscript unit and the played time to the total played time is used as the interruption position; And / or, determining the target explanation position based on the interruption position includes: The interruption position is used as the target explanation position; or the first punctuation mark before the interruption position is found in the explanation text, and the position of the first punctuation mark is used as the target explanation position.

7. The method according to claim 5, characterized in that The explanation interruption instruction is a pause instruction, and the step of playing the video data corresponding to the current manuscript unit and the video data corresponding to the manuscript unit after the current manuscript unit from the target explanation position is performed when the explanation resumption instruction of the user is received; Alternatively, the explanation interruption instruction is a question instruction, and the image frame in the video data includes a commentator who is used to present the audio in the video data; the method further includes: semantically understanding the question instruction to obtain a question text, and using a preset question and answer module to generate an answer text corresponding to the question text, obtaining an answer audio corresponding to the answer text, and using the commentator in the video data to feed back the answer audio to the user; the step of playing the video data corresponding to the current manuscript unit and the video data corresponding to the manuscript unit after the current manuscript unit starting from the target explanation position is performed when the user's explanation resumption instruction is received or it is determined that the conversation corresponding to the question instruction has ended.

8. The method according to claim 2, characterized in that: The synthesizing and playing the video data corresponding to each of the manuscript units includes: Performing synthesis processing on each of the manuscript units in sequence to obtain video data corresponding to each of the manuscript units; The video data corresponding to each of the manuscript units are played in sequence, wherein the video data of a subsequent manuscript unit is played when the video data of the subsequent manuscript unit has been synthesized and the video data of the previous manuscript unit has been played.

9. The method according to claim 8, characterized in that The image frames in the video data include a narrator delivering the audio in the video data; The sequentially playing of the video data corresponding to each of the manuscript units includes: In response to receiving a signal indicating that the explanation of the explainer in the video data corresponding to the previous manuscript unit has ended and determining that the video data corresponding to the next manuscript unit has been synthesized, the video data corresponding to the next manuscript unit is played.

10. The method according to claim 2, characterized in that The image frames in the video data include a narrator delivering the audio in the video data; Synthesizing video data corresponding to the document unit, including: Based on the explanation text corresponding to the manuscript unit, the explainer video data about the explainer explaining the manuscript unit is synthesized, wherein the video data corresponding to the manuscript unit is the explainer video data, or a combination of the explainer video data and the manuscript image corresponding to the manuscript unit.

11. The method according to claim 10, characterized in that The explanation information also includes an acquisition address of a document image corresponding to the document unit; Playing the video data corresponding to the manuscript unit includes: The narrator video data corresponding to the document unit and the acquisition address are sent to the user terminal, so that the user terminal decodes and renders the narrator video data, and uses the acquisition address to obtain the document image corresponding to the document unit, and superimposes the document image on the narrator video data.

12. The method according to claim 2, characterized in that: The document unit is a page.

13. The method according to claim 1, characterized in that The explanation adjustment instruction is a voice instruction; And / or, the presentation includes PPT; And / or, the playing of the video data is achieved through a web page sent to a user terminal.

14. A manuscript explanation device, characterized in that: The device comprises: A script generation module, used to generate an explanation script based on the presentation manuscript, wherein the explanation script includes an explanation text corresponding to the presentation manuscript; A first playback module is used to synthesize and play the video data corresponding to the explanation script, wherein the audio in the video data is generated based on the explanation text; An explanation position determination module is used to receive an explanation adjustment instruction triggered by a user during the playback of the video data, and determine a target explanation position in the explanation script based on the explanation adjustment instruction; The second playing module is used to continue playing the video data corresponding to the explanation script from the target explanation position.

15. An electronic device, characterized in that: comprising a memory and a processor coupled to each other, The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions that can be run by a processor, and the program instructions can be executed by the processor to implement the method according to any one of claims 1 to 13.

Images