Document display method, device, apparatus, and program product
By automatically identifying and highlighting matching text in the target document based on audio and video playback time information, the system solves the problem of users understanding text content in corporate training and remote meetings, realizes real-time linkage between audio and video and documents, and improves users' comprehension efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- KE COM (BEIJING) TECHNOLOGY CO LTD
- Filing Date
- 2026-02-26
- Publication Date
- 2026-06-02
Smart Images
Figure CN122132585A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to a document display method, apparatus, device, and program product. Background Technology
[0002] In various scenarios requiring content sharing and explanation, such as corporate training, contract explanation, and remote conferencing, users can share and explain content via audio and video, or via text documents. When sharing content via text documents, the recipient needs a certain level of reading ability and background knowledge to accurately understand the content; otherwise, misunderstandings or information omissions are likely. Conversely, when sharing content via audio and video, the recipient passively receives the content, often finding it difficult to accurately locate key information and lacking the space for active thinking, making a comprehensive understanding of the content challenging. Summary of the Invention
[0003] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a document display method, apparatus, device, and program product.
[0004] According to one aspect of the embodiments of this disclosure, a document display method is provided, including: Based on the playback time information of audio and video, target text in the target document that matches the playback time information is determined. The content of the target document is consistent with the corresponding content of the audio and video, and the target text of the target document is the text displayed on the screen. Based on the global text string corresponding to the target document, obtain the text index information corresponding to the target text; Based on the text index information, the target region coordinates corresponding to the target text are determined in the global text index table. The global text index table records the mapping relationship between the text index information and text item information of each text character in the global text string. The text item information includes the absolute coordinates of the page. Based on the coordinates of the target region, the target text in the target document is highlighted.
[0005] In some optional implementations, determining the target text in the target document that matches the playback time information based on the audio and video playback time information includes: Based on the playback time update event, obtain the playback time information of the audio and video; Obtain the target text corresponding to the playback time information from the multi-dimensional content mapping file, which records the mapping relationship between the playback time information of the audio and video and the target text.
[0006] In some optional implementations, obtaining the text index information corresponding to the target text based on the global text string corresponding to the target document includes: Obtain the position of the substring that matches the target text within the global text string; Based on the position, the text index information of the starting character and the text index information of the ending character of the substring are determined, and the text index information of the starting character and the text index information of the ending character of the substring are determined as the text index information corresponding to the target text.
[0007] In some optional implementations, determining the target region coordinates corresponding to the target text in the global text index table based on the text index information includes: Based on the text index information of the starting character and the text index information of the ending character, the text item information of the starting character and the text item information of the ending character are determined in the global text index table respectively; Based on the text item information of the starting character and the text item information of the ending character, the coordinates of the target region corresponding to the target text are determined.
[0008] In some optional implementations, determining the target region coordinates corresponding to the target text based on the text item information of the starting character and the text item information of the ending character includes: Determine whether the target text is a single-line text; If the target text is a single-line text, the coordinates of the target region corresponding to the target text are determined based on the single-line rule; If the target text is neither a single-line text nor a multi-page text, the target region coordinates of each line of text corresponding to the target text are determined based on the single-line rule; or, the target region coordinates corresponding to the target text are determined based on the multi-line rule. If the target text is a multi-page text, the target text is divided into at least two paragraphs according to the page, and for any paragraph, the target area coordinates corresponding to the target text are determined based on the single-line rule or the multi-line rule.
[0009] In some optional implementations, determining the target region coordinates corresponding to the target text based on single-line rules includes: Based on the first and second coordinates of the page absolute coordinates of the starting character, the character height, and the second coordinate of the page absolute coordinates of the ending character, the coordinates of the target area corresponding to the target text are determined.
[0010] In some optional implementations, determining the target region coordinates corresponding to the target text based on line crossing rules includes: Based on the page line width of the target document, the first coordinate of the page absolute coordinate of the starting character, and the first coordinate of the page absolute coordinate of the ending character, the coordinates of the target region corresponding to the target text are determined.
[0011] According to another aspect of the present disclosure, a document display device is provided, comprising: The target text determination module is used to determine the target text in the target document that matches the playback time information based on the playback time information of the audio and video. The content of the target document is consistent with the corresponding content of the audio and video, and the target text of the target document is the text displayed on the screen. The text index acquisition module is used to acquire the text index information corresponding to the target text based on the global text string corresponding to the target document; The region coordinate determination module is used to determine the target region coordinates corresponding to the target text in a global text index table based on the text index information. The global text index table records the mapping relationship between the text index information and text item information of each text character in the global text string. The text item information includes absolute page coordinates. The display module is used to highlight the target text in the target document based on the coordinates of the target region.
[0012] In some optional implementations, the target text determination module includes: The first acquisition submodule is used to acquire the playback time information of the audio and video based on the playback time update event; The second acquisition submodule is used to acquire the target text corresponding to the playback time information from the multi-dimensional content mapping file, wherein the multi-dimensional content mapping file records the mapping relationship between the playback time information of the audio and video and the target text.
[0013] In some optional implementations, the text index acquisition module includes: The third acquisition submodule is used to acquire the position of the substring that matches the target text in the global text string; The first determining submodule is used to determine the text index information of the starting character and the text index information of the ending character of the substring based on the position, and to determine the text index information of the starting character and the text index information of the ending character of the substring as the text index information corresponding to the target text.
[0014] In some optional implementations, the region coordinate determination module includes: The second determining submodule is used to determine the text item information of the starting character and the text item information of the ending character based on the text index information of the starting character and the text index information of the ending character, respectively. The third determining submodule is used to determine the target region coordinates corresponding to the target text based on the text item information of the starting character and the text item information of the ending character.
[0015] In some alternative implementations, the third determining submodule is used for: Determine whether the target text is a single-line text; If the target text is a single-line text, the coordinates of the target region corresponding to the target text are determined based on the single-line rule; If the target text is neither a single-line text nor a multi-page text, the target region coordinates of each line of text corresponding to the target text are determined based on the single-line rule; or, the target region coordinates corresponding to the target text are determined based on the multi-line rule. If the target text is a multi-page text, the target text is divided into at least two paragraphs according to the page, and for any paragraph, the target area coordinates corresponding to the target text are determined based on the single-line rule or the multi-line rule.
[0016] In some optional implementations, the third determining submodule is used to determine the target region coordinates corresponding to the target text based on the second coordinate, the first coordinate, and the character height of the page absolute coordinates of the starting character, and the second coordinate and the character width of the page absolute coordinates of the ending character.
[0017] In some optional implementations, the third determining submodule is used to determine the target region coordinates corresponding to the target text based on the page line width of the target document, the first coordinate of the page absolute coordinates of the starting character, and the first coordinate of the page absolute coordinates of the ending character.
[0018] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing computer program instructions that, when executed, implement the above-described document display method.
[0019] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: Memory, used to store computer program products; A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the above-described document display method.
[0020] According to another aspect of the present disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the above-described document display method.
[0021] Based on the embodiments of this disclosure, in scenarios requiring content sharing and explanation, target text matching the playback time information in a target document can be determined based on the playback time information of the audio and video. The content of the target document is consistent with the content of the audio and video, and the target text of the target document can be displayed on the screen. Then, based on the global text string corresponding to the target document, the text index information corresponding to the target text is obtained. Based on the text index information corresponding to the target text, the coordinates of the target area corresponding to the target text are determined. Then, based on the coordinates of the target area, the target text in the target document is highlighted. Thus, the technical solution of this disclosure can realize content sharing and explanation by combining audio and video with documents, and can automatically highlight the matching text in the target document based on the playback time information of the audio and video, realizing real-time linkage between the audio and video playback progress and the text area of the target text in the document. This helps users quickly and accurately locate the content being played in the document, improving the user's understanding efficiency of the text content.
[0022] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0023] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.
[0024] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein: Figure 1 A flowchart illustrating one embodiment of the document display method of this disclosure; Figure 2 A flowchart of step 101 of the document display method of this disclosure; Figure 3 This is a flowchart illustrating the generation of the global text index table in the document display method of this disclosure; Figure 4 A flowchart of step 102 of the document display method of this disclosure; Figure 5 A flowchart for step 103 of the document display method of this disclosure; Figure 6A flowchart of step 132 of the document display method of this disclosure; Figure 7 This is a diagram illustrating the method for highlighting single-line text in the document display method disclosed herein; Figure 8 This is a diagram illustrating the method for highlighting multi-line text in the document display of this disclosure; Figure 9 This is a diagram illustrating the method for highlighting cross-page text in the document display of this disclosure; Figure 10 This is a schematic diagram of the structure of one embodiment of the document display device of this disclosure; Figure 11 This is a schematic diagram of the structure of yet another embodiment of the document display device of this disclosure; Figure 12 This is a structural diagram of an electronic device provided as an illustrative embodiment of the present disclosure. Detailed Implementation
[0025] In related technologies, in scenarios such as corporate training, contract explanation, and remote conferencing, for example, when renovation workers introduce the contents of a renovation contract to customers via video / audio, they may need to frequently compare the contract explanation video with the contract terms in the PDF document. However, due to the lack of an automatic synchronization mechanism, when switching to view the PDF document, it may not be possible to locate the contract terms being played / explained in the contract explanation video in a timely and accurate manner, which can easily lead to misunderstandings and omissions of information.
[0026] To address the technical problems in related technologies, embodiments of this disclosure provide a document display method, apparatus, device, and program product. Through the technical solution of this disclosure, content sharing and explanation can be automatically achieved by combining audio / video and document content. Furthermore, it can automatically highlight matching text in associated target documents based on the playback time information of the audio / video, allowing users to quickly locate the matching text in the document without performing any positioning or searching operations.
[0027] The technical solutions of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.
[0028] Exemplary methods Figure 1 This is a flowchart illustrating one embodiment of the document display method of this disclosure. This document display method can be applied to electronic devices (such as computer systems, servers, etc.). Figure 1 As shown, the document display method includes the following steps: In step 101, based on the playback time information of the audio and video, the target text in the target document that matches the playback time information is determined. The content of the target document is consistent with the content corresponding to the audio and video, and the target text of the target document is the text displayed on the screen.
[0029] The audio and video files can be either audio or video files, such as audiobooks or contract explanation videos. The target document is a document related to the audio and video content. For example, if the audio and video is a contract explanation video, the target document is the contract text; or if the audio and video is an audiobook of a novel, the target document is the text of that novel. The target text indicates a segment of text in the target document that matches the playback time information. This can be the text of the audio stream corresponding to the playback time information in the audio and video within the target document.
[0030] The playback time information of audio and video is used to indicate the current playback time of audio and video. You can add an ontimeupdate (playback time update) event listener to the audio and video in advance, and use the currentTime property in the event callback function to get the current playback time.
[0031] In this embodiment of the disclosure, the mapping relationship between playback time information and target text can be recorded in the subtitle file in advance, so that when the playback time information is obtained, the matching target text can be determined according to the subtitle file.
[0032] For example, see the following code: const videoSubtitles = [ { start: 10.5, / / Start time (seconds) end: 25.3, / / End time (seconds) text: "Explanation of Terms and Conditions in Part One of the Renovation Contract", / / Subtitle Text highlightText: ["First item: Project Overview", "Second item: Project Deadline"] / / Target Text }, / / In the code above, when the playback time is between 10.5 and 25.3 seconds, the target text is "First Project Overview" and "Second Project Deadline".
[0033] In this embodiment of the disclosure, the playback time information can be a point in time. The time period in which the point in time is located can be determined first, and the target text corresponding to the time period can be determined.
[0034] Taking the above code as an example, the target text recorded in the subtitle file for the time period 10.5-25.3 seconds is "First Project Overview" and "Second Project Deadline". Therefore, when the playback time information is any time point between 10.5 and 25.3 seconds in the audio and video, the corresponding time period is 10.5-25.3 seconds, and the target text determined according to the time period is "First Project Overview" and "Second Project Deadline".
[0035] In step 102, the text index information corresponding to the target text is obtained based on the global text string corresponding to the target document.
[0036] The global text string corresponding to the target document is a string formed by concatenating all text characters in the target document in the order they appear in the document. By using a text parsing tool to traverse all the content of the target document, all characters can be obtained. Then, non-text information, such as spaces and punctuation marks, can be removed, and the remaining text characters can be concatenated in order to obtain the global text string corresponding to the target document.
[0037] In this embodiment of the disclosure, non-text characters such as spaces and punctuation marks in the target text can be deleted first to obtain a plain text string, and then the corresponding text index information can be determined based on the plain text string.
[0038] The text index information is used to indicate the index information of the target text in the global text string. For example, if the target text corresponds to the 15th to 81st characters in the global text string, the text index information can be [15, 81].
[0039] In this embodiment of the disclosure, the target text is a continuous string of characters, and the index information of the starting character and the index information of the ending character of the string can be determined as the text index information of the target text.
[0040] In step 103, the target region coordinates corresponding to the target text are determined in the global text index table based on the text index information.
[0041] The global text index table records the mapping relationship between the text index information and text item information of each text character in the global text string. The text item information includes the absolute coordinates of the page.
[0042] Among them, the target region coordinates corresponding to the target text are used to indicate the coordinates of the target text in the target document, which can indicate the horizontal and vertical coordinates of the target text on a specific page in the target document.
[0043] In this embodiment of the disclosure, the coordinates of the target region corresponding to the target text can be determined based on the mapping relationship between text index information and text item information of each text character in the page document. The text item information of each text character includes at least the character's absolute page coordinates, character width, and character height. Therefore, based on the text index information, the absolute page coordinates, character width, and character height of each text character in the target text can be obtained, thereby determining the coordinates of the target region corresponding to the target text.
[0044] The absolute page coordinates of a character are the coordinates of its top-left corner.
[0045] In this embodiment of the disclosure, the method for generating the global text index table can be found in [reference needed]. Figure 3 The specific implementation method for determining the target region coordinates corresponding to the target text based on the text index information in the illustrated embodiment can be found in [reference needed]. Figure 5 The embodiments shown are not described in detail here.
[0046] In step 104, the target text in the target document is highlighted based on the target region coordinates.
[0047] Highlighting refers to marking target text in a document with a specific color or font style to make it more visually striking and easier for users to quickly find.
[0048] Highlighting can include various highlighting methods such as color highlighting, background shading, font style highlighting (e.g., bolding, changing font color, increasing font size), and character / paragraph border display (adding border lines to text or paragraphs through the "Borders and Shading" function).
[0049] In this embodiment of the disclosure, the highlighting display can be achieved by using different colored fluorescent pen effects to highlight the text, such as yellow, green, etc.
[0050] In this embodiment of the disclosure, based on the target region coordinates, a semi-transparent highlight layer can be drawn on the target text of the target document; or, based on the target region coordinates, the font attributes of the target text of the target document can be adjusted, such as adjusting the text size, color, thickness, etc.
[0051] In practice, the highlighting function in the target document can be called, such as the highlightMultipleTexts() function, to achieve highlighting. Alternatively, a highlighting layer can be drawn using a Canvas to achieve highlighting. Various types of document highlighting functions in related technologies can also be used to achieve highlighting. This disclosure does not limit the specific algorithm for achieving highlighting.
[0052] Through steps 101 to 104 above, in scenarios requiring content sharing and explanation, the target text in the target document that matches the playback time information can be determined based on the audio / video playback time information. The content of the target document is consistent with the content of the audio / video, and the target text of the target document can be displayed on the screen. Then, based on the global text string corresponding to the target document, the text index information corresponding to the target text is obtained. Based on the text index information corresponding to the target text, the coordinates of the target area corresponding to the target text are determined. Then, based on the coordinates of the target area, the target text in the target document is highlighted. Thus, the technical solution of this disclosure can realize content sharing and explanation by combining audio / video and documents, and can automatically highlight the matching text in the target document based on the playback time information of the audio / video, realizing real-time linkage between the audio / video playback progress and the text area of the target text in the document. This helps users quickly and accurately locate the content being played in the document, improving the user's understanding efficiency of the text content.
[0053] Figure 2 This is a flowchart illustrating step 101 of the method for demonstrating the present disclosure. (See attached flowchart.) Figure 2 As shown above, in the above Figure 1 Based on the illustrated embodiment, step 101 includes the following steps 111 to 112. The steps are explained below: In step 111, the playback time information of the audio and video is obtained based on the playback time update event.
[0054] The ontimeupdate event is used to track the playback progress of audio and video in real time. When the playback progress changes, this event can be automatically triggered to obtain the current playback time.
[0055] In some optional implementations, playback time update events can be listened to at preset time intervals to obtain audio and video playback time information. The preset time interval can be determined based on the audio and video playback frame rate, or it can be customized according to actual needs, such as an interval of 100 milliseconds to 5 seconds, balancing performance and accuracy to achieve the best performance effect.
[0056] In this embodiment of the disclosure, an ontimeupdate event listener can be added to a video or audio element, and the currentTime property can be used in the event callback to obtain the current playback time information.
[0057] In step 112, the target text corresponding to the playback time information is obtained from the multi-dimensional content mapping file.
[0058] The multi-dimensional content mapping file records the playback time information of audio and video and the mapping relationship between the target text.
[0059] In this embodiment of the disclosure, the playback time information in the mapping relationship can be a time period including multiple time points. For example, the target text corresponding to the time period 10.5-25.3 seconds is "First Project Overview" and "Second Project Deadline".
[0060] In practice, the time period containing the playback time information can be determined from the multi-dimensional content mapping file first, and then the corresponding target text can be determined based on the time period and the mapping relationship.
[0061] In the case of audio and video being video, the multi-dimensional content mapping file can also record the mapping relationship between the playback time information of the audio and video and the subtitle text of the audio and video, so as to determine the subtitle text that needs to be displayed in the video.
[0062] For example, the code for finding the corresponding subtitle interval based on playback time information, obtaining the target text group, and calling the highlighting function to achieve highlighting is as follows: function syncHighlights(currentTime) { const matchedSubtitle = videoSubtitles.find(sub => currentTime>= sub.start&¤tTime<= sub.end ); if (matchedSubtitle) { highlightMultipleTexts(matchedSubtitle.highlightText); } } In the code above, `currentTime` represents the current playback time, `sub.start` and `sub.end` represent the start and end times of the time segment containing the current playback time, `matchedSubtitle` represents the subtitle file, and `highlightText` represents the target text. The `Highlights(currentTime)` method retrieves the target text to be highlighted during the current playback time, while the `highlightMultipleTexts(matchedSubtitle.highlightText)` method applies the highlighting to the identified target text.
[0063] Based on the embodiments of this disclosure, playback time information of audio and video can be obtained by utilizing playback time update events, and then the target text corresponding to the playback time information can be obtained in a multi-dimensional content mapping file. This achieves real-time linkage between audio and video playback progress and the highlighted area of the text document, eliminating the need for users to locate the matching text from the target document. The operation is simple and efficient.
[0064] Figure 3 This is a flowchart illustrating the generation process of the global text index table in the document display method of this disclosure. For example... Figure 3 As shown, the specific process for generating the global text index table may include steps 301 to 303. The following is a description of each step: In step 301, a document parsing tool is used to obtain a set of text items in the target document. The set of text items includes the initial text item information of each character, which includes the relative coordinates of the character and the character transformation matrix.
[0065] The document parsing tool is used to convert various unstructured documents (such as PDFs, scanned documents, images, etc.) into structured data (such as Markdown, JSON, tables, etc.). The text item set includes text item information for each character. The initial text item information includes character content, character relative coordinates, character page index, character transformation matrix, and character width and height.
[0066] In this embodiment of the disclosure, a document parsing tool can be used to traverse the content of each page of the target document using the getTextContent() function to obtain the set of text items in the target document.
[0067] Among them, the character content can be the actual text content displayed in the target document; the character relative coordinates are the relative positions based on the coordinate system of the target document page (such as a PDF page); the character transformation matrix is a transformation matrix used to convert the character relative coordinates into the page absolute coordinates; the character width and character height are used to indicate the size of the character; and the character page index is the identifier of the page where the character is located.
[0068] In step 302, based on the character transformation matrix and the character relative coordinates, the page absolute coordinates of each character are determined, and the page absolute coordinates are added to the initial text item information to obtain the text item information.
[0069] The relative coordinates of characters typically use a Cartesian coordinate system, with the origin (0,0) located at the bottom left corner of the page. The X-axis is positive to the right, and the Y-axis is positive upwards. The direction is different from the regular web page coordinate system. By using a character transformation matrix, the vertical Y-coordinate of the character can be transformed downwards to match the regular web page coordinate system. Through coordinate transformation, the precise position of each character in the target document can be accurately located, and the absolute coordinates of the page can be obtained.
[0070] In this embodiment of the disclosure, after obtaining the absolute page coordinates of each character, the absolute page coordinates can be added to the initial text item information of the character to obtain the text item information of the character. The text item information of the character may include the character content, the absolute page coordinates, the character page index, and the character width and character height.
[0071] In step 303, a global text index table is determined based on the text item information of each text character and the global text string.
[0072] The global text string is a string formed by concatenating the remaining text characters after removing non-text characters such as spaces and punctuation from the target document. The global text index table records the mapping relationship between the text index information and text item information of each text character in the global text string.
[0073] Since the text item information of each character has been obtained through steps 301 to 302, each text character in the generated global text string also has matching text item information. Each text character also has its own text index information in the global text string. For example, if the character 1 is located at the first position in the global text string, the corresponding text item information includes the page absolute coordinates [0, 0], the character width is item.width, and the character height is item.height. Therefore, a mapping relationship is established between the text index information and the text item information of each text character, resulting in a global text index table.
[0074] Based on the embodiments of this disclosure, a set of text items in a target document can be obtained through a document parsing tool. The relative coordinates of characters in the set of text items and the character transformation matrix can be used to obtain the absolute coordinates of characters on the page. This allows for the acquisition of the mapping relationship between the text index information and text item information of each text character in the global text string, supporting fast indexing and locating of the coordinates of the target text.
[0075] Figure 4 This is a flowchart illustrating step 102 of the method for demonstrating the present disclosure. (See attached flowchart.) Figure 4 As shown above, in the above Figure 1 Based on the illustrated embodiment, step 102 includes the following steps 121 to 122. The steps are explained below: In step 121, the position of the substring that matches the target text in the global text string is obtained.
[0076] In this embodiment of the disclosure, the target text to be highlighted can be preprocessed first, for example, by removing spaces, punctuation marks, etc., to obtain a substring that matches the target text.
[0077] In the embodiments of the present disclosure, the position of the substring that matches the target text can be obtained in the global text string by means of exact matching.
[0078] When specifically implemented, the substring that matches the target text can be used as a substring, and the indexOf() method can be used to obtain the position where the substring appears in the global text string. If the substring that matches the target text appears multiple times in the global text string, the position of the first occurrence can be determined as the position of the substring in the global text string.
[0079] Among them, the position of the substring in the global text string can also be determined by various string matching methods such as global search and regular expression matching. The technical solution of the present disclosure does not make specific limitations on this.
[0080] In step 122, based on the position, determine the text index information of the starting character and the ending character of the substring, and determine the text index information of the starting character and the ending character of the substring as the text index information corresponding to the target text.
[0081] Among them, the starting character of the substring is the first character of the substring, and the ending character of the substring is the last character of the substring. For example, if the substring is "The first project overview", the starting character of the substring is "第", and the ending character of the substring is "况".
[0082] The text index information of the starting character of the substring is used to indicate the starting index information of the target text in the global text string, and the text index information of the ending character of the substring is used to indicate the ending index information of the target text in the global text string. Through the starting index information and the ending index information, the text content of the target text can be retrieved. Therefore, the text index information of the starting character and the ending character of the substring can be determined as the text index information corresponding to the target text.
[0083] Based on the embodiments of the present disclosure, the text index information of the target text can be determined by means of string matching, and then the coordinate information of the target text can be quickly indexed and positioned in the global text index table according to the text index information of the target text.
[0084] Figure 5 It is a flowchart of step 103 of the document display method of the present disclosure. As Figure 5 shown, on the basis of the above Figure 1 shown embodiments, step 103 includes the following steps 131 to step 132. The following is an explanation of each step: In step 131, based on the text index information of the start character and the text index information of the end character, the text item information of the start character and the text item information of the end character are determined in the global text index table, respectively.
[0085] In this embodiment of the disclosure, the text item information of the starting character and the text item information of the ending character can be determined based on the mapping relationship between the text index information and text item information of each text character recorded in the global text index table, as well as the text index information of the starting character and the text index information of the ending character.
[0086] In step 132, the coordinates of the target region corresponding to the target text are determined based on the text item information of the start character and the text item information of the end character.
[0087] The text item information includes the page absolute coordinates and the size of the characters. Therefore, based on the page absolute coordinates of the starting character and the ending character, it can be determined whether the target text is a single-line text, and then the target area coordinates corresponding to the target text can be determined according to the corresponding rules (single-line rule or cross-line rule).
[0088] For specific implementation details, please refer to [link / reference]. Figure 6 In the embodiment shown, step 132 may include the following steps 1321 to 1324.
[0089] In step 1321, it is determined whether the target text is a single-line text.
[0090] First, the page index of the starting character and the page index of the ending character can be used to determine whether the starting and ending characters are on the same page. If they are not on the same page, the target text can be determined to be cross-page text. If they are on the same page, the difference between the y-coordinate of the page absolute coordinates of the starting character and the ending character can be used to determine whether it is single-line text. If the difference is very small, such as 0 or close to 0, the target text can be determined to be single-line text; otherwise, it can be determined to be cross-line text.
[0091] Further, if the target text is a single-line text, step 1322 can be executed; if the target text is neither a single-line text nor a multi-page text, step 1323 can be executed; if the target text is a multi-page text, step 1324 can be executed.
[0092] In step 1322, if the target text is a single-line text, the coordinates of the target region corresponding to the target text are determined based on the single-line rule.
[0093] Among them, the single-line rule is a rule used to determine the coordinates of the target region of a single-line target text.
[0094] The method for determining the target region coordinates corresponding to the target text based on single-line rules is as follows: the target region coordinates corresponding to the target text are determined based on the second coordinate, first coordinate, and character height of the page absolute coordinates of the starting character, and the second coordinate and character width of the page absolute coordinates of the ending character.
[0095] See Figure 7 The left boundary of the target area is the x-coordinate (second coordinate, x-coordinate) of the starting character, the upper boundary of the target area is the y-coordinate of the starting character, the width of the target area is the difference between the right boundary of the ending character (the sum of the x-coordinate of the ending character and the width of the ending character) and the left boundary of the starting character (the x-coordinate of the starting character), and the height of the target area is the maximum height of the characters between the starting character and the ending character. By determining the coordinates of the upper left corner of the target area, as well as the area height and area width, the target area coordinates can be determined.
[0096] In step 1323, if the target text is neither a single-line text nor a multi-page text, the target area coordinates of each line of text corresponding to the target text are determined based on the single-line rule; or, the target area coordinates of the target text are determined based on the multi-line rule.
[0097] When the target text is multi-line text on the same page, the target text can be split into lines, and the area coordinates of the target area can be calculated for each line according to the single-line rule; alternatively, the lines of text containing the target text can be merged into a large rectangle as the target area.
[0098] See Figure 8 When splitting the target text line by line, the left boundary of the first line's region is the x-coordinate (second coordinate, x-coordinate) of the starting character, the top boundary is the y-coordinate of the starting character, the width is the difference between the maximum x-coordinate of the text line and the x-coordinate of the starting character, and the height is the height of the starting character. This determines the coordinates of the top left corner, height, and width of the first line's region, thus determining the target region coordinates. The width of the middle line's region is the maximum width of the text segment's line, the height of the middle line's region is the maximum height of the text segment's characters in the line, and the top boundary of the middle line is the first coordinate (y-coordinate) of the middle line's characters. The width of the last line's region is the difference between the right boundary of the ending character (the sum of the ending character's x-coordinate and width) and the x-coordinate of the first character of the text line, the height of the last line's region is the maximum height of the text segment's characters in the line, and the top boundary of the last line is the first coordinate (y-coordinate) of the ending character. This allows for the determination of the target region coordinates by splitting the text line by line.
[0099] In this embodiment of the disclosure, the target region coordinates corresponding to the target text are determined based on the line crossing rule. Specifically, the target region coordinates corresponding to the target text are determined based on the page line width of the target document, the first coordinate of the page absolute coordinates of the starting character, and the first coordinate of the page absolute coordinates of the ending character.
[0100] When merging the lines of text containing the target text into a large rectangle as a target area, the width of the target area is the page line width (the width of the entire line of text), and the height of the target area is obtained by subtracting the sum of the first coordinate (y-coordinate) of the end character and the height of the end character from the first coordinate (y-coordinate) of the start character.
[0101] In step 1324, if the target text is a multi-page text, the target text is divided into at least two paragraphs according to the page, and for any paragraph, the coordinates of the target area corresponding to the target text are determined based on the single-line rule or the multi-line rule.
[0102] When the target text is a multi-page text, such as Figure 9 As shown, if the first character (starting character) and the last character (ending character) are on different pages, the target text can be divided into at least two paragraphs according to the page. For any paragraph, the target area coordinates corresponding to the target text are determined based on the single-line rule or the cross-line rule mentioned above.
[0103] When the target text spans multiple pages, the target area can be rendered separately for each page, and the document can be automatically scrolled to the page corresponding to the current playback time information of the audio or video.
[0104] Based on the embodiments of this disclosure, after determining the text item information of the starting character and the text item information of the ending character based on the text index information of the starting character, the method for determining the target area coordinates of single-line text, multi-line text and multi-page text is realized respectively, which helps users accurately locate the content played by audio and video in the text document.
[0105] Corresponding to the aforementioned embodiments of the document display method, this disclosure also provides embodiments of the document display device.
[0106] Exemplary device Figure 10 This is a schematic diagram illustrating the structure of one embodiment of the document display device disclosed herein. This device is applied to electronic devices (such as computer systems, servers, etc.). Figure 10 As shown, the document display device may include: The target text determination module 110 is used to determine the target text in the target document that matches the playback time information based on the playback time information of the audio and video. The content of the target document is consistent with the corresponding content of the audio and video, and the target text of the target document is the text displayed on the screen. The text index acquisition module 120 is used to acquire the text index information corresponding to the target text based on the global text string corresponding to the target document; The region coordinate determination module 130 is used to determine the target region coordinates corresponding to the target text in the global text index table based on the text index information. The global text index table records the mapping relationship between the text index information and text item information of each text character in the global text string. The text item information includes the absolute coordinates of the page. Display module 140 is used to highlight target text in a target document based on the coordinates of the target region.
[0107] Figure 11 This is a schematic diagram of the structure of yet another embodiment of the document display device of this disclosure. (See diagram below.) Figure 11 As shown, in Figure 10 Based on the illustrated embodiment, the target text determination module 110 may include: The first acquisition submodule 1101 is used to acquire the playback time information of audio and video based on the playback time update event; The second acquisition submodule 1102 is used to obtain the target text corresponding to the playback time information in the multi-dimensional content mapping file. The multi-dimensional content mapping file records the mapping relationship between the playback time information of audio and video and the target text.
[0108] In some alternative embodiments, the document display device may further include: The text item acquisition module 150 is used to acquire a set of text items in the target document using a document parsing tool. The set of text items includes initial text item information for each character, and the initial text item information includes character relative coordinates and character transformation matrix. The absolute coordinate determination module 160 is used to determine the page absolute coordinates of each character based on the character transformation matrix and the character relative coordinates, and add the page absolute coordinates to the initial text item information to obtain the text item information; The index table determination module 170 is used to determine the global text index table based on the text item information of the text characters in each character and the global text string.
[0109] In some alternative implementations, the text index acquisition module 120 may include: The third submodule 1201 is used to obtain the position of the substring that matches the target text in the global text string; The first determining submodule 1202 is used to determine the text index information of the starting character and the ending character of the substring based on the position, and to determine the text index information of the starting character and the ending character of the substring as the text index information corresponding to the target text.
[0110] In some alternative implementations, the region coordinate determination module 130 may include: The second determining submodule 1301 is used to determine the text item information of the starting character and the text item information of the ending character respectively in the global text index table based on the text index information of the starting character and the text index information of the ending character; The third determining submodule 1302 is used to determine the target region coordinates corresponding to the target text based on the text item information of the start character and the text item information of the end character.
[0111] In some alternative implementations, the third determining submodule 1302 is used for: Determine if the target text is a single-line text; If the target text is a single-line text, determine the coordinates of the target region corresponding to the target text based on the single-line rule; If the target text is neither a single-line text nor a multi-page text, determine the target region coordinates of each line of text corresponding to the target text based on the single-line rule; or, determine the target region coordinates of the target text based on the multi-line rule. If the target text is a multi-page text, divide the target text into at least two paragraphs according to the page, and for any paragraph, determine the target area coordinates corresponding to the target text based on single-line rules or multi-line rules.
[0112] In some optional implementations, the third determining submodule 1302 is used to determine the target region coordinates corresponding to the target text based on the first coordinate, the second coordinate, and the character height of the page absolute coordinates of the starting character, and the second coordinate and the character width of the page absolute coordinates of the ending character.
[0113] In some optional implementations, the third determining submodule 1302 is used to determine the target region coordinates corresponding to the target text based on the page line width of the target document, the first coordinate of the page absolute coordinates of the starting character, and the first coordinate of the page absolute coordinates of the ending character.
[0114] In some alternative implementations, the display module 140 is used to draw a semi-transparent highlight layer on the target text of the target document based on the target region coordinates; or, based on the target region coordinates, adjust the font attributes of the target text of the target document.
[0115] The modules and units in this disclosed device can be further divided into finer-grained units according to actual needs, and the specific configuration can be set according to actual needs.
[0116] The apparatus of this disclosure embodiment can be used to implement the methods of the above embodiments of this disclosure. The two correspond to each other in specific implementation, and the specific implementation of related parts can be referred to each other, which will not be repeated here.
[0117] Exemplary electronic devices, computer program products, and computer-readable storage media This disclosure also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, wherein when the computer program is executed, it implements the document display method of any of the above embodiments of this disclosure.
[0118] Below, for reference Figure 12 This describes an electronic device according to embodiments of the present disclosure, wherein apparatus for implementing methods according to embodiments of the present disclosure may be integrated. Figure 12 This is a structural diagram of an electronic device provided in an illustrative embodiment of the present disclosure, such as... Figure 12 As shown, the electronic device includes one or more processors 11, one or more memory 12 of computer-readable storage media, and a computer program stored in the memory and executable on the processor. When the program in the memory 12 is executed, the document display method described above can be implemented.
[0119] Specifically, in practical applications, the electronic device may also include components such as an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). Those skilled in the art will understand that... Figure 12 The structure of the electronic device shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or certain components, or different component arrangements. Wherein: The processor 11 may be a central processing unit (CPU) or other processing unit with error handling and / or instruction execution capabilities. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 12 and calling data stored in the memory 12, thereby providing overall monitoring of the electronic device.
[0120] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the document display methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.
[0121] The input device 13 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
[0122] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.
[0123] The electronic device may also include a power supply for powering various components, which can be logically connected to the processor 11 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and any other components.
[0124] Of course, for the sake of simplicity, Figure 12 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.
[0125] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the document display methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.
[0126] Computer program products can be written in any combination of one or more programming languages to perform the operations of embodiments of this disclosure. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0127] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the document display methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.
[0128] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0129] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0130] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0131] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.
[0132] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the method is for illustrative purposes only, and the steps of the method of this disclosure are not limited to the order specifically described above, unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the method according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the method according to this disclosure.
[0133] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.
Claims
1. A document display method, characterized in that, The method includes: Based on the playback time information of audio and video, target text in the target document that matches the playback time information is determined. The content of the target document is consistent with the content corresponding to the audio and video, and the target text of the target document is the text displayed on the screen. Based on the global text string corresponding to the target document, obtain the text index information corresponding to the target text; Based on the text index information, the target region coordinates corresponding to the target text are determined in the global text index table. The global text index table records the mapping relationship between the text index information and text item information of each text character in the global text string. The text item information includes the absolute coordinates of the page. Based on the coordinates of the target region, the target text in the target document is highlighted.
2. The method according to claim 1, characterized in that, The determination of target text in the target document that matches the playback time information based on audio and video playback time information includes: Based on the playback time update event, obtain the playback time information of the audio and video; Obtain the target text corresponding to the playback time information from the multi-dimensional content mapping file, which records the mapping relationship between the playback time information of the audio and video and the target text.
3. The method according to any one of claims 1-2, characterized in that, The step of obtaining the text index information corresponding to the target text based on the global text string corresponding to the target document includes: Obtain the position of the substring that matches the target text within the global text string; Based on the position, the text index information of the starting character and the text index information of the ending character of the substring are determined, and the text index information of the starting character and the text index information of the ending character of the substring are determined as the text index information corresponding to the target text.
4. The method according to claim 3, characterized in that, The step of determining the target region coordinates corresponding to the target text in the global text index table based on the text index information includes: Based on the text index information of the starting character and the text index information of the ending character, the text item information of the starting character and the text item information of the ending character are determined in the global text index table respectively; Based on the text item information of the starting character and the text item information of the ending character, the coordinates of the target region corresponding to the target text are determined.
5. The method according to claim 4, characterized in that, Determining the target region coordinates corresponding to the target text based on the text item information of the start character and the text item information of the end character includes: Determine whether the target text is a single-line text; If the target text is a single-line text, the coordinates of the target region corresponding to the target text are determined based on the single-line rule; If the target text is neither a single-line text nor a multi-page text, the target region coordinates of each line of text corresponding to the target text are determined based on the single-line rule; or, the target region coordinates corresponding to the target text are determined based on the multi-line rule. If the target text is a multi-page text, the target text is divided into at least two paragraphs according to the page, and for any paragraph, the target area coordinates corresponding to the target text are determined based on the single-line rule or the multi-line rule.
6. The method according to claim 5, characterized in that, The step of determining the target region coordinates corresponding to the target text based on single-line rules includes: Based on the first and second coordinates of the page absolute coordinates of the starting character, the character height, and the second coordinate of the page absolute coordinates of the ending character, the coordinates of the target area corresponding to the target text are determined.
7. The method according to claim 5, characterized in that, The step of determining the target region coordinates corresponding to the target text based on the cross-line rule includes: Based on the page line width of the target document, the first coordinate of the page absolute coordinate of the starting character, and the first coordinate of the page absolute coordinate of the ending character, the coordinates of the target region corresponding to the target text are determined.
8. A document display device, characterized in that, The device includes: The target text determination module is used to determine the target text in the target document that matches the playback time information based on the playback time information of the audio and video. The content of the target document is consistent with the corresponding content of the audio and video, and the target text of the target document is the text displayed on the screen. The text index acquisition module is used to acquire the text index information corresponding to the target text based on the global text string corresponding to the target document; The region coordinate determination module is used to determine the target region coordinates corresponding to the target text in a global text index table based on the text index information. The global text index table records the mapping relationship between the text index information and text item information of each text character in the global text string. The text item information includes absolute page coordinates. The display module is used to highlight the target text in the target document based on the coordinates of the target region.
9. An electronic device, characterized in that, The electronic device includes: Memory, used to store computer program products; A processor for executing a computer program product stored in the memory, wherein when the computer program product is executed, it implements the method described in any one of claims 1-7.
10. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-7.