A note summary generation method, device, equipment and storage medium

Abstract

This invention provides a method, apparatus, device, and storage medium for generating note summaries. The method includes: acquiring a target text image; segmenting several target regions from the target text image and determining the category of each target region, wherein the several target regions include several text regions, each text region being one of the following: the original text region, the user-written region, or the user-marked region; performing text recognition on each segmented text region to obtain recognition results corresponding to each text region; using the recognition results corresponding to the user-written region and / or the recognition results corresponding to the user-marked region as guiding information, combined with the recognition results corresponding to the original text region, generating a note summary for the target user. This invention can automatically generate a user's note summary based on a text image, significantly reducing time consumption and improving the efficiency of note organization compared to manual note-taking methods, while avoiding the influence of human factors.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method, apparatus, device, and storage medium for generating note summaries. Background Technology

[0002] In certain scenarios, users will write things on the text, such as their thoughts on the text, their summaries of the text, or they may also mark certain parts of the text, such as highlighting or marking content they are interested in. For example, in a learning context, students will take notes on the teacher's blackboard writing or insights during class, and they will also mark the content the teacher emphasizes.

[0003] Due to limited energy, users generally do not pay special attention to the organization and neatness of their notes when writing and marking on text. When users want to review the text content, messy notes will have a great impact on the review. Therefore, users need to organize the recorded content. However, organizing the recorded content is extremely time-consuming and laborious. Summary of the Invention

[0004] In view of this, the present invention provides a method, apparatus, device, and storage medium for generating note summaries, to solve the problem that organizing recorded content is extremely time-consuming and laborious for users. The technical solution is as follows:

[0005] A method for generating note summaries, including:

[0006] Obtain the text image containing the target user's notes, and use it as the target text image;

[0007] Several target regions are segmented from the target text image, and the category of each target region is determined. The several target regions include several text regions, and each text region is one of the original text region, user-written region, and user-marked region.

[0008] Perform text recognition on each segmented text region to obtain the recognition results corresponding to the several text regions respectively;

[0009] Using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, and combining them with the recognition results corresponding to the original text area, the target user's note summary is generated.

[0010] Optionally, the target regions may further include image regions;

[0011] The method further includes:

[0012] The aforementioned image regions are incorporated into the generated summary of notes.

[0013] Optionally, segmenting several target regions from the target text image includes:

[0014] Based on the pre-trained image segmentation model, several target regions are segmented from the target text image, and the category of each target region is determined.

[0015] Each target region is one of the following: an image region, an original text region, a user-written region, or a user-marked region. The image segmentation model is trained on an initial image segmentation model using training text images labeled with the positions and categories of several target regions.

[0016] Optionally, the initial image segmentation model includes a feature extraction module;

[0017] The feature extraction module in the initial image segmentation model is trained using training images labeled with image categories to obtain the feature extraction module in the image classification model;

[0018] One of the training image annotations is an image category that is one or two of the following categories: contains only images, contains only text, contains both images and text, has no user notes, or has user notes.

[0019] Optionally, the step of generating the target user's note summary by using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, combined with the recognition results corresponding to the original text area, includes:

[0020] Obtain a first text set and a second text set, wherein the first text set and the second text set are, in turn, a first sentence set and a second sentence set, or, in turn, a third sentence set and a fourth sentence set. The first sentence set includes each original sentence in the recognition result corresponding to the original text region. The second sentence set includes each original sentence in the recognition result corresponding to the user-marked region. The third sentence set includes sentences concatenated with each user-written sentence in the recognition result corresponding to the user-written region and each original sentence in the recognition result corresponding to the original text region. The fourth sentence set includes sentences concatenated with key sentences and / or reminder sentences from the third sentence set. The key sentences are original sentences in the recognition result corresponding to the user-marked region, and the reminder sentences are original sentences corresponding to user-written sentences.

[0021] Obtain the chapter feature vectors corresponding to the first text set and the second text set respectively, wherein the chapter feature vector is the feature vector of a chapter composed of all texts in the corresponding text set;

[0022] Based on the chapter feature vectors corresponding to the first text set and the second text set respectively, a summary of the target user's notes is generated.

[0023] Optionally, obtaining the chapter feature vectors corresponding to the first text set and the second text set respectively includes:

[0024] For the target text set whose corresponding chapter feature vectors need to be determined in the first text set and the second text set:

[0025] Each text in the target text set is encoded at the character level to obtain a sentence representation vector for each text in the target text set;

[0026] The sentence representation vectors of each text in the target text set are fused together, and the fused vector is used as the chapter representation vector of the target text set.

[0027] Sentence-level encoding is performed on the text representation vector of the target text set to obtain the text feature vector corresponding to the target text set.

[0028] Optionally, if the first text set and the second text set are respectively the third sentence set and the fourth sentence set, then the sentence-level encoding of the discourse representation vector of the target text set includes:

[0029] By combining the positional information corresponding to each text in the target text set, sentence-level encoding is performed on the text representation vector of the target text set. The positional information corresponding to a text includes the relative positional information of the two text regions where the two sentences that make up the text are located.

[0030] Optionally, generating the target user's note summary based on the chapter feature vectors corresponding to the first text set and the second text set respectively includes:

[0031] Based on the chapter feature vectors corresponding to the second text set, and combined with the chapter feature vectors corresponding to the first text set, a summary of notes corresponding to each text in the second text set is generated.

[0032] The notes summaries corresponding to each text in the second text set are merged and processed to obtain the notes summary of the target user.

[0033] Optionally, the step of generating note summaries corresponding to each text in the second text set based on the chapter feature vectors corresponding to the second text set and combining them with the chapter feature vectors corresponding to the first text set includes:

[0034] For the target text in the second text set from which the corresponding note summary is to be generated:

[0035] From the chapter feature vectors corresponding to the second text set, obtain the feature vectors related to the target text, and use them as the first feature vectors corresponding to the target text;

[0036] Based on the first feature vector corresponding to the target text and the chapter feature vector corresponding to the first text set, determine the feature vector required to generate the note summary corresponding to the target sentence, and use it as the second feature vector corresponding to the target text;

[0037] Based on the second feature vector corresponding to the target text, a summary of notes corresponding to the target text is generated.

[0038] Optionally, using the recognition results corresponding to the user-marked region as guiding information, and combining them with the recognition results corresponding to the original text region, a summary of the target user's notes is generated, including:

[0039] Based on the pre-trained first note summary generation model, the recognition results corresponding to the user-marked region and the recognition results corresponding to the original text region are processed to obtain a note summary faithful to the original text. The first note summary generation model is trained using multiple training text data labeled with note summaries faithful to the original text. Each training text data includes the recognition results corresponding to the original text region and the recognition results corresponding to the user-marked region segmented from a training text image.

[0040] Using the recognition results corresponding to the user's writing area and the user's marked area as guiding information, and combining them with the recognition results corresponding to the original text area, a summary of the target user's notes is generated, including:

[0041] Based on the pre-trained second note summary generation model, the recognition results corresponding to the user's writing area, the user's marked area, and the original text area are processed to obtain personalized note summaries. The second note summary generation model is trained using multiple training text data labeled with personalized note summaries. Each training text data includes the recognition results corresponding to the user's writing area, the user's marked area, and the original text area, which are segmented from a training text image.

[0042] A note summary generation device includes: an image acquisition module, an image segmentation module, a text recognition module, and a note summary generation module;

[0043] The image acquisition module is used to acquire a text image containing the target user's notes, as the target text image;

[0044] The image segmentation module is used to segment several target regions from the target text image and determine the category of each target region. The several target regions include several text regions, and each text region is one of the original text region, user-written region, and user-marked region.

[0045] The text recognition module is used to perform text recognition on each segmented text region to obtain the recognition results corresponding to the several text regions respectively;

[0046] The note summary generation module is used to generate the target user's note summary by using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, combined with the recognition results corresponding to the original text area.

[0047] A note-taking summary generation device includes: a memory and a processor;

[0048] The memory is used to store programs;

[0049] The processor is used to execute the program to implement each step of the note summary generation method described above.

[0050] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the note summary generation method described in any of the preceding claims.

[0051] The note-generating method, apparatus, device, and storage medium provided by this invention, after obtaining a text image containing the target user's notes (i.e., a target text image), first segments several targets (containing multiple text regions) from the target text image. Then, text recognition is performed on each segmented text region to obtain recognition results corresponding to each text region. Finally, using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, and combining them with the recognition results corresponding to the original text regions, a summary of the target user's notes is generated. The note summary generation method provided by this invention can automatically generate a summary of the target user's notes based on a text image containing the target user's notes. Compared to manual note-taking, this significantly reduces time consumption, improves the efficiency of note-taking, and avoids the influence of human factors. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0053] Figure 1 A flowchart illustrating the note summary generation method provided in an embodiment of the present invention;

[0054] Figure 2 This is an example of a note summary generated based on the note summary generation method provided by the present invention;

[0055] Figure 3 This is a flowchart illustrating the process of generating a target user's note summary by using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, combined with the recognition results corresponding to the original text area, as provided in this embodiment of the invention.

[0056] Figure 4 A flowchart illustrating a specific example of generating a summary of notes faithful to the original text, provided as an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram illustrating the generation of a faithful summary of notes based on a first note summary generation model, as provided in an embodiment of the present invention.

[0058] Figure 6 A flowchart illustrating a specific example of generating personalized note summaries provided in an embodiment of the present invention;

[0059] Figure 7 This is a schematic diagram illustrating the generation of personalized note summaries based on the second note summary generation model provided in an embodiment of the present invention.

[0060] Figure 8 A schematic diagram of the structure of the note summary generation device provided in an embodiment of the present invention;

[0061] Figure 9 A schematic diagram of the structure of the note summary generation device provided in an embodiment of the present invention. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] Given that organizing user notes is extremely time-consuming and laborious, the inventors of this case attempted to propose a method that can automatically generate note summaries. To this end, research was conducted, and it was discovered that there are currently some text summarization methods that can automatically summarize text. Although these methods can automatically generate summaries, most of them only summarize based on the original text. That is, most current text summarization methods only focus on the original text and do not pay attention to the content written or marked by the user. However, at times, users are more concerned with their written and marked content. Obviously, current text summarization methods cannot meet the needs of users.

[0064] In view of this, the inventors of this case conducted research and, through continuous research, finally proposed a note summary generation method. This note summary method uses user-written content and / or user-marked content as guiding information, and automatically generates the user's note summary in combination with the original text.

[0065] Before introducing the note summary generation method provided by this invention, the hardware architecture involved in this invention will be described first.

[0066] In one possible implementation, the hardware architecture involved in this invention may include: electronic devices and servers.

[0067] For example, an electronic device can be any electronic product that allows human-computer interaction with a user through one or more means such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as personal computers, laptops, tablets, mobile phones, smart TVs, learning machines, etc.

[0068] For example, a server can be a single server, a server cluster consisting of multiple servers, or a cloud computing server center. A server may include a processor, memory, and network interfaces, etc.

[0069] For example, an electronic device can establish a connection and communicate with a server through a wireless communication network; for example, an electronic device can establish a connection and communicate with a server through a wired network.

[0070] In one possible implementation, the electronic device receives a text image containing user notes uploaded by the user, sends the obtained text image to the server, the server generates a summary of the user's notes according to the note summary generation method provided by this invention, sends the generated note summary to the electronic device, and the electronic device displays the note summary.

[0071] In another possible implementation, the electronic device has a camera that can capture a text image containing the user's notes, send the captured text image to a server, the server generates a summary of the user's notes according to the note summary generation method provided by this invention, sends the generated note summary to the electronic device, and the electronic device displays the note summary.

[0072] In another possible implementation, the hardware architecture involved in this invention may include an electronic device. The electronic device is one with strong data processing capabilities.

[0073] For example, an electronic device can be any electronic product that can interact with a user through one or more means such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as personal computer, laptop, tablet, mobile phone, smart TV, learning machine, etc.

[0074] In one possible implementation, the electronic device receives a text image containing user notes uploaded by the user, generates a summary of the user's notes according to the note summary generation method provided by the present invention, and then displays the generated note summary.

[0075] In another possible implementation, the electronic device has a camera that can capture a text image containing the user's notes. After acquiring the text image, the electronic device generates a summary of the user's notes according to the note summary generation method provided by this invention, and then displays the generated note summary.

[0076] Those skilled in the art should understand that the above-described electronic devices and servers are merely examples, and other existing or future electronic devices or servers that are applicable to this invention should also be included within the scope of protection of this invention, and are hereby incorporated by reference.

[0077] The note summary generation method provided by the present invention will be described below through the following embodiments.

[0078] Please see Figure 1 The diagram illustrates a flowchart of a note summary generation method provided in an embodiment of the present invention, which may include:

[0079] Step S101: Obtain the target text image.

[0080] The content of the target text image includes the original text and user notes (such as user-written content, user-marked content on the original text, etc.). Optionally, the content of the target text image may also include images related to the original text.

[0081] Step S102: Segment several target regions from the target text image and determine the category of each target region.

[0082] The target regions segmented from the target text image may include only text regions, or they may include both text regions and image regions. Each text region can be one of the original text region, the user-written region, or the user-marked region.

[0083] Step S103: Perform text recognition on each segmented text region to obtain recognition results corresponding to several text regions respectively.

[0084] In one possible implementation, text recognition can be performed on each segmented text region based on a pre-trained text recognition model.

[0085] The text recognition model is trained using multiple training text images labeled with real text content. Optionally, to better recognize the text content in the user's writing area, the multiple training text images of the text recognition model can include images of the text written by the target user (for example, the target user can be asked to write text as required, and then the text written by the target user can be photographed to obtain an image of the text written by the target user).

[0086] It should be noted that this embodiment is not limited to performing text recognition on each text region based on a text recognition model. Other implementation methods that can recognize text from text regions are also applicable to this invention.

[0087] Step S104: Using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, and combining them with the recognition results corresponding to the original text area, generate a summary of the target user's notes.

[0088] Among them, the recognition results corresponding to the original text region contain sentences from the original text, the recognition results corresponding to the user-marked region contain sentences from the original text marked by the user, which are key sentences, and the recognition results corresponding to the user-written region contain sentences from the user-written region.

[0089] In one possible implementation, the recognition results corresponding to the user-marked regions can be used as guidance information, and combined with the recognition results corresponding to the original text regions, a summary of notes faithful to the original text can be generated.

[0090] In another possible implementation, the recognition results corresponding to the user's writing area can be used as guidance information, or the recognition results corresponding to the user's writing area and the recognition results corresponding to the user's marked area can be used as guidance information, combined with the recognition results corresponding to the original text area to generate a more personalized note summary.

[0091] In practical applications, users can select a note-taking summary generation strategy based on their specific needs to obtain the desired summary. Please refer to [link / reference]. Figure 2 , Figure 2 An example of a note summary generated based on the note summary generation method provided by the present invention is shown.

[0092] As mentioned above, the target regions segmented from the target text image may include text regions and image regions. Optionally, if the target regions segmented from the target text image include image regions, then after generating the target user's note summary via step S104, the image regions segmented from the target text image can be incorporated into the generated note summary, and the note summary after incorporating the image regions can be used as the final note summary.

[0093] After obtaining the target user's notes summary, the summary can be displayed. When displaying the summary, it can be presented according to the user's selected display mode (multiple display modes can be set for users to choose from, such as displaying only text or a combination of text and images). Furthermore, users can modify the generated summary, such as adding new insights and ideas, or correcting inaccurate text content.

[0094] The note summary generation method provided in this invention, after obtaining a text image containing the target user's notes (i.e., a target text image), first segments several target regions (containing multiple text regions) from the target text image. Then, it performs text recognition on each segmented text region, obtaining recognition results corresponding to each text region. Finally, using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, and combining them with the recognition results corresponding to the original text regions, it generates a note summary for the target user. This note summary generation method can automatically generate a target user's note summary from a text image containing the target user's notes. Compared to manual note-taking, it significantly reduces time consumption, improves note-taking efficiency, and avoids the influence of human factors.

[0095] In another embodiment of the present invention, the specific implementation process of "step S102: segmenting several target regions from the target text image and determining the category of each target region" in the above embodiment will be described.

[0096] There are multiple ways to segment several target regions from a target text image and determine the category of each target region. This embodiment provides two optional implementation methods.

[0097] First implementation method:

[0098] Based on a pre-trained first image segmentation model, several target regions are segmented from the target text image, and the category of each target region is determined. The target regions segmented by the first image segmentation model are all text regions, and each target region can be one of the following: the original text region, the user-written region, or the user-marked region. Optionally, for user-marked regions, further classifications such as user-emphasized marked regions and user-general marked regions can be made.

[0099] The first image segmentation model is trained using multiple text images labeled with the location and category of text regions. Because it is trained on multiple text images with labeled location and category of text regions, it has the ability to segment text regions from text images and determine the category of each segmented text region.

[0100] The first implementation mentioned above only focuses on the text region in the text image, and does not focus on the image region in the text image.

[0101] The second implementation method:

[0102] The pre-trained second image segmentation model segments several target regions from the target text image and determines the category of each target region. Each target region segmented by the second image segmentation model can be classified as an image region, the original text region, the user-written region, or the user-marked region. Optionally, for user-marked regions, further classifications such as user-emphasized marked regions and user-general marked regions can be made.

[0103] The second image segmentation model is trained using multiple training text images labeled with the locations and categories of both text and image regions. Because this model is trained on these labeled text and image regions, it has the ability to segment text and image regions from text images and determine the category of each segmented region.

[0104] The second implementation method mentioned above not only focuses on the text region in the text image, but also on the image region in the text image.

[0105] To obtain a high-performance image segmentation model, the general approach is to acquire a large number of labeled training text images and train the model using these images. However, in reality, labeling the location and category of each target region in the training text images requires significant human and financial resources. Therefore, it is usually difficult to obtain a large amount of high-quality labeled data. If high-quality but limited labeled data is used directly for training, the resulting image segmentation model will have poor performance. In view of this, this invention proposes the following strategy:

[0106] An image classification model is pre-trained using multiple training images labeled with image categories. This model includes a feature extraction module and a classification task module. After pre-training the image classification model, an initial image segmentation model is constructed based on the feature extraction and image segmentation task modules from the pre-trained model. Then, a small number of training text images with the location and category of the target region annotated are used to train the initial image segmentation model. The image category of each training image is one or two of the following categories: image only, text only, image and text, no user notes, and user notes.

[0107] In summary, this invention first uses coarsely labeled data (training images labeled with image categories) to pre-train a feature extraction module with good feature extraction capabilities. Then, based on the pre-trained feature extraction module and image segmentation task module, an initial image segmentation model is constructed. Next, the constructed image segmentation model is fine-tuned using finely labeled data (training text images labeled with the location and category of each target region). In this way, both time and computing resources are saved, and a high-performance image segmentation model is obtained.

[0108] It should be noted that this embodiment is not limited to segmenting several target regions from a target text image based on an image segmentation model and determining the category of each target region. Other implementation methods that can segment several target regions from a target text image and determine the category of each target region are also applicable to this invention.

[0109] Optionally, the feature extraction model described above can employ the atrous spatial pooling pyramid (ASPP) module.

[0110] ASPP uses multiple parallel atrous / dilated convolution layers with different sampling rates. The features extracted for each sampling rate are further processed in a separate branch and fused to generate the final result. The ASPP module constructs convolution kernels with different receptive fields through different dilation rates (rate is a parameter in atrous convolution that represents the size of the dilation) to obtain multi-scale object information.

[0111] Dilated convolution is used to address the challenge of achieving a large receptive field without excessively reducing the resolution of the feature map, as excessive resolution loss leads to the loss of many detailed information about image boundaries. The actual kernel size K and receptive field r of dilated convolution are... n The calculation formula is as follows:

[0112] K = k + (k-1)(r-1) (1)

[0113]

[0114] Where k is the original convolution kernel size, r is the rate of dilated convolution, and r n For the receptive field of this layer, r n-1 For the receptive field of the previous layer, s i This represents the stride of the i-th convolutional or pooling layer.

[0115] Optionally, to extract more complex and abstract features, the ASPP module can be improved. For example, each dilated convolution can be improved into a deep dilated convolution and a pointwise convolution, and multiple layers of convolution can be used to improve feature extraction capabilities. In addition, the order of inter-channel fusion and multi-scale feature fusion can be switched to reduce the depth operation of multi-scale feature fusion. The computational distribution of convolution can be analyzed and the kernel size can be adjusted to improve module performance without increasing the amount of computation too much.

[0116] It should be noted that the feature extraction module of the image segmentation model extracts relatively complex and abstract features, which enables the task segmentation module to segment each target region more accurately. In other words, the image segmentation model can better separate and distinguish the user-written and user-annotated parts from the original text.

[0117] In another embodiment of the present invention, the specific implementation process of "step S104: using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, and combining them with the recognition results corresponding to the original text area, to generate a summary of the target user's notes" in the above embodiment will be described.

[0118] Please see Figure 3 This diagram illustrates a process for generating a target user's note summary by combining the recognition results corresponding to the user's writing area and / or the user's marked area with the recognition results corresponding to the original text area. The process may include:

[0119] Step S301: Obtain the first text set and the second text set.

[0120] In one possible implementation, the first text set and the second text set are respectively the first sentence set and the second sentence set, wherein the first sentence set includes each original sentence in the recognition result corresponding to the original text region, and the second sentence set includes each original sentence in the recognition result corresponding to the user-marked region.

[0121] For example, if the original text sentences in the recognition results corresponding to the original text region include s1, s2, s3, s4, and s5, then the first sentence set S1 = {s1, s2, s3, s4, and s5}, and the original text sentences in the recognition results corresponding to the user-marked region include s2 and s3, then the second sentence set S2 = {s2 and s3}.

[0122] In another possible implementation, the first text set and the second text set are respectively the third sentence set and the fourth sentence set. The third sentence set includes sentences concatenated from each user-written sentence in the recognition result corresponding to the user-written area and each original sentence in the recognition result corresponding to the original text area. The fourth sentence set includes sentences concatenated from the third sentence set that contain key sentences and / or reminder sentences. The key sentences are the original sentences in the recognition result corresponding to the user-marked area, and the reminder sentences are the original sentences corresponding to the user-written sentences.

[0123] For example, the original sentences in the recognition results corresponding to the original text region include s1, s2, s3, s4, and s5; the original sentences (i.e., key sentences) in the recognition results corresponding to the user-marked region include s2 and s3; the user-written sentences in the recognition results corresponding to the user-written region include q1 and q2; and the original sentences (i.e., key reminder sentences) corresponding to the user-written sentences include s5. Then, the third sentence set S3 = {contact(q1,s1), contact(q1,s2), contact(q1,s3), contact(q1,s5)} ,s4),contact(q1,s5),contact(q2,s1),contact(q2,s2),contact(q2,s3),contact(q2,s4),contact(q2,s5)}, fourth Sentence set S4={contact(q1,s2), contact(q1,s3), contact(q1,s5), contact(q2,s2), contact(q2,s3), contact(q2,s5)}.

[0124] Step S302: Obtain the chapter feature vectors corresponding to the first text set and the second text set respectively.

[0125] Here, the chapter feature vector is the feature vector of a chapter composed of all the texts in the corresponding text set.

[0126] Step S303: Generate a summary of the target user's notes based on the chapter feature vectors corresponding to the first and second text sets respectively.

[0127] Specifically, firstly, based on the feature vectors of the chapters corresponding to the second text set, and combined with the feature vectors of the chapters corresponding to the first text set, a summary of notes corresponding to each text in the second text set is generated. Then, the summary of notes corresponding to each text in the second text set is merged and processed to obtain the summary of notes for the target user.

[0128] More specifically, for the target text in the second text set for which the corresponding note summary is to be generated, firstly, feature vectors related to the target text are obtained from the chapter feature vectors corresponding to the second text set, serving as the first feature vector for the target text. Then, based on the first feature vector for the target text and the chapter feature vectors corresponding to the first text set, the feature vectors required to generate the note summary for the target text are determined, serving as the second feature vector for the target text. Finally, the note summary for the target text is generated based on the second feature vector. After obtaining the note summaries corresponding to each text in the second text set, the note summaries corresponding to each text in the second text set are merged and processed to obtain the note summary for the target user.

[0129] It should be noted that if the first text set and the second text set are the first sentence set and the second sentence set respectively, then a summary of notes faithful to the original text will be generated based on the chapter feature vectors corresponding to the first text set and the second text set respectively. If the first text set and the second text set are the third sentence set and the fourth sentence set respectively, then a personalized summary of notes will be generated based on the chapter feature vectors corresponding to the first text set and the second text set respectively.

[0130] Based on the above implementation process, the following is a specific example of generating notes summaries that are faithful to the original text.

[0131] Please see Figure 4 The diagram illustrates a specific example of generating a summary of notes that faithfully reflects the original text, which may include:

[0132] Step S401: Obtain the first sentence set and the second sentence set.

[0133] The first sentence set can be formed by combining all the original sentences in the recognition results corresponding to the original text region, and the second sentence set can be formed by combining all the original sentences in the recognition results corresponding to the user-marked region.

[0134] Step S402: Perform character-level encoding on each sentence in the first sentence set to obtain the sentence representation vector of each sentence in the first sentence set.

[0135] In one possible implementation, a first note summary generation model, which has been pre-trained, can be used to encode each sentence in the first sentence set at the character level to obtain a sentence representation vector for each sentence in the first sentence set.

[0136] like Figure 5 As shown, the first note summary generation model may include a character-level encoder 501. Each sentence in the first sentence set is input to the character-level encoder 501 for encoding. The character-level encoder 501 performs detailed understanding of the sentence at the character level and outputs the character-level encoding result of the sentence. It should be noted that when encoding each sentence in the first sentence set, a [CLS] flag is generally set at the beginning of the sentence. [CLS] also participates in the encoding. After completing the character-level encoding of each sentence, the representation vector of [CLS] is used as the sentence representation vector of the entire sentence (through encoding, the vector at the [CLS] position will contain the information of the entire sentence).

[0137] Step S403a: Merge the sentence representation vectors of each sentence in the first sentence set, and use the merged vector as the discourse representation vector of the first sentence set.

[0138] Specifically, the sentence representation vectors of each sentence in the first sentence set can be concatenated, and the concatenated vector can be used as the discourse representation vector of the first sentence set.

[0139] Step S403b: Merge the sentence representation vectors of each sentence in the second sentence set, and use the merged vector as the discourse representation vector of the second sentence set.

[0140] Since the sentences in the second sentence set are part of the sentences in the first sentence set, when obtaining the sentence representation vector of each sentence in the first sentence set via step S402, the sentence representation vector of each sentence in the second sentence set is also obtained.

[0141] Specifically, the sentence representation vectors of each sentence in the second sentence set can be concatenated, and the concatenated vector can be used as the discourse representation vector of the second sentence set.

[0142] Step S404a: Perform sentence-level encoding on the discourse representation vector of the first sentence set to obtain the discourse feature vector corresponding to the first sentence set.

[0143] Step S404b: Perform sentence-level encoding on the discourse representation vector of the second sentence set to obtain the discourse feature vector corresponding to the second sentence set.

[0144] The first note summary generation model can be used to perform sentence-level encoding on the text representation vectors of the first sentence set and the second sentence set to obtain the text feature vectors corresponding to the first sentence set and the second sentence set.

[0145] like Figure 5As shown, the first note summary generation model may include a sentence-level encoder 502. The discourse representation vector of the first sentence set is input into the sentence-level encoder 502 for encoding. The sentence-level encoder 502 understands the full text information at the sentence level and then outputs the discourse feature vector corresponding to the first sentence set. Similarly, the discourse representation vector of the second sentence set is input into the sentence-level encoder 502 for encoding to obtain the discourse feature vector corresponding to the second sentence set.

[0146] Step S405: Based on the text feature vector corresponding to the second sentence set, and combined with the text feature vector corresponding to the first sentence set, generate note summaries corresponding to each sentence in the second sentence set.

[0147] Specifically, the model can be generated based on the summary of the first set of notes, using the text feature vector corresponding to the second set of sentences as a basis, and combining the text feature vector corresponding to the first set of sentences to generate the summary of notes for each sentence in the second set of sentences.

[0148] like Figure 5 As shown, the first note summary generation model may include a decoding module 503. The text feature vectors corresponding to the first sentence set and the second sentence set are input into the decoding module 503 for processing. More specifically, the decoding module 503 may include a first cross-attention module 5031, a second cross-attention module 5032, and a note summary prediction module 5033. The text feature vectors corresponding to the second sentence set can be input into the first cross-attention module 5031 for cross-attention calculation. For each sentence in the second sentence set, the first cross-attention module 5031 outputs the attention vector corresponding to that sentence as the first feature vector corresponding to that sentence. The first feature vector corresponding to that sentence and the text feature vector corresponding to the first sentence set are input into the second cross-attention module 5032 for cross-attention calculation. The attention vector output by the second cross-attention module 5032 is used as the second feature vector corresponding to that sentence. Finally, based on the second feature vector corresponding to that sentence, the note summary corresponding to that sentence is predicted by the note summary prediction module 5033.

[0149] Step S406: Merge and process the note summaries corresponding to each text in the second sentence set to obtain note summaries that are faithful to the original text.

[0150] Among them, the notes summary that are faithful to the original text include the important information in the original text, and the final note summary can be in the form of a logical hierarchical structure.

[0151] The aforementioned first note summary generation model is trained using multiple training text data points labeled with notes faithful to the original text. Each training text data point includes the recognition result (original text) corresponding to the original text region segmented from a training text image and the recognition result (key text) corresponding to the user-marked region. It should be noted that when annotating the training text data, the annotators will summarize the original text based on the key text to obtain a note summary that is faithful to the original text and contains important information. The note summary can be in the form of a logical hierarchical structure.

[0152] Specifically, the training process of the first note summary generation model includes: obtaining a first sentence set and a second sentence set based on the training text data (the first sentence set contains all original sentences in the recognition results corresponding to the original text regions segmented from the training text images, and the second sentence set contains all original sentences in the recognition results corresponding to the user-labeled regions segmented from the training text images); processing the first and second sentence sets based on the first note summary generation model to generate note summaries faithful to the original text; determining the prediction loss of the first note summary generation model based on the generated note summaries and the real note summaries labeled in the training text images; and updating the parameters of the first note summary generation model based on the prediction loss. This process is repeated multiple times using different training text data until the training termination condition is met.

[0153] The following is a specific example of generating personalized note summaries.

[0154] Please see Figure 6 The diagram illustrates a specific example of generating personalized note summaries, which may include:

[0155] Step S601: Obtain the third and fourth sentence sets.

[0156] Each user-written sentence in the recognition result corresponding to the user's writing area can be concatenated with each original sentence in the recognition result corresponding to the original text area. The resulting concatenated sentences form a third sentence set. The fourth sentence set consists of concatenated sentences from the third sentence set that contain key sentences and important reminder sentences.

[0157] Step S602: Perform character-level encoding on each concatenated sentence in the third sentence set to obtain the sentence representation vector of each concatenated sentence in the third sentence set.

[0158] In one possible implementation, a pre-trained second note summary generation model can be used to encode each concatenated sentence in the third sentence set at the character level to obtain a sentence representation vector for each concatenated sentence in the third sentence set.

[0159] like Figure 7 As shown, the second note summary generation model may include a character-level encoder 701. Each concatenated sentence in the third sentence set is input to the character-level encoder 701 for encoding. The character-level encoder 701 performs detailed understanding of the concatenated sentences at the character level and outputs the encoded result of the concatenated sentences. It should be noted that when encoding each concatenated sentence in the third sentence set, a [CLS] flag can be set at the beginning of the concatenated sentence. [CLS] will also participate in the encoding. After completing the character-level encoding of each concatenated sentence, the representation vector of [CLS] is used as the sentence representation vector of the entire concatenated sentence (through encoding, the vector at the [CLS] position will contain the information of the entire concatenated sentence).

[0160] Step S603a: Merge the sentence representation vectors of each concatenated sentence in the third sentence set, and use the merged vector as the discourse representation vector of the third sentence set.

[0161] Specifically, the sentence representation vectors of each concatenated sentence in the third sentence set can be concatenated, and the concatenated vector can be used as the discourse representation vector of the third sentence set.

[0162] Step S603b: Merge the sentence representation vectors of each concatenated sentence in the fourth sentence set, and use the merged vector as the discourse representation vector of the fourth sentence set.

[0163] Specifically, the sentence representation vectors of each concatenated sentence in the fourth sentence set can be concatenated, and the concatenated vector can be used as the discourse representation vector of the fourth sentence set.

[0164] Step S604a: Combining the positional information corresponding to each concatenated sentence in the third sentence set, perform sentence-level encoding on the discourse representation vector of the third sentence set to obtain the discourse feature vector corresponding to the third sentence set.

[0165] To achieve better note summary generation results, this embodiment introduces the position information corresponding to each concatenated sentence in the third sentence set when performing sentence-level encoding on the discourse representation vector of the third sentence set.

[0166] It should be noted that the positional information corresponding to the concatenated sentence in the third sentence set includes the relative positional information of the two sentences that make up the concatenated sentence. This invention uses the relative positional information of the two text regions containing the two sentences that make up the concatenated sentence as the relative positional information of the two sentences that make up the concatenated sentence. For example, the positional information corresponding to the concatenated sentence contact(q1,s3) is the relative positional information between the user-written area where q1 is located and the original text area where s3 is located.

[0167] The relative position information of the two text regions containing the two sentences is determined by the positions of the two text regions containing the two sentences in the target text image. The positions of the two text regions in the target text image can be determined when performing target region segmentation on the target text image.

[0168] Step S604b: Combining the positional information corresponding to each concatenated sentence in the fourth sentence set, perform sentence-level encoding on the discourse representation vector of the fourth sentence set to obtain the discourse feature vector corresponding to the fourth sentence set.

[0169] Similar to sentence-level encoding of the discourse representation vector of the third sentence set, when performing sentence-level encoding of the discourse representation vector of the fourth sentence set, the positional information corresponding to each concatenated sentence in the fourth sentence set is introduced. The positional information corresponding to each concatenated sentence in the fourth sentence set includes the relative positional information of the two text regions where the two sentences that make up the concatenated sentence are located.

[0170] Based on the second note summary generation model, and combined with the positional information corresponding to each spliced ​​sentence in the third sentence set, the text representation vector of the third sentence set can be encoded at the sentence level. Similarly, based on the second note summary generation model, and combined with the positional information corresponding to each spliced ​​sentence in the fourth sentence set, the text representation vector of the fourth sentence set can be encoded at the sentence level.

[0171] like Figure 7 As shown, the second note summary generation model may include a sentence-level encoder 702. The position information corresponding to each concatenated sentence in the third sentence set and the discourse representation vector of the third sentence set are input into the sentence-level encoder 702 for encoding. The sentence-level encoder 702 outputs the discourse feature vector corresponding to the third sentence set. Similarly, the position information corresponding to each concatenated sentence in the fourth sentence set and the discourse representation vector of the fourth sentence set are input into the sentence-level encoder 702 for encoding. The sentence-level encoder 702 outputs the discourse feature vector corresponding to the fourth sentence set.

[0172] Understandably, the text written by the user is usually located near the relevant original text; that is, the user writes sentence q. i With the original sentence s j The closer the two, the greater the correlation between them. Therefore, when encoding the discourse representation vector, the sentence-level encoder 702 will base its encoding on the sentence q written by the user. i With the original sentence s j The position determines the concatenation of the sentence contact(q) i ,s j Attention weights w attn (q i ,s j ), q i With s jThe closer the positions, the higher the attention weight w attn (q i ,s j The larger ) is, the smaller q is. i With s j The farther the location, the greater the attention weight w attn (q i ,s j The smaller the value, the better when determining the attention weight w. attn (q i ,s j After that, it can be normalized according to the following formula (3) to transform it into the range [0,1]. In addition, the sentence-level encoder 702 will also determine the concatenated sentence contact(q) according to the input discourse representation vector during encoding. i ,s j Attention score attn (q i ,s j ), in obtaining attention weight w attn (q i ,s j After normalizing the result of w, we can... attn (q i ,s j The normalized result and score attn (q i ,s j Multiply by q, and use the result as the concatenated sentence contact(q) i ,s j The final attention score, considering that normalizing the attention weights might lead to excessively large differences between weights, can be addressed by incrementing the attention weights by 1 before multiplying them by the attention score, i.e., concatenating the sentence contact(q). i ,s j The final attention score can be expressed as (4). After determining the final attention score of each concatenated sentence, the sentence-level encoder 702 will determine the final encoding result based on the final attention score of each concatenated sentence (i.e., determine the encoding result based on the input text representation vector and the final attention score).

[0173]

[0174] score attn ′(q i ,s j )=(1+softmax(w attn (q i ,s j )))*score attn (q i ,sj (4)

[0175] It should be noted that in this invention, each concatenated sentence is input into the second note summary generation model for encoding, so that the model can simultaneously access the original text and the user's written text, thereby obtaining the connection between the two.

[0176] Step S605: Based on the text feature vector corresponding to the fourth sentence set, and combined with the text feature vector corresponding to the third sentence set, generate the note summary corresponding to each spliced ​​sentence in the fourth sentence set.

[0177] Specifically, the note summary generation model can be based on the second note summary, using the text feature vector corresponding to the fourth sentence set as a basis, and combining it with the text feature vector corresponding to the third sentence set to generate note summaries corresponding to each spliced ​​sentence in the fourth sentence set.

[0178] like Figure 7 As shown, the second note summary generation model may include a decoding module 703. The text feature vectors corresponding to the third and fourth sentence sets are input into the decoding module 703 for processing. More specifically, the decoding module 703 may include a first cross-attention module 7031, a second cross-attention module 7032, and a note summary prediction module 7033. The text feature vector corresponding to the fourth sentence set can be input into the first cross-attention module 7031 for cross-attention calculation. For each concatenated sentence in the fourth sentence set, the first cross-attention module 7031 outputs the attention vector corresponding to the concatenated sentence as the first feature vector corresponding to the concatenated sentence. The first feature vector corresponding to the concatenated sentence and the text feature vector corresponding to the third sentence set are input into the second cross-attention module 7032 for cross-attention calculation. The attention vector output by the second cross-attention module 7032 is used as the second feature vector corresponding to the concatenated sentence. Finally, based on the second feature vector corresponding to the concatenated sentence, the note summary corresponding to the concatenated sentence is predicted by the note summary prediction module 7033.

[0179] It should be noted that the input decoder of this invention includes features related to key sentences and key reminder sentences. The input of the features related to key sentences and key reminder sentences enables the decoder to not only focus on the key content in the original text, thus ensuring no omissions, but also to focus on the target user's personal records, thus achieving personalization.

[0180] Step S606: Merge and process the notes summaries corresponding to each spliced ​​sentence in the fourth sentence set to obtain personalized note summaries.

[0181] Personalized note summaries include key information from the original text, information that the target user is personally interested in and writes, and the final note summaries can be in a structured, hierarchical format.

[0182] The aforementioned second note-taking summary generation model is trained using multiple training text data sets annotated with personalized note summaries. Each training text data set includes the recognition results corresponding to the original text region segmented from a training text image, the recognition results corresponding to the user-written region, and the recognition results corresponding to the user-marked region. It should be noted that each training data set, in addition to annotating the personalized note summary, also annotates the original sentences corresponding to the user-written sentences. Furthermore, when annotating the training text data, the annotators read the user-written content and the user-marked content to understand the user's purpose and focus, and annotate the sentences in the original text corresponding to the user-written sentences (i.e., key point reminder sentences). Finally, based on the user-written sentences, the user-marked sentences (key sentences), and the sentences in the original text corresponding to the user-written sentences (key point reminder sentences), and in conjunction with the original text sentences, a personalized note summary is determined. The personalized note summary can include important information from the original text, information about the user's personal focus, and the information written. The personalized note summary can be in a structured, hierarchical format.

[0183] Optionally, the training text data for the second note-taking summary generation model can include multiple clean training text data sets, as well as noisy training text data obtained by adding noise to the clean training text data sets. When adding noise to the clean training text data sets, the user-written parts of the training text data can be noisy (e.g., synonym replacement, word segment deletion, word segment disordering, etc.). Introducing noisy training text data sets can both expand the amount of data and improve the training effect of the model. Of course, the second note-taking summary generation model can also be trained using only multiple clean training text data sets.

[0184] Specifically, the training process of the second note summary generation model includes: obtaining a third sentence set and a fourth sentence set based on training text data (the third sentence set contains concatenated sentences of each sentence from the recognition results corresponding to the user's writing area segmented from the training text image and each sentence from the recognition results corresponding to the original text area segmented from the training text image; the fourth sentence set contains concatenated sentences of the third sentence set including key sentences and reminder sentences); processing the third and fourth sentence sets based on the second note summary generation model to generate personalized note summaries; determining the prediction loss of the second note summary generation model based on the generated note summaries and the real note summaries annotated in the training text image; and updating the parameters of the second note summary generation model based on the prediction loss. The second note summary generation model is iteratively trained multiple times using different training text data according to the above process until the training termination condition is met.

[0185] This invention also provides a note summary generation device. The note summary generation device provided in this invention will be described below. The note summary generation device described below can be referred to in correspondence with the note summary generation method described above.

[0186] Please see Figure 8 The diagram shows a schematic of the note summary generation device provided in an embodiment of the present invention. The note summary generation device may include: an image acquisition module 801, an image segmentation module 802, a text recognition module 803, and a note summary generation module 804.

[0187] Image acquisition module 801 is used to acquire text images containing the target user's notes as target text images.

[0188] Image segmentation module 802 is used to segment several target regions from the target text image and determine the category of each target region. The several target regions include several text regions, and each text region is one of the original text region, user-written region, and user-marked region.

[0189] The text recognition module 803 is used to perform text recognition on each segmented text region to obtain the recognition results corresponding to the several text regions respectively.

[0190] The note summary generation module 804 is used to generate the target user's note summary by combining the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area with the recognition results corresponding to the original text area.

[0191] Optionally, the target regions may also include image regions. The note-taking summary generation apparatus provided in this embodiment may further include an image region adding module.

[0192] The image region adding module is used to incorporate the aforementioned image regions into the note summary generated by the note summary generation module 804.

[0193] Optionally, when segmenting several target regions from the target text image and determining the category of each target region, the image segmentation module 802 is specifically used for:

[0194] Based on the pre-trained image segmentation model, several target regions are segmented from the target text image, and the category of each target region is determined.

[0195] Each target region is one of the following: an image region, an original text region, a user-written region, or a user-marked region. The image segmentation model is trained on an initial image segmentation model using training text images labeled with the positions and categories of several target regions.

[0196] Optionally, the initial image segmentation model includes a feature extraction module; the feature extraction module in the initial image segmentation model is trained using training images labeled with image categories to obtain the feature extraction module in the image classification model; wherein, the image category labeled by a training image is one or two of the following categories: containing only images, containing only text, containing both images and text, without user notes, and with user notes.

[0197] Optionally, when the note summary generation module 804 generates the target user's note summary using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, combined with the recognition results corresponding to the original text area, it is specifically used for:

[0198] Obtain a first text set and a second text set, wherein the first text set and the second text set are, in turn, a first sentence set and a second sentence set, or, in turn, a third sentence set and a fourth sentence set. The first sentence set includes each original sentence in the recognition result corresponding to the original text region. The second sentence set includes each original sentence in the recognition result corresponding to the user-marked region. The third sentence set includes sentences concatenated with each user-written sentence in the recognition result corresponding to the user-written region and each original sentence in the recognition result corresponding to the original text region. The fourth sentence set includes sentences concatenated with key sentences and / or reminder sentences from the third sentence set. The key sentences are original sentences in the recognition result corresponding to the user-marked region, and the reminder sentences are original sentences corresponding to user-written sentences.

[0199] Obtain the chapter feature vectors corresponding to the first text set and the second text set respectively, wherein the chapter feature vector is the feature vector of a chapter composed of all texts in the corresponding text set;

[0200] Based on the chapter feature vectors corresponding to the first text set and the second text set respectively, a summary of the target user's notes is generated.

[0201] Optionally, the note summary generation module 804, when obtaining the chapter feature vectors corresponding to the first text set and the second text set respectively, is specifically used for:

[0202] For the target text set whose corresponding chapter feature vectors need to be determined in the first text set and the second text set:

[0203] Each text in the target text set is encoded at the character level to obtain a sentence representation vector for each text in the target text set;

[0204] The sentence representation vectors of each text in the target text set are fused together, and the fused vector is used as the chapter representation vector of the target text set.

[0205] Sentence-level encoding is performed on the text representation vector of the target text set to obtain the text feature vector corresponding to the target text set.

[0206] If the first text set and the second text set are respectively the third sentence set and the fourth sentence set, then the note summary generation module 804, when performing sentence-level encoding on the discourse representation vector of the target text set, specifically uses the following:

[0207] By combining the positional information corresponding to each text in the target text set, sentence-level encoding is performed on the text representation vector of the target text set. The positional information corresponding to a text includes the relative positional information of the two text regions where the two sentences that make up the text are located.

[0208] Optionally, when the note summary generation module 804 generates the target user's note summary based on the chapter feature vectors corresponding to the first text set and the second text set, it is specifically used for:

[0209] Based on the chapter feature vectors corresponding to the second text set, and combined with the chapter feature vectors corresponding to the first text set, a summary of notes corresponding to each text in the second text set is generated.

[0210] The notes summaries corresponding to each text in the second text set are merged and processed to obtain the notes summary of the target user.

[0211] Optionally, when the note summary generation module 804 generates note summaries for each text in the second text set based on the chapter feature vectors corresponding to the second text set and in combination with the chapter feature vectors corresponding to the first text set, it is specifically used for:

[0212] For the target text in the second text set from which the corresponding note summary is to be generated:

[0213] From the chapter feature vectors corresponding to the second text set, obtain the feature vectors related to the target text, and use them as the first feature vectors corresponding to the target text;

[0214] Based on the first feature vector corresponding to the target text and the chapter feature vector corresponding to the first text set, determine the feature vector required to generate the note summary corresponding to the target text, and use it as the second feature vector corresponding to the target text;

[0215] Based on the second feature vector corresponding to the target text, a summary of notes corresponding to the target sentence is generated.

[0216] Optionally, when the note summary generation module 804 generates the target user's note summary using the recognition results corresponding to the user-marked area as guiding information and combining them with the recognition results corresponding to the original text area, it is specifically used for:

[0217] Based on the pre-trained first note summary generation model, the recognition results corresponding to the user-marked region and the recognition results corresponding to the original text region are processed to obtain a note summary faithful to the original text. The first note summary generation model is trained using multiple training text data labeled with note summaries faithful to the original text. Each training text data includes the recognition results corresponding to the original text region and the recognition results corresponding to the user-marked region segmented from a training text image.

[0218] Optionally, when the note summary generation module 804 generates the target user's note summary using the recognition results corresponding to the user's writing area and the user's marked area as guiding information, combined with the recognition results corresponding to the original text area, it is specifically used for:

[0219] Based on the pre-trained second note summary generation model, the recognition results corresponding to the user's writing area, the user's marked area, and the original text area are processed to obtain personalized note summaries. The second note summary generation model is trained using multiple training text data labeled with personalized note summaries. Each training text data includes the recognition results corresponding to the user's writing area, the user's marked area, and the original text area, which are segmented from a training text image.

[0220] The note summary generation device provided in this invention, after obtaining a text image containing the target user's notes (i.e., a target text image), first segments several targets (containing multiple text regions) from the target text image. Then, it performs text recognition on each segmented text region, obtaining recognition results corresponding to each text region. Finally, using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, and combining them with the recognition results corresponding to the original text regions, it generates a note summary for the target user. The note summary generation device provided in this invention can automatically generate a note summary for a target user based on a text image containing the target user's notes. Compared to manual note-taking, this significantly reduces time consumption, improves note-taking efficiency, and avoids the influence of human factors.

[0221] This invention also provides a note summary generation device; please refer to [link / reference]. Figure 9The diagram shows the structure of the note summary generation device, which may include: a processor 901, a communication interface 902, a memory 903, and a communication bus 904.

[0222] In this embodiment of the invention, the number of processor 901, communication interface 902, memory 903, and communication bus 904 is at least one, and processor 901, communication interface 902, and memory 903 communicate with each other through communication bus 904.

[0223] The processor 901 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0224] The memory 903 may include high-speed RAM, or it may also include non-volatile memory, such as at least one disk storage device;

[0225] The memory stores a program, which the processor can call. The program is used for:

[0226] Obtain the text image containing the target user's notes, and use it as the target text image;

[0227] Several target regions are segmented from the target text image, and the category of each target region is determined. The several target regions include several text regions, and each text region is one of the original text region, user-written region, and user-marked region.

[0228] Perform text recognition on each segmented text region to obtain the recognition results corresponding to the several text regions respectively;

[0229] Using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, and combining them with the recognition results corresponding to the original text area, the target user's note summary is generated.

[0230] Optionally, the refined and extended functions of the program can be found in the description above.

[0231] This invention also provides a readable storage medium that stores a program suitable for execution by a processor, the program being used for:

[0232] Obtain the text image containing the target user's notes, and use it as the target text image;

[0233] Several target regions are segmented from the target text image, and the category of each target region is determined. The several target regions include several text regions, and each text region is one of the original text region, user-written region, and user-marked region.

[0234] Perform text recognition on each segmented text region to obtain the recognition results corresponding to the several text regions respectively;

[0235] Using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, and combining them with the recognition results corresponding to the original text area, the target user's note summary is generated.

[0236] Optionally, the refined and extended functions of the program can be found in the description above.

[0237] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0238] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0239] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating note summaries, characterized in that, include: Obtain the text image containing the target user's notes, and use it as the target text image; Several target regions are segmented from the target text image, and the category of each target region is determined. The several target regions include several text regions, and each text region is one of the original text region, user-written region, and user-marked region. Perform text recognition on each segmented text region to obtain the recognition results corresponding to the several text regions respectively; Using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, and combining them with the recognition results corresponding to the original text area, a summary of the target user's notes is generated; The step of generating the target user's note summary by using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, combined with the recognition results corresponding to the original text area, includes: Using the recognition results corresponding to the user-marked area as guidance information, and combining them with the recognition results corresponding to the original text area, a summary of notes faithful to the original text is generated; Alternatively, at least the recognition results corresponding to the user's writing area can be used as guiding information, combined with the recognition results corresponding to the original text area, to generate personalized note summaries.

2. The note summary generation method according to claim 1, characterized in that, The target regions also include image regions; The method further includes: The aforementioned image regions are incorporated into the generated summary of notes.

3. The note summary generation method according to claim 2, characterized in that, The step of segmenting several target regions from the target text image includes: Based on the pre-trained image segmentation model, several target regions are segmented from the target text image, and the category of each target region is determined. Each target region is one of the following: an image region, an original text region, a user-written region, or a user-marked region. The image segmentation model is trained on an initial image segmentation model using training text images labeled with the positions and categories of several target regions.

4. The note summary generation method according to claim 3, characterized in that, The initial image segmentation model includes a feature extraction module; The feature extraction module in the initial image segmentation model is trained using training images labeled with image categories to obtain the feature extraction module in the image classification model; One of the training image annotations is an image category that is one or two of the following categories: contains only images, contains only text, contains both images and text, has no user notes, or has user notes.

5. The note summary generation method according to any one of claims 1 to 4, characterized in that, The step of generating the target user's note summary by using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, combined with the recognition results corresponding to the original text area, includes: Obtain a first text set and a second text set, wherein the first text set and the second text set are, in turn, a first sentence set and a second sentence set, or, in turn, a third sentence set and a fourth sentence set. The first sentence set includes each original sentence in the recognition result corresponding to the original text region. The second sentence set includes each original sentence in the recognition result corresponding to the user-marked region. The third sentence set includes sentences concatenated with each user-written sentence in the recognition result corresponding to the user-written region and each original sentence in the recognition result corresponding to the original text region. The fourth sentence set includes sentences concatenated with key sentences and / or reminder sentences from the third sentence set. The key sentences are original sentences in the recognition result corresponding to the user-marked region, and the reminder sentences are original sentences corresponding to user-written sentences. Obtain the chapter feature vectors corresponding to the first text set and the second text set respectively, wherein the chapter feature vector is the feature vector of a chapter composed of all texts in the corresponding text set; Based on the chapter feature vectors corresponding to the first text set and the second text set respectively, a summary of the target user's notes is generated.

6. The note summary generation method according to claim 5, characterized in that, The step of obtaining the chapter feature vectors corresponding to the first text set and the second text set respectively includes: For the target text set whose corresponding chapter feature vectors need to be determined in the first text set and the second text set: Each text in the target text set is encoded at the character level to obtain a sentence representation vector for each text in the target text set; The sentence representation vectors of each text in the target text set are fused together, and the fused vector is used as the chapter representation vector of the target text set. Sentence-level encoding is performed on the text representation vector of the target text set to obtain the text feature vector corresponding to the target text set.

7. The note summary generation method according to claim 6, characterized in that, If the first text set and the second text set are respectively the third sentence set and the fourth sentence set, then the sentence-level encoding of the discourse representation vector of the target text set includes: By combining the positional information corresponding to each text in the target text set, sentence-level encoding is performed on the text representation vector of the target text set. The positional information corresponding to a text includes the relative positional information of the two text regions where the two sentences that make up the text are located.

8. The note summary generation method according to claim 5, characterized in that, The step of generating the target user's note summary based on the chapter feature vectors corresponding to the first text set and the second text set respectively includes: Based on the chapter feature vectors corresponding to the second text set, and combined with the chapter feature vectors corresponding to the first text set, a summary of notes corresponding to each text in the second text set is generated. The notes summaries corresponding to each text in the second text set are merged and processed to obtain the notes summary of the target user.

9. The note summary generation method according to claim 8, characterized in that, The step of generating note summaries for each text in the second text set based on the text feature vectors corresponding to the second text set and combining them with the text feature vectors corresponding to the first text set includes: For the target text in the second text set from which the corresponding note summary is to be generated: From the chapter feature vectors corresponding to the second text set, obtain the feature vectors related to the target text, and use them as the first feature vectors corresponding to the target text; Based on the first feature vector corresponding to the target text and the chapter feature vector corresponding to the first text set, determine the feature vector required to generate the note summary corresponding to the target text, and use it as the second feature vector corresponding to the target text; Based on the second feature vector corresponding to the target text, a summary of notes corresponding to the target text is generated.

10. The note summary generation method according to claim 1, characterized in that, Using the recognition results corresponding to the user-marked region as guiding information, and combining them with the recognition results corresponding to the original text region, a summary of the target user's notes is generated, including: Based on the pre-trained first note summary generation model, the recognition results corresponding to the user-marked region and the recognition results corresponding to the original text region are processed to obtain a note summary faithful to the original text. The first note summary generation model is trained using multiple training text data labeled with note summaries faithful to the original text. Each training text data includes the recognition results corresponding to the original text region and the recognition results corresponding to the user-marked region segmented from a training text image. Using the recognition results corresponding to the user's writing area and the user's marked area as guiding information, and combining them with the recognition results corresponding to the original text area, a summary of the target user's notes is generated, including: Based on the pre-trained second note summary generation model, the recognition results corresponding to the user's writing area, the user's marked area, and the original text area are processed to obtain personalized note summaries. The second note summary generation model is trained using multiple training text data labeled with personalized note summaries. Each training text data includes the recognition results corresponding to the user's writing area, the user's marked area, and the original text area, which are segmented from a training text image.

11. A note-taking summary generation device, characterized in that, include: Image acquisition module, image segmentation module, text recognition module, and note summary generation module; The image acquisition module is used to acquire a text image containing the target user's notes, as the target text image; The image segmentation module is used to segment several target regions from the target text image and determine the category of each target region. The several target regions include several text regions, and each text region is one of the original text region, user-written region, and user-marked region. The text recognition module is used to perform text recognition on each segmented text region to obtain the recognition results corresponding to the several text regions respectively; The note summary generation module is used to generate the target user's note summary by using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, combined with the recognition results corresponding to the original text area. When the note summary generation module generates a note summary for the target user by using the recognition results corresponding to the user's writing area and / or the recognition results corresponding to the user's marked area as guiding information, and combining them with the recognition results corresponding to the original text area, it is specifically used to generate a note summary faithful to the original text by using the recognition results corresponding to the user's marked area as guiding information, and combining them with the recognition results corresponding to the original text area; or, at least, to generate a personalized note summary by using the recognition results corresponding to the user's writing area as guiding information, and combining them with the recognition results corresponding to the original text area.

12. A note-taking summary generation device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the note summary generation method as described in any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements each step of the note summary generation method as described in any one of claims 1 to 10.

Images