A text segment retrieval method, an electronic device, and a storage medium

Abstract

This invention provides a method, electronic device, and storage medium for recalling text fragments, relating to the field of text fragment recall technology. The method includes: acquiring a target semantic vector T0 and a preset text fragment semantic vector library T; determining initial text fragment semantic vectors; clustering text fragment semantic vectors with the same article number into the same cluster and arranging them sequentially according to the paragraph order corresponding to the text fragment semantic vectors to obtain a cluster list A; and, based on f(j), retrieving text fragments from N... j A was identified in the corresponding text fragment. j The corresponding number of recalled text fragments; if N j For the pre-defined multi-event article numbering, then based on f(j) and N j The degree of correlation between any two text segments in the corresponding text fragment, from N j A was identified in the corresponding text fragment. j The invention provides several corresponding recall text fragments; it can significantly improve the actual relevance between the recalled text fragments and the target question, thereby increasing the accuracy of subsequent answer generation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text segment recall, in particular to a text segment recall method, an electronic device and a storage medium. BACKGROUND

[0002] In the field of natural language processing, accurate recall of text segments is a core link to realize intelligent question answering, information retrieval and other tasks, and the recall quality directly affects the accuracy of subsequent answer generation. In the prior art, the recall of text segments mostly depends on semantic similarity calculation: first, the target problem text and the text segment in the preset text segment library are respectively converted into semantic vectors, and the cosine similarity, Euclidean distance and other indicators between the vectors are calculated to select the top N text segments with the highest similarity as the recall results.

[0003] However, this recall method based only on semantic similarity has obvious defects: the event type difference of the article to which the text segment belongs is not considered. In actual application, articles can be divided into single event type (such as news reports focusing on a specific event) and multiple event type (such as review articles covering multiple independent events). For single event articles, the text segments contained therein usually revolve around the same event, and the correlation between the segments is strong. At this time, only according to the similarity, key supplementary information may be missed; while for multiple event articles, different text segments may correspond to completely independent events, and the correlation between the segments is weak. If only the similarity is recalled, other event segments unrelated to the target problem may be mixed in. The above situations will all lead to insufficient actual relevance of the recalled text segments to the target problem, and further reduce the correctness of subsequent answer generation. SUMMARY

[0004] To solve the above technical problems, the technical solution adopted by the present application is as follows:

[0005] According to a first aspect of the present application, a text segment recall method is provided, which comprises the following steps:

[0006] S100, obtaining a target semantic vector T0 corresponding to a target problem text and a preset text segment semantic vector library T; T includes a plurality of preset text segment semantic vectors, each text segment semantic vector corresponding to an article number;

[0007] S200, determining the text segment semantic vectors corresponding to the n largest text segment semantic similarities between T0 and each text segment semantic vector in T as initial text segment semantic vectors;

[0008] S300, according to the article number corresponding to the initial text segment semantic vector, the text segment semantic vectors with the same article number are clustered into the same cluster and arranged in turn according to the paragraph order corresponding to the text segment semantic vector, and a cluster list A=(A1, A2, …, A j , …, A m ) is obtained, j=1, 2, …, m; A j is the jth cluster obtained by clustering, and m is the number of clusters obtained by clustering; A j =(A j，1 , A j，2 , …, A j，p , …, A j，f（j） ), p=1, 2, …, f(j); A j，p is the pth text segment semantic vector in the jth cluster obtained by clustering, and f(j) is the number of text segment semantic vectors in the jth cluster obtained by clustering;

[0009] S400, if the article number N j corresponding to A j is a preset single event article number, then according to f(j), a plurality of recall text segments corresponding to A j are determined from the text segments corresponding to N j .

[0010] S500, if N j is a preset multi-event article number, then according to f(j) and the association degree between any two text segments in the text segments corresponding to N j , a plurality of recall text segments corresponding to A j are determined from the text segments corresponding to N j .

[0011] According to another aspect of the present application, a non-transitory computer readable storage medium is also provided, the storage medium storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by a processor to implement the above-mentioned text segment recall method.

[0012] According to another aspect of the present application, an electronic device is also provided, comprising a processor and the above-mentioned non-transitory computer readable storage medium.

[0013] The present application has at least the following beneficial effects:

[0014] The recall method of the text segment of the present application, for a single event article, the text segments of such articles usually describe the same event from different angles and different stages, and there is a close logical relationship between them. When recalling according to the number of text segments in the cluster, it means that more segments related to the event can be included. For example, an article reporting a sports match, different text segments may describe the cause of the match, the key goal in the process, the final result, etc. By recalling a sufficient number of such segments, key supplementary information around the same event can be fully included, avoiding missing important content due to selecting only a small number of high similarity segments, such as a segment with slightly lower similarity that explains the key reason for the match result.

[0015] For multi-event articles, the text segments they contain may belong to multiple unrelated events. If only similarity is used for recall, it is likely that other event segments that are semantically similar to the target problem but actually irrelevant will also be recalled. When the number of text segments in the cluster and the correlation between the segments are combined to determine the recalled segments, when the correlation between two segments is low, it can be judged that they may belong to different events, so that irrelevant segments belonging to other independent events can be accurately excluded. By such differentiated processing of different types of event articles, the actual correlation of the recalled text segments with the target problem is significantly improved. The high correlation of the recalled segments can provide more accurate and effective information basis for subsequent answer generation, thereby improving the correctness of subsequent answer generation and providing more reliable foundation support for natural language processing tasks such as intelligent question answering and information retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0017] Figure 1 The flowchart of the text segment recall method provided by the embodiments of the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0019] It is noted that, based on the present disclosure, one skilled in the art will appreciate that one aspect described herein can be implemented independently of any other aspects and that two or more of these aspects can be combined in any suitable manner. For example, an apparatus can be implemented and / or a method can be practiced using any number of the aspects set forth herein. In addition, such an apparatus can be implemented and / or such a method can be practiced using other structure and / or functionality in addition to or other than one or more of the aspects set forth herein.

[0020] A text segment retrieval method will be introduced below with reference to the flowchart of the text segment retrieval method shown in the figure. Figure 1

[0021] The text segment retrieval method can include the following steps:

[0022] S100, obtaining a target semantic vector T0 corresponding to a target question text and a preset text segment semantic vector library T; T includes a plurality of preset text segment semantic vectors, and each text segment semantic vector corresponds to an article number.

[0023] In this embodiment, the target question text can be input to a pre-trained language model (such as BERT, Word2Vec, GloVe, etc.) to encode the processed text. Taking the BERT model as an example, it will convert each word in the text into a corresponding word vector, and dynamically adjust the word vector in combination with the context information, and finally obtain the semantic vector T0 of the entire target question text through the output layer (usually the vector corresponding to the [CLS] position) of the model.

[0024] The construction of the preset text segment semantic vector library T needs to go through a systematic text processing and vector generation process, and the specific process is as follows:

[0025] Firstly, the source of the text is determined and the segment is divided. Collect various related articles (such as news, papers, encyclopedias, etc.) as the original text library, and then divide each article into a plurality of text segments according to paragraphs, sentence groups or semantic units (such as continuous text around a certain point of view or event details), and mark the unique number of the article (i.e. the "article number" hereinafter) for each text segment, to ensure that the correspondence between the segment and the source article can be traced back.

[0026] Secondly, the divided text segments are preprocessed. The processing method is consistent with the target question text, including removing punctuation marks, stop words (such as "ed", "in", etc.), performing word segmentation for Chinese segments, performing morphological reduction for English segments, etc., eliminating redundant information in the text, and making the core semantics of the segment more prominent.

[0027] ​Finally, the pre-trained language model is used to generate semantic vectors. The same pre-trained model as T0 (such as BERT, Word2Vec, etc.) is used to encode each pre-processed text segment, converting the segment into a fixed-dimensional semantic vector. All generated text segment semantic vectors and their corresponding article numbers are integrated and stored, forming a pre-set text segment semantic vector library T.

[0028] This construction ensures the semantic space consistency of the vectors in the library and the target problem vectors, providing a reliable foundation for subsequent similarity calculation between T0 and the vectors in the library. At the same time, through the association of article numbers, it lays a data foundation for subsequent recall strategies based on article clustering and event type differentiation.

[0029] S200, the maximum n text segment semantic similarity between T0 and each text segment semantic vector in T is determined as the initial text segment semantic vector.

[0030] In this embodiment, the similarity between the target semantic vector T0 and each text segment semantic vector in the pre-set text segment semantic vector library T is calculated, and the text segment semantic vectors corresponding to the maximum n text segment semantic similarity are selected as the initial text segment semantic vectors.

[0031] This step first selects a batch of text segments that are most semantically close to the target problem text, ensuring that the initial range of recall has high semantic relevance. Compared with the existing technology of directly fixing the number of recall, this step provides a high-quality candidate set for subsequent more accurate screening combined with article event types, reducing the mixing of irrelevant fragments due to improper initial range.

[0032] S300, according to the article number corresponding to the initial text segment semantic vector, the text segment semantic vectors with the same article number are clustered into the same cluster and arranged in order according to the paragraph order corresponding to the text segment semantic vector, obtaining a cluster list A=(A1,A2,…,A j ,…,A m ),j=1,2,…,m;A j is the jth cluster obtained by clustering, and m is the number of clusters obtained by clustering; A j =(A j，1 ,A j，2 ,…,A j，p ,…,A j，f（j） ),p=1,2,…,f(j);A j，p is the pth text segment semantic vector in the jth cluster obtained by clustering, and f(j) is the number of text segment semantic vectors in the jth cluster obtained by clustering.

[0033] In this embodiment, the generation of the text segment is in the unit of the paragraph of the article, that is, each paragraph corresponds to a text segment; through this step, the classification and arrangement of the text segment are realized, and the segments belonging to the same article are gathered together.

[0034] Since the same article usually develops around a specific event, this clustering manner enables subsequent differential processing for the event type of the same article. Meanwhile, the arrangement in the order of the paragraphs guarantees the logical coherence between the segments, which helps to more comprehensively obtain the event-related information for the single-event article, avoids the incomplete information acquisition caused by the scattered distribution of the segments, and provides structured data support for the effective implementation of the subsequent steps.

[0035] S400, if A j corresponds to the article number N j is the preset single-event article number, then A j is determined from the N j corresponding text segments according to f(j).

[0036] In this embodiment, the text segments of the single-event article are around the same event and have strong relevance, and the quantity f(j) to some extent reflects the relevance degree of the article to the target problem and the richness of the information. According to f(j), sufficient number of relevant segments can be ensured to be included, different angles and stages of the event information can be fully obtained, and the problem of missing important supplementary information due to only recalling according to the semantic similarity in the prior art is avoided. For example, in the sports match report, sufficient number of segments can cover the cause, process and result of the match, and the relevance and information integrity of the recalled segments to the target problem are improved.

[0037] Further, the step S400 can include the following steps:

[0038] S410, if A j corresponds to the article number N j is the preset single-event article number and f(j) = 1, then A j，1 corresponds to the paragraph number D j，1 .

[0039] When there is only one initially screened relevant text segment in the single-event article, the corresponding paragraph number thereof is determined, which provides accurate positioning for subsequent expansion to obtain more relevant information. The paragraphs of the single-event article are usually developed in a logical order around the same event, and each paragraph can contain different details of the event. After the paragraph number of the initial segment is determined, the starting point of the expanded information is found, blind searching for relevant content in the entire article is avoided, and the efficiency and pertinence of the subsequent steps are improved.

[0040] S411, obtaining N j adjacent to D j，1 The text segment corresponding to the first preset number NUM1 of adjacent paragraph numbers is determined as A j The corresponding several recall text segments.

[0041] Further, NUM1 is in the range of [6, 10].

[0042] In this embodiment, since the paragraphs of a single event article have strong coherence and relevance, the paragraphs adjacent to the initial relevant paragraphs often contain supplementary information closely related to the event.

[0043] For example, in an article reporting an earthquake event, if the initial relevant paragraph describes the time and location of the earthquake, the adjacent paragraphs may involve important information such as the magnitude of the earthquake and the loss caused. By selecting the text segments corresponding to the adjacent NUM1 paragraphs, the information range of the recall can be effectively expanded, and the information deficiency problem that may exist only by relying on the initial single text segment can be made up.

[0044] The beneficial effect of this step is that it not only ensures the high relevance of the recall text segment to the target problem (because the adjacent paragraphs belong to the same event), but also enriches the content of the recall information. At the same time, NUM1 can ensure that the number of recalled text segments is not too many, reducing the computational power consumption; avoid the information one-sidedness that may be caused by relying only on the initial similar segment in the prior art, further improve the completeness, practicality and computational efficiency of the recalled text segment, and provide strong support for generating more accurate and comprehensive answers subsequently.

[0045] Further, step S400 can further include the following steps:

[0046] S420, if A j The article number N j is a preset single event article number, and f(j) > 1, then A j，f（j） The paragraph number D j，f（j） corresponding to A j，1 The paragraph number D j，1 The paragraph number difference ΔNUM between A

[0047] By calculating the paragraph number difference, the distribution range of the multiple relevant text segments initially selected in the article is determined. The text segments of a single event article develop around the same event, and the paragraph number difference can reflect the event information range covered by these segments. Determining ΔNUM provides a key basis for subsequent differential recall strategies according to different range conditions, avoids the problem of too much or insufficient information that may be caused by using a unified recall method for all conditions when f(j) > 1, and enhances the flexibility and accuracy of recall.

[0048] S421, if ΔNUM = NUM1-1, then D j，f（j） D j，1 and D j，f（j） With D j，1 The text segment corresponding to the paragraph number between them is determined to be A. j The corresponding recall text fragments.

[0049] In this embodiment, when ΔNUM=NUM1-1, it means that from D j，1 To D j，f（j） The total number of paragraphs covered is exactly equal to NUM1. For example, if NUM1 is set to 6, then ΔNUM = 5, meaning from the first paragraph D... j，1 To the last paragraph D j，f（j） It contains 5 interval paragraphs, which, together with the first and last paragraphs, make up 6 paragraphs, perfectly matching the first preset quantity NUM1.

[0050] This step fully leverages the logical coherence and close informational connections between paragraphs in a single event article. Since the number of paragraphs within this range is exactly the preset number, no additional additions or deletions are needed, allowing for the complete and minimal retrieval of all event-related text fragments within this range. These fragments collectively constitute a complete information unit surrounding the target question, ensuring that information is not interrupted by the omission of a paragraph, nor is irrelevant content introduced due to exceeding the range. This maximizes the completeness, accuracy, and high relevance to the target question of the retrieved text fragments, laying a solid foundation for generating accurate and comprehensive answers based on these fragments.

[0051] S422, if ΔNUM < NUM1-1, then D j，f（j） D j，1 and D j，f（j） With D j，1 The text segment corresponding to the paragraph number between them is determined to be A. j Several corresponding recall text fragments, and will be associated with D j，f（j） Or D j，1 The text segment corresponding to several adjacent paragraph numbers is determined as A. j The corresponding number of recalled text fragments is such that the number of identified recalled text fragments is NUM1.

[0052] In this step, the initially selected fragments cover a relatively small range. Retrieving only these fragments may not meet the needs for event information. By supplementing with text fragments from adjacent paragraphs, the coherence of paragraphs in a single event article can be used to ensure the relevance of the new fragments to the event. This also ensures that the number of retrieved information reaches the preset value, enriching the retrieved content, avoiding one-sidedness of information, and providing more support for generating comprehensive answers in the future.

[0053] S423, if ΔNUM>NUM1-1, determine the text segment corresponding to the semantic similarity of the NUM1 text segments with the largest similarity to T0 as A j corresponding to the several recall text segments.

[0054] In this embodiment, when ΔNUM>NUM1-1, it indicates that the multiple text segments initially screened out are widely distributed in the article, and if all the segments and the paragraphs between them are recalled, the number will far exceed NUM1, and some information with low relevance to the target problem may be included. By selecting NUM1 text segments with the largest similarity to T0, the information with the closest semantic relevance to the target problem can be preferentially retained, ensuring the high quality of the recalled segments. In this way, while ensuring the number of recalls, the introduction of excessive irrelevant information due to the wide range is avoided, and the core relevance of the recalled segments to the target problem is improved, providing more accurate key information for subsequent answer generation.

[0055] S500, if N j is a preset multi-event article number, then according to f(j) and N j , the relevance between any two text segments in the corresponding text segments is determined, from N j , the text segment A j corresponding to the several recall text segments.

[0056] Further, step S500 can include the following steps:

[0057] S510, if N j is a preset multi-event article number and f(j)=1, then A j，1 corresponding to the paragraph number D j，1 .

[0058] The multi-event article contains multiple independent events, and a single initial relevant segment may belong to only one event. By determining the paragraph number D j，1 , the position of the segment in the article can be accurately located, providing a clear starting point for subsequent analysis of the event relevance of the segment to the preceding and following paragraphs. This step avoids blindly expanding the recall range in multi-event articles, lays a foundation for subsequent screening based on relevance, and improves the targeting of recall.

[0059] S511, in the order of adjacent paragraphs from near to far, D j，1 corresponding to the text segment and D j，1 corresponding to the article located before D j，1 corresponding to the text segment, the first relevance of each text segment to the same event is determined, to obtain the first relevance list η= (η1, η2, …, η j，1 corresponding to the first relevance list.a ,..., η b ), a = 1, 2,..., b; η a is D j，1 corresponding to D j，1 corresponding to D j，1 corresponding to D j，1 corresponding to D

[0060] In this embodiment, the relevance between the text segments corresponding to any two paragraphs in each article for the same event can be obtained by preprocessing, and the specific method is as follows:

[0061] First, the core event elements of the text segments are extracted, including subjects (such as persons, organizations), time, place, behavior actions, and event results. The overlap ratio of two segments on these elements is calculated: for example, if two segments involve the same subject and place, the time range has an intersection, and the described behavior develops around the same core action, the element overlap degree is high, and the basic score of the relevance degree is high; if there is no overlap of elements (such as different subjects, distant places, and no intersection of time), the basic score is low.

[0062] Second, the relevance degree judgment is strengthened by semantic similarity calculation. A pre-trained language model (such as BERT) is used to encode two text segments to obtain segment-level semantic vectors, and the cosine similarity between the vectors is calculated to reflect the closeness of the semantic expressions of the segments. If the semantic similarity is high (such as describing different stages of the same event), the relevance degree is added; if the semantic difference is large (such as describing two independent events), the relevance degree is reduced.

[0063] Finally, the logical relevance is modified. For segments with causal relationships and progressive relationships (such as “event occurrence → development → result”), even if there are slight differences in some elements (such as time), they are still determined to have high relevance; for segments without logical connection (such as one describing a sports event and one describing a technological breakthrough), even if there are individual co-occurring words, they are still determined to have low relevance.

[0064] Through the weighted calculation of the above multi-dimensional indicators (such as event element overlap degree accounting for 60%, semantic similarity accounting for 30%, and logical relevance accounting for 10%), a relevance value between 0 and 1 can be obtained, and the closer the value is to 1, the higher the possibility that the two segments belong to the same event. This quantitative method not only ensures the objectivity of the judgment, but also accurately distinguishes different segments of the same event and different segments of different events, providing a reliable relevance basis for the text segment recall of multi-event articles. The generation of the list η provides a quantitative basis for subsequent screening, facilitating the accurate identification of prequel supplementary information of the same event.

[0065] Further, if D j，1 is the paragraph number corresponding to the first paragraph of the corresponding article, this step can be skipped.

[0066] S512, in the order of the proximity of the paragraphs, sequentially obtain D j，1 , the second correlation degree between the corresponding text segment and D j，1 , the second correlation degree between the corresponding text segment and D j，1 , the second correlation degree between the corresponding text segment and D j，1 , the second correlation degree between the corresponding text segment and D c , the second correlation degree between the corresponding text segment and D d , c=1, 2, …, d; η c is the paragraph number corresponding to the first paragraph of the corresponding article, this step can be skipped. j，1 , the second correlation degree between the corresponding text segment and D j，1 , the second correlation degree between the corresponding text segment and D j，1 , the second correlation degree between the corresponding text segment and D j，1 , the second correlation degree between the corresponding text segment and D

[0067] As in the preceding paragraph, the subsequent paragraph may also belong to different events. By generating the subsequent correlation list λ, the event correlation of the target segment and the context can be comprehensively mastered. Combined with η and λ, the extension range of the event to which the target segment belongs in the article can be completely outlined, avoiding missing key information of the same event due to only focusing on the single direction paragraph, and providing a comprehensive quantitative reference for subsequent screening.

[0068] Further, if D j，1 is the paragraph number corresponding to the last paragraph of the corresponding article, this step can be skipped.

[0069] S513, obtaining a first preset value Q=0; entering S514.

[0070] S514, if η Q+1 ≥RE and Q

[0071] The core of steps S513-S514 is to screen out the number Q of the preceding segment belonging to the same event as the target segment. The threshold value RE is the critical value for distinguishing the same event and different events. When the correlation degree ≥RE, it can be determined as the same event. Through loop counting, the maximum number of segments belonging to the same event in the preceding direction can be accurately determined, avoiding the inclusion of the preceding segments of other events in the recall range, and ensuring the event consistency of the preceding recalled segments.

[0072] S515, obtaining a second preset value M=0; entering S516.

[0073] S516, if η M+1 ≥ RE and M < d, obtaining M=M+1, entering S516; otherwise, exiting the current processing.

[0074] Consistent with the calculation logic of Q, the two steps are used to determine the maximum number of segments M belonging to the same event as the target segment in the subsequent direction. Through quantitative screening, the same event segment in the subsequent direction is accurately locked, further excluding the interference of other events, ensuring the event relevance of the subsequent recall segment, complementing the previous screening, and completely enclosing the extension range of the same event before and after.

[0075] S517, if Q+M≤ NUM1-1, determining D j，1 corresponding text segment and the previous Q and the following M text segments adjacent to D j，1 corresponding text segment as a j corresponding number of recall text segments.

[0076] In this embodiment, when Q+M≤ NUM1-1, D j，1 corresponding text segment and the previous Q and the following M text segments adjacent to D

[0077] At this time, the total number of the front and rear related segments of the same event does not exceed the preset number NUM1-1 (plus the target segment, Q+M+1≤ NUM1), and all recalls can completely retain the relevant information of the event. Since these segments are confirmed as the same event through the correlation degree screening, the problem of mixing other event segments in multi-event articles is avoided, while ensuring the sufficiency of the recall number, improving the event cohesion and information integrity of the recall segment.

[0078] S518, if Q+M> NUM1-1, determining D j，1 corresponding text segment and the previous Q and the following M text segments adjacent to D j，1 corresponding text segment as a j corresponding number of recall text segments.

[0079] In this embodiment, when Q+M> NUM1-1, D j，1 corresponding text segment and the previous Q and the following M text segments adjacent to D

[0080] When the total number of related fragments for the same event exceeds NUM1, the NUM1-1 fragments closest to the target fragment are selected first (because the closer the distance, the higher the correlation). This ensures that the core information of the same event is retained to the maximum extent possible, while not exceeding a preset limit. This process controls the number of recalled fragments, ensures high correlation between fragments and the target question through the distance-first principle, avoids information overload, and maintains the consistency of the recalled fragments, providing accurate and appropriate information for subsequent answer generation.

[0081] Furthermore, step S500 may also include the following steps:

[0082] S520, if N j If the predefined multi-event article number is given and f(j) > 1, then obtain D. j，1 The corresponding text fragment and D j，f（j） Previously and D j，1 The third degree of relevance between each subsequent text segment.

[0083] In multi-event articles, when multiple initial text fragments related to the target question exist, these fragments may belong to different parts of the same event or to different events. By calculating D... j，1 The corresponding text fragment and its surroundings (including D) j，f（j） Previously with D j，1 The third relevance of each text fragment can clearly understand the relationship between the initial fragment and other fragments in terms of event affiliation, providing a quantitative basis for judging whether these fragments belong to the same event, avoiding blind integration due to the large number of fragments, and laying the foundation for accurate screening in the future.

[0084] S521, obtain D j，f（j） The corresponding text fragment and D j，f（j） Previously and D j，1 The fourth degree of relevance between each subsequent text segment.

[0085] This step complements S520, from D j，f（j） The correlation between the corresponding text fragment and surrounding text fragments is calculated. This is achieved by simultaneously analyzing two key initial fragments (D...). j，1 and D j，f（j） By examining the relationships between multiple initial segments and other segments, we can gain a more comprehensive understanding of the event relationships between multiple initial segments and between them and other segments, thereby improving the accuracy of event attribution judgments and preventing the omission of important related information.

[0086] S522, if every third degree of association and every fourth degree of association is greater than or equal to RE, then D will be... j，1 Corresponding text fragment, D j，f（j） The corresponding text fragment and Dj，1 and D j，f（j） The text segment corresponding to the paragraph number between them is determined to be A. j The corresponding recall text fragments.

[0087] When all third and fourth association degrees are not lower than RE, it indicates that D j，1 and D j，f（j） The corresponding text fragments and all the text fragments between them belong to the same event. Recalling all these fragments at this time allows for the complete acquisition of relevant information about the event. This fully utilizes the correlation between fragments of the same event in multi-event articles, avoids missing parts of the same event due to misjudgment, ensures the completeness of the recalled fragments and the consistency of the event, and improves the relevance to the target issue.

[0088] S523, if the text segment corresponding to the first third relevance degree less than RE is adjacent or identical to the text segment corresponding to the first fourth relevance degree less than RE, then the text segment corresponding to each third relevance degree before the first third relevance degree less than RE and each fourth relevance degree before the first fourth relevance degree less than RE is determined as A. j The corresponding recall text fragments.

[0089] In multi-event articles, when a text fragment has a relevance score less than RE, it indicates that the corresponding text fragments may belong to different events. When the text fragments corresponding to the first third and fourth relevance scores less than RE are adjacent or identical, it signifies an event boundary at that location. In this case, selecting text fragments with the required relevance score before the boundary for recall can accurately eliminate fragments belonging to other events, ensuring that all recalled text fragments belong to the same event. This effectively avoids mixing fragments from different events, improves the accuracy and effectiveness of the recalled fragments, and provides a high-quality information foundation for subsequent answer generation.

[0090] Furthermore, after step S523, the method further includes the following steps:

[0091] S524, if A is determined j If the number of corresponding recalled text fragments is greater than NUM1, then based on each recalled text fragment and A j The text fragments in the semantic vector are filtered based on their distance from each other, so that the number of recalled text fragments is NUM1.

[0092] In this embodiment, "distance" refers to the distance of the text fragment to the paragraph position in the article. The closer the distance, the closer the fragment is to the initially selected A. j The closer the text fragments (those with high semantic similarity to the target question) are to the same event, the more directly they may contain key information. For example, if Aj If the text segments in the article are located in paragraph 5 and paragraph 10, then in the recalled segments, the paragraphs 6, 7, 8, and 9 that are relatively close are more likely to contain core content related to the target question than the paragraphs 2 and 3 that are relatively far away.

[0093] The beneficial effect of this step is that when the number of recalled text segments of the same event is too large, the distance filtering can, under the premise of ensuring the consistency of the event, preferentially retain the core information that is more closely associated with the initial high similarity segment, avoid information redundancy caused by too many segments, and at the same time ensure that the number of final recalled segments meets the preset NUM1, which not only controls the information size and reduces the computing power consumption, but also improves the core relevance of the recalled segments, provides simplified and key information support for subsequent answer generation, and further improves the efficiency and accuracy of answer generation.

[0094] In this embodiment, for a single event article, the text segments of such articles are usually described from different angles and different stages around the same event, and they have close logical connections. When recalling according to the number of text segments in the cluster, it means that more segments related to the event can be included. For example, an article about a sports match may describe the cause of the match, the key goal in the process, and the final result. Recalling a sufficient number of such segments can fully include the key supplementary information around the same event, and avoid missing important content due to selecting only a small number of high similarity segments. For example, a segment with slightly lower similarity may explain the key reason for the match result.

[0095] For multi-event articles, the text segments contained therein may belong to multiple independent events. If only similarity is used for recall, it is likely that segments that are semantically similar to the target question but actually irrelevant to the target question will also be recalled. When the correlation degree of two segments is low, it can be judged that they may belong to different events, so that irrelevant segments belonging to other independent events can be accurately excluded. Through such differentiated processing of different types of event articles, the actual relevance of the recalled text segments to the target question is significantly improved. The high correlation degree of the recalled segments can provide a more accurate and effective information basis for subsequent answer generation, thereby improving the correctness of subsequent answer generation and providing a more reliable foundation for intelligent question answering, information retrieval, and other natural language processing tasks.

[0096] Moreover, although individual steps of the methods in the present disclosure are described in a particular order in the figures, this is not required or implied as to the order of execution of the steps, nor is it required that all of the steps be executed for the desired results. Additionally or alternatively, certain steps can be omitted, combined into a single step, broken into multiple steps, and / or executed in a different order than shown.

[0097] Embodiments of the present application also provide a non-transitory computer readable storage medium, which can be arranged in an electronic device to save at least one instruction or at least one program related to a method in the method embodiments, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided by the above-mentioned embodiments.

[0098] The program product can employ any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0099] The computer readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave, in which the readable program code is embodied. Such propagated data signal can take multiple forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable signal medium can also be any readable medium other than the readable storage medium, which can send, propagate, or transmit the program for use by or in connection with an instruction execution system, apparatus, or device.

[0100] The program code contained on the readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, and the like, or any suitable combination of the above.

[0101] The program code may be implemented in any of various ways, including procedure-based, object-based, and / or class-based environments. Software implemented in such ways includes program instructions that implement the methods of the present application. Program code can be implemented in a high level procedural or object-oriented programming and / or scripting languages, database programming languages, forth-based languages, and / or machine languages. It will be appreciated that the software can be implemented in any combination of firmware, hardware, and / or software. Program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device (for example, through the Internet using an Internet Service Provider).

[0102] Embodiments of the present application also provide an electronic device including a processor and the aforementioned non-transitory computer readable storage medium.

[0103] Electronic device is merely an example, and should not bring any limitation to the function and use range of embodiments of the present application.

[0104] The electronic device is in the form of a general computing device. Components of the electronic device can include, but are not limited to, the aforementioned at least one processor, the aforementioned at least one memory, a bus connecting different system components, including the memory and the processor.

[0105] The memory stores program code that can be executed by the processor, so that the processor performs steps in various embodiments described in the specification.

[0106] The memory can include a readable medium in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and can further include read only memory (ROM).

[0107] The memory can also include program / utility programs having a set of (at least one) program modules, including but not limited to an operating system, one or more application programs, other programs / modules, and program data, each of which or some combination thereof can include implementation of a network environment.

[0108] The bus can be one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or local bus using any of a variety of bus structures, and the like.

[0109] The electronic device can also communicate with one or more external devices such as a keyboard or a pointing device, through an I / O interface. The electronic device can communicate with one or more devices that enable a user to interact with it through a communication interface. The electronic device can also communicate with one or more devices or networks (e.g., LANs, WANs, and / or the Internet) through a network adapter. The network adapter can communicate with the other modules of the electronic device through the bus. It should be appreciated that other hardware and / or software modules can be used in conjunction with the electronic device. For example, a microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc. can be used with the electronic device.

[0110] From the above description of the embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to make a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) execute the method according to the embodiments of the present disclosure.

[0111] The embodiments of the present disclosure also provide a computer program product, which includes program codes for causing an electronic device to perform the steps in the method according to various example embodiments of the present disclosure described above in the specification when the program product is run on the electronic device.

[0112] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art should understand that the above examples are only for illustration, and are not intended to limit the scope of the present disclosure. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present disclosure.

Claims

1. A method for recalling text fragments, characterized in that, The method includes the following steps: S100, obtain the target semantic vector T0 corresponding to the target question text and the preset text fragment semantic vector library T; T includes several preset text fragment semantic vectors, and each text fragment semantic vector corresponds to an article number; S200, determine the initial text segment semantic vectors as the text segment semantic vectors corresponding to the n text segment semantic similarities with the largest similarity between T0 and each text segment semantic vector in T; S300, based on the article number corresponding to the initial text segment semantic vector, cluster the text segment semantic vectors with the same article number into the same cluster and arrange them sequentially according to the paragraph order corresponding to the text segment semantic vector, to obtain the cluster list A = (A1, A2, ..., A...). j A m ), j=1,2,…,m; A j Let A be the j-th cluster obtained from clustering, and m be the number of clusters obtained from clustering; j = (A j，1 A j，2 A j，p A j，f（j） ), p=1,2,…,f(j);A j，p Let f(j) be the semantic vector of the p-th text segment within the j-th cluster obtained by clustering, and let f(j) be the number of semantic vectors of the text segments within the j-th cluster obtained by clustering. S400, if A j Corresponding article number N j If the article number is a preset single event, then according to f(j), from N j A was identified in the corresponding text fragment. j The corresponding recall text fragments; S500, if N j For the pre-defined multi-event article numbering, then based on f(j) and N j The degree of correlation between any two text segments in the corresponding text fragment, from N j A was identified in the corresponding text fragment. j The corresponding recall text fragments.

2. The text fragment recall method according to claim 1, characterized in that, Step S400 includes the following steps: S410, if A j Corresponding article number N j Given a pre-defined single event article number and f(j) = 1, then obtain A. j，1 Corresponding paragraph number D j，1 ; S411, N j Middle and D j，1 The text segment corresponding to the first preset number NUM1 of adjacent paragraph numbers is determined as A. j The corresponding recall text fragments.

3. The text fragment recall method according to claim 2, characterized in that, Step S400 also includes the following steps: S420, if A j Corresponding article number N j Given a pre-defined single event article number, and f(j) > 1, then obtain A. j，f（j） Corresponding paragraph number D j，f（j） With A j，1 Corresponding paragraph number D j，1 The difference in paragraph numbers between them is ΔNUM; S421, if ΔNUM = NUM1-1, then D j，f（j） D j，1 and D j，f（j） With D j，1 The text segment corresponding to the paragraph number between them is determined to be A. j The corresponding recall text fragments; S422, if ΔNUM < NUM1-1, then D j，f（j） D j，1 and D j，f（j） With D j，1 The text segment corresponding to the paragraph number between them is determined to be A. j Several corresponding recall text fragments, and will be associated with D j，f（j） Or D j，1 The text segment corresponding to several adjacent paragraph numbers is determined as A. j The corresponding number of recalled text fragments makes the number of identified recalled text fragments NUM1; S423, if ΔNUM > NUM1-1, then the text segment corresponding to the NUM1 text segments with the highest semantic similarity to T0 is determined as A. j The corresponding recall text fragments.

4. The text fragment recall method according to claim 3, characterized in that, Step S500 includes the following steps: S510, if N j Given a pre-defined multi-event article number and f(j) = 1, then obtain A. j，1 Corresponding paragraph number D j，1 ; S511, obtain D sequentially according to the adjacent order of paragraphs from nearest to farthest. j，1 The corresponding text fragment and D j，1 The corresponding article is located in D j，1 The first degree of relevance between each text fragment preceding the corresponding text fragment is the same as the degree of relevance for the same event, in order to obtain D. j，1 The corresponding first degree of association list η = (η1, η2, ..., η) a , ..., η b ), a=1,2,…,b; η a D j，1 The corresponding text fragment and D j，1 The corresponding article is located in D j，1 The first correlation between the corresponding text segment and the a-th text segment preceding it, b is the correlation between the segments located at D. j，1 The number of text segments preceding the corresponding text segment; S512, obtain D sequentially according to the adjacent order of paragraphs from nearest to farthest. j，1 The corresponding text fragment and D j，1 The corresponding article is located in D j，1 The second degree of relevance between each text fragment following the corresponding text fragment is the same event, to obtain D. j，1 The corresponding second degree of association list λ = (λ1, λ2, ..., λ) c , …, λ d ), c=1,2,…,d; η c D j，1 The corresponding text fragment and D j，1 The corresponding article is located in D j，1 The second correlation between the c-th text segment following the corresponding text segment, where d is the correlation between the segments located at D. j，1 The number of text segments preceding the corresponding text segment; S513, Obtain the first preset value Q=0; Proceed to S514; S514, if η Q+1 If RE ≥ b and Q < b, then obtain Q = Q + 1 and proceed to S514; otherwise, exit the current processing; RE is the preset correlation threshold. S515, Obtain the second preset value M=0; Proceed to S516; S516, if η M+1 If ≥RE and M<d, then obtain M=M+1 and proceed to S516; otherwise, exit the current processing. S517, if Q+M≤NUM1-1, then D j，1 The corresponding text fragment and its relation to D j，1 The preceding Q and following M text segments of the corresponding text segment are determined as A. j The corresponding recall text fragments; S518, if Q+M>NUM1-1, then D j，1 The corresponding text fragment and its relation to D j，1 The corresponding text segment is identified as A, with the NUM1-1 adjacent text segments being A. j The corresponding recall text fragments.

5. The text fragment recall method according to claim 4, characterized in that, Step S500 also includes the following steps: S520, if N j If the predefined multi-event article number is given and f(j) > 1, then obtain D. j，1 The corresponding text fragment and D j，f（j） Previously and D j，1 The third degree of relevance between each subsequent text segment; S521, obtain D j，f（j） The corresponding text fragment and D j，f（j） Previously and D j，1 The fourth degree of relevance between each subsequent text segment; S522, if every third degree of association and every fourth degree of association is greater than or equal to RE, then D will be... j，1 Corresponding text fragment, D j，f（j） The corresponding text fragment and D j，1 and D j，f（j） The text segment corresponding to the paragraph number between them is determined to be A. j The corresponding recall text fragments; S523, if the text segment corresponding to the first third relevance degree less than RE is adjacent or identical to the text segment corresponding to the first fourth relevance degree less than RE, then the text segment corresponding to each third relevance degree before the first third relevance degree less than RE and each fourth relevance degree before the first fourth relevance degree less than RE is determined as A. j The corresponding recall text fragments.

6. The text fragment recall method according to claim 5, characterized in that, Following step S523, the method further includes the following steps: S524, if A is determined j If the number of corresponding recalled text fragments is greater than NUM1, then based on each recalled text fragment and A j The text fragments in the semantic vector are filtered based on their distance from each other, so that the number of recalled text fragments is NUM1.

7. The text fragment recall method according to claim 2, characterized in that, The value range of NUM1 is [6, 10].

8. A non-transitory computer-readable storage medium, wherein the storage medium stores at least one instruction or at least one program segment, characterized in that, The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the text segment recall method as described in any one of claims 1-7.

9. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 8.

Images