A script long text semantic segmentation and key information extraction system

Abstract

The present application relates to the technical field of semantic processing, and particularly relates to a script long text semantic segmentation and key information extraction system, which comprises a memory, a processor and a computer program stored in the memory and running on the processor, and the processor implements the following steps when executing the computer program: extracting global plot semantic anchor points, and dynamically segmenting the script according to the processor display memory constraint and the maximum input length of the model; combining the script structure identification and local semantic changes to determine the candidate boundary, calculating the intra-segment semantic consistency and cross-boundary semantic consistency under the constraint of the global plot semantic anchor points, adaptively determining the segmentation threshold according to the kernel density distribution of all formed mutation parameters and reconstructing the plot semantic segment; extracting the characters, locations, times, events, key items and character relationships, and performing cross-segment entity and event merging, thereby reducing the plot context rupture caused by hardware segmentation and improving the stability of script long text semantic segmentation and structured key information extraction.

Description

Technical Field

[0001] This invention relates to the field of semantic processing technology, and in particular to a system for semantic segmentation and key information extraction of long script texts. Background Technology

[0002] In film and television production, script review, content retrieval, and intelligent assisted creation scenarios, automatically dividing a long script into semantically complete plot segments and further extracting characters, locations, times, events, key items, and character relationships from the segments is the foundation for subsequent plot analysis, character relationship construction, and content retrieval. Existing technologies typically employ fixed-length segmentation, rule-based segmentation based on line breaks or scene markers, or directly utilize pre-trained language models to perform semantic representation and information extraction on text segments.

[0003] In practical applications, long script texts often contain scene titles, time and location descriptions, character dialogues, action descriptions, narration, transition information, and plot events that continue across scenes. Their text length significantly exceeds the single input window of a conventional pre-trained model. When performing forward propagation, models with multi-head self-attention mechanisms incur significant computational and memory overhead as the length of the input discrete identifier sequence increases. Directly inputting the entire script into the model at once can easily exceed the available memory of the underlying processor or the maximum input length of the model. Therefore, it is usually necessary to first divide the script into multiple short text segments.

[0004] However, segmentation methods based on fixed length or a single line break rule cannot simultaneously take into account both hardware security boundaries and plot semantic boundaries: fixed-length segmentation may split continuous dialogue in the same scene, the cause and effect of the same event, or changes in character relationships into different processing blocks; relying solely on scene titles or line break markers may mistake formatted line breaks in the same plot event for semantic boundaries, while omitting actual plot boundaries that have changed in time, place, core characters, or event objectives but are not clearly formatted.

[0005] Furthermore, there are significant differences in dialogue density, action description density, scene length, and narrative rhythm among different types of scripts. If a uniform fixed threshold is used to judge semantic changes, it is easy to cause excessive segmentation in scripts with frequent dialogue, or to miss the real boundaries in long-scene narrative scripts. On the other hand, when information is extracted independently from each local segment, the name, abbreviation, title, or pronoun of the same character in different segments may be identified as different entities, and interconnected events may also lose their connection due to hardware slicing.

[0006] Therefore, a processing system is needed that can safely segment long script texts under the constraints of the underlying processor's processing power, while maintaining semantic connections across segments using global plot information, adaptively determining the true segmentation boundaries based on the semantic changes of the current script itself, and finally completing the unified merging of key information across segments. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide a semantic segmentation and key information extraction system for long script texts to solve the problems that are easy to cause the plot context to break when long script texts are segmented at a fixed length, that a uniform segmentation threshold is difficult to adapt to the semantic fluctuations of different scripts, and that the independent extraction of local segments leads to the separation of cross-segment entities and event relationships.

[0008] This invention provides a system for semantic segmentation and key information extraction of long script texts, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it performs the following steps:

[0009] Extract the global plot semantic anchor sequence from the long script text to be processed, and perform text segmentation and reconstruction operation based on the underlying processor memory usage constraints and the maximum input length of the pre-trained semantic evaluation model to obtain multiple local benchmark text blocks;

[0010] Based on the script structure identifiers and local semantic changes, candidate segment positions are determined in each local baseline text block, and a front context window and a back context window are constructed around each candidate segment position.

[0011] Using a pre-trained semantic evaluation model, forward propagation is performed on the preceding and following context windows under the constraint of the global plot semantic anchor sequence to parse out intra-segment semantic consistency features and cross-boundary semantic consistency features.

[0012] Based on algebraic operations of intra-segment semantic consistency features and cross-boundary semantic consistency features, semantic boundary mutation parameters are extracted.

[0013] All semantic boundary mutation parameters corresponding to the long text of the script are projected into a continuous kernel density distribution curve. Based on the kernel density distribution curve, the adaptive segmentation threshold corresponding to the current script is determined. The real semantic boundary is determined according to the relationship between the semantic boundary mutation parameters and the adaptive segmentation threshold. Multiple plot semantic segments are reconstructed.

[0014] Extract characters, locations, times, events, key items, and character relationships from each plot semantic segment. Based on the global plot semantic anchor point sequence, normalize and merge the same entity and related events across plot semantic segments to output a set of structured key information for the script.

[0015] Preferably, the extraction of the global plot semantic anchor point sequence includes:

[0016] The long script text to be processed is parsed into an independent set of discrete identifiers, and character name identifiers, scene identifiers, time identifiers, location identifiers, scene transition identifiers, and key item identifiers in the script are identified.

[0017] Count the frequency of each discrete identifier in the current script long text, multiply the frequency by the corresponding value in the inverse document frequency dictionary, and output the feature weight distribution array of each discrete identifier.

[0018] The discrete identifiers are sorted in descending order according to the numerical value of the feature weight distribution array, and the character name identifier, location identifier, key item identifier, and a preset number of core discrete identifiers with the highest feature weight ranking are determined as plot semantic anchors.

[0019] The global plot semantic anchor sequence is constructed by concatenating a one-dimensional array according to the order of the first occurrence of each plot semantic anchor in the long text of the script to be processed, and retaining the same plot semantic anchor only once.

[0020] Preferably, the text segmentation and reconstruction operation includes:

[0021] Read the maximum available remaining GPU memory of the current underlying processor, the hidden layer feature dimension of the pre-trained semantic evaluation model, and the maximum allowed input sequence length of the pre-trained semantic evaluation model;

[0022] Based on the available maximum remaining memory and the hidden layer feature dimension, the maximum discrete identifier sequence length under memory constraints is determined, and the smaller value between the maximum discrete identifier sequence length under memory constraints and the maximum input sequence length is determined as the maximum discrete identifier sequence length allowed by the single forward propagation theory.

[0023] Starting from the theoretical truncation position of the current interval to be processed and moving towards the starting position of the current interval to be processed, a backward traversal is performed within a preset maximum backoff range that is less than the maximum allowable discrete identifier sequence length. The end-of-session marker, transition marker, paragraph line break marker, and sentence end marker are matched in sequence. If a corresponding marker is matched, the safety context backoff distance is locked. If no corresponding marker is matched, the safety context backoff distance is set to zero.

[0024] The actual dynamic slice length is obtained by subtracting the security context backoff distance from the maximum allowed discrete identifier sequence length. The actual dynamic slice length is used to perform cyclic cutting, and hardware slice boundaries are written between adjacent local reference text blocks. The local reference text blocks are then output.

[0025] Preferably, determining the candidate segment position includes:

[0026] Scan the scene titles, time and location markers, dialogue start markers, transition markers and paragraph line break markers in the local reference text blocks, and read the hardware slice boundaries generated by text segmentation and reconstruction between adjacent local reference text blocks, and write the corresponding positions into the structure candidate boundary set.

[0027] The similarity between local semantic representations is calculated using two adjacent text windows of a preset length as a unit. When the similarity is lower than the preset candidate change limit, the corresponding window boundary position is written into the semantic candidate boundary set.

[0028] Perform position deduplication and neighbor position merging on the structural candidate boundary set and the semantic candidate boundary set to obtain the candidate segment position set; take the candidate segment position as the center and extract a preset number of discrete identifiers before and after it to obtain the front context window and the back context window.

[0029] Preferably, parsing the intra-segment semantic consistency features and cross-boundary semantic consistency features includes:

[0030] Based on the maximum input length allowed by the pre-trained semantic evaluation model and the maximum available remaining GPU memory of the current underlying processor, length constraints are applied to the global plot semantic anchor sequence, the front context window, and the back context window to ensure that any evaluation sequence satisfies the model input length constraint and GPU memory usage constraint.

[0031] The length-constrained global plot semantic anchor sequence is concatenated to the input headers of the front and back context windows respectively, forming a two-sided evaluation sequence constrained by the global plot.

[0032] The logarithmic mean of the conditional probabilities within the front context window and the back context window is calculated using a pre-trained semantic evaluation model. The arithmetic mean of the results on both sides is then calculated to obtain the semantic consistency feature within the segment.

[0033] The front context window is used as the historical context, and the back context window is used as the sequence to be evaluated. Under the joint constraint of the global plot semantic anchor sequence with length constraints, the logarithmic mean of the conditional probability of each discrete identifier in the back context window is calculated to obtain the cross-boundary semantic consistency feature.

[0034] Preferably, the extraction of the semantic boundary mutation parameters includes:

[0035] Intra-segment semantic consistency features are used as the first semantic level parameter, and cross-boundary semantic consistency features are used as the second semantic level parameter.

[0036] The semantic splitting difference parameter is obtained by performing a subtraction operation with the first semantic level parameter as the minuend and the second semantic level parameter as the subtrahend.

[0037] The semantic complexity basis parameters are obtained by performing arithmetic addition on the first semantic level parameters and the second semantic level parameters.

[0038] Using the semantic splitting difference parameter as the numerator and the sum of the absolute value of the semantic complexity basis parameter and the preset positive stability constant as the denominator, a division operation is performed to obtain the semantic boundary mutation parameter.

[0039] Preferably, determining the adaptive segmentation threshold includes:

[0040] Extract the semantic boundary mutation parameters generated by all candidate segment positions corresponding to the long text of the script to be processed, and form a one-dimensional observation point set; call the Gaussian kernel density evaluation module, introduce the Gaussian kernel function, and perform continuous smooth expansion on the one-dimensional observation point set to form the kernel density distribution curve of the semantic boundary mutation parameters;

[0041] Search for the main density peak and the secondary density region on the side of high mutation parameter in the nuclear density distribution curve, and fix the abscissa corresponding to the local density valley between the two as the adaptive segmentation threshold of the current script.

[0042] When no distinguishable sub-density region is formed, the preset high quantile of the semantic boundary mutation parameter set is used as the adaptive segmentation threshold.

[0043] Preferably, the reconstructed plot semantic segment includes:

[0044] Read the semantic boundary mutation parameters corresponding to the candidate segment positions one by one;

[0045] When the semantic boundary mutation parameter is greater than or equal to the adaptive segmentation threshold, the corresponding candidate segmentation position is marked as the true semantic boundary.

[0046] The text is segmented according to the position order of the true semantic boundaries in the long script text to be processed, and adjacent local reference text blocks that are generated by text segmentation and reconstruction based on the underlying processor memory usage constraints and the maximum input length of the pre-trained semantic evaluation model but are not marked as true semantic boundaries are reassembled to obtain multiple plot semantic segments.

[0047] Preferably, the step of extracting characters, locations, times, events, key items, and character relationships from each plot semantic segment, and performing cross-segment entity and event merging, includes:

[0048] Identify character names, locations, times, and key items in each plot semantic segment;

[0049] Identify character behaviors, event trigger words, event objects, event results, and the interaction relationships between characters;

[0050] Based on the global plot semantic anchor sequence, entity name similarity, character co-occurrence relationship, contextual reference relationship and the time order of adjacent events, entities pointing to the same object in different plot semantic segments are uniformly identified, and event nodes with sequential relationships are associated.

[0051] A structured set of key information for the script is generated based on semantic segment identifiers, character identifiers, location identifiers, time identifiers, event identifiers, key item identifiers, and character relationship identifiers.

[0052] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows:

[0053] 1. This invention combines the global plot semantic anchor extraction mechanism with the underlying processor memory limit. After determining the theoretical boundary that simultaneously satisfies the underlying processor memory constraint and the maximum input length of the pre-trained semantic evaluation model, the script structure identifier is used to perform context semantic backoff cutting, so that the generated local benchmark text block takes into account both hardware security and plot structure integrity, and the global plot semantic anchor is used to compensate for the lack of span information caused by long text slices.

[0054] 2. This invention does not equate line breaks with actual semantic boundaries. Instead, it constructs context windows on both sides of the candidate position and quantifies plot semantic mutations by the relative changes in intra-segment semantic consistency and cross-boundary semantic consistency. This allows positions where time, location, main character combinations, or event targets undergo substantial changes to receive higher boundary responses, reducing pseudo-boundary interference caused by dialogue line breaks, action description line breaks, and hardware slicing.

[0055] 3. This invention utilizes the relative mapping between semantic parameter differences and sums to weaken the basis drift caused by different script language complexities, and determines the adaptive segmentation threshold based on the kernel density distribution of semantic boundary mutation parameters of the entire script itself, so that the segmentation scale changes with the dialogue density and narrative rhythm of the current script.

[0056] 4. After completing the semantic segmentation of the plot, this invention performs structured extraction of characters, locations, times, events, key items and character relationships in each plot semantic segment, and uses global plot semantic anchors to perform cross-segment entity unification and event chain merging, reducing the situation where the same character is repeatedly modeled due to changes in aliases, titles or pronouns, and continuous events are broken due to hardware fragmentation. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart of a method for semantic segmentation and key information extraction of long script text provided in Embodiment 1 of the present invention. Detailed Implementation

[0059] The embodiments of this disclosure are described in detail below. The embodiments described below are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0060] It should be noted that the terms "first," "second," etc., used in this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0061] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0062] This invention provides a system for semantic segmentation and key information extraction of long script texts, including a processor and a memory. The processor executes a computer program stored in the memory to implement a method for semantic segmentation and key information extraction of long script texts. The method includes the following steps:

[0063] Step S101: Extract the global plot semantic anchor sequence from the long script text to be processed, and perform text segmentation and reconstruction operation based on the underlying processor memory usage constraints and the maximum input length of the pre-trained semantic evaluation model to obtain multiple local benchmark text blocks.

[0064] It should be noted that pre-trained language models typically employ a neural network architecture with a multi-head self-attention mechanism, capturing contextual features through operations between the query matrix, key matrix, and value matrix. As the length of the input discrete identifier sequence increases, the self-attention-related intermediate tensors and attention matrix lead to a significant increase in computational and memory overhead. For extremely long sequences formed by the entire script, directly loading them into the model's memory at once can easily exceed the model's context window or the available memory of the current underlying processor. Therefore, this invention first determines the single processing length based on the available memory of the underlying processor and the maximum input length of the pre-trained semantic evaluation model, and then backs away from this length based on script structure identifiers, ensuring that the safe slice position falls as close as possible to the text boundary with a complete plot structure.

[0065] Specifically, the server computing node receives the long text of the script to be processed through the network communication interface, loads it into system memory, and calls the natural language segmentation component to parse the script into a set of independent discrete identifiers. Simultaneously, it calls the script structure parsing interface to identify scene titles, character names, time and location descriptions, dialogue start markers, transition markers, and key item candidate entities. The system calls the background corpus inverse document frequency dictionary in local non-volatile memory, inputs the discrete identifiers into the word frequency and inverse document frequency feature calculation module, counts the frequency of each discrete identifier in the current script, and multiplies it by the corresponding inverse document frequency to form a feature weight distribution array.

[0066] The system sorts the feature weight distribution array in descending order of value, selects a preset number of high-weight core discrete identifiers, and retains the main character names, high-frequency locations, key items, and recurring event target words identified by the script structure parsing interface, using them as unified plot semantic anchors. For the same plot semantic anchor that appears repeatedly, it is retained only once, and a one-dimensional array is concatenated according to the order of the first appearance of each plot semantic anchor in the long text of the script to be processed, resulting in a global plot semantic anchor sequence. This sequence is repeatedly called in each subsequent local block semantic evaluation and cross-segment merging process to maintain a unified reference for the core plot variables of the entire script in each local processing cycle.

[0067] In one embodiment, the preset number of high-weight core discrete identifiers ranges from 20 to 50, preferably 30. Character names, scene locations, and key items are not limited by this preset number of high-weight core discrete identifiers; they are written into the global plot semantic anchor sequence when they meet the structural recognition confidence requirements. It should be noted that this global plot semantic anchor sequence can be completely retained when used as a cross-segment merging reference. When used as the input context of a pre-trained semantic evaluation model, it must, together with the current context window, satisfy the model's maximum input length and the underlying processor's memory constraints. If the combined input exceeds these constraints, the anchor content actually participating in this model evaluation is truncated according to the feature weights of the plot semantic anchors from high to low.

[0068] Furthermore, to determine the allowed input length for a single forward propagation, let the maximum remaining GPU memory currently available for model inference on the underlying processor be . The hidden layer feature dimension of the pre-trained semantic evaluation model is Considering the engineering memory overhead coefficients corresponding to the attention matrix and intermediate tensors. Then the length of a single input is The approximate memory constraint expression for time is:

[0069]

[0070] In the formula, This represents the estimated video memory requirement for the current input length. Indicates the length of the discrete identifier sequence; Indicates the dimension of the hidden layer features; The project memory overhead factor is greater than zero, and its value is determined according to... and The memory capacity unit used is calibrated and determined by the model structure, number of attention heads, data precision, batch size, intermediate tensor retention strategy, and inference framework.

[0071] Will Not greater than As a memory safety constraint, rounding down the calculated sequence length yields the maximum discrete identifier sequence length under the memory constraint. :

[0072]

[0073] Furthermore, the maximum allowed input sequence length of the pre-trained semantic evaluation model is read. The smaller of the maximum discrete identifier sequence length under memory constraints and the maximum input sequence length of the model is determined as the maximum discrete identifier sequence length allowed by the single forward propagation theory. :

[0074]

[0075] It should be noted that the above expression is used to obtain the engineering safety benchmark that simultaneously satisfies the processor's video memory and the model's input window under the current running state. In the specific implementation, the benchmark can be adjusted based on the video memory statistics of the model used. Perform offline calibration and reserve a safety margin.

[0076] Furthermore, the system sets a data addressing cursor in the memory-mapped area of ​​the long script text to be processed, and in each processing cycle, moves this cursor from the theoretical truncation position of the current processing interval. Traverse backwards from the starting position of the current interval to be processed, one discrete identifier at a time; during the traversal, match the end of the scene, transition, paragraph newline, period, question mark and exclamation mark in sequence.

[0077] The preset maximum yield range is less than If a script structure boundary is matched within this range, the traversal stops and the discrete identifier offset from the theoretical truncation position to that boundary position is recorded, thus obtaining the safety context backoff distance. If no available structural boundary is found, then... Set to zero.

[0078] Actual dynamic slice length for:

[0079]

[0080] The system follows Perform text segmentation for the current loop cycle and write "hardware slice boundary" markers at the boundaries of adjacent local reference text blocks; when At that time, the corresponding cutting position is The corresponding hard truncation position. The hardware slice boundary only represents the underlying computation slice and is not directly used as the actual semantic boundary. Instead, it undergoes semantic judgment during the subsequent candidate boundary evaluation process.

[0081] For example, using a suspense script containing 120 scenes and approximately 100,000 words as the long text to be processed, the system parses out key characters, locations, items, and event words such as Lin Qi, Captain Zhou, the old warehouse, the recording pen, and the missing persons case, and combines them with other high-weight discrete identifiers to form a global plot semantic anchor sequence. The theoretical maximum length, determined by the maximum length constrained by the memory and the maximum input sequence length of the pre-trained semantic evaluation model, is 2048 discrete identifiers. The 2048th position is located in the middle of the dialogue. The system scans forward and finds a transition marker at the 2017th position, so the safe context backoff distance is 31, and the actual dynamic slice length is 2017. After slicing, the system uses the 2018th position as the starting point for subsequent loop processing.

[0082] Step S102: Determine the candidate segment positions based on script structure identifiers and local semantic changes, and construct a front context window and a back context window around each candidate segment position.

[0083] It should be noted that format boundaries in a script are not necessarily equivalent to semantic boundaries in the plot. For example, dialogue is usually broken by character, but consecutive dialogue may still revolve around the same event; conversely, some scripts may use narrative prompts such as "a moment later" or "on the other side" to indicate a shift in time or focus within the same scene, but without a new scene number. Therefore, this invention uses structural identifiers and local semantic changes only to generate candidate positions, and then subsequent bilateral semantic evaluation determines whether they constitute a true boundary.

[0084] Specifically, the candidate boundary construction module sequentially scans the local baseline text block, reads the scene title, time and location markers, dialogue start markers, transition markers, paragraph line break markers, and hardware slice boundaries written in stage S101, and writes these positions into the structural candidate boundary set; among them, a minimum interval constraint can be set for continuous dialogue line breaks to avoid each line of dialogue directly forming an independent candidate point.

[0085] Furthermore, the system calls the semantic encoder to obtain local semantic representations using two adjacent text windows of preset length as units, representing the left window of the candidate position as a vector. The right window is represented as a vector. And calculate the cosine similarity between the two:

[0086]

[0087] When the similarity Below the preset candidate change limit When the window boundary is reached, the corresponding boundary position is written into the semantic candidate boundary set. The candidate change boundary is only used to improve the recall rate and is not used as the final segmentation threshold.

[0088] The system merges the set of structural candidate boundaries and the set of semantic candidate boundaries, performs clustering on multiple candidate locations falling within a preset proximity distance range, and retains the locations with the highest structural priority or the lowest local similarity, thus obtaining the final set of candidate segmentation locations. For each candidate segment position The system extracts discrete identifiers of a preset length from its left side to form the front context window. And extract discrete identifiers of the same or similar length from its right side to form the back context window. When a window capture encounters the boundary of a local reference text block, it can read continuous content from adjacent text blocks, so that hardware fragmentation does not block the semantic evaluation of candidate positions.

[0089] Step S103: Using a pre-trained semantic evaluation model, forward propagation is performed on the context of both sides of the candidate boundary under the constraint of global plot semantic anchor point to parse out the semantic consistency features within the segment and the semantic consistency features across the boundary.

[0090] It should be noted that different script fragments have varying vocabulary difficulty and syntactic forms, and directly comparing the single absolute probability of different candidate positions is easily affected by language style. Therefore, this invention constructs two evaluation dimensions simultaneously at the same candidate position: whether the internal components on both sides of the boundary are continuous, and whether continuity remains after crossing from the front to the back. This allows the relative changes in the two-dimensional results at the same position to determine plot breakpoints.

[0091] Specifically, candidate positions are set. The front context window contains Discrete identifiers The back context window contains Discrete identifiers Before constructing the model evaluation sequence, the system performs length checks on the global plot semantic anchor sequence, the preceding context window, and the following context window to ensure that the total number of discrete identifiers in any actual evaluation sequence does not exceed [a certain value]. When the combined input exceeds First, based on the feature weights of the plot semantic anchor points, the anchor point content actually participating in this evaluation is extracted from high to low, and then, if it still exceeds [a certain threshold], [the remaining content is considered]. The context window will shorten accordingly. The global plot semantic anchor sequence follows below. This represents the anchor point content that is actually added to the model evaluation sequence after the above length constraints.

[0092] The system uses a pre-trained semantic evaluation model to calculate the single-step conditional probability of each discrete identifier within the front and back windows under its preceding context, and takes the natural logarithmic mean. The internal consistency scores for the front and back windows are as follows:

[0093]

[0094]

[0095] and These represent the sequence of preceding discrete identifiers within the corresponding context window.

[0096] The candidate positions are obtained by arithmetically averaging the internal consistency scores of both sides. Intra-segment semantic consistency features :

[0097]

[0098] Furthermore, the system will link the global plot semantic anchor sequence G with the preceding context window. Together, they serve as the prior context of the subsequent context window. The log-mean conditional probability of each discrete identifier in the subsequent context is calculated to obtain the cross-boundary semantic consistency feature. :

[0099]

[0100] When both sides of a candidate position belong to the same continuous plot event, the preceding context can provide effective semantic constraints for the dialogue, actions, or event outcomes of the following position. and The differences are relatively small; when there are substantial changes in the time, location, main character combination, or event objective of the candidate position, the conditional support of the preceding context for the following content weakens. Compared to This resulted in a more significant decrease. Global narrative semantic anchor points. To maintain consistency in both evaluation methods, the model should avoid losing information about the core characters and plot themes of the entire script due to hardware slicing of long texts.

[0101] Step S104: Based on the spatial anisotropic deviation and cotropic basis of intra-segment semantic consistency features and cross-boundary semantic consistency features, calculate the semantic boundary mutation parameters.

[0102] It should be noted that the absolute logarithmic probability level output by the pre-trained model may fall within different ranges for dialogue-dense slice-of-life scripts and terminology-dense industry scripts; even within the same script, the language of action descriptions and character dialogue can alter the probability basis. Therefore, this invention does not directly rely on... Instead of using the absolute value to determine the threshold, we first calculate the difference between the two evaluation dimensions for the same candidate position, and then perform relative normalization based on their common basis.

[0103] Specifically, the semantic splitting difference parameter is calculated by using intra-segment semantic consistency features as the first semantic level parameter and cross-boundary semantic consistency features as the second semantic level parameter. :

[0104]

[0105] The first semantic level parameter and the second semantic level parameter are summed by arithmetic addition to form the semantic complexity basis parameter, which represents the overall language decoding difficulty of the current candidate position. :

[0106]

[0107] Then use absolute value pairs Normalization is performed to obtain semantic boundary mutation parameters. :

[0108]

[0109] In the formula, Indicates candidate position Semantic boundary mutation parameters; This indicates the difference between intra-segment semantic consistency and cross-boundary semantic consistency. This represents the semantic complexity basis formed by both factors. If both sides of a candidate position belong to the same plot chain, then the cross-boundary evaluation is close to the bilateral intra-position evaluation. Smaller The position is close to the low end; if the candidate position constitutes a real plot transition, then the cross-boundary consistency is significantly reduced. Increase, thus Shift towards higher value regions.

[0110] Step S105: Project all semantic boundary mutation parameters into a continuous kernel density distribution curve, adaptively determine the segmentation threshold corresponding to the current script, and reconstruct multiple plot semantic segments based on the threshold.

[0111] It should be noted that different scripts vary in narrative pacing, dialogue frequency, and scene length. Using uniform, fixed values ​​as... The boundary judgment threshold is prone to cross-script mismatch. Therefore, this invention utilizes the boundary judgment threshold of all candidate positions in the current script itself. The distribution determines the judgment boundary.

[0112] Specifically, suppose the current script has n candidate segment positions, and their semantic boundary mutation parameters form a one-dimensional set of observation points. The system calls the Gaussian kernel density evaluation module, on the continuous horizontal axis. Estimating the parameter density function:

[0113]

[0114]

[0115] In the formula, This represents the kernel bandwidth that is greater than zero. This represents the Gaussian kernel function. Most candidate positions, formed solely by formatting breaks, normal dialogue switching, or hardware slicing, are typically concentrated in lower... The regions form a main density peak, while the locations where there are substantial changes in scene, time, character combinations, or event objectives form relatively higher density peaks. And may form a sub-density region on the high-value side.

[0116] The system moves from the location of the main density peak to higher... In the directional search, when a sub-density region with a preset minimum peak significance is detected, the x-coordinate corresponding to the local density valley between the main density peak and the sub-density region is read, and this x-coordinate is fixed as the adaptive segmentation threshold of the current script. When the sample size is too small or the kernel density distribution does not form a resolvable subdensity region, the system will... Preset high quantile of the set As ,in It can take values ​​in the range of 0.80 to 0.95.

[0117] Finally, the system reads the corresponding candidate positions one by one. ,when Greater than or equal to At that time, the candidate positions Mark it as a true semantic boundary; otherwise, treat it as a non-true boundary. After completing the judgment of all candidate positions, perform segmentation according to the order of the true semantic boundaries in the long text of the script to be processed; for hardware slice boundaries in S101 that are generated by text segmentation reconstruction but are not marked as true semantic boundaries, the system reassembles the text before and after them into the same plot semantic segment, thereby decoupling the hardware-level segmentation from the final semantic-level segmentation.

[0118] For example, in the same scene, the continuous dialogue of "Lin Qi checks the recording pen - Captain Zhou continues to question - Lin Qi provides supplementary explanation" maintains a high degree of cross-boundary semantic consistency even with multiple line breaks. It is in the low-value zone; however, when the text subsequently enters a new scene "the next morning, outside the old warehouse," the time, location, and characters all change, corresponding to... Significantly increased and exceeded The system identifies this location as the semantic boundary of the actual plot.

[0119] Step S106: Extract characters, locations, times, events, key items, and character relationships from each plot semantic segment, and perform cross-segment entity and event merging to generate a set of structured key information for the script.

[0120] It should be noted that semantic segmentation only addresses the boundary issues of the plot structure. If each plot semantic segment continues to be extracted independently and output directly, the same character may still be repeatedly modeled due to its full name, abbreviation, job title, or pronoun, or a continuous event may be broken down into unrelated nodes in adjacent plot segments. Therefore, this invention performs cross-segment unification after extraction within a segment.

[0121] Specifically, after the reconstruction of the plot semantic segments is completed, the corresponding text content is read in sequence according to the order of each plot semantic segment in the long script text to be processed, the character names, locations, times and key items in each plot semantic segment are identified, and the plot semantic segment to which each identification result belongs and its text position in the corresponding plot semantic segment are recorded.

[0122] Furthermore, the character behaviors, event trigger words, event objects, and event results in each plot semantic segment are identified. Based on the semantic association between character names and corresponding behaviors, event trigger words, and event objects, the events in which the characters participate and the interaction relationships between the characters are determined, thereby forming character information, location information, time information, event information, key item information, and character relationship information corresponding to each plot semantic segment.

[0123] Furthermore, after extracting key information within each plot semantic segment, the global plot semantic anchor point sequence constructed in stage S101 is read, and plot semantic anchor points such as character names, locations, and key items are used as the basis for cross-plot semantic segment information comparison. Cross-segment correspondence verification is performed on the entity information extracted from different plot semantic segments.

[0124] Specifically, for entities with different names or titles in different plot semantic segments but potentially pointing to the same object, joint verification is performed based on entity name similarity, character co-occurrence relationships, contextual referential relationships, location continuity, and the chronological order of adjacent events. When an entity to be verified can be mapped to the same plot semantic anchor point based on its name or contextual referential relationships, or when it can be determined that it points to the same plot object by combining character co-occurrence relationships, location continuity, and event chronological order, the corresponding entities are unified under the same entity identifier; when it is impossible to determine that different entities point to the same plot object based on the above information, the corresponding entities remain independent of each other, and forced merging is not performed.

[0125] For example, when the character name "Lin Qi" appears in the current plot semantic segment, and "Xiao Lin" and "Dr. Lin" appear in subsequent plot semantic segments, the system checks the correspondence between "Xiao Lin" and "Dr. Lin" in the global plot semantic anchor sequence based on the co-occurrence relationship of the characters, the contextual reference relationship, and the temporal sequence relationship of adjacent events in the text where "Xiao Lin" and "Dr. Lin" are located. When all of the above titles can uniquely point to "Lin Qi", "Lin Qi", "Xiao Lin" and "Dr. Lin" are unified under the same character identifier, thereby avoiding the same character being recorded repeatedly due to the use of different titles across plot semantic segments.

[0126] Furthermore, cross-segment associations are performed on events extracted from different plot semantic segments. For events that are adjacent in plot time sequence or have a continuity relationship of event objects, the participating characters, event objects, key items, and the event results of the previous event are compared. When a subsequent event uses the same character, event object, or key item as the processing object as the previous event, or uses the event result of the previous event as its trigger condition, the corresponding event is identified as an event node with a sequential relationship, and an event association relationship is established.

[0127] For example, if the previous plot semantic segment extracts the event "Lin Qi discovers the recording pen", the next plot semantic segment extracts the event "Lin Qi plays the recording", and the subsequent plot semantic segment extracts the event "investigate the warehouse based on the recording content", then since the recording information involved in the next event comes from the key item discovered in the previous event, and there is no conflict in the time sequence between the events, the above event nodes are sequentially established to establish a sequential relationship.

[0128] Finally, after unifying cross-segment entities and associating events, the key information corresponding to each plot semantic segment is structured according to plot semantic segment identifiers, character identifiers, location identifiers, time identifiers, event identifiers, key item identifiers, and character relationship identifiers, generating a set of structured key information for the script corresponding to the long script text to be processed.

[0129] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A system for semantic segmentation and key information extraction of long script texts, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it performs the following steps: Extract the global plot semantic anchor sequence from the long script text to be processed, and perform text segmentation and reconstruction operation based on the underlying processor memory usage constraints and the maximum input length of the pre-trained semantic evaluation model to obtain multiple local benchmark text blocks; Based on the script structure identifiers and local semantic changes, candidate segment positions are determined in each local baseline text block, and a front context window and a back context window are constructed around each candidate segment position. Using a pre-trained semantic evaluation model, forward propagation is performed on the preceding and following context windows under the constraint of the global plot semantic anchor sequence to parse out intra-segment semantic consistency features and cross-boundary semantic consistency features. Based on algebraic operations of intra-segment semantic consistency features and cross-boundary semantic consistency features, semantic boundary mutation parameters are extracted. All semantic boundary mutation parameters corresponding to the long text of the script are projected into a continuous kernel density distribution curve. Based on the kernel density distribution curve, the adaptive segmentation threshold corresponding to the current script is determined. The real semantic boundary is determined according to the relationship between the semantic boundary mutation parameters and the adaptive segmentation threshold. Multiple plot semantic segments are reconstructed. Extract characters, locations, times, events, key items, and character relationships from each plot semantic segment. Then, based on the global plot semantic anchor point sequence, perform normalization and merging on the same entity and related events across plot semantic segments to output a set of structured key information for the script.

2. The system for semantic segmentation and key information extraction of long script texts according to claim 1, characterized in that, The extraction of the global plot semantic anchor point sequence includes: The long script text to be processed is parsed into an independent set of discrete identifiers, and character name identifiers, scene identifiers, time identifiers, location identifiers, scene transition identifiers, and key item identifiers in the script are identified. Count the frequency of each discrete identifier in the current script long text, multiply the frequency by the corresponding value in the inverse document frequency dictionary, and output the feature weight distribution array of each discrete identifier. The discrete identifiers are sorted in descending order according to the numerical value of the feature weight distribution array, and the character name identifier, location identifier, key item identifier, and a preset number of core discrete identifiers with the highest feature weight ranking are determined as plot semantic anchors. The global plot semantic anchor sequence is constructed by concatenating a one-dimensional array according to the order of the first occurrence of each plot semantic anchor in the long text of the script to be processed, and retaining the same plot semantic anchor only once.

3. The system for semantic segmentation and key information extraction of long script texts according to claim 1, characterized in that, The text fragment reconstruction operation includes: Read the maximum available remaining GPU memory of the current underlying processor, the hidden layer feature dimension of the pre-trained semantic evaluation model, and the maximum allowed input sequence length of the pre-trained semantic evaluation model; Based on the available maximum remaining memory and the hidden layer feature dimension, the maximum discrete identifier sequence length under memory constraints is determined, and the smaller value between the maximum discrete identifier sequence length under memory constraints and the maximum input sequence length is determined as the maximum discrete identifier sequence length allowed by the single forward propagation theory. Starting from the theoretical truncation position of the current interval to be processed and moving towards the starting position of the current interval to be processed, a backward traversal is performed within a preset maximum backoff range that is less than the maximum allowable discrete identifier sequence length. The end-of-session marker, transition marker, paragraph line break marker, and sentence end marker are matched in sequence. If a corresponding marker is matched, the safety context backoff distance is locked. If no corresponding marker is matched, the safety context backoff distance is set to zero. The actual dynamic slice length is obtained by subtracting the security context backoff distance from the maximum allowed discrete identifier sequence length. The actual dynamic slice length is used to perform cyclic cutting, and hardware slice boundaries are written between adjacent local reference text blocks. The local reference text blocks are then output.

4. The system for semantic segmentation and key information extraction of long script texts according to claim 1, characterized in that, Determining the candidate segment position includes: Scan the scene titles, time and location markers, dialogue start markers, transition markers and paragraph line break markers in the local reference text blocks, and read the hardware slice boundaries generated by text segmentation and reconstruction between adjacent local reference text blocks, and write the corresponding positions into the structure candidate boundary set. The similarity between local semantic representations is calculated using two adjacent text windows of a preset length as a unit. When the similarity is lower than the preset candidate change limit, the corresponding window boundary position is written into the semantic candidate boundary set. Perform position deduplication and neighbor position merging on the structural candidate boundary set and the semantic candidate boundary set to obtain the candidate segment position set; take the candidate segment position as the center and extract a preset number of discrete identifiers before and after it to obtain the front context window and the back context window.

5. A system for semantic segmentation and key information extraction of long script texts according to claim 4, characterized in that, The analysis of intra-segment semantic consistency features and cross-boundary semantic consistency features includes: Based on the maximum input length allowed by the pre-trained semantic evaluation model and the maximum available remaining GPU memory of the current underlying processor, length constraints are applied to the global plot semantic anchor sequence, the front context window, and the back context window to ensure that any evaluation sequence satisfies the model input length constraint and GPU memory usage constraint. The length-constrained global plot semantic anchor sequence is concatenated to the input headers of the front and back context windows respectively, forming a two-sided evaluation sequence constrained by the global plot. The logarithmic mean of the conditional probabilities within the front context window and the back context window is calculated using a pre-trained semantic evaluation model. The arithmetic mean of the results on both sides is then calculated to obtain the semantic consistency feature within the segment. The front context window is used as the historical context, and the back context window is used as the sequence to be evaluated. Under the joint constraint of the global plot semantic anchor sequence with length constraints, the logarithmic mean of the conditional probability of each discrete identifier in the back context window is calculated to obtain the cross-boundary semantic consistency feature.

6. The system for semantic segmentation and key information extraction of long script texts according to claim 5, characterized in that, Extracting the semantic boundary mutation parameters includes: Intra-segment semantic consistency features are used as the first semantic level parameter, and cross-boundary semantic consistency features are used as the second semantic level parameter. The semantic splitting difference parameter is obtained by performing a subtraction operation with the first semantic level parameter as the minuend and the second semantic level parameter as the subtrahend. The semantic complexity basis parameters are obtained by performing arithmetic addition on the first semantic level parameters and the second semantic level parameters. Using the semantic splitting difference parameter as the numerator and the sum of the absolute value of the semantic complexity basis parameter and the preset positive stability constant as the denominator, a division operation is performed to obtain the semantic boundary mutation parameter.

7. A system for semantic segmentation and key information extraction of long script texts according to claim 1, characterized in that, Determining the adaptive segmentation threshold includes: Extract the semantic boundary mutation parameters generated by all candidate segment positions corresponding to the long text of the script to be processed, and form a one-dimensional observation point set; call the Gaussian kernel density evaluation module, introduce the Gaussian kernel function, and perform continuous smooth expansion on the one-dimensional observation point set to form the kernel density distribution curve of the semantic boundary mutation parameters; Search for the main density peak and the secondary density region on the side of high mutation parameter in the nuclear density distribution curve, and fix the abscissa corresponding to the local density valley between the two as the adaptive segmentation threshold of the current script. When no distinguishable sub-density region is formed, the preset high quantile of the semantic boundary mutation parameter set is used as the adaptive segmentation threshold.

8. A system for semantic segmentation and key information extraction of long script texts according to claim 7, characterized in that, The reconstructed plot semantic segment includes: Read the semantic boundary mutation parameters corresponding to the candidate segment positions one by one; When the semantic boundary mutation parameter is greater than or equal to the adaptive segmentation threshold, the corresponding candidate segmentation position is marked as the true semantic boundary. The text is segmented according to the position order of the true semantic boundaries in the long script text to be processed, and adjacent local reference text blocks that are generated by text segmentation and reconstruction based on the underlying processor memory usage constraints and the maximum input length of the pre-trained semantic evaluation model but are not marked as true semantic boundaries are reassembled to obtain multiple plot semantic segments.

9. A system for semantic segmentation and key information extraction of long script texts according to claim 1, characterized in that, The extraction of characters, locations, times, events, key items, and character relationships from each plot semantic segment, and the merging of cross-segment entities and events, includes: Identify character names, locations, times, and key items in each plot semantic segment; Identify character behaviors, event trigger words, event objects, event results, and the interaction relationships between characters; Based on the global plot semantic anchor sequence, entity name similarity, character co-occurrence relationship, contextual reference relationship and the time order of adjacent events, entities pointing to the same object in different plot semantic segments are uniformly identified, and event nodes with sequential relationships are associated. A structured set of key information for the script is generated based on semantic segment identifiers, character identifiers, location identifiers, time identifiers, event identifiers, key item identifiers, and character relationship identifiers.

Images