An advertisement material multi-modal content conflict identification method

Abstract

The application discloses an advertisement material multi-modal content conflict identification method, and particularly relates to the technical field of advertisement material multi-modal content detection. The target visual object is determined according to the subtitle description or the voice description in the advertisement video. The visual state of the same object is obtained according to the continuous frames, and the range and direction of the conversion from the first visual state to the second visual state are determined. When the description interval crosses the conversion range boundary, the state content or the state change content in the description and the corresponding conversion stage are determined, the visual part participating in the consistency judgment is limited, and the visual state not belonging to the corresponding conversion stage but time overlapping is excluded. Whether the content is supported by the actual visual state is judged within the limited range, and the description change direction and the actual conversion direction are checked when the state changes, so that the cross-modal content conflict result is formed, and the misjudgment caused by the stage mismatch in the normal state conversion process is reduced.

Description

Technical Field

[0001] This invention relates to the field of multimodal content detection technology for advertising materials, specifically a method for identifying multimodal content conflicts in advertising materials. Background Technology

[0002] With the widespread application of digital advertising formats such as short video ads, feed ads, and image / video ads, advertising materials typically contain multiple information simultaneously, including continuous video footage, subtitles, and audio. To automatically detect different modalities of content before the official launch of an ad, existing technologies usually extract features from video, audio, and text separately, and then establish correspondences between different modalities through methods such as temporal location, spatial location, semantic association, or similarity. This allows for ad content identification, anomaly detection, or review. For example, patent CN120451872A discloses semantic segmentation of advertising videos, extracting visual, audio, and textual information, and then combining this with knowledge graphs and large language models for fine-grained analysis. Patent CN121660749A discloses extracting multimodal features from video streams, audio streams, and auxiliary text, and using methods such as similarity matching, shot switching, audio abrupt changes, and text clustering to determine ad segments and monitor content. All of the above-mentioned solutions involve processing such as multimodal feature extraction, video segment localization, or association of information from different modalities.

[0003] In actual advertising videos, the visual state of the same product, device, or component may change with consecutive frames, such as gradually changing from closed to open, or from folded to unfolded. The subtitles or audio associated with this visual object usually have a certain duration, and their start and end times do not necessarily coincide with the start and end positions of the visual state change. Therefore, a single state description may simultaneously cover parts of the screen before, during, and after the state change. Existing multimodal processing methods can establish associations between text, audio, and video based on temporal overlap, segment position, or overall semantic relationships. However, in the aforementioned dynamic state change scenarios, judging consistency solely based on local frames or instantaneous visual states within the described time interval may treat the visual states of the same target visual object, which actually exist at different stages, as contradictory information.

[0004] Especially when the continuous interval of subtitles or audio crosses the state transition boundary of the same target visual object, and the description content actually refers to a specific stage before, during, or after the state transition, or within a complete state change, multiple temporally valid but semantically distinct visual states may exist simultaneously within the description time range. If the state stage being described is not accurately distinguished, and static state comparisons are directly performed based on overlapping time frames, normal object state changes may be interpreted as inconsistencies in multimodal content, potentially leading to different judgment results in adjacent frames or at different sampling positions. Therefore, accurately judging situations where the same target visual object undergoes continuous state changes and the description interval crosses the state transition boundary during multimodal content detection of advertising materials remains a technical problem that requires further resolution. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for identifying multimodal content conflicts in advertising materials, thereby resolving the problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for identifying multimodal content conflicts in advertising materials, comprising: S1. Determine the target visual object based on at least one of the subtitle description and voice description of the advertising video, obtain the visual state of the same object based on consecutive frames, and determine the range and direction of its transition from the first visual state to the second visual state. S2. When the description interval crosses the boundary of the transformation range, determine at least one of the state content and state change content for the target visual object, and determine that the content corresponds to one of the following categories: before transformation, during transformation, after transformation, and complete transformation process. S3. Based on the correspondence, determine the corresponding parts involved in the consistency judgment in the first visual state, the transition process, and the second visual state, and exclude visual states that overlap with the description time but do not belong to the corresponding transition stage. S4. Determine whether the content is supported by the actual visual state in the corresponding section, and verify the description of the change direction and the actual conversion direction when state changes are involved; S5. If the content is supported and the direction is consistent when state changes are involved, the conflict is not confirmed based on the local inconsistency of the excluded visual state; otherwise, a cross-modal content conflict result is formed.

[0007] Furthermore, the time semantics of the description before, during, and after the conversion are determined based on the temporal semantics that have a direct modifying relationship with the state words or state change actions in the description; A complete transition process is defined when the description clearly specifies both the initial and final states and expresses the state changes between them. Before the conversion corresponds to the first visual state, during the conversion corresponds to the conversion range and the actual conversion direction, and after the conversion corresponds to the second visual state. The complete conversion process corresponds to an ordered state change consisting of the first visual state, the conversion range, and the second visual state.

[0008] Furthermore, when the same state label of the same target visual object remains consistent in consecutive frames to reach a set number of frames, the corresponding consecutive frames and the adjacent consecutive frames with the same state label are determined as the stable state range. When different state labels appear after a stable state range and then another stable state range is formed, the first visual state and the second visual state are determined respectively. The transition range starts from the first frame after the state change after the previous stable state range and ends at the adjacent video frame before the starting frame of the second visual state.

[0009] Furthermore, based on the object region and appearance information in adjacent frames, the same object identifier is maintained, and only frames with consecutive and consistent object identifiers are used to form the visual state sequence. If the identity of the object cannot be confirmed to be consistent, the previous and subsequent visual states will not be spliced ​​together as the state transition of the same target visual object.

[0010] Furthermore, the visual state of the target visual object is represented as a canonical state label, synonymous state words in the description are mapped to canonical state names consistent with the canonical state labels, and state change actions are mapped to ordered state pairs with start and end states. The ordered state pair is compared with the ordered state pair formed by the first visual state and the second visual state. If the two are consistent, it is determined that the described change direction is consistent with the actual conversion direction.

[0011] Furthermore, when describing the corresponding transformation, there is a temporal overlap between the description interval and the state transformation range, and the state change is determined to be supported by the actual visual state when the described change direction is consistent with the actual transformation direction; State transitions that have already ended in the previous description range are not used to support the current description.

[0012] Furthermore, any description in the caption or audio description is supported by the visual state and state transition range corresponding to the target visual object it is specifically associated with; the same state or state change of other objects in the picture are not used as the supporting basis for the description.

[0013] Furthermore, when the same target visual object undergoes multiple state transitions, each transition is composed of the corresponding first visual state, transition range, and second visual state, forming independent state transition information. When describing multiple separate state transitions, each state transition is judged separately, and multiple state transitions are combined into one transition without crossing intermediate stable states.

[0014] Furthermore, when the same advertisement segment contains both subtitles and audio, the subtitle content and display time are recorded separately from the audio recognition text and audio time. When both are pointing to the same target visual object, the first visual state, conversion range, second visual state, and conversion direction of the target visual object are read respectively. When they are pointing to different target visual objects, the first visual state, conversion range, second visual state, and conversion direction of each target visual object are read respectively.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. By recognizing the visual states of consecutive frames of the same target visual object, the first visual state, the state transition range, the second visual state, and the actual transition direction are determined. Based on the subtitle or voice description corresponding to the pre-transition, during-transition, post-transition, or complete transition process, the visual content participating in the consistency judgment is limited. This avoids directly judging conflicts based solely on local visual states that overlap in time, thereby reducing stage mismatches and misjudgments caused by the description interval crossing the transition boundary during normal state transitions.

[0016] 2. By unifying the state content in the description to standardized state names, and forming ordered state pairs consisting of a start state and an end state when state changes are involved, and verifying them against the actual visual state and transition direction of the target visual object, conflict judgment is simultaneously constrained by object identity, semantic stage, actual state, and change direction, thereby improving the consistency and interpretability of dynamic state description and visual change relationship determination.

[0017] 3. By processing the subtitle description and voice description separately according to their respective time intervals and target visual objects, and forming corresponding first visual state, transition range and second visual state when the same target visual object has multiple state transitions, without splicing different transition processes across intermediate stable states, the cross-modal content conflict results can be mapped to the actual target visual object and the corresponding description time range. Attached Figure Description

[0018] Figure 1 This is a flowchart of the multimodal content conflict identification method for advertising materials according to the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example: Combined with Appendix Figure 1 This embodiment provides a method for identifying multimodal content conflicts in advertising materials, including: S1. Determine the target visual object based on at least one of the subtitle description and voice description of the advertising video, obtain the visual state of the same object based on consecutive frames, and determine the range and direction of its transition from the first visual state to the second visual state. S2. When the description interval crosses the boundary of the transformation range, determine at least one of the state content and state change content for the target visual object, and determine that the content corresponds to one of the following categories: before transformation, during transformation, after transformation, and complete transformation process. S3. Based on the correspondence, determine the corresponding parts involved in the consistency judgment in the first visual state, the transition process, and the second visual state, and exclude visual states that overlap with the description time but do not belong to the corresponding transition stage. S4. Determine whether the content is supported by the actual visual state in the corresponding section, and verify the description of the change direction and the actual conversion direction when state changes are involved; S5. If the content is supported and the direction is consistent when state changes are involved, the conflict is not confirmed based on the local inconsistency of the excluded visual state; otherwise, a cross-modal content conflict result is formed.

[0021] This implementation method is applied to multimodal content inspection of advertising videos before their official release. The implementation platform is a material processing device capable of reading advertising videos, subtitle tracks, and audio tracks, and performing image and text processing. The material processing device takes existing video frames and frame time information from the advertising video, as well as actual data from subtitle content, subtitle display time, audio content, and audio occurrence time, as input. The advertising video is read in continuous frame format, with each frame having a time position on the video timeline; when subtitles are present, the subtitle text and its start and end times are directly read; when audio is present, the audio content is recognized to obtain the audio text and its corresponding start and end times. Both subtitle and audio descriptions are organized into description records containing description content, start time, and end time, and processed according to their respective time intervals.

[0022] This implementation targets visual objects in advertising videos whose state changes can be directly observed from consecutive frames. Examples include products, devices, or components with distinguishable visual states such as open / closed, unfolded / folded, empty / loaded, stopped / running, etc. When a current description can establish a definite association with an actual visual object in the frame, that visual object is designated as the target visual object, and its object identity is maintained across consecutive frames. If the same description can only determine the object category but not the specific object instance among multiple instances of the same type in the same frame, the states of multiple instances are not concatenated into a single target visual object's state change process. In this implementation, the state transition judgment is based on the target visual object whose specific object instance can already be determined as the processing unit.

[0023] In advertising videos, target visual objects are associated with the object names in the description and the object recognition results in the video frames. The material processing device first obtains the object names from the description text, such as "screen," "bottle cap," "stand," and "body," and then inputs the video frames and object names into the visual object localization program. The visual object localization program outputs the object region and object category corresponding to the object name. When the object region corresponding to the current description can be identified as a single object instance, that object region is used as the initial region of the target visual object. When multiple instances of the same category exist in the same frame, each instance is assigned an object identifier by the video object association program. Only object instances that have been identified as associated with the current description enter the state change analysis. The internal network structure of the visual object localization program does not participate in subsequent visual stage constraints and conflict judgments.

[0024] The identity of the same object across consecutive frames is maintained through a video object association program. This program reads the object region and appearance information from adjacent frames and assigns the same object identifier to consecutively corresponding objects. Only frames with consecutively consistent object identifiers are used to form the visual state sequence of the same target visual object; if the object identity cannot be determined to be the same object within a certain consecutive frame range, that consecutive frame range is not directly concatenated with the states of different objects before and after it. The object association program only outputs the object identifier and corresponding object region used for state sequence; subsequent conflict judgment uses the visual state, time range, and state transition direction of the target visual object as the basis for judgment.

[0025] To ensure a consistent timeframe for subsequent state determinations, video frame times, subtitle times, and audio times are all converted to relative times following the start of the advertisement video. Video frame times are based on the decoded frame times; subtitle times use the start and end times recorded in the subtitle file or video subtitle track; and audio descriptions use the start and end times corresponding to the speech recognition results. The term "description interval" refers to the continuous time range from the start to the end of a subtitle or audio description, and the term "visual state time range" refers to the frame time range within which the same target visual object maintains its corresponding visual state across consecutive frames. With a unified timeline, whether a description crosses a visual state transition boundary can be directly determined by the sequence and overlap of the two time ranges.

[0026] The material processing equipment reads video frames covering the time interval of each subtitle or audio description containing an object's state expression. It then continuously reads adjacent video frames along the same target visual object, before and after the description's start and end times, until a visual state meeting the conditions for stable state continuity is obtained on the corresponding side. This confirms the existence of stable visual states on both sides of the description interval. Adjacent consecutive frames are only used to observe the state continuity of the same target visual object near the description interval and do not change the time range of the subtitles or audio themselves. If a visual state meeting the conditions for stable state continuity is not obtained on the corresponding side even after reading to the start or end position of the advertisement video, then a stable visual state is not formed on that side, and the complete state transition relationship of the first visual state, state transition range, and second visual state is not established based on the video frames on that side. In a directly executable implementation path, the advertisement video is read at the original frame rate; when the original video is 25 frames / s, the target visual object area and object identifier are obtained frame by frame for consecutive frames.

[0027] Each target visual object region is input into a visual state recognition program. This program is a visual classification program that has already configured a set of standardized state labels for the corresponding object category before executing this method. It takes the identified target visual object region and its object category in the current frame as input and outputs a standardized state label corresponding to the object in the current frame. For example, for a device with an open / closed state, the state label set includes "closed" and "open"; for a foldable stand, the state label set includes "folded" and "unfolded". The state label set is set based on the distinct visual states of the processed object that can be directly identified from the image. The state labels represent the actual observable object state in the image and are not modified in reverse based on the description in the advertising copy.

[0028] The state labels of the same target visual object in consecutive frames are arranged according to frame time to form a visual state sequence. To avoid single-frame recognition jumps being directly interpreted as actual state transitions, this embodiment uses a continuous maintenance rule to form stable states. Taking a video of 25 frames / second as an example, when the same state label remains consistent in 5 consecutive frames, these 5 frames and adjacent consecutive frames that maintain the same state label are classified into the same stable state range. 5 consecutive frames correspond to approximately 0.2 seconds, and this value is used in this embodiment to illustrate the formation process of stable states; at other frame rates, the required number of consecutive frames can be calculated according to the same time length, so that the stable state judgment is still based on continuous visual evidence rather than single-frame results.

[0029] When a continuous visual state different from a stable state appears after a stable state range, and the new state label subsequently also satisfies the stable state maintenance condition, it is determined that the same target visual object has transitioned from a first visual state to a second visual state. The first visual state corresponds to the continuously maintained state before the transition, and the second visual state corresponds to the continuously maintained state after the transition. The state transition range starts from the first frame after the last stable continuous frame of the first visual state where a state change occurs, and ends at the adjacent video frame before the starting frame of the second visual state; when the second visual state is continuously maintained to meet the stable state maintenance condition, the first frame constituting the continuous maintenance interval is determined as the starting frame of the second visual state. Within the state transition range, there are allowed to be gradual changes from the first visual state to the second visual state, temporarily unstable state recognition, and continuous visual images between two stable states, but they must be connected to the already confirmed first and second visual states respectively.

[0030] For example, in a 25 frames / second device advertisement, the target device is identified as "off" for consecutive frames from 2.00s to 3.00s, and then as "on" starting at 3.04s, with consecutive frames from 3.60s onwards. Therefore, the first visual state is "off," with a stable time range covering at least 2.00s to 3.00s; the state transition range is 3.04s to 3.56s. The second visual state is "on," with a stable time range starting from 3.60s. The ordered relationship from the first visual state "off" to the second visual state "on" is the actual transition direction. Subsequent text state judgments use this actual transition direction, rather than simply judging whether the "off" and "on" states appeared in the video.

[0031] For object state changes that take a long time to complete, such as slow unfolding or gradual filling, the state transition range can include more consecutive frames. The processing device still uses the same formation rule: first, determine the first visual state that is continuously maintained before the transition, then determine the second visual state that is continuously maintained after the transition, and the continuous changes in the images between the two constitute the state transition range. Therefore, the state transition range itself originates from the continuous images of the same target visual object, and does not rely on words such as "currently" or "already" in the subtitles to determine the visual boundary.

[0032] When only a single stable visual state appears in consecutive frames without forming two different stable visual states before and after, the state transition range required by this implementation method is not formed. Such descriptions do not fall under the processing case of "describing a time interval that crosses the state transition boundary of the same object" and can continue to be processed according to the general content consistency method of existing advertising materials. Only target visual objects that have formed a first visual state, a state transition range, a second visual state, and an actual transition direction are subject to description judgment that crosses the state transition boundary.

[0033] To maintain consistency in the visual state results, the first visual state, the state transition range, and the second visual state are all bound to the same object identifier. If an object is briefly occluded during a state transition, but the object association procedure can maintain the same object identifier before and after the occlusion, the state transition range can still be continuously preserved. If the occlusion causes the object identifier to be inconsistent, the state of the occluded object will not be directly used for the state transition judgment of the previous object. This process ensures that continuous state analysis always targets the same target visual object and does not incorporate the visual state of another object into the state transition judgment of the current object.

[0034] After determining the first visual state, the state transition range, the second visual state, and the actual transition direction, each target visual object that meets the state change conditions forms a set of state transition information that can be directly read subsequently. This includes the object identifier, the first visual state, the time range corresponding to the first visual state, the state transition range, the second visual state, the time range corresponding to the second visual state, and the transition direction from the first visual state to the second visual state. This set of information uses the timeline of the advertisement video as a unified benchmark, and subsequent description interval comparisons, semantic stage correspondences, and state support judgments all use these pre-formed states and time ranges.

[0035] After the subtitle or audio description forms a description record, the material processing device compares the description start time and end time with the time range of the first visual state, the state transition range, and the second visual state for the same target visual object. Descriptions that fall within only one stable state range, or that end before a state transition begins, or that begin after a state transition is complete, are not considered descriptions that cross state transition boundaries. This implementation focuses on handling cases where the description range simultaneously covers visual content on both sides of a state transition boundary.

[0036] Specifically, if the description starts earlier than the end time of the state transition range and ends later than the start time of the state transition range, and the description interval also covers at least one side of the first visual state and the second visual state, then the description is considered to overlap with the state transition in time. If the description interval includes both the image before the start position of the state transition and the image after the completion position of the state transition, then the description interval is confirmed to span the complete state transition; if the description starts from the middle of the state transition range and continues into the second visual state, or from the first visual state and continues into the middle of the state transition range, then the description is also confirmed to cross the corresponding state transition boundary. The determination of time overlap and boundary crossing is based on the time range of the first visual state, the state transition range, the time range of the second visual state, and the description interval.

[0037] After confirming that the described time interval crosses the state transition boundary of the target visual object, at least one of the state content and state change content for the target visual object is extracted from the description text. The main implementation path uses textual correspondence rules for object words, state words, and time semantic words for extraction. First, the object name that has been associated with the target visual object is located in the description; then, state words that can directly express the visual state and action words that can express the change of state from one state to another are searched in the predicate phrases containing the object name. State words and visual state labels use the same standardized state names, such as "closed," "open," "folded," and "expanded"; synonymous expressions in the text are mapped to the same standardized state names before comparison.

[0038] To standardize state terms, this embodiment establishes a state term list corresponding to object categories. Taking a device with open and closed states as an example, "on," "open," and "in open state" uniformly correspond to the standardized state "open," and "closed," "closed," and "in closed state" uniformly correspond to the standardized state "closed." When "open" and "open" are used as change actions, they indicate a state change from "closed" to "open," and when "closed" and "closed" are used as change actions, they indicate a state change from "open" to "closed." The state term list is only used to convert textual state expressions into standardized states that can be directly compared with visual state labels; it does not participate in generating new visual states.

[0039] The temporal semantic location is determined by the textual structure in the description that directly reflects the stage of the state occurrence. The main implementation rule divides temporal semantics into four categories: before the transition, during the transition, after the transition, and the complete transition process. When the description contains time expressions such as "before" or "previously" indicating that the target object's state is before the state transition occurs, and can simultaneously identify the explicit state content before the transition, the state description is determined to correspond to the before transition; when words such as "currently," "starting," "gradually," or "in process" appear and directly modify the state change action, it is determined to correspond to the during transition; when structures such as "already," "completed," "after opening," "after closing," or "after unfolding" indicate that the state change has been completed, it is determined to correspond to the after transition; when structures such as "from the first state to the second state," "switch from the first state to the second state," or "transform from the first state to the second state" simultaneously specify the starting and ending states, it is determined to correspond to the complete transition process.

[0040] Temporal semantic words are combined with state words or state change actions according to the direct modification relationship within the same predicate phrase, and are not determined by whether a particular temporal word appears alone in a sentence. For example, in "display parameters after the screen is turned on", "after turning on" specifies the time position after the screen is turned on, so the state "on" corresponds to the state after the transition; in "the screen is turning on", "is turning on" directly modifies the action "turning on", so it corresponds to the state during the transition; in "keep the device off before turning on", "before turning on" indicates that the off state corresponds to the state before the transition, and "off" is the explicit state content before the transition, so the actual first visual state is read in subsequent comparisons; when the actual first visual state is "off", the state before the transition is visually supported; "the device changes from off to on" gives the starting state, the change action, and the ending state, corresponding to the complete transition process.

[0041] When no time-related semantic terms appear explicitly in the description, but the state expression itself has a clear meaning of completion, the semantic position is determined according to the relationship between the state expression and the sentence structure. For example, "the device is in the on state" only expresses a stable state and does not contain any meaning of a state change process. If the description interval crosses the state transition boundary and it cannot be uniquely determined from the text whether the stable state corresponds to before or after the transition, then the description is not used for the stage exclusion process in this implementation method to avoid forcibly assigning uncertain semantics to a certain visual stage. The description can still be retained for general multimodal consistency detection.

[0042] The state content and temporal semantic position of the speech description are determined according to the same text rules as the subtitles. The speech recognition program outputs the speech text along with its start and end times in the audio, which are then converted to the unified timeline of the advertising video. The speech recognition program itself is only responsible for converting speech into processable text and providing the time position; it does not modify the recognition result based on the visual image. After obtaining the speech text, the processing of object words, state words, state change words, and temporal semantic words remains consistent with the subtitle description.

[0043] A cross-boundary description can contain only a single state or a complete state change. For example, "Parameters can be displayed after the device is turned on" mainly represents the target object being in the "on" state and the corresponding state after the transition; "Device is turning on" represents the state change from "off" to "on" and the transition in progress; "Device changes from off to on" represents the complete state change from "off" to "on" and the complete transition process. Subsequent visual comparisons no longer directly calculate an overall consistency result by comparing the entire description with all time-overlapping frames, but instead use the already determined state content, state change direction, and semantic temporal position to determine the actual visual range that can participate in the comparison.

[0044] Through this processing, each description that satisfies the cross-boundary conditions and clearly defines the semantic location of the state forms the necessary information for subsequent determination: the target visual object, the normalized state content or normalized state change content, the description start time, the description end time, and a clear relationship between the normalized state content or normalized state change content and the state before, during, after, or during the complete transformation process. All information comes from existing descriptions in the advertising video and the state transformation time range already formed for the same target visual object, without introducing external factual data.

[0045] The material processing device simultaneously reads the first visual state, state transition range, second visual state, and the determined semantic temporal position of the cross-boundary description of the same target visual object, and limits the visual content used for consistency judgment of the current description according to a fixed stage correspondence. This limitation occurs before the state consistency judgment, and its function is to first determine which visual states have a temporal semantic basis for direct state comparison with the current description.

[0046] When describing the state before the transition, the visual content involved in the consistency judgment is limited to the continuous frames corresponding to the first visual state; when describing the state transition process, the visual content involved in the consistency judgment is limited to the continuous frames within the state transition range and the actual transition direction from the first visual state to the second visual state; when describing the state after the transition, the visual content involved in the consistency judgment is limited to the continuous frames corresponding to the second visual state; when describing the complete state change, the visual content involved in the consistency judgment is limited to the ordered state change consisting of the first visual state, the state transition range, and the second visual state.

[0047] The phase correspondence is directly determined by the temporal semantic position of the description. For example, if the subtitle is displayed from 2.40s to 5.80s, the target visual object remains "off" from 2.00s to 3.00s, transitions from off to on from 3.04s to 3.56s, and remains "on" after 3.60s. If the subtitle content is "Real-time parameters can be displayed after the device is turned on," then the visual state that actually participates in the consistency judgment of this state is the second visual state "on" formed after 3.60s. Although the subtitle started displaying from 2.40s and overlapped with the "off" screen from 2.40s to 3.00s, this "off" state belongs to the pre-transition state and is not the semantically applicable phase of the "after on" state description. Therefore, this off state is not used as direct evidence of conflict with the "after on" description.

[0048] For descriptions like "opening device," the comparison object is not the single "open" state after the transition, but rather the range of continuous state transitions of the target visual object from the first visual state to the second visual state. In this case, the start and end times of the state transition range and the actual transition direction from the first visual state to the second visual state are retained for subsequent judgment on whether the change in the description is supported by the image. The first visual state that is stable before the transition and the second visual state that is stable after the transition can serve as the basis for confirming the states at both ends of the transition, but a discrepancy between either stable state and the literal "opening" state does not necessarily create a conflict.

[0049] For a complete transition description like "from closed to open," a continuous, ordered relationship is required, consisting of a first visual state, a transition range, and a second visual state. The first visual state must precede the transition range, the second visual state must follow the transition range, and the actual transition direction must be from "closed" to "open." If "closed" and "open" do appear simultaneously in the video, but the actual order is "open" followed by "closed," then the description cannot be considered visually supported simply because both states appear around the same time as the description.

[0050] After defining the visual scope for each cross-boundary description, visual states that overlap with the description's time but do not belong to the corresponding semantic stage are excluded from the direct conflict comparison range of that description. This exclusion only changes the visual input used for state conflict judgment in the current description; it does not delete video frames or alter the state transition information already formed by the target visual object. The excluded visual states remain in the original timeline of the advertisement video and the state sequence of the target visual object, used to illustrate the complete process of actual state changes, but they will not be interpreted as conflicting simply because they differ from the literal state of the current description.

[0051] After defining the visual comparison range according to the semantic temporal position of the description, when the same long subtitle spans multiple visual states, the processing device will not simply treat the entire subtitle time interval as a static visual phase. For example, if the subtitle starts displaying from when the device is off, goes through the device's opening process, and continues until the device is fully open, if the subtitle describes the state as "after opening," then only the visual state after opening is completed is used to determine whether the description is supported; if the subtitle describes "opening," then only the actual state transition range is used for judgment; if the subtitle explicitly describes "changing from off to on," then the entire ordered state change is used for judgment. The visual comparison range is thus determined by the semantic position of the described state, and is not directly replaced by the entire time overlap range between the subtitle and the video frame.

[0052] After the constraints are established, the material processing device retains the target visual object area, corresponding frame time, standardized visual state, and necessary state transition directions that are actually required for consistency judgment in the current description. Subsequent state support judgments only read these already constrained visual contents and do not reuse excluded inapplicable states as the basis for conflict in the current state description.

[0053] State consistency is determined using standardized state names. If a cross-boundary description represents a stable state, the processing device reads the visual state of the target visual object within the semantic stage corresponding to the description and compares the standardized state name (after unifying the textual state terms) with the standardized state label output by the visual state recognition program. If they match, the state content is supported by the actual visual state; otherwise, the state content is not supported by the visual state within its semantic scope.

[0054] For example, if the description states "parameters can be displayed after the device is turned on," the defined standard state is "on," and the semantic location is after the transformation; if the target visual object's second visual state is also "on," then the described state receives visual support. Even if the first visual state "off" overlaps with the first half of the description's display time, it is not considered relevant due to semantic limitations and therefore does not participate in the state support judgment. If the second visual state is "off," then the required transformed "on" state lacks visual support, leading to a conflicting result.

[0055] If a cross-boundary description expresses a state change, the processing device not only compares the names of the states involved in the change, but also compares the described direction of change with the actual conversion direction of the target visual object. The described direction of change is determined by the initial and final states in the text; the actual conversion direction is determined by the first and second visual states. Only when the described initial state is consistent with the first visual state, and the described final state is consistent with the second visual state, is the described direction of change considered to be consistent with the actual conversion direction.

[0056] Taking "the device changes from off to on" as an example, the text indicates a change direction of "off → on". If the first visual state in the visual state transition information is "off" and the second visual state is "on", then the change direction is consistent; if the actual visual state is "on → off", even if both "off" and "on" states appear in the advertisement video, the change direction is inconsistent. The direction comparison is based on the order of state transitions, not on whether the two states exist simultaneously within the same descriptive time frame.

[0057] For descriptions like "opening," which only express the action of transition without listing the start and end states in the text, the direction of change is formed based on the defined transition relationship corresponding to that action in the target object's state vocabulary. For example, for an open / closed object with two standard states, "closed" and "open," "open" as a state change action corresponds to "closed → open"; and "closed" as a state change action corresponds to "open → closed." The state vocabulary remains fixed within the same object category, ensuring that the same action words have the same state direction meaning in similar advertising materials.

[0058] For the description of the transition, it is also required that there is a temporal overlap between the described interval and the actual state transition range. When the description is "opening", if the actual "closed → open" state transition of the target visual object occurs within the transition range covered by the description interval, then both directional consistency and the transition process occurring in the semantic stage corresponding to the description are satisfied. If the "closed → open" transition was completed in the image at an earlier time, but the image corresponding to the current description interval only remains in the open state without a transition in progress, then the completed historical transition will not be used to support the current "opening" description.

[0059] State support judgments retain the correspondence between the object identifier and the time interval of the description of the current target visual object. A subtitle or voice description only uses the visual state corresponding to its already identified target visual object, and does not use another object in the picture that happens to have the same state to replace the visual support of the current object. For example, if the text describes that the target device A is turned on, but device A is turned off and device B is turned on in the picture, the state of device B cannot be used as the support result of the description of device A.

[0060] The implementation path using direct comparison of state labels does not require the introduction of a new comprehensive score. The visual state recognition program has already output standardized state labels to characterize the current object's state, and text processing has also unified the state expressions in the description to the same standardized state names. Therefore, the state can be directly determined by whether the state labels are the same; state changes can be directly determined by whether the start state, end state, and direction are consistent. If the visual state recognition program used in the specific deployment also outputs confidence scores, the confidence scores only serve as the basis for confirming the state labels within the visual state recognition program and do not enter into the cross-modal conflict scoring of this implementation method, nor do they change the applicable rules for each stage.

[0061] The processed data forms state support results for the current cross-boundary description. For stable state descriptions, the state support results indicate whether the actual visual state is consistent with the textual state within the semantically applicable phase; for state change descriptions, the state support results also indicate whether the actual state transition direction is consistent with the textual change direction. These support results directly serve as the basis for confirming cross-modal content conflicts.

[0062] The material processing device confirms conflicts based on the current state support results of the cross-boundary description. If the stable state expressed in the description is supported by the actual visual state in its semantically applicable visual stage, the current description will not conflict due to the difference between other excluded visual states and the text state. For example, when the description "after opening" is supported by the second visual state "open", the overlap of the subtitle start stage and the first visual state "closed" is a stage difference in the normal state change process and is not used as a basis for conflict.

[0063] If the description involves a state change, and the direction of the state change in the description is consistent with the actual transformation direction of the target visual object, and the corresponding state transformation occurs within the scope of the transformation process corresponding to the description, then the state change is considered to be supported by the actual visual process. A static state before or after the transformation that differs from the literal meaning of the action word does not constitute a conflict on its own. For example, if "opening" is supported by the actual "closed → open" transformation process, a conflict between the description and the visual is not considered to occur simply because a short segment of the screen before the transformation begins is still "closed" or a short segment after the transformation is complete is already "open."

[0064] A cross-modal content conflict arises when the stable state described in a description lacks a corresponding visual state in the semantically applicable visual phase. For example, if the description explicitly states "after opening," but the target visual object actually remains "closed" after the transition, the required state after the transition lacks visual support, resulting in a conflict. Similarly, a conflict arises if the described direction of change differs from the actual direction of the state transition. For instance, if the description states "from closed to open," but the actual state transitions to "open → closed," even though both states appear in the video, the opposite direction of the transition creates a conflict.

[0065] Cross-modal content conflict results must correspond at least to the current ad video, the target visual object, and the current caption or audio description. To facilitate material review, the time range corresponding to the description that caused the conflict, as well as the time range of the actual visual state, should be retained, so that confirmed conflicts can be mapped back to specific objects and time locations in the ad creative.

[0066] When a description is supported by the applicable visual stage, the processing device makes a current judgment of "no conflict confirmed" and ends the state conflict check for that description under this method; when the description is not supported or the direction of state change is inconsistent, a cross-modal content conflict result is formed. Each subtitle description and audio description is judged separately according to their respective time intervals. When there are multiple descriptions in the same advertising video, they are processed one by one in the order of description time. The processing of each description reads the actual visual state of the corresponding target visual object within the corresponding time range.

[0067] The following is an example of a device advertisement video illustrating the continuous processing from the state transition of the target visual object to the confirmation of cross-modal content conflicts. The advertisement video is 8.00 seconds long and has a frame rate of 25 frames per second. The video shows the same device gradually turning on from a closed state and remaining on. The subtitle "Real-time parameters can be displayed after the device is turned on" appears from 2.40 seconds to 5.80 seconds. The object name "device" in the subtitle corresponds to the target visual object in the current advertisement video, "turned on" is the state expression for this target visual object, and "after turning on" indicates the temporal semantics of the completion of the state change corresponding to the turned-on state.

[0068] The device object area is obtained from consecutive frames of the advertising video, maintaining the same object identifier. The device's status label remains "off" for consecutive frames from 2.00s to 3.00s; from 3.04s onwards, the device's appearance begins to change, completing the on / off process by 3.56s; after 3.60s, the status label remains "on" for consecutive frames. This forms the first visual state "off," the state transition range from 3.04s to 3.56s, and the second visual state "on," with the actual transition direction being "off → on."

[0069] The subtitle interval from 2.40s to 5.80s covers the latter half of the first visual state, the complete state transition range, and the first half of the second visual state. Therefore, the subtitle time interval spans the state transition boundary. The state of the subtitle text is extracted: "Open" is uniformly standardized as "Open," and "After" indicates that the state description corresponds to the stage after the transition is completed. Therefore, the state content expressed by this subtitle is "Open," and its semantic time position is "After the transition."

[0070] Based on the semantic position after the conversion, the visual state actually used for consistency judgment is limited to the second visual state "on". The subtitles were displayed between 2.40s and 3.00s, but the device was still in the "off" state. This "off" state overlapped with the subtitles in time, but belonged to the pre-conversion state, not the semantic stage described in the "after on" state description, and therefore was not used as a direct object of comparison for the current subtitles. The on action between 3.04s and 3.56s is a state transition process, and similarly, the intermediate screen during the transition is not used to directly negate the stable state description of "after on".

[0071] The second visual state after 3.60 seconds is read, and the visual state label is "On," consistent with the "On" state in the subtitle specification. Therefore, the subtitle state is supported by the actual visual state. The current subtitle does not conflict due to the difference between the closing screen (2.40 seconds to 3.00 seconds) and the literal "On" state, nor does it conflict due to the opening action (3.04 seconds to 3.56 seconds). This completes the cross-modal state conflict judgment for this subtitle.

[0072] If another piece of advertising material still uses the subtitle "Real-time parameters can be displayed after the device is turned on," but the target device actually changes from an "on" state to a "off" state, resulting in a second visual state of "off," then the "on" in the subtitle is not supported as the transformed state in the second visual state. The conflict arises from the inconsistency between the "on" state required by the subtitle and the actual "off" state in the second visual state, rather than from literal differences in the wording of the subtitle compared to other overlapping scenes.

[0073] In another embodiment, the advertising video shows a foldable stand gradually unfolding from a folded state. The video frame rate is 25 frames / s, and the audio duration corresponding to the voice "Stand is unfolding" is from 4.10s to 5.30s. The voice recognition result forms the text "Stand is unfolding", with the object name "Stand", "in progress" indicating a transition, and "unfolding" as a state change action corresponding to "folded → unfolded" in the stand state vocabulary.

[0074] In the continuous frames of the video, the same support remains "folded" from 3.60s to 4.20s, continuously unfolds from 4.24s to 5.04s, and remains "unfolded" after 5.08s. Therefore, the first visual state is "folded," the second visual state is "unfolded," and the actual transition direction is "folded → unfolded." The speech interval from 4.10s to 5.30s spans the start and end of the state transition, meeting the conditions for cross-boundary description.

[0075] Because the visual content involved in the consistency judgment during the "unfolding" transition is limited to the state transition range of 4.24s to 5.04s, rather than directly using the folded image from 4.10s to 4.20s and the fully unfolded image from 5.08s to 5.30s as the basis for action conflict, the actual state transition direction is "folding → unfolding," which is consistent with the direction of change described in the speech. Furthermore, the actual transition range overlaps with the speech interval, therefore, this speech state change is supported by the visual process.

[0076] If the advertisement video actually shows the support frame gradually "folding" from "unfolding," while the audio still says "the support frame is unfolding," then the actual transition direction is "unfolding → folding," which is the opposite of the text change direction "folding → unfolding." In this case, even if the state transition does occur within the 4.10s to 5.30s audio range, it cannot be assumed that the audio is supported simply because there is a dynamic change; rather, it results in a cross-modal content conflict.

[0077] For advertising copy that clearly describes a complete state change, the visual comparison range covers the ordered relationship formed by the first visual state, the state transition range, and the second visual state. For example, the caption "The hood changes from closed to open" is displayed from 1.80s to 4.80s. The same hood remains closed until 2.00s, opens between 2.04s and 3.10s, and remains open after 3.14s. The caption "changes from closed to open" simultaneously provides the starting and ending states, confirming a complete transition process, with the text change direction being "closed → open".

[0078] At this point, check sequentially whether the same cover is in the first visual state of "off" within the time range covered by the subtitles, whether there is a continuous transition from the first visual state to the second visual state, whether the second visual state is "on", and check whether the actual transition direction is "off → on". When the first visual state, continuous transition, second visual state, and transition direction are all true, the complete state change is supported by the visual content.

[0079] If the video first shows the cover "open," then performs a closing action and ultimately remains "closed," although both "closed" and "open" states appear in the subtitle area, the actual transition direction is "open → closed." The processing device determines the inconsistency in direction based on the order of the states, resulting in a cross-modal content conflict. This judgment avoids misinterpreting the disordered existence of two states as support for a complete description of the change.

[0080] This method uses the same state processing chain for both subtitle and audio descriptions. Subtitle text directly uses the subtitle track content; audio text is obtained from speech recognition. Before entering state processing, both are recorded as descriptions containing text, start time, and end time. Therefore, subsequent cross-boundary judgments, state word unification, temporal semantic location determination, visual range limitation, and state support judgments all use the same rules. If the same advertisement segment contains both subtitles and audio, the subtitle and audio descriptions are judged as independent description records, and the text of the two descriptions is not forcibly merged into a new description.

[0081] The specific category of the target visual object is not limited to opening and closing devices. As long as the object has different visual states that can be directly identified from consecutive video frames, and the state changes can form a clear sequential relationship, the first visual state, the state transition range, and the second visual state can be formed using the same processing method. For example, a container changes from an empty state to a loaded state, a folded structure changes from a folded state to an unfolded state, and a display screen changes from an off state to an on state. For each object category, the visual state recognition program uses the corresponding standardized state label for that category, and the textual state vocabulary uses state names consistent with these standardized state labels, enabling direct comparison between the visual state and the descriptive state.

[0082] For advertisements containing multiple target visual objects, the overall state of the image is not used to substitute for the state of each object. Each target visual object forms its own visual state sequence with an independent object identifier. For example, if the main device and the accessory device appear simultaneously in the image, and the subtitles clearly describe the main device as "opening," only the visual state and state transition range corresponding to the main device's object identifier are read; even if the accessory device performs the opening action at the same time, it is not used to support the text description of the main device. This object constraint ensures that state transitions, semantic stages, and conflict judgments always point to the same actual visual object.

[0083] When a description lacks textual relationships that uniquely identify the transition from pre-transformation to post-transformation or the complete transformation process, the semantic stage is not forcibly inferred using stage exclusion rules. For example, if a subtitle simply states "device on," and the subtitle time spans three visual stages—closing, turning on, and turning on—while the text lacks expressions like "currently," "already," or "after" to determine the semantic location, then this description cannot uniquely determine whether the visual state during or after transformation should be used according to this implementation method. Such descriptions retain their original text and time range, without subjectively designating a particular stage as the semantically applicable stage.

[0084] When a state description uniquely identifies the semantic location, all subsequent judgments maintain the same consistency. Once a description is determined to be the state description after the transformation, subsequent judgments will not change to the original description even if the image and text before the transformation have low similarity. Similarly, once a "opening" description is determined to be a state change during transformation, subsequent judgments will not use a second visual state to replace the transformation process even if the visual state after opening is clearer. This consistent approach ensures that the semantic location truly fulfills its role in defining visual input.

[0085] The visual state recognition program requires only the object region and object category already associated with the target visual object as input. When cropping the object region from the video frame, the program preserves the complete visible area in the original image that reflects the object's state, without altering the meaning of the state during image compositing. If the advertising video is scaled up to meet the input size requirements of the visual recognition program, the scaling only changes the pixel size, not the frame time, object identifier, or the sequence of states. The state labels output by the visual state recognition program are written into the visual state sequence of the target visual object according to the video frame time.

[0086] The textual state lexicon and visual state labels use standardized names. Once the object category is determined, the standardized state set corresponding to that category is read, and synonymous state words in the description are mapped to this set. For example, "on," "lit," and "on" are standardized to "on," and "off" and "display off" are standardized to "off." This textual standardization only addresses the issue of different natural language expressions for the same actual state; it does not merge two semantically different states. If a descriptive state word cannot be mapped to any observable visual state of the current object category, then that state cannot be directly confirmed by visual state comparison using this method.

[0087] The direction of state change actions is also represented using a set of canonical states. For each change action with two definite states, the action word is mapped to an ordered pair of states. For example, "unfold" corresponds to "fold → unfold", "fold" corresponds to "unfold → fold", "light up" corresponds to "light up → light up", and "light up" corresponds to "light up → light up". When processing the direction of text changes, this ordered pair of states is read; when processing the direction of visual changes, the ordered pair formed by the first visual state and the second visual state is read. The direction is confirmed to be consistent only when the two ordered pairs are completely identical.

[0088] Temporal semantic terms only determine which visual stage the state description should be compared to; they do not directly change the visual state itself. Expressions like "currently" indicate the actual range of state transitions; expressions like "already," "completed," "after activation," and "after unfolding" indicate the second visual state; expressions like "before" and "previously," indicating the target state is before the state transition, indicate the first visual state, and are compared with the first visual state by the explicit state content in the description; complete transition sentences specifying the previous, changed, and subsequent states indicate a complete and ordered state change. When multiple temporal semantic terms exist in the description, those directly modifying the current state term or state change action are used; temporal terms related to another object or action are not used to replace the semantic position of the current object.

[0089] To enable direct rule execution, the description can be divided into sentences during text preprocessing based on punctuation and speech pauses. Within each sentence, the corresponding predicate phrase is located based on the target object name. The current state description is formed when the target object name, state word, and time semantic word are located in the same predicate phrase. If a sentence describes multiple objects simultaneously, the state content is extracted from the predicate phrase corresponding to each object name, forming its own description processing object; the time semantic word of one object is not transferred to another. This processing still relies on the existing textual relationships in the advertising copy.

[0090] Once both the visual and textual states have been standardized, subsequent judgments do not require recompressing the entire image and text into a unified similarity score. State support judgments only compare the definite relationships between the current target visual object, the current semantic application stage, and the current standardized state. This process allows conflict judgments to directly trace back to which object, which state, and which state transition direction lacked visual support.

[0091] The state transition time boundary is determined using the temporal position of consecutive video frames. After the last frame satisfying the continuity condition of the first visual state, when the target visual object begins to show a continuous sequence of images transitioning to the second visual state, the time of the first frame of this transition is taken as the starting point of the state transition range. When the second visual state is continuously maintained for a predetermined number of frames, the first frame constituting this continuous maintenance interval is determined as the starting frame of the second visual state, and the adjacent video frames before the starting frame of the second visual state are determined as the ending frames of the state transition range. The time range of the second visual state begins at the time corresponding to the starting frame of the second visual state. The time range of the first visual state terminates before the starting point of the state transition range. Therefore, the time ranges of the first visual state, the state transition range, and the second visual state are arranged consecutively according to the actual temporal order of the video frames, and the same video frame cannot simultaneously serve as the starting point of the state transition range and the second visual state time range.

[0092] The description of boundary crossing is determined by directly comparing the start and end times with the time ranges of the first visual state, the state transition range, and the second visual state. Taking the state transition range of 3.04s to 3.56s as an example, the description from 2.40s to 5.80s covers both the transition range and the states before and after it, thus crossing the state transition boundary; the description from 3.20s to 4.20s extends from during the transition to after the transition, also crossing the transition completion boundary; the description from 2.40s to 3.30s extends from before the transition to during the transition, crossing the transition start boundary. When the description from 4.00s to 5.00s is entirely within the second visual state, it does not fall under the cross-state transition boundary case addressed by this method.

[0093] The start and end times of subtitles are based on the actual display time of the subtitle track and are not affected by the extraction or skipping of a particular frame; the start and end times of audio are based on the time of the sentences obtained from speech recognition. Both visual state timing and description timing use the same video timeline, therefore, there is no need to manually estimate the sequence of events between different sources. If the video itself has a unified media timestamp, the media timestamp is directly used to convert it to relative time; if the reading program outputs frame numbers, the frame numbers are converted to video time positions according to the video frame rate.

[0094] When a video uses a variable frame rate, the actual time range of continuous states is determined by the inherent display time of each frame, rather than by fixed frame number intervals replacing the actual media time. Stable states are still formed by the continuous frame states, and the start and end positions of state transitions are ultimately recorded as the actual time of the corresponding frames. Therefore, subtitles, audio, and visual states can still be aligned to a unified time reference.

[0095] When an advertising video enters the material processing device, the video frames and frame times are first read, along with the subtitle descriptions; if audio descriptions are present, both the audio text and audio time are obtained. For each description, the name of the target object is extracted, and this name is associated with the object recognition results in the video to obtain the target visual object. The same object in consecutive video frames maintains consistency through object identifiers, and each object region in each frame receives a standardized visual state label, which is then used to form a visual state time sequence based on frame time.

[0096] If a visual state sequence shows a continuous maintenance of a first normative state, followed by continuous changes and a continuous maintenance of another normative state, then the first visual state, the state transition range, the second visual state, and the actual transition direction are determined. The described start and end times are compared with the state transition range. Only when the described time interval crosses the state transition start boundary, transition completion boundary, or the complete transition range, and the text actually contains an expression of the current target visual object's state or state change, does it enter the stage limitation processing.

[0097] For cross-boundary descriptions, canonical state words or canonical state change directions are extracted. Based on temporal semantic words directly related to the state expression, it is determined whether the expression belongs to the pre-transformation, during-transformation, post-transformation, or complete transformation process. Pre-transformation descriptions use the first visual state as direct state evidence; during-transformation descriptions use the actual state transformation range and actual transformation direction; post-transformation descriptions use the second visual state; and complete transformation descriptions use the ordered relationship formed by the first visual state, state transformation, and second visual state. Visual states that coincide with the description time but do not belong to the semantic stage corresponding to the current state expression are not used as direct conflict comparison inputs for the current description.

[0098] Within the defined visual range, the described canonical state is compared with the visual canonical state. If the stable states are the same, state support is formed; if the stable states are different, no state support is formed. When state changes are involved, the described ordered state pair is compared with the ordered state pair formed by the actual visual first and second states. If they are the same, the direction is consistent; if the order is reversed or either end of the state is different, the direction is inconsistent. All comparisons are directed towards the same target visual object.

[0099] When a state is supported by the applicable visual stage and the direction of the state change description is consistent with the actual transition direction, a conflict is not formed based on the local inconsistency of the excluded visual state. If a state lacks visual support in the semantic application stage, or if the direction of the state change is inconsistent with the actual transition direction, a cross-modal content conflict result is formed. The resulting conflict result corresponds to the current ad video, target visual object, descriptive text, and descriptive time, enabling the material processing device to locate the actual ad content where the conflict occurred. Thus, a description completes continuous processing from the original video and description input to the cross-modal conflict result.

[0100] In a display screen advertisement, a video shows the screen gradually lighting up from an off state. The subtitle "Screen remains off before lighting up" appears from 1.60s to 3.80s, where the target visual object is the "screen," "off" is the standard state, and "before lighting up" specifies the state before the transition from off. In the continuous frames of the video, the screen remains "off" from 1.20s to 2.40s, changes from off to on from 2.44s to 2.96s, and remains "on" after 3.00s. The subtitle time range simultaneously covers the screen before, during, and after the transition, constituting a description that crosses the boundary of the state transition.

[0101] Since the subtitles explicitly state "before it lights up," the visual content used for consistency judgment of the "off" state is limited to the first visual state. The visual state of the same display screen from 1.20s to 2.40s is read, and the standardized state label is "off," consistent with the subtitle state; therefore, this state content receives visual support. After 2.44s, the display screen gradually brightens and eventually lights up. Although these images are still within the subtitle display range, their temporal semantics no longer belong to "before it lights up." Therefore, the description of the pre-transition state by the transitioning and post-transition lighting images is not negated by the subtitles.

[0102] If another advertising material contains the subtitle "The screen remains off until it is turned on," while the target display screen actually transitions from an "on" state to an "off" state, and the subtitle's time interval crosses the boundary of this actual state transition, then the actual first visual state before the transition is "on." The subtitle requires the state before the transition to be "off," which is inconsistent with the first visual state. Therefore, the state content lacks visual support during the semantic application phase, resulting in a cross-modal content conflict. This judgment only uses the actual visual state before the transition and does not use the "off" state after the transition to support the description before the transition.

[0103] The pre- and post-conversion expressions employ the same state-stage limitation mechanism, but their semantic temporal positions differ. The pre-conversion expression limits the comparison scope to the first visual state, while the post-conversion expression limits the comparison scope to the second visual state. Both first determine the semantically applicable visual stage before performing state support judgments. This processing is suitable for situations where the advertising description describes the initial state before the state change and the resulting state after the state change.

[0104] When the same advertisement segment contains both subtitles and audio, the subtitle content and its display time form one description record, and the audio recognition text and its audio timing form another description record. Both description records share the same advertisement video timeline. When the subtitle and audio descriptions point to the same target visual object, the state transition information already formed for that target visual object is read separately. When the subtitle and audio descriptions point to different target visual objects, the state transition information corresponding to their respective target visual objects is read separately. The subtitle text and audio text are not concatenated into a new state description. Each description separately determines whether it crosses the state transition boundary of the corresponding target visual object, extracts the state content and temporal semantic position, and limits its own visual range for consistency judgment.

[0105] For example, if the device undergoes a "closed → open" state transition between 5.00s and 5.60s, the subtitle "Data can be viewed after the device is turned on" appears from 4.60s to 6.40s, and the voice message "Device is turning on" continues from 4.90s to 5.50s. Both the subtitle and the voice message involve the same device. The subtitle time interval spans the start and end positions of the state transition, while the voice time interval extends from the first visual state into the state transition range and crosses the boundary of the state transition start. Specifically, the subtitle state "open" corresponds to the state after the transition, and the voice message "turning on" corresponds to the state in progress. The processing device uses the second visual state as direct state evidence for the subtitle and uses the actual overlapping state transition image between the 5.00s and 5.50s voice time intervals and the actual transition direction of "closed → open" as direct evidence for the voice message.

[0106] In scenarios where the device transitions from "off" to "on" and both subtitles and audio are present, although the subtitles and audio share a common time range of 4.90s to 5.50s, they do not substitute for each other as judgment criteria simply because they respectively express "the device is on" and "the device is being turned on." Subtitle judgment focuses on whether the "on" state after the transition is valid, while audio judgment focuses on whether the transition process and the actual direction of the transition are valid. When both descriptions are supported by corresponding visual evidence, they each result in a non-conflict conclusion; when either description lacks visual support within its semantic scope, only that description results in a cross-modal content conflict.

[0107] If the subtitles and audio descriptions are in the same state, the boundary crossing condition is still determined separately based on their respective start and end times. The subtitle display time may be earlier than the audio start time, and the audio end time may be earlier than the subtitle disappearance time; this method does not require that the two have exactly the same time interval. As long as a description crosses the state transition boundary of the target visual object and the semantic position of the state can be determined, the visual state comparison can be completed according to the same stage constraint rules.

[0108] When the material processing device performs a state conflict identification for a description, it retains the intermediate information required to complete the visual stage constraint and state support judgment in the current processing task. The continuous state information of the target visual object includes at least the object identifier, the video time corresponding to each state, the first visual state, the second visual state, and the state transition range and direction; the descriptive information includes at least the descriptive text, the start and end times of the description, the target visual object, the standard state content or state change content, and the determined semantic time position. After the continuous state information and descriptive information of the target visual object are formed, they are directly used as input for the visual stage constraint and state support judgment.

[0109] When an advertisement video contains multiple descriptions, they are processed sequentially from beginning to end according to their start time. For each description, the temporal sequence of visual states of the same target visual object within its time interval is read. When the target visual object undergoes multiple state transitions in a long video, each state transition is composed of the corresponding first visual state, the state transition range, and the second visual state, forming independent state transition information. When the current description overlaps with the range of one of these state transitions, the boundary judgment is performed based on that state transition information, without deleting the intermediate stable state and concatenating the two separate state transitions into a single transition. When the time interval of the current description simultaneously covers two separate state transitions, the two state transitions are treated as independent processing objects, and each forms a corresponding judgment result based on the state content, semantic temporal position, and actual transition direction of the current description.

[0110] For an advertisement segment featuring consecutive "close → open → close" transitions to the same target visual object, two state transitions with different directions are formed on the timeline: "close → open" and "open → close". When the description of "opening" overlaps with the previous "close → open" transition time, the previous transition process and direction provide support for the judgment. When the description time actually covers the subsequent "open → close" process, the actual transition direction is opposite to the "close → open" direction required by "opening", resulting in a conflict. The judgment results of the two state transitions correspond to their respective state transition time ranges, and the earlier "open" state is not used as support for the later description state across intermediate transitions.

[0111] After each description is evaluated, the correspondence between the current conflict identification result and the advertising video, target visual object, and description time is retained. When processing another description, the state transition information of the target visual object corresponding to the description time is read; the "support" judgment or conflict result formed by the previous description is not used as the visual state input for another description. Each description shares the objective visual state timeline of the same advertising video, but the state semantic position and conflict judgment are formed independently by their respective text content and time interval.

[0112] The material processing equipment can ultimately summarize the conflict results of each description in chronological order of the advertisement video. The summary only arranges the results that have been formed, without changing the judgment rules of individual descriptions. For descriptions that have not formed a conflict, the current judgment supported by visual information can be retained; for descriptions that have formed a conflict, the target visual object, the corresponding description, and its time range are retained for subsequent material inspection and positioning. The entire method starts from the initial input of continuous video frames, subtitles, and audio, and continuously forms the final cross-modal content conflict results through the state transformation of the same target visual object, cross-boundary description semantic positioning, visual comparison range limitation, state and direction support judgment.

[0113] It should be noted that the methods, systems, devices, or equipment described in the above embodiments can be implemented in software, hardware, or a combination of both, depending on the actual application scenario. When implemented in software, the corresponding program instructions can be stored in a computer-readable storage medium and executed by a processor to achieve the processing steps or functions described in the above embodiments. For the specific processing procedures of each functional module in the system, device, or equipment, refer to the corresponding processing procedures in the foregoing method embodiments.

[0114] The above description is merely a specific embodiment of the present invention, used to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Without departing from the technical concept of the present invention, any equivalent substitutions, reasonable modifications, or combinations made by those skilled in the art based on the disclosure of this specification should be determined according to the claims to determine whether they fall within the scope of protection of the present invention.

Claims

1. A method for identifying multimodal content conflicts in advertising materials, characterized in that, include: S1. Determine the target visual object based on at least one of the subtitle description and voice description of the advertising video, obtain the visual state of the same object based on consecutive frames, and determine the range and direction of its transition from the first visual state to the second visual state. S2. When the description interval crosses the boundary of the transformation range, determine at least one of the state content and state change content for the target visual object, and determine that the content corresponds to one of the following categories: before transformation, during transformation, after transformation, and complete transformation process. S3. Based on the correspondence, determine the corresponding parts involved in the consistency judgment in the first visual state, the transition process, and the second visual state, and exclude visual states that overlap with the description time but do not belong to the corresponding transition stage. S4. Determine whether the content is supported by the actual visual state in the corresponding section, and verify the description of the change direction and the actual conversion direction when state changes are involved; S5. If the content is supported and the direction is consistent when state changes are involved, the conflict is not confirmed based on the local inconsistency of the excluded visual state; otherwise, a cross-modal content conflict result is formed.

2. The method for identifying multimodal content conflicts in advertising materials according to claim 1, characterized in that, The time semantic expression before, during, and after the conversion is determined based on the direct modifying relationship between the state words or state change actions in the description; A complete transition process is defined when the description clearly specifies both the initial and final states and expresses the state changes between them. Before the conversion corresponds to the first visual state, during the conversion corresponds to the conversion range and the actual conversion direction, and after the conversion corresponds to the second visual state. The complete conversion process corresponds to an ordered state change consisting of the first visual state, the conversion range, and the second visual state.

3. The method for identifying multimodal content conflicts in advertising materials according to claim 1, characterized in that, When the same state label of the same target visual object remains consistent in consecutive frames to reach a set number of frames, the corresponding consecutive frames and the adjacent consecutive frames with the same state label are determined as the stable state range. When different state labels appear after a stable state range and then another stable state range is formed, the first visual state and the second visual state are determined respectively. The transition range starts from the first frame after the state change after the previous stable state range and ends at the adjacent video frame before the starting frame of the second visual state.

4. The method for identifying multimodal content conflicts in advertising materials according to claim 3, characterized in that, Based on the object region and appearance information in adjacent frames, the same object identifier is maintained, and only frames with consecutive and consistent object identifiers are used to form the visual state sequence. If the identity of the object cannot be confirmed to be consistent, the previous and subsequent visual states will not be spliced ​​together as the state transition of the same target visual object.

5. The method for identifying multimodal content conflicts in advertising materials according to claim 2, characterized in that, The visual state of the target visual object is represented as a canonical state label, the synonymous state words in the description are mapped to canonical state names consistent with the canonical state labels, and the state change action is mapped to an ordered state pair with a start state and an end state. The ordered state pair is compared with the ordered state pair formed by the first visual state and the second visual state. If the two are consistent, it is determined that the described change direction is consistent with the actual conversion direction.

6. The method for identifying multimodal content conflicts in advertising materials according to claim 2, characterized in that, When describing the corresponding transformation, the description interval and the state transformation range overlap in time, and the state change is determined to obtain actual visual state support when the described change direction is consistent with the actual transformation direction. State transitions that have already ended in the previous description range are not used to support the current description.

7. The method for identifying multimodal content conflicts in advertising materials according to claim 1, characterized in that, Each description in the subtitle and audio description is supported by the visual state and state transition range corresponding to the target visual object that it is specifically associated with. The same state or state change of other objects in the picture are not used as the supporting basis for the description.

8. The method for identifying multimodal content conflicts in advertising materials according to claim 3, characterized in that, When the same target visual object undergoes multiple state transitions, each transition is formed by the corresponding first visual state, the transition range, and the second visual state, creating independent state transition information. When describing multiple separate state transitions, each state transition is judged separately, and multiple state transitions are combined into one transition without crossing intermediate stable states.

9. The method for identifying multimodal content conflicts in advertising materials according to claim 1, characterized in that, When the same advertisement segment contains both subtitles and audio, the subtitle content and display time are recorded separately from the audio recognition text and audio time. When both are pointing to the same target visual object, the first visual state, conversion range, second visual state, and conversion direction of the target visual object are read respectively. When they are pointing to different target visual objects, the first visual state, conversion range, second visual state, and conversion direction of each target visual object are read respectively.

Images