Interactive video generation system with image sequence consistency and method thereof

Abstract

This invention provides a multimodal interactive 2D video system and method with image sequence consistency processing: Using at least one reference image as the start / end pose of the interactive target, multiple action segments are generated offline, and image sequence consistency (transition frames, motion compensation reconstruction, geometric alignment, fusion, color correction, or any combination thereof) is performed on the beginning and end of the segments. The system establishes interactive area data (e.g., polygon pixel coordinates) that can be updated over time. During execution, it receives trigger signals (including touch input or sound features) to determine and select a response segment, and simultaneously outputs single-sentence subtitles and pre-generated audio messages. Subsequent triggers can be ignored during playback, and the system returns to standby loop after playback ends. The interactive target is not limited to a person; a person is used as an example in the following description.

Claims

1. An interactive video generation method with image sequence consistency executed by an interactive video generation system, comprising: the interactive video generation system providing a standby state, wherein the standby state is playing a standby loop video, and the start frame and end frame of the standby loop video are visually aligned to the same reference image through image sequence consistency processing; the interactive video generation system preparing a plurality of reaction video segments, wherein each reaction video segment is processed to align the visual features of its start frame and end frame to the reference image; the system obtaining interactive area definition data corresponding to the display screen and changing over time; the system receiving a trigger signal and determining whether the trigger signal hits the range specified by the interactive area definition data corresponding to the current time; if it is determined to hit, the system obtaining an area identification based on the interactive area definition data corresponding to the current time, selecting a target segment from the plurality of reaction video segments based on the area identification and playing the target segment by the system; after the target segment finishes playing, the system switches back to the standby state.

2. The method as described in claim 1, wherein the image sequence consistency processing includes at least one of: generating intermediate transition frames, estimating motion fields and performing motion compensation reconstruction, geometric alignment, image fusion, and color correction, to reduce visual differences at segment switching points.

3. The method as described in claim 1, wherein the interactive area definition data includes time series data corresponding to different time points, and the time series data is at least one of the following: polygon coordinate sequence, pixel mask sequence, or bounding box sequence.

4. The method as described in claim 1 further includes: obtaining text prompt data and audio data corresponding to the target segment by the interactive video generation system; and during the playback of the target segment, synchronously displaying the text prompt data on the display screen and playing the audio data by the system.

5. An interactive video generation system with image sequence consistency, comprising: a display unit for displaying a screen; an input unit for receiving a trigger signal; a storage unit for storing a reference image, a plurality of standby / response video segments, interactive area definition data corresponding to different time points, and correspondence data between area identification and response video segments; wherein each of the standby / response video segments undergoes image sequence consistency processing to align the visual features of its start and end with the reference image; and a processing unit coupled to the display unit, the input unit, and the storage unit, the processing unit being configured to perform the following steps: controlling the display unit to present a standby state, wherein the standby state is playing a standby loop video; determining whether the trigger signal falls within the range specified by the interactive area definition data corresponding to the current time; if so, obtaining an area identification based on the interactive area definition data corresponding to the current time, and selecting the corresponding response video segment for playback based on the correspondence data between the area identification and the response video segment; and controlling the system state to switch back to the standby state when playback ends.

Images