Method, system, device and storage medium for generating video from a single image based on interest points

Abstract

The present invention provides a method, system, device and storage medium for generating videos based on a single image of a point of interest, the method comprising: collecting a large amount of videos of the point of interest; preprocessing the large amount of videos, using the preprocessed videos as a training data set, and using a large language model to describe the video to generate language description information; training a large image-generated video model based on the video and the language description information, in which the video is at least split into a number of video frames; obtaining a target image of the point of interest; inputting the target image into the trained large image-generated video model, identifying the object in the target image, and obtaining language description information based on the image, the language description information at least includes the objects identified in the image, the spatial positions between the objects and the camera motion trajectory, thereby generating a video showing the point of interest. The present invention can realize the rapid generation of dynamic effect videos from a single image, greatly reduce labor costs, optimize the content display of hotels, scenic spots, etc., and improve user experience.

Description

Background Art

[0002] In the current Internet environment, images and videos are two important information display media. Compared with images, videos can dynamically reflect the development and change trends of things, contain more information than pictures, and display more vivid content. On the OTA platform, you can see a large amount of various image data and some video data, but for some hotels and tourist attractions, there is a lack of video display data, and it takes a lot of manpower and time to collect video data. Therefore, as an OTA platform, these image data can be converted into video data to improve the user experience. Turning simple images into dynamic videos not only expands the corresponding video display data, but also provides consumers with a novel, interesting, and immersive display effect, bringing a visual impact and user experience that comprehensively surpasses two-dimensional images.

[0003] The OTA platform has a huge amount of hotel and tourist attraction image data, but the amount of video data is relatively small. If the relevant video data needs to be expanded, a lot of manpower and time will be required to collect it.

[0004] Therefore, the present invention provides a method, system, device and storage medium for generating a video from a single image based on points of interest. Summary of the invention

[0005] In response to the problems in the prior art, the purpose of the present invention is to provide a method, system, device and storage medium for generating videos from a single image based on points of interest, which overcomes the difficulties of the prior art and can quickly generate dynamic effect videos from a single image, significantly reduce labor costs, optimize the content display of hotels, scenic spots, etc., and enhance user experience.

[0006] An embodiment of the present invention provides a method for generating a video from a single image based on interest points, comprising the following steps:

[0007] S110, collecting massive videos of points of interest;

[0008] S120, preprocessing the massive amount of videos, using the preprocessed videos as training data sets, and using a large language model to describe the videos to generate language description information;

[0009] S130, training a large image-generated video model based on the video and the language description information, wherein the video is split into at least a plurality of video frames in the large image-generated video model;

[0010] S140, obtaining a target image of a point of interest;

[0011] S150, inputting the target image into a trained image-generated video model, identifying objects in the target image, and obtaining language description information based on the image, wherein the language description information at least includes the objects identified in the image, the spatial positions between the objects, and the camera motion trajectory based on the objects and the spatial positions;

[0012] S160: Generate a point of interest display video based on the object, the spatial positions between the objects, and the camera motion trajectory.

[0013] Preferably, the step S120 includes:

[0014] S121, preprocessing the massive amount of videos, including filtering based on video resolution, video length, video aesthetic score, video motion score, and portrait recognition;

[0015] S122: Use the preprocessed video as a training data set, and use a large language model to perform video description on each video in the training data set to generate language description information.

[0016] Preferably, in step S130, the image-generated video model includes a video encoder, a text encoder, a diffusion model, and a video decoder. The video encoder is used to compress and extract input video features in spatial and temporal dimensions; the text encoder is used to compress and extract input video description features; the main function of the diffusion model is to gradually add noise and recover the target sample from the noise; the video decoder decodes the video content generated by the diffusion model into watchable video content.

[0017] Preferably, in step S140, the target image of the point of interest obtained includes the image with the most likes or the most reposts based on the point of interest, or the image uploaded by the user.

[0018] Preferably, the step S160 includes generating a virtual space based on the objects and the spatial positions between the objects, and generating a motion perspective in the virtual space according to the camera motion trajectory to generate a point of interest display video.

[0019] Preferably, the massive videos include official hotel videos, hotel room videos, and travel videos uploaded by users.

[0020] Preferably, the points of interest include any one of hotels, natural attractions, and landmark buildings.

[0021] An embodiment of the present invention further provides a system for generating a video from a single image based on points of interest, which is used to implement the above-mentioned method for generating a video from a single image based on points of interest. The system for generating a video from a single image based on points of interest includes:

[0022] Video acquisition module, which collects massive videos of points of interest;

[0023] A language description information module pre-processes the massive videos, uses the pre-processed videos as training data sets, and uses a large language model to describe the videos to generate language description information;

[0024] A model training module, which trains a large image-generated video model based on the video and the language description information, wherein the video is split into at least a plurality of video frames in the large image-generated video model;

[0025] Target image module, obtains the target image of the point of interest;

[0026] A model input module, inputting the target image into a trained image-generated video model, identifying objects in the target image, and obtaining language description information based on the image, wherein the language description information at least includes the objects identified in the image, the spatial positions between the objects, and the camera motion trajectory based on the objects and the spatial positions;

[0027] The video output module generates a point of interest display video based on the object, the spatial position between the objects and the camera motion trajectory.

[0028] An embodiment of the present invention further provides a device for generating a video based on a single image of an interest point, comprising:

[0029] processor;

[0030] a memory storing executable instructions of the processor;

[0031] The processor is configured to execute the steps of the above-mentioned method for generating a video from a single image based on points of interest by executing the executable instructions.

[0032] An embodiment of the present invention further provides a computer-readable storage medium for storing a program, which, when executed, implements the steps of the above-mentioned method for generating a video from a single image based on interest points.

[0033] The purpose of the present invention is to provide a method, system, device and storage medium for generating videos from a single image based on points of interest, which can realize the rapid generation of dynamic effect videos from a single image, greatly reduce labor costs, optimize the content display of hotels, scenic spots, etc., and improve user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Other features, objectives and advantages of the present invention will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0035] Figure 1 It is a flow chart of the method for generating video from a single image based on interest points of the present invention.

[0036] Figure 2 It is a schematic diagram of the implementation process of the method for generating video from a single image based on interest points of the present invention.

[0037] Figure 3 It is a data preprocessing flow chart of the method for generating video from a single image based on interest points of the present invention.

[0038] Figure 4 It is a flowchart of fine-tuning a large model in the method for generating a video from a single image based on interest points of the present invention.

[0039] Figure 5 It is a structural schematic diagram of the system for generating video based on a single image of interest points of the present invention.

[0040] Figure 6 It is a structural schematic diagram of a single-image video generation device based on interest points of the present invention.

[0041] Figure 7 It is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the present application. The present application can also be implemented or applied through other different specific implementation methods, and the details in the present application can also be modified or changed in various ways according to different viewpoints and application systems without departing from the spirit of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0043] The following is a detailed description of the embodiments of the present application with reference to the accompanying drawings so that those skilled in the art can easily implement the present application. The present application can be embodied in a variety of different forms and is not limited to the embodiments described herein.

[0044] In the representations of the present application, the representations with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean that the specific features, structures, materials or characteristics represented in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the represented specific features, structures, materials or characteristics may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples represented in the present application and the features of the different embodiments or examples, unless they contradict each other.

[0045] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the representation of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0046] In order to clearly describe the present application, components irrelevant to the description are omitted, and the same reference symbols are given to the same or similar components throughout the specification.

[0047] Throughout the specification, when a device is said to be "connected" to another device, this includes not only the case of "direct connection" but also the case of "indirect connection" by placing other elements therebetween. In addition, when a device is said to "include" a certain component, unless otherwise stated, it does not exclude other components, but means that other components may be included.

[0048] When a device is said to be "on" another device, it may be directly on the other device, but there may also be other devices between it. In contrast, when a device is said to be "directly" on another device, there are no other devices between it.

[0049] Although the terms first, second, etc. are used to represent various elements in some examples, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first interface and the second interface, etc. are represented. Moreover, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising" and "including" indicate the existence of features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Only when the combination of elements, functions, steps or operations is inherently mutually exclusive in some way, will there be an exception to this definition.

[0050] The technical terms used herein are only used to refer to specific embodiments and are not intended to limit the present application. The singular form used herein also includes the plural form unless the sentence clearly indicates the contrary meaning. The meaning of "including" used in the specification is to specify specific characteristics, regions, integers, steps, operations, elements and / or components, and does not exclude the existence or addition of other characteristics, regions, integers, steps, operations, elements and / or components.

[0051] Although not defined differently, all terms, including technical and scientific terms used herein, have the same meaning as those generally understood by those skilled in the art to which this application belongs. Terms defined in commonly used dictionaries are additionally interpreted as having meanings consistent with the relevant technical literature and the contents of the current disclosure, and unless defined, shall not be overly interpreted as ideal or very formal meanings.

[0052] Figure 1 : is a flow chart of the method for generating a video based on a single image of interest points of the present invention. Figure 1 As shown, the method for generating a video based on a single image of interest points of the present invention comprises the following steps:

[0053] S110, collecting massive videos of points of interest.

[0054] S120, preprocessing the massive amount of videos, using the preprocessed videos as training data sets, and using a large language model to describe the videos to generate language description information.

[0055] S130. Training a large image-generated video model based on the video and the language description information, wherein the video is split into at least a plurality of video frames in the large image-generated video model.

[0056] S140: Obtain a target image of the point of interest.

[0057] S150, inputting the target image into a trained image-generated video model, identifying objects in the target image, and obtaining language description information based on the image, wherein the language description information at least includes objects identified in the image, spatial positions between objects, and camera motion trajectories based on the objects and spatial positions.

[0058] S160: Generate a point of interest display video based on the object, the spatial positions between the objects, and the camera motion trajectory.

[0059] The problem to be solved by the present invention is to establish an algorithm model in view of the current lack of video display data on the OTA platform, use a single image to generate videos with motion effects and camera movements, expand relevant video display data such as hotels and scenic spots, and improve user experience. The present invention reduces the difficulty and cost of manually collecting videos by utilizing the large model method of image-generated video, and automatically generates videos using existing image data to expand the video coverage of the OTA platform. In recent years, image-generated videos have developed rapidly, and a single image can be used to generate stereoscopic videos with a fusion of multiple motion effects. This article introduces how to use a text-based controllable method to quickly generate videos from a single image.

[0060] In a preferred embodiment, the step S120 includes:

[0061] S121, pre-processing the massive amount of videos, including filtering based on video resolution, video length, video aesthetic score, video motion score, and portrait recognition.

[0062] S122: Using the preprocessed video as a training data set, and using a large language model to perform video description on each video in the training data set to generate language description information, but the present invention is not limited thereto.

[0063] In a preferred embodiment, in step S130, the image-generated video model includes a video encoder, a text encoder, a diffusion model, and a video decoder. The video encoder is used to compress and extract input video features in spatial and temporal dimensions. The text encoder is used to compress and extract input video description features. The main function of the diffusion model is to gradually add noise and recover the target sample from the noise. The video decoder decodes the video content generated by the diffusion model into watchable video content, but is not limited to this.

[0064] In a preferred embodiment, in step S140, the target image of the point of interest obtained includes the image with the most likes or the most reposts based on the point of interest, or the image uploaded by the user, but is not limited thereto.

[0065] In a preferred embodiment, the step S160 includes generating a virtual space based on the objects and the spatial positions between the objects, and generating a motion perspective in the virtual space according to the camera motion trajectory to generate a point of interest display video, but is not limited to this.

[0066] In a preferred embodiment, the massive amount of videos includes official hotel videos, hotel room videos, and travel videos uploaded by users, but is not limited thereto.

[0067] In a preferred embodiment, the points of interest include any one of hotels, natural attractions, and landmark buildings, but are not limited thereto.

[0068] In a preferred embodiment, step S170 is also included, which regularly converts the recent popular scenic spot pictures or hotel pictures on the OTA website into short videos of the same length and width, and replaces the original pictures, so as to further enhance the promotion effect of the points of interest, but is not limited to this.

[0069] The present invention establishes an algorithm model based on the existing OTA platform video data to realize the rapid generation of dynamic effect videos from a single image, which can greatly reduce labor costs, optimize the content display of hotels, scenic spots, etc., improve user experience, and build the brand image of OTA.

[0070] Specific embodiments of the present invention include:

[0071] Collect a large number of videos of points of interest, including official hotel videos, hotel room videos, and travel videos uploaded by users. The points of interest include any one of hotels, natural attractions, and landmark buildings. Preprocess the massive videos, including filtering based on video resolution, video length, video aesthetic score, video motion score, and portrait recognition. Use the preprocessed videos as training data sets, and use a large language model to describe each video in the training data set to generate language description information. Train a large graph-generated video model based on the video and language description information. In the large graph-generated video model, the video is at least split into several video frames. The large graph-generated video model includes a video encoder, a text encoder, a diffusion model, and a video decoder. The video encoder is used to compress and extract input video features in spatial and temporal dimensions. The text encoder is used to compress and extract input video description features. The main function of the diffusion model is to gradually add noise and recover the target sample from the noise. The video decoder decodes the video content generated by the diffusion model into watchable video content. Obtain target images of points of interest, including images with the most likes or the most forwarded numbers based on the points of interest, or images uploaded by users. Input the target image into a trained large model of image-generated video, identify the objects in the target image, and obtain language description information based on the image, the language description information at least includes the objects identified in the image, the spatial positions between the objects, and the camera motion trajectory based on the objects and the spatial positions. Generate a virtual space based on the objects and the spatial positions between the objects, and generate a motion perspective in the virtual space according to the camera motion trajectory to generate a point of interest display video. Regularly convert the recent popular scenic spot images or hotel images on the OTA website into short videos of the same length and width, and replace the original images, so as to further enhance the publicity effect of the OTA points of interest, especially users who have liked and forwarded. After seeing the original images upgraded to videos, they will generate higher order rates and better recognition for related products and OTA websites, further improving the user experience.

[0072] Figure 2 The present invention proposes a universal image-to-video method for OTA scenarios based on text controllable OTA scenarios. The implementation flow chart of the scheme is shown in the attached figure. Figure 2 The specific implementation is through the following technical solution, which includes the following steps:

[0073] S01: Data collection and integration, collecting original video data related to hotels and attractions on the OTA platform, including official hotel videos, hotel room videos, travel videos, etc.

[0074] S02: Data preprocessing: Through a series of cleaning processes such as video resolution, video length, video beauty score, video motion score, portrait recognition, and OCR text recognition, we automatically select OTA online videos with high beauty, high clarity, high motion intensity, and no portraits or text to produce training data sets, and use a large language model to generate video descriptions.

[0075] S03: Training of large image-generated video models: Use the cleaned high-quality video data to train large image-generated video models and optimize model parameters.

[0076] S04: Manual intervention and screening to generate the best videos.

[0077] Among them, for S01, in the implementation of this case, a large amount of video data is collected from the OTA platform database as the initial data set, and the video data types include hotel official videos, hotel room videos, travel videos, scenic spot videos, etc. The videos cover scenes such as hotels and scenic spots.

[0078] Figure 3 This is a data preprocessing flow chart of the method for generating a video based on a single image of interest points of the present invention. Figure 3 As shown, in the implementation of this case, firstly, the videos with video frame numbers between 75 and 200 frames and video resolutions greater than 720P are screened out from the initial data set based on the two basic conditions of video frame number and video resolution; then, the video segmentation model is used to identify the transitions in the original video and divide the original video into independent video clips; then, the video beauty recognition model, video motion recognition model, portrait recognition model and OCR text recognition model are used to score and identify the segmented video clips respectively, and the video beauty score greater than 4 points and the video motion score greater than 2 points are screened out, and the video clips with portrait recognition probability greater than 0.8 and OCR text recognition probability greater than 0.2 are filtered out; finally, the multimodal large language model is used to generate descriptions for the screened video clips, and the generated descriptions include the main content in the video and the camera motion trajectory in the video, and finally a high-quality video text pair data set is obtained.

[0079] As for S03, in the implementation of this case, the image-generated video model mainly includes a video encoder, a text encoder, a diffusion model, a video decoder, etc. The video encoder is used to compress and extract input video features in spatial and temporal dimensions, the text encoder is mainly used to compress and extract input video description features, the diffusion model is mainly used to generate high-quality video content by gradually adding noise and recovering the target sample from the noise, and the video decoder decodes the video content generated by the diffusion model into watchable video content. Figure 4This is a flowchart of fine-tuning a large model in the method of generating a video from a single image based on interest points of the present invention. Figure 4 As shown, the specific steps are:

[0080] Step 1: Model building

[0081] For the cleaned high-quality video-text pair training data, the video is first masked to retain only the first frame of the video. Then, the masked video and mask are compressed in spatial and temporal dimensions using a video encoder to extract video features and mask features. The video description is extracted as text features using a text encoder. The video features, mask features, and random noise are then concatenated and sent to the diffusion model. The text features are fused in the diffusion model through the attention mechanism. Finally, the resulting features output by the diffusion model are sent to the video decoder to restore the final video results.

[0082] Step 2: Model fine-tuning

[0083] In order to improve training efficiency, reduce model training difficulty, and enhance the model's image-generated video effect in OTA scenarios, LoRA (Low-Rank Adaptation) technology was used in this case to fine-tune the large image-generated video model. The target modules for fine-tuning mainly include the attention module and the fully connected module in the diffusion model. These key modules are mainly responsible for processing videos and text descriptions.

[0084] Step 3: Video generation

[0085] After fine-tuning training, given a picture and picture description, the image-generated video model can understand the language content in the picture description, accurately process the correspondence between the picture description and the content in the picture, so that the model has the ability to generate dynamic effects such as water movement and cloud movement. Then, by understanding the camera motion trajectory description in the picture description, the motion perspective in the generated video is controlled, and finally the video result generated based on text control is obtained.

[0086] For S04, the generated motion effect videos will be manually screened and scored and annotated, and finally videos with higher scores and better generated effects will be selected for online display.

[0087] Based on the limited online hotel and scenic spot video data of the OTA platform, the present invention realizes a text-controllable OTA scene general image-to-video method, which can automatically convert a single image and image description into a dynamic video, effectively control the video to generate motion effects, and expand the online video display data, which greatly reduces labor costs and improves user experience.

[0088] In this embodiment, the large model for generating image content description and camera motion trajectory description is an independent model, which is only used to generate image content description and camera motion trajectory description in the whole process. In the process of training the large model for video generation, the training data is the video and the video description, so the fine-tuning method of LoRA in the training process will also train the modules for video generation and text description processing.

[0089] Figure 5 Schematic diagram of the structure of the system for generating video based on a single image of interest points of the present invention. Figure 5 As shown, an embodiment of the present invention further provides a single-image video generation system based on interest points, which is used to implement the above-mentioned single-image video generation method based on interest points. The single-image video generation system based on interest points 5 includes:

[0090] The video acquisition module 51 acquires a large amount of video of points of interest.

[0091] The language description information module 52 pre-processes the massive videos, uses the pre-processed videos as training data sets, and uses a large language model to describe the videos to generate language description information.

[0092] The model training module 53 trains a large image-generated video model based on the video and language description information, in which the video is divided into at least a number of video frames.

[0093] The target image module 54 obtains the target image of the point of interest.

[0094] The model input module 55 inputs the target image into the trained image-generated video model, identifies the objects in the target image, and obtains language description information based on the image, wherein the language description information at least includes the objects identified in the image, the spatial positions between the objects, and the camera motion trajectory based on the objects and the spatial positions.

[0095] The video output module 56 generates a point of interest display video based on the objects, the spatial positions between the objects and the camera motion trajectory.

[0096] In a preferred embodiment, the language description information module 52 is configured to pre-process the massive video, including filtering based on video resolution, video length, video aesthetic score, video motion score, and portrait recognition, using the pre-processed video as a training data set, and using a large language model to generate language description information for each video in the training data set, but not limited to this.

[0097] In a preferred embodiment, the model training module 53 is configured as the image-generated video model includes a video encoder, a text encoder, a diffusion model, and a video decoder. The video encoder is used to compress and extract input video features in spatial and temporal dimensions. The text encoder is used to compress and extract input video description features. The main function of the diffusion model is to gradually add noise and recover the target sample from the noise. The video decoder decodes the video content generated by the diffusion model into watchable video content, but is not limited to this.

[0098] In a preferred embodiment, the target picture module 54 is configured to obtain the target pictures of the point of interest including pictures with the most likes or the most reposts based on the point of interest, or pictures uploaded by users, but not limited thereto.

[0099] In a preferred embodiment, the video output module 56 is configured to generate a virtual space based on the objects and the spatial positions between the objects, and to generate a motion perspective in the virtual space according to the camera motion trajectory to generate a point of interest display video, but is not limited to this.

[0100] In a preferred embodiment, the massive amount of videos includes official hotel videos, hotel room videos, and travel videos uploaded by users, but is not limited thereto.

[0101] In a preferred embodiment, the points of interest include any one of hotels, natural attractions, and landmark buildings, but are not limited thereto.

[0102] The single-image video generation system based on points of interest of the present invention can realize the rapid generation of dynamic effect videos from a single image, greatly reduce labor costs, optimize the content display of hotels, scenic spots, etc., and improve user experience.

[0103] The embodiment of the present invention also provides a device for generating a video based on a single image of interest points, comprising a processor and a memory in which executable instructions of the processor are stored. The processor is configured to execute the steps of the method for generating a video based on a single image of interest points by executing the executable instructions.

[0104] As shown above, the single-image video generation device based on points of interest in this embodiment of the present invention can quickly generate dynamic effect videos from a single image, greatly reduce labor costs, optimize the content display of hotels, scenic spots, etc., and improve user experience.

[0105] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0106] Figure 6 This is a schematic diagram of the structure of the device for generating video based on a single image of interest points of the present invention. Figure 6 The electronic device 600 according to this embodiment of the present invention is described. Figure 6 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0107] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0108] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0109] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0110] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0111] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0112] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0113] The embodiment of the present invention also provides a computer-readable storage medium for storing a program, and the steps of the method for generating a video from a single image based on interest points are implemented when the program is executed. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above method section of this specification.

[0114] As shown above, the single-image video generation system based on points of interest in this embodiment of the present invention can quickly generate dynamic effect videos from a single image, greatly reduce labor costs, optimize the content display of hotels, scenic spots, etc., and improve user experience.

[0115] Figure 7 Schematic diagram of the structure of the computer-readable storage medium of the present invention. Figure 7 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0116] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0117] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0118] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0119] In summary, the purpose of the present invention is to provide a method, system, device and storage medium for generating videos from a single image based on points of interest, which can realize the rapid generation of dynamic effect videos from a single image, greatly reduce labor costs, optimize the content display of hotels, scenic spots, etc., and enhance user experience.

[0120] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A method for generating a video from a single image based on interest points, characterized in that: The following steps are involved: S110, collecting massive videos of points of interest; S120, preprocessing the massive amount of videos, using the preprocessed videos as training data sets, and using a large language model to describe the videos to generate language description information; S130, training a large image-generated video model based on the video and the language description information, wherein the video is split into at least a plurality of video frames in the large image-generated video model; S140, obtaining a target image of a point of interest; S150, inputting the target image into a trained image-generated video model, identifying objects in the target image, and obtaining language description information based on the image, wherein the language description information at least includes the objects identified in the image, the spatial positions between the objects, and the camera motion trajectory based on the objects and the spatial positions; S160: Generate a point of interest display video based on the object, the spatial positions between the objects, and the camera motion trajectory.

2. The method for generating a video from a single image based on interest points as claimed in claim 1, characterized in that: The step S120 includes: S121, preprocessing the massive amount of videos, including filtering based on video resolution, video length, video aesthetic score, video motion score, and portrait recognition; S122: Use the preprocessed video as a training data set, and use a large language model to perform video description on each video in the training data set to generate language description information.

3. The method for generating a video from a single image based on interest points as claimed in claim 1, characterized in that: In step S130, the image-generated video model includes a video encoder, a text encoder, a diffusion model, and a video decoder, and the video encoder is used to compress and extract input video features in spatial and temporal dimensions; The text encoder is used to compress and extract input video description features; The main function of the diffusion model is to gradually add noise and recover the target sample from the noise; the video decoder decodes the video content generated by the diffusion model into watchable video content.

4. The method for generating a video from a single image based on interest points as claimed in claim 1, characterized in that: In the step S140, the target pictures of the points of interest are obtained, including pictures with the most likes or the most reposts based on the points of interest, or pictures uploaded by users.

5. The method for generating a video from a single image based on interest points as claimed in claim 1, characterized in that: The step S160 includes generating a virtual space based on the objects and the spatial positions between the objects, and generating a motion perspective in the virtual space according to the camera motion trajectory to generate a point of interest display video.

6. The method for generating a video from a single image based on interest points as claimed in claim 1, characterized in that: The massive amount of videos include official hotel videos, hotel room videos, and travel videos uploaded by users.

7. The method for generating a video from a single image based on interest points as claimed in claim 1, characterized in that: The method also includes step S170, which regularly converts the recent popular scenic spot pictures or hotel pictures on the OTA website into short videos with the same length and width dimensions, and replaces the original pictures.

8. A system for generating videos from a single image based on points of interest, used to implement the method for generating videos from a single image based on points of interest as claimed in claim 1, characterized in that: include: Video acquisition module, which collects massive videos of points of interest; A language description information module pre-processes the massive videos, uses the pre-processed videos as training data sets, and uses a large language model to describe the videos to generate language description information; A model training module, which trains a large image-generated video model based on the video and the language description information, wherein the video is split into at least a plurality of video frames in the large image-generated video model; Target image module, obtains the target image of the point of interest; A model input module, inputting the target image into a trained image-generated video model, identifying objects in the target image, and obtaining language description information based on the image, wherein the language description information at least includes the objects identified in the image, the spatial positions between the objects, and the camera motion trajectory based on the objects and the spatial positions; The video output module generates a point of interest display video based on the object, the spatial position between the objects and the camera motion trajectory.

9. A device for generating video based on a single image of interest points, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to execute the steps of the method for generating a video based on a single image of interest points as described in any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the method for generating a video based on a single image of interest points according to any one of claims 1 to 7 are implemented.

Images