Video generation method and device, electronic equipment and storage medium

Abstract

The invention provides a video generation method and device, electronic equipment and a storage medium. The invention relates to the technical field of artificial intelligence, in particular to the technical field of vehicle-mounted image processing, and can be applied to application scenes such as an automatic driving assistance system, vehicle-mounted panoramic Vlog shooting and panoramic video generation. The specific implementation scheme is as follows: acquiring a plurality of target videos shot by different cameras; analyzing each target video to obtain a multi-frame image sequence and a timestamp corresponding to each target video; generating a multi-frame panoramic image based on the multi-frame image sequence corresponding to the plurality of target videos and the timestamp of each frame of image; performing image understanding analysis on each frame of panoramic image, and identifying and extracting composition elements and overall description of each frame of panoramic image; generating light music and subtitle streams based on the composition elements and the overall description of each frame of panoramic image; and generating a panoramic video based on each frame of panoramic image and the corresponding light music and subtitle stream in combination with the timestamps of the multiple frames of panoramic images.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the field of vehicle-mounted image processing technology. The present disclosure can be used in application scenarios such as automatic driving assistance systems, vehicle-mounted panoramic Vlog shooting, and panoramic video generation, and specifically relates to a video generation method, device, electronic device, and storage medium. Background Art

[0002] With the development of vehicle-mounted camera technology, driving video recording and autonomous driving assistance have gradually become important components of smart cars. Traditional driving recording relies on a single camera perspective, which is prone to information loss, and video production requires manual music and editing, which is inefficient. Therefore, how to automatically generate panoramic videos with multiple perspectives and complete information has become an urgent problem to be solved. Summary of the invention

[0003] The present disclosure provides a video generation method, device, electronic device and storage medium.

[0004] According to one aspect of the present disclosure, a video generation method is provided, the method comprising:

[0005] Acquire multiple target videos captured by different cameras, where the multiple target videos cover different perspectives of the same scene;

[0006] Parse each target video to obtain a multi-frame image sequence corresponding to each target video and a timestamp of each frame image;

[0007] Generate a multi-frame panoramic image based on a multi-frame image sequence corresponding to a plurality of target videos and a timestamp of each frame image;

[0008] Perform image understanding analysis on each panoramic image frame to identify and extract the composition elements and overall description of each panoramic image frame;

[0009] Based on the composition elements and overall description of each panoramic image frame, light music and subtitle streams matching each panoramic image frame are generated;

[0010] Based on each frame of panoramic image and its corresponding light music and subtitle stream, combined with the timestamps of multiple frames of panoramic images, a panoramic video is generated.

[0011] According to another aspect of the present disclosure, a video generating device is provided, the device comprising:

[0012] A video acquisition module is used to acquire multiple target videos shot by different cameras, where the multiple target videos cover different perspectives of the same scene;

[0013] An image parsing module is used to parse each target video to obtain a multi-frame image sequence corresponding to each target video and a timestamp of each frame image;

[0014] A panoramic generation module, used to generate a multi-frame panoramic image based on a multi-frame image sequence corresponding to a plurality of target videos and a timestamp of each frame image;

[0015] An image analysis module is used to perform image understanding and analysis on each panoramic image frame, identify and extract the composition elements and overall description of each panoramic image frame;

[0016] A music subtitle module, for generating light music and subtitle streams matching each panoramic image frame based on the composition elements and overall description of each panoramic image frame;

[0017] The video synthesis module is used to generate a panoramic video based on each frame of the panoramic image and its corresponding light music and subtitle stream, combined with the timestamps of multiple frames of the panoramic image.

[0018] Image Generation Model According to another aspect of the present disclosure, there is provided an electronic device, including:

[0019] at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0022] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0023] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0024] By adopting the solution disclosed in the present invention, multi-view videos can be automatically processed to generate complete and high-quality panoramic videos, thereby improving the efficiency of panoramic video production.

[0025] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0027] Figure 1is a flowchart of a video generation method according to an embodiment of the present disclosure;

[0028] Figure 2 is a schematic diagram of a process of panoramic image synthesis according to an embodiment of the present disclosure;

[0029] Figure 3 It is a schematic diagram of the overall process of generating a panoramic Vlog according to an embodiment of the present disclosure;

[0030] Figure 4 is a structural schematic diagram of a video generating device according to an embodiment of the present disclosure;

[0031] Figure 5 is a schematic diagram of a scene of a video generation method according to an embodiment of the present disclosure;

[0032] Figure 6 It is a structural schematic diagram of an electronic device used to implement the video generating method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0034] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0035] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0036] Before introducing the technical solutions of the embodiments of the present disclosure, the technical terms that may be used in the present disclosure are further explained:

[0037] Timestamp: refers to the specific time point when each frame of image appears in the video. It is used to mark the time sequence of image frames and ensure the synchronization of multiple frames of images and audio and video content.

[0038] Subtitle stream: refers to the subtitle content sequence generated based on image understanding, which is used to match the video screen display and provide viewers with text description or explanation of the video.

[0039] Light music: refers to background music generated by an algorithm, which usually has a light and soft rhythm. It is used to enhance the viewing experience of the video and is consistent with the style of the video content.

[0040] Vlog: The abbreviation of “Video Log”, is a form of recording personal life, travel or other activities through video, usually including pictures, sound effects and music.

[0041] In related technologies, traditional driving records rely on a single camera perspective, which can easily cause information loss, and video production requires manual music and editing, which is inefficient. Therefore, how to automatically generate panoramic videos with multiple perspectives and complete information becomes the key.

[0042] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, the present disclosure proposes a vehicle-mounted panoramic V1og generation solution based on multimodal information integration, which automatically processes multi-view videos, generates complete and high-quality panoramic videos, and improves the production efficiency of panoramic V1og.

[0043] The present disclosure provides a video generation method. Figure 1 1 is a flow chart of a video generation method according to an embodiment of the present disclosure, and the video generation method can be applied to a video generation device. The video generation device is located in an electronic device. The electronic device includes but is not limited to a fixed device and / or a mobile device. For example, a fixed device includes but is not limited to a server, and the server can be a cloud server or an ordinary server. For example, a mobile device includes but is not limited to: a mobile phone, a tablet computer, and a vehicle-mounted terminal. In some possible implementations, the video generation method can also be implemented by a processor calling a computer-readable instruction stored in a memory. Figure 1 As shown, the video generation method includes:

[0044] S101, acquiring multiple target videos shot by different cameras, where the multiple target videos cover different perspectives of the same scene;

[0045] S102, parsing each target video to obtain a multi-frame image sequence corresponding to each target video and a timestamp of each frame image;

[0046] S103, generating a multi-frame panoramic image based on a multi-frame image sequence corresponding to a plurality of target videos and a timestamp of each frame image;

[0047] S104, performing image understanding analysis on each panoramic image frame, identifying and extracting the composition elements and overall description of each panoramic image frame;

[0048] S105, generating light music and subtitle streams matching each frame of the panoramic image based on the composition elements and overall description of each frame of the panoramic image;

[0049] S106: Generate a panoramic video based on each frame of the panoramic image and its corresponding light music and subtitle stream, combined with timestamps of multiple frames of the panoramic image.

[0050] In the disclosed embodiments, composition elements refer to key visual elements in an image, which generally include visual information such as objects, people, landscapes, colors, shapes, light and shadow, etc. These elements constitute the main content and structure of the image. For example, in a panoramic image, composition elements may include specific objects such as vehicles, buildings, trees, road signs, etc., as well as their relative positions and proportions in the picture.

[0051] In the disclosed embodiments, the overall description refers to a comprehensive semantic description of the entire image content, which not only includes a simple listing of the composition elements, but also combines the context, scene atmosphere and emotional tone of the image to provide a higher-level interpretation of the image. For example, the overall description of a street panoramic image may be "a busy city street with pedestrians passing by, vehicles driving, and tall buildings and blue sky and white clouds in the distance."

[0052] In some embodiments, obtaining multiple target videos shot by different cameras includes: collecting the captured video data from multiple cameras by controlling the camera device or related software system. Including but not limited to batch or real-time importing video files from multiple cameras through a specific data acquisition module or interface. Different cameras refer to camera devices that are different in physical location, model, lens focal length or shooting angle. Including but not limited to specific brands, models, lens types (wide-angle, fisheye, standard lens, etc.), and cameras with different shooting parameters (such as frame rate, resolution) to ensure that different viewing angles can be covered.

[0053] In some embodiments, multiple target videos cover different perspectives of the same scene, including: multiple perspectives refer to multiple video perspectives of the same scene shot from different positions or angles. Including but not limited to front perspective (front perspective), rear perspective (back perspective), side perspective (left and right perspective), top perspective (high angle perspective), upward perspective (low angle perspective), and other possible special shooting angles to ensure that different parts of the scene are fully covered. The target video refers to the original video data shot by these different cameras. The present disclosure does not limit the video format of the target video.

[0054] In some embodiments, a multi-frame image sequence corresponding to each target video and a timestamp of each frame of the image are obtained, including: parsing the target video frame by frame through video decoding technology (such as a high-efficiency video coding and decoding algorithm), converting it into an image sequence, and extracting the appearance time of each frame of the image in the video, and marking it in the form of a timestamp. Frame extraction is usually based on a fixed frame rate (such as 30 frames per second) to ensure that the image sequence can fully reflect the dynamic changes of the video content. The timestamp of each frame of the image is the specific appearance time of the frame in the video, usually in seconds or milliseconds, to ensure the timing synchronization and consistency of subsequent processing (such as subtitles, music, etc.).

[0055] In some embodiments, generating a multi-frame panoramic image includes: aligning and synthesizing a multi-frame image sequence from different cameras frame by frame using an image stitching algorithm to ensure that each frame of the image is consistent in time. First, a feature matching technique (such as Scale-Invariant Feature Transform (SIFT) or feature point extraction and feature point description (Oriented FAST and Rotated BRIEF, ORB)) is used to identify common feature points between adjacent images at different perspectives, and then a random sampling (RANSAC) algorithm is used to remove mismatched points, and the perspective transformation matrix between perspectives is further calculated. Next, based on these transformation matrices, a stitching algorithm (such as finding the best stitching seam (Seam Finding) or multi-band fusion (Multi-Band)) is used to stitch the images to form a seamless panoramic image. Each frame of the image retains its timestamp when it is synthesized to ensure that the image can maintain temporal consistency with the original video content during the panoramic stitching process, thereby generating a multi-frame panoramic image covering the complete perspective.

[0056] In some embodiments, identifying and extracting the composition elements and overall description of each panoramic image frame includes: analyzing the panoramic image frame by frame through a deep learning model (such as a convolutional neural network (CNN) or a deep neural network (Transformer) model based on a self-attention mechanism). First, a pre-trained image understanding model (such as a visual-language pre-training model (Bootstrapping Language-Image Pre-training, BLIP2) or a multimodal learning model (Contrastive Language-Image Pre-training, CLIP)) is used to extract features of the visual content of the image to identify key composition elements in the image, such as people, buildings, natural landscapes, etc. Then, based on these features, a natural language generation model is used to semantically encode the identified composition elements to generate suitable keywords and short descriptions. These descriptions not only cover the objects and scenes in the image, but also extract the main theme or emotional tone of the image according to the overall picture layout, and generate an overall description in combination with contextual information, so as to fully express the composition elements and overall description of each panoramic image frame.

[0057] In some embodiments, light music and subtitle streams matching each frame of the panoramic image are generated, including: extracting keywords that match the emotion and theme of the picture by semantically analyzing the composition elements and overall description of the panoramic image. Based on these keywords, an audio generation model (such as a diffusion model) is used to generate light music that matches the content of the picture. The generation of light music takes into account factors such as the rhythm of the image, the change in color brightness, and the overall atmosphere, so that the emotional expression of the music is consistent with the video content. For example, recognizing the natural landscape in the image will generate soft and soothing background music, while busy city scenes will generate music with a strong sense of rhythm. At the same time, the generation of the subtitle stream rewrites and beautifies the composition elements and the overall description of the panoramic image through the semantic understanding of the panoramic image, combined with the pre-trained language model. The model generates suitable keywords by analyzing the composition elements of each frame of the panoramic image, and then generates subtitles based on the overall description. The subtitle stream not only describes the objects and scenes in the picture, but also interprets the emotional tone of the image to ensure that the subtitles are consistent with the panoramic video content and are natural and smooth.

[0058] For example, suppose there is a set of on-board cameras installed on the front, back, left, right and roof of the vehicle to capture the environment around the vehicle and generate multiple target videos covering different perspectives of the same scene. Through H.264 decoding technology, these videos are parsed, each frame image and its timestamp are extracted, and the video frames of each perspective are converted into image sequences. SIFT feature matching technology is used, combined with the RANSAC algorithm to eliminate errors, and multiple frames of panoramic images from these different perspectives are generated. Then, based on a deep learning model (such as BLIP2), the key composition elements in the panoramic image, such as visual elements such as pedestrians, vehicles, and buildings in the image, are analyzed, and corresponding keywords and overall descriptions are generated. Through the audio generation model, the system generates light music that matches the emotion of the picture, such as generating light and soft background music in a tree-lined street scene, and automatically generates a subtitle stream based on the image content to describe the street scene and surrounding activities. Finally, all the generated elements are combined, and multiple frames of panoramic images, music and subtitles are synchronized by timestamp to generate a complete on-board panoramic video log.

[0059] The technical solution of the disclosed embodiment can generate high-quality panoramic videos covering the entire scene through panoramic stitching and image semantic analysis of multi-view videos, thereby improving the integrity of video information. The use of automatically generated light music and subtitle streams significantly improves the efficiency and synchronization of Vlog production and reduces the workload of manual editing. In addition, through the analysis of composition elements and overall description, it ensures that the generated content is highly consistent with the emotional tone of the picture, thereby enhancing the quality of the panoramic video.

[0060] In some embodiments, obtaining multiple target videos captured by different cameras further includes:

[0061] Obtain original videos shot by different cameras at multiple viewing angles;

[0062] A plurality of target videos are extracted from original videos shot by different cameras based on video selection parameters.

[0063] In the disclosed embodiment, the video selection parameters include time period, road section, video format, video quality and other parameters.

[0064] In the disclosed embodiment, multiple cameras shoot the same object or scene from different angles. These cameras are located at different positions of the vehicle to capture various aspects of the object or scene.

[0065] In some embodiments, by controlling multiple camera devices of different brands, models and specifications, these cameras may have different shooting parameters such as focal length, resolution, frame rate, etc., and are installed in different physical locations to ensure coverage of multiple perspectives of the same scene. Through the relevant software system or hardware interface, the video data from each camera is imported into the central processing system in real time or in batches to ensure adaptation to different processing requirements. The original video recorded by each camera not only needs to cover different parts of the scene, but also needs to ensure time synchronization. Through the synchronization control system or timestamp technology, the time consistency of multi-view videos is achieved so that in the subsequent panoramic stitching process, the video frames of each view can be accurately aligned.

[0066] For example, suppose a vehicle is equipped with multiple cameras of different brands and models, located in the front, back, left, right and top of the vehicle. Each camera has a different resolution and frame rate to ensure full coverage of the vehicle's surroundings. These cameras are started at the same time to capture video of the vehicle in motion. The system imports the video data captured by each camera into the central processing system in real time through control signals. To ensure the consistency of video timing, each camera video is timestamped to facilitate the precise alignment of subsequent video frames. Based on the preset selection parameters, the system automatically extracts key target video clips from these original videos, such as video clips of a certain time period or a certain road section, to generate panoramic videos covering all perspectives, ultimately achieving seamless splicing and multi-perspective display of videos.

[0067] In this way, by controlling multiple cameras, full coverage of the same scene is achieved, ensuring the integrity and multi-view of video data. At the same time, the method of extracting target videos based on video selection parameters optimizes the accuracy and efficiency of video processing and avoids unnecessary redundant data. Time synchronization and automatic alignment technology ensure seamless splicing of multi-view videos, significantly improving the quality and coherence of panoramic video generation.

[0068] In some embodiments, based on the multi-frame image sequences corresponding to the multiple target videos and the timestamp of each frame image, a multi-frame panoramic image is generated, including:

[0069] Determine the perspective transformation matrix between multiple frame images at each timestamp;

[0070] Determine the optimal seam position between multiple frame images at each timestamp;

[0071] Based on the perspective transformation matrix and the optimal seam position of the multiple frame images at each time stamp, a panoramic image at each time stamp is generated.

[0072] In the disclosed embodiment, the perspective transformation matrix refers to the geometric relationship between images taken by different cameras calculated by a feature point matching algorithm (such as SIFT), which is used to align images from different perspectives to the same plane.

[0073] In the disclosed embodiment, the seam position refers to the fusion boundary between images determined during the stitching process, which is usually calculated through an algorithm to make the stitched image look natural and avoid obvious stitching marks.

[0074] In some embodiments, determining the perspective transformation matrix between multiple frames of images at each timestamp includes: first, SIFT feature matching is used to extract local feature points from each image. These feature points are scale invariant and rotation invariant, and can remain consistent under different viewing angles, scaling or rotation conditions. The SIFT algorithm detects extreme points (such as corners or edges) in the image and generates a descriptor for each feature point to describe its local pixel gradient distribution. In multi-view images, SIFT feature matching identifies common feature points of different viewing angle images in the same scene by comparing these descriptors. Then, the RANSAC algorithm is used to eliminate erroneous feature matching points. In this step, RANSAC estimates the geometric transformation model between images by randomly selecting a set of initial matching points, and then verifies whether other matching points conform to the model. Through repeated iterations, RANSAC will eliminate those matching points that do not conform to the model (wrong matching), and finally retain a set of reliable feature matching point pairs. Finally, based on the retained SIFT feature matching point pairs, the direct linear transformation (DLT) algorithm is applied to calculate the perspective transformation matrix between the viewing angles. The DLT algorithm solves the geometric transformation problem between multi-view images through a set of linear equations and calculates the transformation matrix used to describe the perspective relationship between image planes. This matrix is ​​used to map images from different perspectives onto a common plane, thereby achieving image alignment. DLT establishes equations by matching point pairs and solves the homography matrix between images, so that images from different perspectives can be seamlessly connected during panorama stitching.

[0075] In some embodiments, determining the best seam position between multiple frames of images at each timestamp includes: first, by calculating the overlapping area between images of different perspectives, using image features such as gradient analysis to identify areas where the image content changes less. This is because stitching in areas with less changes can avoid obvious color, brightness or structural differences and reduce stitching traces. Next, the Seam Finding algorithm is used to determine the specific seam position. The algorithm analyzes the pixel gradient and color changes in the overlapping area to find the seam path that can minimize the image difference. For example, uniform areas in the image (such as the sky or background) are often the best seam positions because their content consistency is high and it is difficult to detect stitching traces. In addition, the image at the seam position is further optimized in combination with a multi-band fusion algorithm. The algorithm decomposes the image into different frequency bands and processes low-frequency information (such as illumination or color) and high-frequency information (such as details, texture) separately. By performing smooth fusion in the low-frequency area and retaining image details in the high-frequency area, Multi-Band Blending can effectively eliminate the transition discontinuity of the splicing edge, making the image fusion more natural.

[0076] In some embodiments, generating a panoramic image at each timestamp includes: first, mapping multiple frames of images under different camera perspectives to the same plane through the calculated perspective transformation matrix. This step ensures that the images of each perspective can be accurately aligned in terms of geometric structure. The perspective transformation matrix rotates, scales, and translates each frame of the image so that the images can be seamlessly connected in the same panoramic space. Then, using the previously determined optimal seam position, the images are fused and spliced ​​in the overlapping area. The optimal seam position ensures a seamless transition of the image in the overlapping area and reduces the stitching traces. During the stitching process, the algorithm selects those areas with consistent content and minimal differences as seams to avoid obvious visual fragmentation at the edge of the image or in the complex structure area. Finally, the Multi-Band Blending algorithm is applied to further process the seam area. The algorithm decomposes the image into multiple frequency bands, and by fusing the images of different frequency bands separately, it ensures that the panoramic image can achieve natural visual effects in both low-frequency areas (such as color and lighting transitions) and high-frequency areas (such as edges and details). Multi-band fusion not only smoothes the color changes in the seam area, but also preserves the details of the image, ensuring that the stitched image has high-quality visual continuity.

[0077] For example, suppose that multiple cameras are installed on the front, back, left and right sides of a car to capture the environment around the car. The SIFT algorithm is used to extract feature points in the same scene captured by different cameras, and the RANSAC algorithm is used to remove incorrect matching points. Finally, the DLT algorithm is used to calculate the perspective transformation matrix of each perspective image. Then, the Seam Finding algorithm is used to select the seam position with less change in the overlapping area of ​​the image, and the Multi-Band Blending algorithm is used to perform image fusion in the overlapping area. The panoramic image generated in this way is visually seamless and natural, and maintains the consistency of details and lighting.

[0078] In this way, this embodiment can effectively align and seamlessly stitch multi-view images, reducing visual fragmentation or ghosting caused by stitching. Through multi-band fusion technology, the visual effect of the image seam area is further optimized, ensuring the naturalness of high-frequency details and low-frequency transitions, and improving the quality and coherence of the panoramic image.

[0079] In some embodiments, determining the perspective transformation matrix of multiple frames of images at each timestamp further includes:

[0080] Perform feature matching on multiple frames of images from different perspectives to find corresponding feature matching point pairs;

[0081] Based on the feature matching point pairs at each timestamp, the perspective transformation matrix between each view image is calculated.

[0082] In some embodiments, feature matching is performed on multiple frames of images from different perspectives to find corresponding feature matching point pairs, including: first, extracting local feature points in each frame of the image by the SIFT algorithm. The SIFT algorithm detects key points (such as corners and edges) in the image and generates a 128-dimensional feature vector for each feature point to describe its local gradient information. These feature points are scale-invariant and rotation-invariant, and can be consistent under different perspectives, scaling or rotation conditions. Then, these feature vectors are compared using Euclidean distance or other similarity measurement methods to find similar feature points in images from different perspectives. In order to improve the accuracy of matching, the nearest neighbor ratio test (Ratio Test) is used to filter out low-quality matching points, that is, only those points with large differences between the best match and the suboptimal match are retained to reduce the occurrence of mismatching. Then, the erroneous matching points are further eliminated by the RANSAC algorithm. RANSAC randomly samples a set of matching points, fits a geometric model (such as a homography matrix), and verifies whether other matching points conform to the model. Through multiple iterations, RANSAC will eliminate erroneous matching points that do not conform to the model, and finally retain a set of reliable feature matching point pairs.

[0083] In some embodiments, based on the feature matching point pairs at each timestamp, the perspective transformation matrix between each perspective image is calculated, including: first, using the retained feature matching point pairs to construct the geometric relationship between the two images. The matching point pairs extracted by SIFT contain the spatial correspondence between the images at different perspectives. Then, the DLT algorithm is used to calculate the perspective transformation matrix. The DLT algorithm solves the homography matrix between the images through a set of linear equations, which is used to map the image at one perspective to the image plane at another perspective. DLT establishes at least four equations through matching point pairs to describe the mapping relationship between the two image planes, and this process includes rotation, scaling, translation and projection transformation. The algorithm first normalizes the matching points to reduce the numerical instability caused by scale and position differences, and then solves the homography matrix through a set of linear equations. The perspective transformation matrix describes the geometric change relationship between the two perspectives, so that the images can be accurately aligned. Finally, based on the matching points of different perspectives at each timestamp, the DLT algorithm calculates the corresponding perspective transformation matrix to ensure that the image can be correctly transformed in space during panoramic stitching, and achieve seamless alignment and fusion between perspectives.

[0084] For example, suppose there are two cameras installed at the front and rear of a vehicle, each capturing a sequence of images of the vehicle's surroundings. The SIFT algorithm is used to extract feature points in each frame, and key points in the same scene captured by the front and rear cameras, such as road signs and vehicle outlines, are detected. Next, the RANSAC algorithm is used to remove incorrect matching points, ensuring that the retained feature matching points accurately reflect the geometric relationship between the two perspectives. Based on these matching points, the DLT algorithm is used to calculate the perspective transformation matrix of the front and rear cameras, and the images of both are projected onto the same plane to ensure that the images are seamlessly aligned during panoramic stitching.

[0085] In this way, through precise feature matching and geometric transformation, efficient alignment of images from different perspectives is achieved, reducing errors in image stitching. The combination of RANSAC and DLT algorithms ensures the effective elimination of mismatched points and correct perspective transformation, making the final generated panoramic image coherent and natural, enhancing the quality of the visual experience.

[0086] In some embodiments, performing feature matching on multiple frames of images from different viewing angles to find corresponding feature matching point pairs further includes:

[0087] Obtain feature points in multiple frames of images at each timestamp, and calculate feature descriptors of the feature points in the multiple frames of images;

[0088] Use feature descriptors to perform preliminary matching to obtain preliminary feature matching point pairs;

[0089] Eliminate the wrong matching point pairs from the preliminary feature matching point pairs to obtain the correct feature matching point pairs.

[0090] In some implementations, the specific process of obtaining feature points in multiple frames of images at each timestamp and calculating feature descriptors of feature points in multiple frames of images includes: first, detecting key feature points in the image through the SIFT algorithm. The SIFT algorithm traverses different scales and positions in the image to find significant image areas, such as corners, edges, etc., which are usually stable under different viewing angles, lighting, and scaling. Each frame of the image will generate multiple feature points in this process, which are markers for areas in the image with strong local changes and rich information. Next, for each detected feature point, SIFT generates a feature descriptor based on the pixel gradient information of its neighboring areas. Specifically, the SIFT algorithm calculates the direction and magnitude of the pixel gradient in the area around the feature point, and organizes this information into a 128-dimensional vector to describe the local structure of the feature point. This feature descriptor is scale-invariant and rotation-invariant, and can remain consistent under different viewing angles and scaling conditions.

[0091] In some embodiments, feature descriptors are used for preliminary matching to obtain preliminary feature matching point pairs, including: first, using the feature descriptors of each feature point generated by the SIFT algorithm, the feature vectors of the feature points in different images are compared. The Euclidean distance is usually used to measure the similarity between two feature descriptors, and the smaller the distance, the more similar the feature points are. In order to improve the accuracy of the matching, for each feature point, the two closest feature points in the other image are found, which are called the nearest neighbor and the second nearest neighbor, respectively. Then, the nearest neighbor ratio test is used to filter out unreliable matching points. The specific method is to compare the distance of the nearest neighbor with the distance of the second nearest neighbor, and only when the ratio of these two distances is less than a certain threshold (usually 0.75), the matching point is retained. In this way, those mismatched points can be effectively eliminated, the occurrence of mismatching can be reduced, and the retained feature matching point pairs can be ensured to have higher reliability.

[0092] In some embodiments, erroneous matching point pairs are eliminated from preliminary feature matching point pairs to obtain correct feature matching point pairs, including: first, using the RANSAC algorithm to eliminate erroneous matching points that do not conform to the geometric model. The working principle of RANSAC is to randomly select a group of feature matching points from the preliminary matching point pairs, and use them to fit a geometric model, such as a homography matrix (Homography Matrix) or a fundamental matrix (Fundamental Matrix), which can describe the geometric relationship between the two images. Then, the RANSAC algorithm verifies other matching points to determine whether these points conform to the fitted geometric model. Points that conform to the model are regarded as "inner points", that is, correct feature matching point pairs; points that do not conform to the model are regarded as "outer points", that is, erroneous matching points. RANSAC continuously selects new initial matching points for model fitting through multiple iterations, and finally retains the models with the most inliers.

[0093] For example, suppose two cameras capture a scene of a city square from different angles. First, the SIFT algorithm is used to detect feature points in the image, such as the corners of buildings and the edges of street lamps, and the feature descriptors of these feature points are calculated. Then, the Euclidean distance is used to compare these feature descriptors to find similar feature points in different images. Most of the mismatched points are filtered out through the nearest neighbor ratio test. Next, the RANSAC algorithm is used to build a geometric model based on these feature point pairs, eliminate the erroneous matching points that do not conform to the perspective relationship, and finally obtain the correct feature matching point pairs.

[0094] In this way, stable and accurate feature points can be effectively found under different viewing angles, and false matches can be eliminated to ensure higher accuracy and consistency when stitching images, ultimately generating a seamless panoramic image.

[0095] In some embodiments, determining the optimal seam position between multiple frames of images at each timestamp further includes:

[0096] Construct an energy map based on multiple frames of images at each timestamp;

[0097] In the energy diagram, find the minimum energy path from one end to the other as the optimal seam location.

[0098] In the disclosed embodiments, the energy map refers to a two-dimensional matrix generated by calculating the gradient, brightness change, texture complexity and other characteristics of the pixels of the image, and the value of each pixel represents the energy or importance of the pixel in the image. Generally, the edges, areas with complex textures or parts with obvious brightness changes in the image will be assigned higher energy values, while relatively uniform or smooth areas in the image (such as the sky, walls, etc.) will be assigned lower energy values.

[0099] In some embodiments, the process of constructing an energy map based on multiple frames of images at each timestamp first requires calculating the gradient information of the image, that is, the brightness or color changes around each pixel. The gradient information can be obtained by an edge detection algorithm (such as a discrete difference (Sobel) operator or an optimized edge detection (Canny) algorithm). Pixels with large gradients are usually located at the edge of the image or in a complex texture area, representing a higher energy value, while pixels with small gradients are located in a smooth area of ​​the image, representing a lower energy value. Then, in addition to the gradient information, the energy map can also be combined with the brightness change and color consistency of the image. For example, in a complex texture area in an image, the color changes are large and the light and dark contrast is strong, and these pixels usually have higher energy values. In uniform areas of the image (such as a blue sky or a smooth wall, etc.), the color and brightness changes are small, and the corresponding energy values ​​are low. By calculating the above features for all pixels in each frame of the image, an energy map at each timestamp is constructed, and each pixel value of this map represents the importance of the position in the image. This energy map will be used to find the optimal seam position when stitching images, ensuring that the seam is located in an area with lower energy, thereby reducing stitching traces and improving the visual coherence of the panoramic image.

[0100] In some embodiments, the process of finding the minimum energy path from one end to the other end in the energy map as the optimal seam position includes: first using dynamic programming or the shortest path algorithm to calculate the minimum energy path of the image. The process starts from one end of the image (such as the top or left side), gradually traverses the energy value of each pixel, and finds a path with the minimum sum of energy values, and finally reaches the other end of the image (such as the bottom or right side). Specifically, the energy value of each pixel is compared with the energy value of its adjacent pixels, and the system selects the pixel with the smallest energy value on the path for connection, thereby forming a low-energy continuous path. This path preferentially avoids areas with higher energy in the image, such as edges, complex textures, or parts with large brightness changes, ensuring that the seam is located in a smooth area of ​​the image, such as parts with small color or brightness changes, and usually these areas have less visual conflicts. Finally, the minimum energy path found is used as the seam position for splicing two frames of images, ensuring that the splicing is more natural and smooth visually, reducing obvious seam marks, thereby improving the quality of the final generated panoramic image.

[0101] For example, suppose two cameras take pictures of a city landscape from different angles. The Sobel operator is used to calculate the gradient information in the image and generate an energy map. Complex areas such as building edges, pedestrians, and vehicles in the image are assigned higher energy values, while the sky and uniform wall areas are assigned lower energy values. Then, a dynamic programming algorithm is used to find the minimum energy path from top to bottom on the energy map, which is used as the stitching seam of the two images. Since the seam is located in a lower energy area, such as the sky or the wall, the stitched image is almost seamless visually, and there is no obvious sense of separation.

[0102] In this way, by combining the energy map and the minimum energy path, the stitching position is ensured to be located in an area with less change, reducing the stitching traces and making the panoramic image more natural and coherent. This method effectively improves the quality of stitching, especially in complex scenes, ensuring the visual consistency of the image.

[0103] In some embodiments, image understanding is performed on each panoramic image frame, and composition elements and an overall description of each panoramic image frame are output, including:

[0104] The panoramic image is input into a pre-trained vision-language pre-training model for image understanding, and the composition elements and overall description of the panoramic image output by the vision-language pre-training model are obtained.

[0105] In some embodiments, the composition elements and overall description of the panoramic image output by the visual-language pre-training model are obtained, including: first, the input panoramic image is passed through the model's visual encoder to extract the feature representation in the image. The visual encoder is usually based on the architecture of CNN or Transformer, which can capture local and global features in the image. These feature representations cover information such as objects, scenes, colors, shapes, textures, etc. in the image. Then, the language decoder in the visual-language pre-training model maps the feature representation of the image to corresponding composition elements and overall description. Composition elements refer to important visual elements in the image, such as people, buildings, trees, etc., and the language decoder generates keywords for these elements. The overall description is a semantic summary of the entire image content, combining the context, emotional tone and scene atmosphere of the image to generate a natural language description. For example, in a panoramic city street view image, the composition elements may include "vehicles", "pedestrians", "high-rise buildings", etc., and the overall description may be "busy city streets, vehicles shuttle, pedestrians walk, and high-rise buildings stand in the distance". Ultimately, the output of the visual-language pre-trained model will contain the compositional elements and overall semantic description of the image, providing a basis for subsequent caption generation, content analysis or other tasks.

[0106] For example, suppose the system processes a panoramic image of a busy city street. The image is input into a pre-trained vision-language pre-trained model. The model's visual encoder first extracts features from the image, such as vehicles, pedestrians, and tall buildings on the street. Next, the language decoder maps these visual features into compositional elements, generates keywords such as "vehicles", "pedestrians", "buildings", etc., and semantically summarizes the entire image, outputting an overall description of "busy city streets, with vehicles flowing and pedestrians passing by". These compositional elements and overall descriptions can then be used for caption generation or other tasks.

[0107] In this way, the key elements and semantic information of the image can be quickly and accurately extracted in an automated way, reducing the time of manual processing. The combination of vision and language ensures the integrity of the image description and improves the accuracy of the generated content, which is particularly suitable for scenarios such as automatic subtitle generation and content analysis.

[0108] In some embodiments, based on the composition elements and overall description of each panoramic image frame, light music matching each panoramic image frame is generated, further comprising:

[0109] The composition elements and overall description of each panoramic image are encoded by a text encoder;

[0110] The encoding result and timestamp information are input into a pre-trained audio generation model to obtain light music output by the audio generation model that matches each frame of the panoramic image.

[0111] In some embodiments, the composition elements and overall description of each panoramic image are encoded by a text encoder, including: first, the composition elements and overall description of each panoramic image are processed using a pre-trained text encoder (such as a pre-trained language model or a multi-layer bidirectional Transformer encoder (Bidirectional Encoder Representations from Transformers, BERT)). The encoder converts the key visual elements of the panoramic image (such as "vehicles", "pedestrians", "buildings", etc.) and the overall semantic description (such as "busy city streets with a constant stream of vehicles") into corresponding high-dimensional vector representations. The text encoder maps these descriptions into a semantic space by semantically parsing natural language so that they can be passed to subsequent models in a structured manner. This vectorized representation retains the semantic information of the composition elements and compresses complex language descriptions, making it easier to further process.

[0112] In some embodiments, the encoding results and timestamp information are input into a pre-trained audio generation model, including: the vectorized results of the encoded composition elements and overall description are combined with the timestamp information of each frame of the panoramic image to form a multimodal input. Then, these inputs are sent to a pre-trained audio generation model (such as a Diffusion Transformer model), and the model generates background music or sound effects that match the emotional tone of the panoramic image based on the input semantic vector and time information. During the audio generation process, the model takes into account the emotional atmosphere, context, and timestamp of the image to ensure that the generated music is synchronized with the video screen, and is consistent with the image content in terms of rhythm and emotion.

[0113] For example, suppose the system processes an image of a panoramic city street. The text encoder first semantically encodes the composition elements such as "vehicles", "pedestrians", "high-rise buildings" and the overall description of "busy city streets" to generate a high-dimensional vector representation. These encoding results are then combined with the timestamp of the image and input into the audio generation model. Based on this input information, the audio generation model generates light music that matches the emotional tone of the image, such as background music with a brisk rhythm to reflect the busy atmosphere of the city. Ultimately, the generated music is synchronized with the image in terms of emotion and time, enhancing the overall look and feel of the video.

[0114] In this way, by converting the semantic content of the image into the audio model input, we can generate highly matching background music and reduce human intervention. The generated music is consistent with the emotion of the image, making the atmosphere of the final video more coherent and enhancing the user's immersion and experience.

[0115] In some embodiments, based on the composition elements and overall description of each frame of the panoramic image, generating a subtitle stream matching each frame of the panoramic image further includes:

[0116] The composition elements and overall description of the panoramic image are input into the large language model, and the preset prompts are used to guide the large language model to generate a subtitle stream that matches each frame of the panoramic image based on the understanding of the image content.

[0117] In some embodiments, the composition elements and overall description of the panoramic image are input into the large language model, including: first, extracting the composition elements and overall description of each frame of the panoramic image (such as key elements such as "vehicles", "pedestrians", "high-rise buildings" and scene descriptions such as "busy city streets with a constant flow of vehicles"), and inputting this information into a pre-trained large language model. The model is able to understand the visual semantic information of these inputs and use them for further reasoning during the language generation process. The large language model converts the composition elements and overall descriptions into understandable semantic expressions through natural language processing technology, thereby providing a basis for the subsequent generation of subtitles.

[0118] In some embodiments, a preset prompt is used to guide the large language model to generate a subtitle stream that matches each panoramic image frame based on the understanding of the image content, including: through a designed prompt, such as "generate appropriate subtitles for this image" or "describe the core content of this scene", guiding the large language model to generate subtitles based on the composition elements and overall description of the panoramic image. The prompts help the model understand how to process these visual semantic information and convert them into natural language subtitles. Based on the input image description and prompts, the large language model generates a subtitle stream that is consistent with the emotion and content of each panoramic image frame based on the understanding of the image content. For example, for a street panoramic image, the subtitles may be "busy city streets, vehicles are constantly flowing, and pedestrians are shuttling between the roads."

[0119] For example, the system processes a panoramic image of a city square and extracts compositional elements such as "pedestrians," "fountains," and "buildings," as well as an overall description of "a sunny square with tourists strolling leisurely." This information is input into the large language model and combined with preset prompts such as "generate subtitles for this image." After the model understands these visual semantics, it generates subtitles such as "the square is sunny, tourists stroll leisurely, and water splashes in the center of the fountain." These subtitles naturally correspond to the image content and accurately describe the scene.

[0120] In this way, guided by the large language model, the system can automatically generate subtitles that highly match the image content, reducing manual intervention. The generated subtitles can not only be synchronized with the image, but also accurately reflect the emotions of the scene, giving the audience a stronger sense of immersion.

[0121] In some embodiments, based on each panoramic image frame and the light music and subtitle stream corresponding to each panoramic image frame, combined with the timestamp corresponding to each panoramic image frame, a panoramic video is generated, including:

[0122] Sort each frame of the panoramic image and its corresponding light music and subtitle stream according to their respective timestamps;

[0123] Perform video encoding processing on each frame of the panoramic image;

[0124] Perform audio processing on the light music corresponding to each frame of the panoramic image;

[0125] Embed the subtitle stream corresponding to each frame of the panoramic image into the video stream;

[0126] The encoded video stream, the processed audio stream, and the embedded subtitle stream are combined into a complete panoramic video.

[0127] In some embodiments, based on each panoramic image frame and the light music and subtitle stream corresponding to each panoramic image frame, combined with the timestamp corresponding to each panoramic image frame, a panoramic video is generated, including: first, each panoramic image frame and the light music and subtitle stream corresponding to each panoramic image frame are sorted according to their respective timestamps. This step ensures synchronization between images, audio and subtitles, and ensures that they can be seamlessly combined in time order.

[0128] In some embodiments, each panoramic image frame and its corresponding light music and subtitle stream are sorted according to their respective timestamps, including: according to the timestamp information of each frame of the image, the system arranges each frame of the image and its associated light music and subtitle stream in chronological order to ensure that the generated video content maintains a consistent timing. This step can ensure that the multimodal data remains synchronized during the synthesis process.

[0129] In some embodiments, each panoramic image frame is subjected to video encoding, including: the system performs video encoding on each panoramic image frame, usually using a video encoding standard such as H.264 or H.265, to compress the image sequence into a video stream. This step can not only reduce storage space, but also ensure efficient transmission and decoding.

[0130] In some embodiments, audio processing is performed on the light music corresponding to each frame of the panoramic image, including: audio processing is performed on the light music corresponding to each frame of the panoramic image, which may involve encoding of the audio format and optimization of the audio signal to ensure the audio quality and timing consistency with the video stream.

[0131] In some implementations, audio processing is performed on the light music corresponding to each panoramic image frame, including: obtaining a complete subtitle stream generated for all panoramic image frames; generating complete light music based on the complete subtitle stream, and embedding the complete light music into the video stream.

[0132] In some embodiments, embedding the subtitle stream corresponding to each panoramic image frame into the video stream includes: embedding the subtitle stream into the generated video stream, using a common subtitle embedding standard. This process ensures that the subtitles are synchronized with the image and audio content and can be displayed correctly when the video is played.

[0133] In some embodiments, when the subtitle stream corresponding to each frame of the panoramic image is embedded into the video stream, a character string comparison is performed on the subtitle stream corresponding to each frame of the panoramic image. If the number of overlapping characters between the subtitles of the Nth frame of the panoramic image and the subtitles of the N-1th frame of the panoramic image is greater than a preset threshold such as 80%, the subtitle stream of the Nth frame of the panoramic image shall be based on the subtitle stream of the N-1th frame of the panoramic image, that is, the subtitle stream of the N-1th frame of the panoramic image shall be mapped to the Nth frame of the panoramic image.

[0134] In some embodiments, the encoded video stream, the processed audio stream, and the embedded subtitle stream are synthesized into a complete panoramic video, including: the system synthesizes the encoded video stream, the processed audio stream, and the embedded subtitle stream. The synthesized panoramic video contains complete image, audio, and subtitle information, ensuring a seamless combination of visual, auditory, and subtitle experiences, and providing high-quality multimodal video content.

[0135] For example, consider processing a panoramic video of a city trip. First, each panoramic image frame and the corresponding light music and subtitle streams are sorted according to timestamps to ensure that the image, audio, and subtitles are played synchronously. Then, the system encodes each panoramic image frame, processes the light music in a preset format, and embeds the subtitle stream into the video stream. Finally, after the synthesis of the video, audio, and subtitle streams, a complete panoramic video is generated, showing the full view of the city streets, with background music that matches the atmosphere of the scene and the subtitle stream naturally embedded.

[0136] In this way, the precise synchronization of images, audio and subtitles is ensured, and high-quality panoramic videos are generated through the seamless combination of multimodal data. The entire process is automated, which improves the efficiency of video generation, and the final multimodal experience enhances the audience's immersion and viewing effect.

[0137] In some embodiments, the video generation method may further include:

[0138] Detecting the panoramic video according to multiple detection items;

[0139] When the detection result is that the panoramic video does not meet the expected detection effect, the panoramic video is repaired or adjusted.

[0140] In some embodiments, the panoramic video is tested according to a plurality of test items, including: the system comprehensively analyzes the generated panoramic video based on a series of preset test items. The test items may include image clarity, synchronization of audio and video, correct display position of subtitles, image stitching effect, and overall smoothness of the video. Through automated tools or models, the system evaluates these indicators of the video to ensure that the panoramic video meets the expected quality standards.

[0141] In some embodiments, when the detection result shows that the panoramic video does not meet the expected detection effect, the panoramic video is repaired or adjusted, including: if the detection result shows that the panoramic video does not meet the expected effect in certain indicators, the system will automatically or through user prompts to repair or adjust. For example, if the image quality is found to be blurry, the system may readjust the encoding parameters; if the audio and video are not synchronized, the system will realign them based on the timestamp; if the subtitle position is wrong, the system will correct the display position of the subtitle. After repair or adjustment, the video will be regenerated to meet the expected effect.

[0142] For example, a panoramic video of a sunset at the seaside was generated. The detection system analyzed the video and found that the audio and picture of the video were out of sync, and the subtitle position deviated from the predetermined display area. Based on the detection results, the system automatically adjusted the synchronization of audio and video, realigned the timestamps, and corrected the position of the subtitles. After the repair was completed, the video was regenerated to ensure smooth picture, synchronized audio, and accurate subtitle display.

[0143] In this way, the automatic detection and repair steps ensure that the generated video meets the expected effect. The automatic adjustment of audio, video and subtitles improves the efficiency and quality of video generation, reduces the time of manual correction, and significantly improves the user's viewing experience.

[0144] Figure 2 A schematic diagram of the process of panoramic image synthesis is shown, as shown in Figure 2 As shown, the process includes:

[0145] S201: Perform SIFT feature matching on the input multi-view image;

[0146] S202: using the RANSAC algorithm to eliminate erroneous feature matching and obtain a set of SIFT feature matching point pairs;

[0147] S203: using the DLT algorithm to calculate the perspective transformation matrix between the images of each viewing angle;

[0148] S204: combining the multi-view images and calculating the best seam position between the images of each view through the Seam Finding algorithm;

[0149] S205: Perform image fusion according to the perspective transformation matrix and the optimal seam position to obtain a panoramic image.

[0150] Figure 3 The overall process diagram of panoramic Vlog generation is shown in FIG. Figure 3 As shown, the process includes:

[0151] S301: The user selects the video and range shot by the multi-view camera for generation, and outputs multi-view frame-by-frame images and corresponding timestamps through video information analysis;

[0152] S302: Perform panoramic image synthesis on the multi-view images frame by frame;

[0153] S303: Use the BLIP2 model to perform multimodal image understanding on the panoramic image, and output the composition elements and overall description of the panoramic image;

[0154] S304: Encode the composition elements and the overall description through a text encoder, and use the timestamp information of the input video as a generation condition to drive the audio model to generate light music;

[0155] S305: inputting the composition elements and the overall description into the large language model, so that the large language model rewrites and beautifies the text based on the overall description of the picture according to the main element keywords of the picture, and outputs a subtitle stream;

[0156] S306: Integrate the generated light music, panoramic image, and subtitle stream to generate a panoramic Vlog.

[0157] In this way, based on panoramic video generation technology, combined with image understanding, text rewriting and keywords, as well as text description-driven audio generation technology, the intelligent generation of in-vehicle multimodal panoramic Vlog is realized, enriching the integrity of Vlog information and improving production efficiency.

[0158] The present disclosure provides a video generation device, such as Figure 4 As shown, the video generation device may include: a video acquisition module 401, which is used to acquire multiple target videos shot by different cameras, and the multiple target videos cover different perspectives of the same scene; an image parsing module 402, which is used to parse each target video to obtain a multi-frame image sequence corresponding to each target video and a timestamp of each frame; a panoramic generation module 403, which is used to generate a multi-frame panoramic image based on the multi-frame image sequence corresponding to the multiple target videos and the timestamp of each frame; an image analysis module 404, which is used to perform image understanding analysis on each frame of the panoramic image, identify and extract the composition elements and overall description of each frame of the panoramic image; a music subtitle module 405, which is used to generate light music and subtitle streams matching each frame of the panoramic image based on the composition elements and overall description of each frame of the panoramic image; a video synthesis module 406, which is used to generate a panoramic video based on each frame of the panoramic image and its corresponding light music and subtitle stream, combined with the timestamps of the multiple frames of the panoramic images.

[0159] In some embodiments, the video acquisition module 401 includes:

[0160] An acquisition submodule, used to acquire original videos shot by the different cameras at multiple viewing angles;

[0161] The extraction submodule is used to extract the multiple target videos from the original videos shot by the different cameras based on the video selection parameters.

[0162] In some embodiments, the panorama generation module 403 includes:

[0163] A first determination submodule is used to determine the perspective transformation matrix between multiple frame images at each time stamp;

[0164] The second determination submodule is used to determine the optimal seam position between multiple frame images at each time stamp;

[0165] The generating submodule is used to generate a panoramic image at each time stamp based on the perspective transformation matrix and the optimal seam position of the multiple frame images at each time stamp.

[0166] In some embodiments, the first determining submodule is used to:

[0167] Perform feature matching on multiple frames of images from different perspectives to find corresponding feature matching point pairs;

[0168] Based on the feature matching point pairs at each timestamp, the perspective transformation matrix between each view image is calculated.

[0169] In some embodiments, the first determining submodule is used to:

[0170] Acquire feature points in multiple image frames at each timestamp, and calculate feature descriptors of the feature points in the multiple image frames;

[0171] Use feature descriptors to perform preliminary matching to obtain preliminary feature matching point pairs;

[0172] Eliminate erroneous matching point pairs from the preliminary feature matching point pairs to obtain correct feature matching point pairs.

[0173] In some embodiments, the second determining submodule is used to:

[0174] Construct an energy map based on multiple frames of images at each timestamp;

[0175] The minimum energy path from one end to the other end is found in the energy diagram as the optimal seam position.

[0176] In some embodiments, the image analysis module 404 is used to:

[0177] The panoramic image is input into a pre-trained vision-language pre-training model for image understanding, and the composition elements and overall description of the panoramic image output by the vision-language pre-training model are obtained.

[0178] In some embodiments, the music subtitle module 405 includes:

[0179] The light music generation submodule is used to encode the composition elements and overall description of each frame of the panoramic image through a text encoder; input the encoding result and timestamp information into a pre-trained audio generation model to obtain light music output by the audio generation model that matches each frame of the panoramic image.

[0180] In some embodiments, the music subtitle module 405 includes:

[0181] The subtitle stream generation submodule is used to input the composition elements and overall description of the panoramic image into the large language model, and guide the large language model through preset prompts to generate a subtitle stream matching each frame of the panoramic image based on understanding the image content.

[0182] In some embodiments, the video synthesis module 406 includes:

[0183] A sorting submodule is used to sort each frame of the panoramic image and its corresponding light music and subtitle stream according to their respective timestamps;

[0184] A video encoding submodule, used for performing video encoding processing on each frame of the panoramic image;

[0185] An audio processing submodule, used for performing audio processing on the light music corresponding to each frame of the panoramic image;

[0186] A subtitle stream embedding submodule is used to embed the subtitle stream corresponding to each frame of the panoramic image into the video stream;

[0187] The synthesis submodule is used to synthesize the encoded video stream, the processed audio stream and the embedded subtitle stream into a complete panoramic video.

[0188] In some embodiments, the apparatus further comprises:

[0189] A detection module, used to detect the panoramic video according to multiple detection items;

[0190] The optimization module is used to repair or adjust the panoramic video when the detection result shows that the panoramic video does not meet the expected detection effect.

[0191] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0192] The video generation device of the disclosed embodiment can automatically process multi-view videos, generate complete and high-quality panoramic videos, and improve the production efficiency of panoramic videos.

[0193] The present disclosure provides a scene diagram of a video generation method, such as Figure 5 shown.

[0194] As mentioned above, the video generation method provided by the embodiment of the present disclosure is applied to electronic devices. Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices.

[0195] Specifically, the electronic device can perform the following operations:

[0196] Acquire multiple target videos captured by different cameras, where the multiple target videos cover different perspectives of the same scene;

[0197] Parse each target video to obtain a multi-frame image sequence corresponding to each target video and a timestamp of each frame image;

[0198] Generate a multi-frame panoramic image based on a multi-frame image sequence corresponding to a plurality of target videos and a timestamp of each frame image;

[0199] Perform image understanding analysis on each panoramic image frame to identify and extract the composition elements and overall description of each panoramic image frame;

[0200] Based on the composition elements and overall description of each panoramic image frame, light music and subtitle streams matching each panoramic image frame are generated;

[0201] Based on each frame of panoramic image and its corresponding light music and subtitle stream, combined with the timestamps of multiple frames of panoramic images, a panoramic video is generated.

[0202] It should be understood that Figure 5 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 5 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0203] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0204] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0205] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0206] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0207] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0208] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a video generation method. For example, in some embodiments, the video generation method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the video generation method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the video generation method in any other appropriate manner (eg, by means of firmware).

[0209] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0210] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0211] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0212] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0213] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0214] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0215] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0216] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video generation method, comprising: Acquire multiple target videos captured by different cameras, where the multiple target videos cover different perspectives of the same scene; Parse each target video to obtain a multi-frame image sequence corresponding to each target video and a timestamp of each frame image; Generate a multi-frame panoramic image based on the multi-frame image sequences corresponding to the multiple target videos and the timestamp of each frame of the image; Perform image understanding analysis on each panoramic image frame to identify and extract the composition elements and overall description of each panoramic image frame; Based on the composition elements and overall description of each frame of the panoramic image, generating light music and subtitle streams matching each frame of the panoramic image; Based on each frame of panoramic image and its corresponding light music and subtitle stream, combined with the timestamps of multiple frames of panoramic images, a panoramic video is generated.

2. The method according to claim 1, wherein: The obtaining of multiple target videos shot by different cameras includes: Acquire original videos shot by the different cameras at multiple viewing angles; The multiple target videos are extracted from the original videos shot by the different cameras based on the video selection parameters.

3. The method according to claim 1, wherein: The generating of a multi-frame panoramic image based on the multi-frame image sequences respectively corresponding to the multiple target videos and the timestamp of each frame of the image comprises: Determine the perspective transformation matrix between multiple frame images at each timestamp; Determine the optimal seam position between multiple frame images at each timestamp; Based on the perspective transformation matrix and the optimal seam position of the multiple frames of images at each time stamp, a panoramic image at each time stamp is generated.

4. The method according to claim 3, wherein: Determine the perspective transformation matrix of multiple frames of images at each timestamp, including: Perform feature matching on multiple frames of images from different perspectives to find corresponding feature matching point pairs; Based on the feature matching point pairs at each timestamp, the perspective transformation matrix between each view image is calculated.

5. The method according to claim 4, wherein: The performing feature matching on multiple frames of images from different viewing angles to find corresponding feature matching point pairs includes: Acquire feature points in multiple image frames at each time stamp, and calculate feature descriptors of the feature points in the multiple image frames; Use feature descriptors to perform preliminary matching to obtain preliminary feature matching point pairs; Eliminate erroneous matching point pairs from the preliminary feature matching point pairs to obtain correct feature matching point pairs.

6. The method according to claim 3, wherein: The step of determining the optimal seam position between multiple frames of images at each time stamp comprises: Construct an energy map based on multiple frames of images at each timestamp; The minimum energy path from one end to the other end is found in the energy diagram as the optimal seam position.

7. The method according to claim 1, wherein: The image understanding of each panoramic image frame and outputting the composition elements and overall description of each panoramic image frame include: The panoramic image is input into a pre-trained vision-language pre-training model for image understanding, and the composition elements and overall description of the panoramic image output by the vision-language pre-training model are obtained.

8. The method according to claim 1, wherein: The step of generating light music matching each frame of the panoramic image based on the composition elements and the overall description of each frame of the panoramic image comprises: The composition elements and overall description of each panoramic image are encoded by a text encoder; The encoding result and the timestamp information are input into a pre-trained audio generation model to obtain light music output by the audio generation model that matches each frame of the panoramic image.

9. The method according to claim 1, wherein: The method of generating a subtitle stream matching each frame of the panoramic image based on the composition elements and the overall description of each frame of the panoramic image comprises: The composition elements and overall description of the panoramic image are input into a large language model, and the large language model is guided by preset prompts to generate a subtitle stream matching each frame of the panoramic image based on understanding the image content.

10. The method according to claim 1, wherein: The method of generating a panoramic video based on each panoramic image frame and the light music and subtitle stream corresponding to each panoramic image frame, combined with the timestamp corresponding to each panoramic image frame, includes: Sort each frame of the panoramic image and its corresponding light music and subtitle stream according to their respective timestamps; Perform video encoding processing on each frame of the panoramic image; Perform audio processing on the light music corresponding to each frame of the panoramic image; Embed the subtitle stream corresponding to each frame of the panoramic image into the video stream; The encoded video stream, the processed audio stream, and the embedded subtitle stream are combined into a complete panoramic video.

11. The method according to claim 1, further comprising: Detecting the panoramic video according to multiple detection items; When the detection result is that the panoramic video does not meet the expected detection effect, the panoramic video is repaired or adjusted.

12. A video generating device, comprising: A video acquisition module is used to acquire multiple target videos shot by different cameras, where the multiple target videos cover different viewing angles of the same scene; An image parsing module is used to parse each target video to obtain a multi-frame image sequence corresponding to each target video and a timestamp of each frame image; A panoramic generation module, used for generating a plurality of panoramic images based on the plurality of image sequences corresponding to the plurality of target videos and the timestamp of each image frame; An image analysis module is used to perform image understanding and analysis on each panoramic image frame, identify and extract the composition elements and overall description of each panoramic image frame; A music subtitle module, used for generating light music and subtitle streams matching each frame of the panoramic image based on the composition elements and overall description of each frame of the panoramic image; The video synthesis module is used to generate a panoramic video based on each frame of the panoramic image and its corresponding light music and subtitle stream, combined with the timestamps of multiple frames of the panoramic image.

13. The device according to claim 12, wherein: The video acquisition module comprises: An acquisition submodule, used to acquire original videos shot by the different cameras at multiple viewing angles; The extraction submodule is used to extract the multiple target videos from the original videos shot by the different cameras based on the video selection parameters.

14. The device according to claim 12, wherein: The panorama generation module comprises: A first determination submodule is used to determine the perspective transformation matrix between multiple frame images at each time stamp; The second determination submodule is used to determine the optimal seam position between multiple frame images at each time stamp; The generating submodule is used to generate a panoramic image at each time stamp based on the perspective transformation matrix and the optimal seam position of the multiple frame images at each time stamp.

15. The device according to claim 14, wherein: The first determining submodule is used to: Perform feature matching on multiple frames of images from different perspectives to find corresponding feature matching point pairs; Based on the feature matching point pairs at each timestamp, the perspective transformation matrix between each view image is calculated.

16. The device according to claim 15, wherein: The first determining submodule is used to: Acquire feature points in multiple image frames at each timestamp, and calculate feature descriptors of the feature points in the multiple image frames; Use feature descriptors to perform preliminary matching to obtain preliminary feature matching point pairs; Eliminate erroneous matching point pairs from the preliminary feature matching point pairs to obtain correct feature matching point pairs.

17. The device according to claim 15, wherein: The second determining submodule is used to: Construct an energy map based on multiple frames of images at each timestamp; The minimum energy path from one end to the other end is found in the energy diagram as the optimal seam position.

18. The device according to claim 12, wherein: The image analysis module is used for: The panoramic image is input into a pre-trained vision-language pre-training model for image understanding, and the composition elements and overall description of the panoramic image output by the vision-language pre-training model are obtained.

19. The device according to claim 12, wherein: The music subtitle module includes: The light music generation submodule is used to: encode the composition elements and overall description of each frame of the panoramic image through a text encoder; input the encoding result and timestamp information into a pre-trained audio generation model to obtain light music output by the audio generation model that matches each frame of the panoramic image.

20. The device according to claim 12, wherein: The music subtitle module includes: The subtitle stream generation submodule is used to input the composition elements and overall description of the panoramic image into the large language model, and guide the large language model through preset prompts to generate a subtitle stream matching each frame of the panoramic image based on understanding the image content.

21. The device according to claim 12, wherein: The video synthesis module comprises: A sorting submodule is used to sort each frame of the panoramic image and its corresponding light music and subtitle stream according to their respective timestamps; A video encoding submodule, used for performing video encoding processing on each frame of the panoramic image; An audio processing submodule, used for performing audio processing on the light music corresponding to each frame of the panoramic image; A subtitle stream embedding submodule is used to embed the subtitle stream corresponding to each frame of the panoramic image into the video stream; The synthesis submodule is used to synthesize the encoded video stream, the processed audio stream and the embedded subtitle stream into a complete panoramic video.

22. The apparatus according to claim 12, further comprising: A detection module, used to detect the panoramic video according to multiple detection items; The optimization module is used to repair or adjust the panoramic video when the detection result shows that the panoramic video does not meet the expected detection effect.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are for causing a computer to perform a method according to any one of claims 1-11.

25. A computer program product comprising a computer program stored on a storage medium, the computer program implementing the method according to any one of claims 1 to 11 when executed by a processor.

Images