Video synthesis method, device, electronic device and storage medium

Abstract

This invention provides a video synthesis method, apparatus, electronic device, and storage medium. The method includes: acquiring video segments recorded in the cloud and target object features of the video to be synthesized; extracting frames from each video segment to obtain frame-by-frame images of the corresponding video segments; performing similarity filtering on the frame-by-frame images of each video segment to obtain filtered frame-by-frame images; marking objects on each filtered frame-by-frame image and extracting object features from the marked objects on each frame-by-frame image; matching the target object features with the extracted object features from each frame-by-frame image to obtain successfully matched frame-by-frame images; and synthesizing all successfully matched frame-by-frame images to obtain the synthesized video. This invention utilizes the characteristic of uniform resolution in videos, compares pixels at the same location using image processing technology, and calculates similarity based on the comparison results, removing duplicate images, reducing computational resources, and improving the efficiency of video synthesis.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a video synthesis method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of network technology, network technology is increasingly being applied in various technical fields. Video surveillance systems have been widely used by businesses and households, playing a significant role in maintaining public order and protecting property.

[0003] In related technologies, cloud-based home surveillance enables video monitoring, allowing users to view surveillance equipment footage in real time and ensure the security of homes or offices. Cloud-based home surveillance captures video clips through cameras, extracts video frame information from these clips, and labels the extracted frames. Finally, based on the labeling results, a filtering strategy is used to generate condensed, high-quality video clips. However, since the differences in video frame information extracted from a single video may be small, extracting and labeling video frame information from all frames in a video clip would increase computing resources and reduce synthesis efficiency. Summary of the Invention

[0004] This invention provides a video synthesis method, apparatus, electronic device, and storage medium to at least solve the problem in related technologies where, due to the potentially small differences in video frame information extracted within a single video, extracting and labeling video frame information for all video frames in a video segment leads to increased computational resources and reduced synthesis efficiency. The technical solution of this invention is as follows:

[0005] According to a first aspect of the present invention, a video synthesis method is provided, comprising:

[0006] Acquire video clips recorded in the cloud, as well as the features of the target objects in the video to be synthesized;

[0007] Frames are extracted from each video segment to obtain the corresponding frame-extracted images of the video segment;

[0008] The similarity of the extracted frames from each video segment is used to obtain the filtered extracted frames.

[0009] The objects on each of the selected frame images are marked, and the object features of the marked objects on each frame image are extracted.

[0010] The target object features are matched with the object features extracted from each frame image to obtain successfully matched frame images.

[0011] All successfully matched extracted frames are combined to obtain the synthesized video.

[0012] Optionally, before synthesizing all successfully matched frame images, the method further includes:

[0013] Based on the set length of the miniature composite video, select the required number of target video segments and the required number of frame images for each selected target video segment from all successfully matched frame images;

[0014] The selected target video segments and the number of extracted frames in each target video segment are combined to obtain the synthesized video.

[0015] Optionally, the method further includes:

[0016] When compositing all the selected target video segments and the number of frame images in each target video segment, background music is added to obtain a composite video with background music.

[0017] The synthesized video with background music can be pushed to users or uploaded to the cloud.

[0018] Optionally, the step of performing similarity filtering on the frame-by-frame images of each video segment to obtain the filtered frame-by-frame images includes:

[0019] The extracted frames of each video segment are reduced in dimensionality and converted to grayscale to obtain the grayscale extracted frames.

[0020] Each position on the grayscale extracted image is converted into a pixel matrix, and the color in the pixel matrix is ​​represented by 0 and 1;

[0021] The pixel matrices of the first and second adjacent frame-picked images are XORed sequentially at the same positions to obtain the difference matrix.

[0022] The similarity between the second frame-by-frame image and the first frame-by-frame image is determined based on the number of elements with a value of 1 in the difference matrix.

[0023] If the similarity is greater than the set threshold, the second frame-extracting image is removed;

[0024] If the similarity is not greater than the set threshold, the second extracted frame image is retained, resulting in a filtered set of extracted frame images.

[0025] According to a second aspect of the present invention, a video synthesis method is provided, comprising:

[0026] Obtain video clips recorded in the cloud;

[0027] Frames are extracted from each video segment to obtain the corresponding frame-extracted images of the video segment;

[0028] The similarity of the extracted frames from each video segment is used to obtain the filtered extracted frames.

[0029] The objects in each of the filtered frame images are labeled to obtain the object labels for each frame image;

[0030] Obtain the object characteristics of the region corresponding to each object label;

[0031] The object features are matched with the object features in the object database, and the frame images of object features that do not match are marked.

[0032] The extracted frames of the marked but unmatched object features are synthesized to obtain the synthesized video.

[0033] Optionally, the method further includes:

[0034] Based on the set synthetic video length, select the required number of target video segments and the required number of frame images in each selected target video segment from the frame images corresponding to the features of the unmatched objects.

[0035] The selected target video segments and the number of extracted frames in each target video segment are combined to obtain the synthesized video.

[0036] Optionally, the method further includes:

[0037] When compositing all selected target video segments and the number of frame images in each target video segment, background music is added to obtain a composite video with background music.

[0038] The synthesized video with background music can be pushed to users or uploaded to the cloud.

[0039] Optionally, the method further includes:

[0040] Pre-defined object features in the object database, including: facial features or living creature features.

[0041] Optionally, the step of performing similarity filtering on the frame-by-frame images of each video segment to obtain the filtered frame-by-frame images includes:

[0042] The extracted frames of each video segment are reduced in dimensionality and converted to grayscale to obtain the grayscale extracted frames.

[0043] Each position on the grayscale extracted image is converted into a pixel matrix, and the color in the pixel matrix is ​​represented by 0 and 1;

[0044] The pixel matrices of the first and second adjacent frame-picked images are XORed sequentially at the same positions to obtain the difference matrix.

[0045] The similarity between the second frame-by-frame image and the first frame-by-frame image is determined based on the number of 1s in the difference matrix.

[0046] If the similarity is greater than the set threshold, the second frame-extracting image is removed;

[0047] If the similarity is not greater than the set threshold, the second extracted frame image is retained, resulting in a filtered set of extracted frame images.

[0048] According to a third aspect of the present invention, a video synthesis apparatus is provided, comprising:

[0049] The acquisition module is used to acquire video clips recorded in the cloud, as well as the target object features of the video to be synthesized;

[0050] The frame extraction module is used to extract frames from each video segment to obtain the corresponding frame images of the video segment.

[0051] The filtering module is used to filter the extracted frames of each video segment based on similarity, and obtain the filtered extracted frames.

[0052] The feature extraction module is used to label the objects on each frame of the filtered images and extract the object features of the labeled objects on each frame.

[0053] The matching module is used to match the target object features with the object features extracted from each frame image to obtain successfully matched frame images.

[0054] The compositing module is used to combine all the successfully matched extracted frames to obtain the composite video.

[0055] Optionally, the device further includes:

[0056] The selection module is used to select the required number of target video segments and the required number of frame images in each selected target video segment from all the successfully matched frame images before the synthesis module synthesizes all the successfully matched target frame images, according to the set synthesis video length.

[0057] The synthesis module is further configured to synthesize all target video segments selected by the selection module and the number of frame images in each target video segment to obtain a synthesized video.

[0058] Optionally, the device further includes:

[0059] An addition module is used to add background music when the synthesis module synthesizes all selected target video segments and the number of frame images in each target video segment, so as to obtain a synthesized video with background music.

[0060] The calculation module is used to push the video with background music added by the adding module to the user or upload it to the cloud.

[0061] Optionally, the filtering module includes:

[0062] The grayscale processing module is used to reduce the dimensionality of each video segment's frame images and convert them to grayscale, resulting in grayscale frame images.

[0063] A conversion module is used to convert each position on the grayscale frame-by-frame image into a pixel matrix, wherein the color in the pixel matrix is ​​represented by 0 and 1;

[0064] The XOR module is used to perform XOR operations on the same positions of the pixel matrices of the first and second adjacent frame-picked images in sequence to obtain the difference matrix.

[0065] A similarity determination module is used to determine the similarity between the second frame-picking image and the first frame-picking image based on the number of 1s in the difference matrix.

[0066] The removal module is used to remove the second frame-picking image when the similarity determined by the similarity determination module is greater than a set threshold.

[0067] The retention module is used to retain the second frame-picking image when the similarity determined by the similarity determination module is not greater than a set threshold, thereby obtaining a set of filtered frame-picking images.

[0068] According to a fourth aspect of the present invention, a video synthesis apparatus is provided, comprising:

[0069] The acquisition module is used to acquire video clips recorded in the cloud;

[0070] The frame extraction module is used to extract frames from each video segment to obtain the corresponding frame images of the video segment.

[0071] The filtering module is used to filter the extracted frames of each video segment based on similarity, and obtain the filtered extracted frames.

[0072] The first labeling module is used to label the objects on each of the filtered frame images to obtain the object labels on each frame image.

[0073] The extraction module is used to extract the object features of the region corresponding to each object label;

[0074] The matching module is used to match the object features with object features in the object database;

[0075] The second labeling module is used to label the extracted frames of images containing unmatched object features;

[0076] The compositing module synthesizes the extracted frames of all unmatched object features to obtain the synthesized video.

[0077] Optionally, the device further includes:

[0078] The selection module is used to select the required number of target video segments and the number of frame images required for each selected target video segment from frame images that mark all object features that did not match successfully, according to the set synthetic video length.

[0079] The selected target video segments and the number of extracted frames in each target video segment are combined to obtain the synthesized video.

[0080] Optionally, the device further includes:

[0081] Add a module to add background music when compositing all selected target video segments and the number of frame images in each target video segment, so as to obtain a video with background music;

[0082] The push module is used to push the video with background music added by the adding module to the user or upload it to the cloud.

[0083] Optionally, the device further includes:

[0084] The settings module is used to pre-set object characteristics in the object database, including: facial features or living creature features.

[0085] Optionally, the filtering module includes:

[0086] The grayscale processing module is used to reduce the dimensionality of each video segment's frame images and convert them to grayscale, resulting in grayscale frame images.

[0087] A conversion module is used to convert each position on the grayscale frame-by-frame image into a pixel matrix, wherein the color in the pixel matrix is ​​represented by 0 and 1;

[0088] The XOR module is used to perform XOR operations on the same positions of the pixel matrices of the first and second adjacent frame-picked images in sequence to obtain the difference matrix.

[0089] The similarity determination module determines the similarity between the second frame-picked image and the first two frame-picked images based on the number of 1s in the difference matrix.

[0090] The removal module is used to remove the second frame-by-frame image when the similarity is greater than a set threshold.

[0091] The retention module is used to retain the second frame-picking image when the similarity determined by the similarity determination module is not greater than a set threshold, thereby obtaining a set of filtered frame-picking images.

[0092] According to a fifth aspect of the present invention, an electronic device is provided, comprising:

[0093] processor;

[0094] Memory used to store the processor's executable instructions;

[0095] The processor is configured to execute the instructions to implement the video synthesis method described above.

[0096] According to a sixth aspect of the present invention, a computer-readable storage medium is provided, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the video synthesis method as described above.

[0097] According to a seventh aspect of the present invention, a computer program product is provided, comprising a computer program or instructions that, when executed by a processor of an electronic device, implement the video synthesis method described above.

[0098] The technical solutions provided by the embodiments of the present invention bring at least the following beneficial effects:

[0099] In this embodiment of the invention, video clips recorded in the cloud and the target object features of the video to be synthesized are acquired; frames are extracted from each video clip to obtain frame-by-frame images of the corresponding video clips; similarity filtering is performed on the frame-by-frame images of each video clip to obtain filtered frame-by-frame images; objects on each filtered frame-by-frame image are marked, and object features of the marked objects on each frame-by-frame image are extracted; the target object features are matched with the extracted object features on each frame-by-frame image to obtain successfully matched frame-by-frame images; all successfully matched frame-by-frame images are synthesized to obtain the synthesized video. In other words, this embodiment of the invention utilizes the characteristic of uniform video resolution, compares pixels at the same location using image processing technology, and calculates similarity based on the comparison results to remove duplicate images, completing the filtering before matching, reducing computational resources, improving the efficiency of video synthesis, and providing a better user experience.

[0100] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0101] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present invention and, together with the description, serve to explain the principles of the invention. They do not constitute an undue limitation of the invention. To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0102] Figure 1 This is a flowchart of a video synthesis method provided in an embodiment of the present invention.

[0103] Figure 2 This is a schematic diagram of a similarity filtering method provided in an embodiment of the present invention.

[0104] Figure 3 This is a schematic diagram of selecting segments and synthesizing video according to the total duration of the synthesized video provided by this invention.

[0105] Figure 4 This is another flowchart of a video synthesis method provided in an embodiment of the present invention.

[0106] Figure 5 This is a block diagram of a video synthesis device provided in an embodiment of the present invention.

[0107] Figure 6 This is another block diagram of a video synthesis device provided in an embodiment of the present invention.

[0108] Figure 7 This is another block diagram of a video synthesis device provided in an embodiment of the present invention.

[0109] Figure 8 This is a block diagram of an electronic device provided in an embodiment of the present invention.

[0110] Figure 9 This is a block diagram of an apparatus for video synthesis provided in an embodiment of the present invention. Detailed Implementation

[0111] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0112] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0113] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement.

[0114] Technical terms:

[0115] Frame extraction: A video processing technique that involves extracting specific frames (i.e., still images) from a video sequence. Video is essentially composed of a series of consecutive still images; when these images are played back quickly, they create a dynamic effect. Frame extraction selectively extracts certain frames (still images) from this sequence.

[0116] Confidence level: A concept in statistics used to represent the degree of certainty in a statistical inference. In statistics, confidence level is typically used to describe the probability that an interval estimate (such as a confidence interval) contains a population parameter. In other words, it represents how confident we are in our parameter estimation.

[0117] Feature matching: A key technique in computer vision used to find and match similar feature points between different images. This process includes three steps: feature extraction, feature description, and feature matching.

[0118] Based on the understanding of the above technical terms, please also refer to the following embodiments.

[0119] Please see Figure 1 This is a flowchart of a video synthesis method provided in an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0120] Step 101: Obtain the video clips recorded in the cloud, as well as the target object features of the video to be synthesized.

[0121] Step 102: Extract frames from each video segment to obtain the corresponding frame-extracted image of the video segment.

[0122] Step 103: Perform similarity filtering on the extracted frames of each video segment to obtain the filtered extracted frames.

[0123] Step 104: Mark the objects on each of the filtered frame images and extract the object features of the marked objects on each frame image.

[0124] Step 105: Match the target object features with the object features extracted from each frame image to obtain successfully matched frame images.

[0125] Step 106: Combine all the successfully matched frame images to obtain the combined video.

[0126] The video synthesis method described in this invention can be applied to terminals, servers, etc., without limitation. The terminal implementation device can be an electronic device such as a smartphone, laptop, tablet, desktop computer, personal digital assistant (PDA), and wearable device. The server can be an independent server, a server cluster, or a server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, or big data and artificial intelligence platforms, etc., without limitation.

[0127] The following is combined with Figure 1The specific implementation steps of a video synthesis method provided in this embodiment of the invention will be described in detail.

[0128] In step 101, video clips recorded in the cloud and the target object features of the video to be synthesized are obtained.

[0129] In this step, various video clips can be retrieved from the cloud. These video clips can be recorded by cloud-based home security systems, or they can be recorded surveillance video clips, etc. Of course, this embodiment can also retrieve video clips from other servers, applications, etc., and this embodiment does not impose any restrictions. The content of the video clips can include data from different scenarios, user behavior data, etc.

[0130] The target object features of the video to be synthesized can be the facial features of a user, or the features of an animal, etc., and this embodiment does not impose any restrictions.

[0131] In step 102, each video segment is frame-by-frame extracted to obtain the corresponding frame-by-frame image of the video segment.

[0132] In this step, each video segment is frame-by-frame extracted, and the information of each extracted frame image and the time information of the extracted frame image in the video are recorded.

[0133] For example, if a single video is about 1 minute long, the video segment is divided into 30 frames, each lasting 2 seconds. After frame extraction, it is necessary to record the image information of each frame and the time information of each frame within the video segment. The specific process of frame extraction for the video segment is well known to those skilled in the art and will not be described in detail here.

[0134] In step 103, the similarity of the extracted frames for each video segment is filtered to obtain the filtered extracted frames.

[0135] In this step, the frame-by-frame images of each video segment are reduced in dimensionality and converted to grayscale to obtain grayscale frame-by-frame images. Each position on the grayscale frame-by-frame image is converted into a pixel matrix, where the color is represented by 0 and 1. The same positions of the converted pixel matrices of the first and second adjacent frame-by-frame images are XORed sequentially to obtain a difference matrix. The similarity between the second and first frame-by-frame images is determined based on the number of 1s in the difference matrix. If the similarity is greater than a set threshold, the second frame-by-frame image is removed; or if the similarity is not greater than the set threshold, the second frame-by-frame image is retained, resulting in a filtered set of frame-by-frame images.

[0136] Specifically, in this embodiment, to save computing resources and reduce algorithm calls, before marking objects on each frame-by-frame image, all frame-by-frame images are first scaled (e.g., to 64*64), then grayscale processed. Each pixel position in the grayscale frame-by-frame image is then mapped to each element in a two-dimensional matrix, with each element having a value of 0 or 1. By comparing each position in the two-dimensional matrix of two adjacent frame-by-frame images, such as through an XOR operation, a difference matrix between adjacent frame-by-frame images is obtained. Adjacent frame-by-frame images can be represented as a1, a2...an. The number of elements with a value of 1 in the difference matrices of a1 and a2 is counted, denoted by N. The similarity is calculated based on N, i.e., the similarity calculation formula is: N / (64*64)*100%.

[0137] If the similarity value is not greater than a set threshold (e.g., 80%, the specific threshold can be set according to implementation needs, and this embodiment does not impose any restrictions), then a1 and a2 are retained; if the similarity value is greater than the threshold, then a2 is discarded and a1 is retained; continue to compare a1 and a3. If the similarity value of a1 and a3 is greater than the set threshold, then a3 is discarded; otherwise, if the similarity value of a1 and a3 is not greater than the set threshold, then a3 is retained; continue to compare a3 and a4, and so on, until the similarity of all extracted frames has been processed and compared, and the target image set is obtained.

[0138] Specifically, such as Figure 2 As shown, Figure 2 This is a schematic diagram of a similarity filtering method provided by an embodiment of the present invention. As shown in Figure 2, this embodiment takes two frame-by-frame images of cats as an example, still denoted as a1 and a2, but in specific applications, it is not limited to this.

[0139] In this embodiment, a1 and a2 are adjacent original frame images, and these original frame images are usually color images. The similarity comparison process specifically includes:

[0140] 1) Reduce the dimensionality of the original frame-sampling images a1 and a2 respectively (e.g., to 64*64) and convert them to grayscale to obtain grayscale frame-sampling images a11 and a21.

[0141] 2) Convert each position of the grayscale frame-drawn images a11 and a21 into a pixel matrix, where the colors in the pixel matrix are represented by 0 and 1, i.e., black and white.

[0142] 3) Perform an XOR operation on the same positions of the pixel matrices of the two extracted frames a11 and a21 to obtain the difference matrix.

[0143] 4) Count the number of 1s in the difference matrix, denoted by N, and calculate the similarity based on N. The calculation formula is N / (64*64)*100%.

[0144] 5) Determine if the similarity value is greater than the set threshold. If the similarity value is greater than the set threshold, proceed to step 6); otherwise, proceed to step 7).

[0145] In this embodiment, a similarity threshold of 80% is set as an example. However, in practical applications, it is not limited to this.

[0146] 6) Discard a2 and keep a1.

[0147] In other words, frames with a similarity value greater than a set threshold are discarded directly without being called by the algorithm, thus reducing algorithm scheduling and saving computing resources.

[0148] 7) Keep a1 and a2 to obtain the set of extracted frames after similarity filtering.

[0149] This process is repeated until the similarity of all adjacent frame images has been processed, resulting in the set of frame images after similarity filtering, which is the target image set.

[0150] This embodiment uses similarity to filter out some duplicate frame-by-frame images in advance, reducing subsequent algorithm calls and improving the difference between frame-by-frame images within the same video.

[0151] In step 104, the objects on each of the filtered frame images are marked, and the object features of the marked objects on each frame image are extracted.

[0152] In this step, the detection algorithm service is invoked to label the objects (such as human faces, animals, etc.) on each frame of the filtered images, and the label corresponding to each image is obtained. Of course, the confidence information corresponding to the label can also be included. Then, the object features of the labeled objects on each frame of the image are extracted.

[0153] It should be noted that, for human faces, face detection technology can be used to mark the objects on each of the selected frame images and extract the object features of the marked objects on each frame image.

[0154] For objects such as animals, object detection algorithms can be used. This method is also applicable to objects like faces; typically, large-scale computer vision (CV) models are used to detect objects in images, such as animals and faces. The training methods for these CV models employ general training methods, including:

[0155] 1) Obtain the dataset and divide it: In this embodiment, the labeled log dataset is divided into a training set, a validation set and a test set.

[0156] The training set is used for model training, the validation set is used to adjust model parameters and select the optimal model, and the test set is used to evaluate model performance.

[0157] 2) Model training:

[0158] A large model is trained using a training set, and the parameters of the large model are adjusted using optimization algorithms (such as gradient descent) to minimize the loss function.

[0159] Gradient descent is an iterative method used to find the minimum value of a function in machine learning and optimization problems. It works by calculating the gradient of the function at the current point (i.e., the direction of change of the function at that point), and then updating the parameters in the opposite direction of the gradient (i.e., the direction in which the function decreases the fastest), thus gradually approaching the minimum value of the function.

[0160] In this embodiment, various strategies can be used during training to improve the performance of large models, such as data augmentation, early stopping, and learning rate adjustment.

[0161] 3) Model Evaluation and Optimization

[0162] 31) Performance evaluation:

[0163] This embodiment uses a test set to evaluate the trained large model, calculating metrics such as accuracy, recall, and F1 score to measure model performance. Based on the evaluation results, it analyzes the model's strengths and weaknesses.

[0164] 42) Model optimization:

[0165] To address the shortcomings of large-scale models, optimizations and improvements can be made based on the validation set, such as adjusting the model structure, increasing training data, and improving annotation strategies. Alternatively, model fusion and ensemble learning methods can be used to further enhance model performance.

[0166] It should be noted that the process of marking objects in an image and extracting the features of the marked objects is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0167] In this process, the features of the labeled objects can be extracted by using algorithms to analyze the labels and generate corresponding label information. Then, the faces in the corresponding regions of the labels are extracted by drawing frames to obtain the object features on each frame of the image.

[0168] In step 105, the target object features are matched with the object features extracted from each frame image to obtain successfully matched frame images.

[0169] In this step, the target object features (such as facial features) obtained in step 101 are matched with the object features on each frame-by-frame image extracted in this step. If a match is successful, the frame-by-frame image that has been matched is marked. This process continues until all matches are completed and all successfully matched frame-by-frame images are obtained.

[0170] It should be noted that the target object in this embodiment can be a pre-defined target object or a target object in a frame-by-frame image; this embodiment does not impose any specific restrictions.

[0171] In step 106, all successfully matched frame images are combined to obtain the combined video.

[0172] In this step, all successfully matched extracted frames can be sorted by time and then synthesized to obtain the synthesized video. The video synthesis can be performed using streaming media technology.

[0173] Furthermore, the synthesized video can be pushed to users.

[0174] In this embodiment of the invention, video clips recorded in the cloud and the target object features of the video to be synthesized are acquired; frames are extracted from each video clip to obtain frame-by-frame images of the corresponding video clips; similarity filtering is performed on the frame-by-frame images of each video clip to obtain filtered frame-by-frame images; objects on each filtered frame-by-frame image are marked, and object features of the marked objects on each frame-by-frame image are extracted; the target object features are matched with the extracted object features on each frame-by-frame image to obtain successfully matched frame-by-frame images; all successfully matched frame-by-frame images are synthesized to obtain the synthesized video. In other words, this embodiment of the invention utilizes the characteristic of uniform video resolution, compares pixels at the same location using image processing technology, and calculates similarity based on the comparison results to remove duplicate images, completing the filtering before matching, reducing computational resources, improving the efficiency of video synthesis, and providing a better user experience.

[0175] Optionally, in another embodiment, based on the above embodiment, before synthesizing all successfully matched frame-by-frame images, the method may further include: selecting the required number of target video segments and the required number of frame-by-frame images in each selected target video segment from all successfully matched frame-by-frame images according to a set miniature synthesized video length; and synthesizing all selected target video segments and the number of frame-by-frame images in each target video segment to obtain the synthesized video.

[0176] In this step, the total duration N of the miniature composite video needs to be set in advance, the video duration M and the sampling interval S are selected. Then, based on the total duration N, the total number of video segments N / 2 and the number of segments N / 2 / S required for each video segment interval are calculated. The selected video frequency bands and the number of images in each video segment are then synthesized using streaming media technology in chronological order to obtain the composite video.

[0177] like Figure 3 The figure shows a schematic diagram of selecting segments and compositing videos according to a set total duration of the synthesized video, provided by an embodiment of the present invention. As shown, it includes:

[0178] 1) Based on the total synthesis duration N, select the start and end times of the video, query all segments from the parsing results, that is, select the required number of target video segments and the required number of frame images in each selected target video segment from all successfully matched frame images;

[0179] 2) Divide all segments according to the sampling interval S, with each interval lasting M / S;

[0180] 3) Within each interval, use a random algorithm to select N / 2 / S candidate segments and the number of images in each candidate segment;

[0181] 4) Sort all candidate segments and the number of images in each candidate segment according to time sequence, and use multimedia technology to synthesize a video.

[0182] In this embodiment of the invention, a multi-interval sampling method is used to ensure the diversity of video segment sources and enhance the richness of cross-video synthesis. That is, this embodiment determines the required number of target segments based on the length of the synthesized video, and applies a candidate segmentation and further filtering strategy to the target time period to complete the final segment selection, improving the differences between segments, ensuring that the video segments come from multiple independent videos, and enriching the final synthesized video effect.

[0183] Optionally, in another embodiment, based on the above embodiment, the method may further include: adding background music when synthesizing all selected target video segments and the number of frame images in each target video segment to obtain a synthesized video with background music; and pushing the synthesized video with background music to the user.

[0184] In this embodiment of the invention, when synthesizing all the selected target segments, background music is added, and then the video can be uploaded to the cloud; or the synthesized video can be pushed to the user at regular intervals so that the user can play it online or download and share it to social media software, etc.

[0185] This invention provides a strategy for filtering repeated frame-by-frame images before video synthesis, avoiding sending all frame-by-frame images to the algorithm for detection, saving computing resources, and increasing the richness of content retained after single video parsing. During synthesis, this invention employs a segmented sampling strategy to ensure that segments originate from multiple original video segments, improving the overall content richness of the synthesized video. Furthermore, background music is added to the synthesized video to provide the best user experience.

[0186] This invention utilizes a camera to collect raw data, extracts frames, and uses a similarity filtering algorithm to reduce algorithm calls and save computing resources; in the synthesis stage, a segmented sampling mode is used to increase the richness of the video.

[0187] By employing the technical solution described in the embodiments of the present invention, duplicate images can be removed during the parsing stage, and segmented extraction can be performed during the synthesis stage. This avoids situations where the fragment source is singular or there are too many similar fragments, thereby achieving the best synthesis effect and providing the best user experience.

[0188] Please also see Figure 4 The following is another flowchart of a video synthesis method provided by the present invention, the method comprising:

[0189] Step 401: Obtain the video clip recorded in the cloud;

[0190] In this step, various video clips can be obtained from the cloud. These video clips can be video clips recorded by cloud home security, or recorded surveillance video clips, etc. Of course, in this embodiment, video clips can also be obtained from other servers, applications, etc. This embodiment does not impose any restrictions.

[0191] Step 402: Extract frames from each video segment to obtain the corresponding frame-extracted images of the video segment;

[0192] In this step, each video segment is frame-by-frame extracted, and the information of each extracted frame image and the time information of the extracted frame image in the video are recorded.

[0193] For example, if a single video is approximately 2 minutes long, the video segment is divided into 60 frames, each lasting 2 seconds. After frame extraction, it is necessary to record the image information of each extracted frame and its time information within the video segment. The specific process of extracting frames from a video segment is well-known to those skilled in the art and will not be elaborated upon here.

[0194] Step 403: Perform similarity filtering on the frame-by-frame images of each video segment to obtain the filtered frame-by-frame images;

[0195] In this step, the frame-by-frame images of each video segment are reduced in dimensionality and converted to grayscale to obtain grayscale frame-by-frame images; each position on the grayscale frame-by-frame image is converted into a pixel matrix, where the color is represented by 0 and 1; the corresponding positions of the converted pixel matrices of the first and second adjacent frame-by-frame images are XORed sequentially to obtain a difference matrix; the similarity between the second and first frame-by-frame images is determined based on the number of 1s in the difference matrix; if the similarity is greater than a set threshold, the second frame-by-frame image is removed. Alternatively, if the similarity is not greater than a set threshold, retain the second frame image and continue comparing the similarity between the first and third frame images. If the similarity between the first and third frame images is greater than the set threshold, discard the third frame image; otherwise, if the similarity between the first and third frame images is not greater than the set threshold, retain the third frame image. Continue comparing the third and fourth frame images, and so on, until the similarity of all frame images has been processed and compared, thus obtaining the target image set.

[0196] Step 404: Mark the objects on each of the filtered frame images to obtain object labels for each frame image;

[0197] In this step, the detection algorithm service is invoked to label the objects (such as human faces, animals, etc.) on each frame of the filtered images, and the label corresponding to each image is obtained. Of course, the confidence information corresponding to the label can also be included. Then, the object features of the labeled objects on each frame of the image are extracted.

[0198] It should be noted that, for human faces, face detection technology can be used to mark the objects on each of the selected frame images and extract the object features of the marked objects on each frame image.

[0199] For objects such as animals, object detection algorithms can be used. Of course, this method is also applicable to objects such as faces. That is, large computer vision (CV) models are usually used to detect objects in images, such as animals and faces. The training method for large computer vision (CV) models is a general training method, which is detailed above and will not be repeated here.

[0200] It should be noted that the process of marking objects in an image and extracting the features of the marked objects is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0201] Step 405: Obtain the object features of the region corresponding to each object label;

[0202] Extract the object features of the region corresponding to each object label to obtain the object features of the object label on each frame image.

[0203] The specific feature extraction process is already well-known to those skilled in the art, and will not be elaborated here.

[0204] Step 406: Match the object features with the object features in the object database, and mark the frame images of object features that do not match.

[0205] In this step, the extracted object features (such as facial features) are matched with the object features in the object database. If a match is unsuccessful, the frame images that did not match are marked. This process continues until all matches are completed and all unmatched frame images are obtained.

[0206] It should be noted that the object features of the object database in this embodiment are pre-set object features. For example, the facial images of family members are obtained in advance, features are extracted from the facial images to obtain the facial features of the corresponding members, and the facial features of each family member are stored in the object database.

[0207] If the video retrieved from the cloud is home surveillance video, and you want to check if a burglar has entered the house, after following the above series of steps, if the match is successful, it means that all the individuals are family members and no burglar has entered the house; if the match is unsuccessful, it means that an outsider, such as a burglar, has entered the house.

[0208] Step 407: Combine the extracted frames of the marked unmatched object features to obtain the composite video.

[0209] In this step, all unmatched extracted frames can be sorted by time and synthesized to obtain the synthesized video. The video synthesis can be performed using streaming media technology.

[0210] Furthermore, based on the above embodiments, the method may further include: pushing the synthesized video to the user.

[0211] In this embodiment of the invention, video clips recorded in the cloud are acquired; frames are extracted from each video clip to obtain frame-by-frame images of the corresponding video clip; similarity filtering is performed on the frame-by-frame images of each video clip to obtain filtered frame-by-frame images; objects on each filtered frame-by-frame image are marked to obtain object tags on each frame-by-frame image; object features corresponding to each object tag are acquired; the object features are matched with object features in the object database, and frame-by-frame images of unmatched object features are marked; the marked frame-by-frame images of unmatched object features are synthesized to obtain the synthesized video. In other words, this embodiment of the invention utilizes the characteristic of uniform video resolution, compares pixels at the same location using image processing technology, calculates similarity based on the comparison results, removes duplicate images, completes filtering before matching, reduces computational resources, improves the efficiency of video synthesis, and provides a better user experience.

[0212] Optionally, in another embodiment, based on the above embodiments, the method may further include: selecting the required number of target video segments and the required number of frame-drawing images in each selected target video segment from the frame-drawing images corresponding to the unmatched object features according to a set synthetic video length; and synthesizing all the selected target video segments and the number of frame-drawing images in each target video segment to obtain the synthesized video.

[0213] In this step, the length of the synthesized video is determined by pre-setting the total duration N of the miniature synthesized video, selecting the video duration M and the sampling interval S, and then calculating the total number of video segments N / 2 and the number of segments N / 2 / S required for each video segment based on the total duration N, the selected video duration M and the sampling interval S. The selected video frequency bands and the number of images in each video segment are then synthesized using streaming media technology in chronological order to obtain the synthesized video.

[0214] It should be noted that the process of video synthesis according to the total duration of the synthesized video in this embodiment is detailed above and will not be repeated here.

[0215] Optionally, in another embodiment, based on the above embodiment, the method may further include: adding background music when synthesizing all selected target video segments and the number of frame images in each target video segment to obtain a synthesized video with background music; and pushing the synthesized video with background music to the user or uploading it to the cloud.

[0216] In this embodiment of the invention, when synthesizing all the selected target segments, background music is added, and then the video can be uploaded to the cloud; or the synthesized video can be pushed to the user at regular intervals so that the user can play it online or download and share it to social media software, etc.

[0217] Optionally, in another embodiment, based on the above embodiments, the method may further include: pre-setting object features in an object database, the object features including: facial features or living creature features.

[0218] A specific application example includes:

[0219] 1) Users install cloud-based home security cameras and activate the Time Capsule service, initially recording facial image information into the database.

[0220] 2) Obtain video clips recorded by Cloud Home Security from cloud storage. Each video clip is about 1 minute long, and each device has about 60 video clips per day (the device provides basic recognition capabilities, such as human detection and liveness detection).

[0221] 3) The terminal's parsing module can be used to extract frames from each video segment (one frame every 2 seconds), generating 30 images from a single video.

[0222] 4) This parsing module performs similarity filtering on the extracted frames to remove duplicate images.

[0223] 5) This parsing module performs algorithmic analysis and labeling on the remaining images (i.e., the remaining images after removing duplicate images) to generate corresponding label information; and extracts bounding boxes for faces, performs face labeling through face feature extraction and comparison, and generates video-image frame-image feature parsing result data.

[0224] 6) The compositing module of the terminal calculates the required number of segments N / 2 based on the set total duration N of the miniature compositing, the selected video duration M, and the sampling interval S. The number of segments required for each time period is N / 2 / S.

[0225] 7) The synthesis module randomly selects images with matching tags from each time segment and extracts the corresponding segments within 1 second before and after in the video.

[0226] 8) This synthesis module synthesizes all the selected target segments, adds background music, and uploads them to the cloud address.

[0227] 9) Scheduled push of synthesized videos to users, who can then play them online or download and share them on social media.

[0228] This invention utilizes a camera to collect raw data and extract frames. After extraction, a similarity filtering algorithm is used to remove duplicate extracted frames, reducing algorithm calls and saving computing resources. In the synthesis stage, a segmented sampling mode is used to add background music, thereby improving the richness of the video.

[0229] The embodiments of this invention may be applied in various scenarios in the future, including both indoor smart home market miniature mode and outdoor B2B market miniature mode, store miniature mode, etc.

[0230] This invention can also use video analysis results to tag videos, making it easier to filter videos based on tags (such as video querying and location based on tags).

[0231] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the present invention.

[0232] Figure 5 This is a block diagram of a video synthesis device provided in an embodiment of the present invention. The device includes: an acquisition module 501, a frame extraction module 502, a filtering module 503, a feature extraction module 504, a matching module 505, and a synthesis module 506, wherein...

[0233] The acquisition module 501 is used to acquire video clips recorded in the cloud and the target object features of the video to be synthesized;

[0234] The frame extraction module 502 is used to extract frames from each video segment to obtain the frame-extracted image of the corresponding video segment.

[0235] The filtering module 503 is used to perform similarity filtering on the frame images of each video segment to obtain the filtered frame images.

[0236] The feature extraction module 504 is used to mark the objects on each frame of the filtered images and extract the object features of the marked objects on each frame of the images.

[0237] The matching module 505 is used to match the target object features with the object features extracted from each frame image to obtain successfully matched frame images.

[0238] The compositing module 506 is used to combine all the successfully matched extracted frames to obtain the composite video.

[0239] Optionally, in another embodiment, based on the above embodiments, the device further includes: a selection module 601, the structural block diagram of which is shown below. Figure 6 As shown, where,

[0240] The selection module 601 is used to select the required number of target video segments and the required number of frame images in each selected target video segment from all the successfully matched frame images before the synthesis module 506 synthesizes all the successfully matched target frame images, according to the set synthesis video length.

[0241] The synthesis module 506 is further configured to synthesize all the target video segments selected by the selection module and the number of frame images in each target video segment to obtain the synthesized video.

[0242] Optionally, in another embodiment, based on the above embodiments, the apparatus further includes:

[0243] An addition module is used to add background music when the synthesis module synthesizes all selected target video segments and the number of frame images in each target video segment, so as to obtain a synthesized video with background music.

[0244] The calculation module is used to push the video with background music added by the adding module to the user or upload it to the cloud.

[0245] Optionally, in another embodiment, based on the above embodiments, the filtering module includes:

[0246] The grayscale processing module is used to reduce the dimensionality of each video segment's frame images and convert them to grayscale, resulting in grayscale frame images.

[0247] A conversion module is used to convert each position on the grayscale frame-by-frame image into a pixel matrix, wherein the color in the pixel matrix is ​​represented by 0 and 1;

[0248] The XOR module is used to perform XOR operations on the same positions of the pixel matrices of the first and second adjacent frame-picked images in sequence to obtain the difference matrix.

[0249] A similarity determination module is used to determine the similarity between the second frame-picking image and the first frame-picking image based on the number of 1s in the difference matrix.

[0250] The removal module is used to remove the second frame-picking image when the similarity determined by the similarity determination module is greater than a set threshold.

[0251] The retention module is used to retain the second frame-picking image when the similarity determined by the similarity determination module is not greater than a set threshold, thereby obtaining a set of filtered frame-picking images.

[0252] Please also see Figure 7 This is another structural block diagram of a video synthesis apparatus provided in an embodiment of the present invention. The apparatus includes: an acquisition module 701, a frame extraction module 702, a filtering module 703, a first marking module 704, an extraction module 705, a matching module 706, a second marking module 707, and a synthesis module 708, wherein...

[0253] Module 701 is used to acquire video clips recorded in the cloud;

[0254] The frame extraction module 702 is used to extract frames from each video segment to obtain the frame-extracted image of the corresponding video segment.

[0255] The filtering module 703 is used to perform similarity filtering on the frame images of each video segment to obtain the filtered frame images.

[0256] The first labeling module 704 is used to label the objects on each frame of the filtered images to obtain the object labels on each frame of the images.

[0257] Extraction module 705 is used to extract object features of the region corresponding to each object label;

[0258] The matching module 706 is used to match the object features with object features in the object database;

[0259] The second marking module 707 is used to mark the frame images of objects whose features were not successfully matched.

[0260] The compositing module 708 synthesizes the extracted frames of all marked unmatched object features to obtain the synthesized video.

[0261] Optionally, in another embodiment, based on the above embodiments, the apparatus further includes:

[0262] The selection module is used to select the required number of target video segments and the number of frame images required for each selected target video segment from frame images that mark all object features that did not match successfully, according to the set synthetic video length.

[0263] The selected target video segments and the number of extracted frames in each target video segment are combined to obtain the synthesized video.

[0264] Optionally, in another embodiment, based on the above embodiments, the apparatus further includes:

[0265] Add a module to add background music when compositing all selected target video segments and the number of frame images in each target video segment, so as to obtain a video with background music;

[0266] The push module is used to push the video with background music added by the adding module to the user or upload it to the cloud.

[0267] Optionally, in another embodiment, based on the above embodiments, the apparatus further includes:

[0268] The settings module is used to pre-set object characteristics in the object database, including: facial features or living creature features.

[0269] Optionally, in another embodiment, based on the above embodiments, the filtering module includes:

[0270] The grayscale processing module is used to reduce the dimensionality of each video segment's frame images and convert them to grayscale, resulting in grayscale frame images.

[0271] A conversion module is used to convert each position on the grayscale frame-by-frame image into a pixel matrix, wherein the color in the pixel matrix is ​​represented by 0 and 1;

[0272] The XOR module is used to perform XOR operations on the same positions of the pixel matrices of the first and second adjacent frame-picked images in sequence to obtain the difference matrix.

[0273] The similarity determination module determines the similarity between the second frame-picked image and the first two frame-picked images based on the number of 1s in the difference matrix.

[0274] The removal module is used to remove the second frame-by-frame image when the similarity is greater than a set threshold.

[0275] The retention module is used to retain the second frame-picking image when the similarity determined by the similarity determination module is not greater than a set threshold, thereby obtaining a set of filtered frame-picking images.

[0276] Optionally, embodiments of the present invention also provide an electronic device, comprising:

[0277] processor;

[0278] Memory used to store the processor's executable instructions;

[0279] The processor is configured to execute the instructions to implement the video synthesis method described above.

[0280] Optionally, embodiments of the present invention also provide a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the video synthesis method described above.

[0281] Optionally, embodiments of the present invention also provide a computer program product, including a computer program or instructions, which, when executed by a processor of an electronic device, implement the video synthesis method described above.

[0282] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0283] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0284] Figure 8 This is a block diagram of an electronic device 800 provided in an embodiment of the present invention. For example, the electronic device 800 can be a mobile terminal or a server; in this embodiment, a mobile terminal is used as an example for explanation. For example, the electronic device 800 can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0285] Reference Figure 8 The electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0286] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0287] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0288] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0289] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0290] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0291] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0292] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0293] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0294] In an embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the video synthesis method described above.

[0295] In this embodiment, a computer-readable storage medium is also provided, which, when executed by a processor of an electronic device, enables the electronic device 800 to perform the video synthesis method described above. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0296] In one embodiment, a computer program product is also provided, including a computer program or instructions that, when executed by the processor 820 of the electronic device 800, cause the electronic device 800 to perform the video synthesis method described above.

[0297] Figure 9 This is a block diagram of an apparatus 900 for video synthesis provided in an embodiment of the present invention. For example, apparatus 900 may be provided as a server. See also... Figure 9 The apparatus 900 includes a processing component 922, which further includes one or more processors, and memory resources represented by memory 932 for storing instructions, such as application programs, that can be executed by the processing component 922. The application programs stored in memory 932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 922 is configured to execute instructions to perform the methods described above.

[0298] The device 900 may also include a power supply component 926 configured to perform power management of the device 900, a wired or wireless network interface 950 configured to connect the device 900 to a network, and an input / output (I / O) interface 958. The device 900 can operate on an operating system stored in memory 932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0299] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0300] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A video synthesis method, characterized in that, include: Acquire video clips recorded in the cloud, as well as the features of the target objects in the video to be synthesized; Frames are extracted from each video segment to obtain the corresponding frame-extracted images of the video segment; The similarity of the extracted frames from each video segment is used to obtain the filtered extracted frames. The objects on each of the selected frame images are marked, and the object features of the marked objects on each frame image are extracted. The target object features are matched with the object features extracted from each frame image to obtain successfully matched frame images. All successfully matched extracted frames are combined to obtain the synthesized video.

2. The video synthesis method according to claim 1, characterized in that, Before synthesizing all successfully matched frame images, the method further includes: Based on the set length of the miniature composite video, select the required number of target video segments and the required number of frame images for each selected target video segment from all successfully matched frame images; The selected target video segments and the number of extracted frames in each target video segment are combined to obtain the synthesized video.

3. The video synthesis method according to claim 2, characterized in that, The method further includes: When compositing all the selected target video segments and the number of frame images in each target video segment, background music is added to obtain a composite video with background music. The synthesized video with background music can be pushed to users or uploaded to the cloud.

4. The video synthesis method according to any one of claims 1 to 3, characterized in that, The step of performing similarity filtering on the extracted frame images of each video segment to obtain the filtered extracted frame images includes: The extracted frames of each video segment are reduced in dimensionality and converted to grayscale to obtain the grayscale extracted frames. Each position on the grayscale extracted image is converted into a pixel matrix, and the color in the pixel matrix is ​​represented by 0 and 1; The pixel matrices of the first and second adjacent frame-picked images are XORed sequentially at the same positions to obtain the difference matrix. The similarity between the second frame-by-frame image and the first frame-by-frame image is determined based on the number of elements with a value of 1 in the difference matrix. If the similarity is greater than the set threshold, the second frame-extracting image is removed; If the similarity is not greater than the set threshold, the second extracted frame image is retained, resulting in a filtered set of extracted frame images.

5. A video synthesis method, characterized in that, include: Obtain video clips recorded in the cloud; Frames are extracted from each video segment to obtain the corresponding frame-extracted images of the video segment; The similarity of the extracted frames from each video segment is used to obtain the filtered extracted frames. The objects in each of the filtered frame images are labeled to obtain the object labels for each frame image; Obtain the object characteristics of the region corresponding to each object label; The object features are matched with the object features in the object database, and the frame images of object features that do not match are marked. The extracted frames of the marked but unmatched object features are synthesized to obtain the synthesized video.

6. The video synthesis method according to claim 5, characterized in that, The method further includes: Based on the set synthetic video length, select the required number of target video segments and the required number of frame images in each selected target video segment from the frame images corresponding to the features of the unmatched objects. The selected target video segments and the number of extracted frames in each target video segment are combined to obtain the synthesized video.

7. The video synthesis method according to claim 6, characterized in that, The method further includes: When compositing all selected target video segments and the number of frame images in each target video segment, background music is added to obtain a composite video with background music. The synthesized video with background music can be pushed to users or uploaded to the cloud.

8. A video synthesis apparatus, characterized in that, include: The acquisition module is used to acquire video clips recorded in the cloud, as well as the target object features of the video to be synthesized; The frame extraction module is used to extract frames from each video segment to obtain the corresponding frame images of the video segment. The filtering module is used to filter the extracted frames of each video segment based on similarity, and obtain the filtered extracted frames. The feature extraction module is used to label the objects on each frame of the filtered images and extract the object features of the labeled objects on each frame. The matching module is used to match the target object features with the object features extracted from each frame image to obtain successfully matched frame images. The compositing module is used to combine all the successfully matched extracted frames to obtain the composite video.

9. A video synthesis apparatus, characterized in that, include: The acquisition module is used to acquire video clips recorded in the cloud; The frame extraction module is used to extract frames from each video segment to obtain the corresponding frame images of the video segment. The filtering module is used to filter the extracted frames of each video segment based on similarity, and obtain the filtered extracted frames. The first labeling module is used to label the objects on each of the filtered frame images to obtain the object labels on each frame image. The extraction module is used to extract the object features of the region corresponding to each object label; The matching module is used to match the object features with object features in the object database; The second labeling module is used to label the extracted frames of images containing unmatched object features; The compositing module synthesizes the extracted frames of all unmatched object features to obtain the synthesized video.

10. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the video synthesis method as described in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the video synthesis method as described in any one of claims 1 to 7.

Images