Video synthesis method and device, electronic equipment and storage medium

Abstract

The invention provides a video synthesis method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining a video clip recorded at a cloud end and a target object feature of a to-be-synthesized video; performing frame extraction on each video clip to obtain a frame extraction picture of the corresponding video clip; performing similarity screening on the frame extraction picture of each video clip to obtain a screened frame extraction picture; marking an object on each screened frame extraction picture, and extracting an object feature of the marked object on each frame extraction picture; matching the target object feature with the object feature extracted from each frame extraction picture to obtain a successfully matched frame extraction picture; and synthesizing all the successfully matched frame extraction pictures to obtain a synthesized video. According to the method, the characteristic that the resolution of the same video is uniform is utilized, the pixels at the same position are compared through an image processing technology, similarity calculation is performed according to the comparison result, repeated images are removed, computing power resources are reduced, and the video synthesis efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a video synthesis method, device, electronic equipment and computer-readable storage medium. Background Art

[0002] With the rapid development of network technology, network technology is increasingly used in various technical fields. Video surveillance systems have been widely used by enterprises and families, playing a huge role in maintaining public security and property safety.

[0003] In the related art, cloud home surveillance can be performed to view the images of monitoring equipment in real time to ensure the safety of home or office. Cloud home surveillance collects video clips through cameras, extracts video frame information from the video clips, and marks the extracted video frame information. Finally, a screening strategy is used to generate concentrated and essential video clips based on the marking results. However, since the difference in video frame information extracted from a single video may be small, if all video frames in the video clip are extracted and marked, it will increase computing resources and reduce synthesis efficiency. Summary of the invention

[0004] The present invention provides a video synthesis method, device, electronic device and storage medium to at least solve the problem in the related art that since the difference of video frame information extracted in a single video may be small, if all video frames in a video clip are extracted and marked with video frame information, the computing resources will increase and the synthesis efficiency will decrease. The technical solution of the present invention is as follows:

[0005] According to a first aspect of an embodiment of the present invention, a video synthesis method is provided, including:

[0006] Obtain the video clips recorded in the cloud and the features of the target object to be synthesized into the video;

[0007] Extract frames from each video clip to obtain an extracted frame image of the corresponding video clip;

[0008] Perform similarity screening on the extracted frame images of each video clip to obtain the screened extracted frame images;

[0009] Mark the objects on each of the filtered frame images, and extract the object features of the marked objects on each frame image;

[0010] Matching the target object features with the object features extracted from each frame-by-frame image to obtain a successfully matched frame-by-frame image;

[0011] All successfully matched frame images are synthesized to obtain a synthesized video.

[0012] Optionally, before synthesizing all successfully matched frame extraction pictures, the method further includes:

[0013] According to the set epitome synthesized video length, the required number of target video segments and the required number of frame-drawing pictures in each selected target video segment are selected from all successfully matched frame-drawing pictures;

[0014] The number of all selected target video clips and the number of extracted frame images in each target video clip are synthesized to obtain a synthesized video.

[0015] Optionally, the method further includes:

[0016] When synthesizing the number of all selected target video clips and the number of extracted frame images in each target video clip, background music is added to obtain a synthesized video with background music;

[0017] Push the synthesized video with background music to the user or upload it to the cloud.

[0018] Optionally, the similarity screening of the extracted frame pictures of each video clip to obtain the screened extracted frame pictures includes:

[0019] Perform dimensionality reduction and grayscale conversion on the frame-sampling images of each video clip to obtain the grayscale frame-sampling images;

[0020] Convert each position on the grayscaled frame-sampling image into a pixel matrix, where the colors in the pixel matrix are represented by 0 and 1;

[0021] Sequentially perform XOR operations on the same positions of the pixel matrices of the first and second adjacent extracted frame images after conversion to obtain a difference matrix;

[0022] Determine the similarity between the second frame-drawing picture and the first frame-drawing picture according to the number of elements with a value of 1 in the difference matrix;

[0023] When the similarity is greater than a set threshold, the second extracted frame image is removed;

[0024] When the similarity is not greater than the set threshold, the second frame-dropped picture is retained to obtain a filtered frame-dropped picture set.

[0025] According to a second aspect of an embodiment of the present invention, a video synthesis method is provided, including:

[0026] Get video clips recorded in the cloud;

[0027] Extract frames from each video clip to obtain an extracted frame image of the corresponding video clip;

[0028] Perform similarity screening on the extracted frame images of each video clip to obtain the screened extracted frame images;

[0029] Mark the objects on each filtered frame image to obtain the object label on each frame image;

[0030] Get the object features of the area corresponding to each object label;

[0031] Matching the object features with the object features in the object database, and marking the extracted frames of the object features that are not successfully matched;

[0032] The frame-by-frame images of the marked object features that have not been successfully matched are synthesized to obtain a synthesized video.

[0033] Optionally, the method further includes:

[0034] According to the set length of the synthetic video, the required number of target video segments and the required number of frame pictures in each selected target video segment are selected from the frame pictures corresponding to the object features marked as unsuccessfully matched;

[0035] The number of all selected target video clips and the number of extracted frame images in each target video clip are synthesized to obtain a synthesized video.

[0036] Optionally, the method further includes:

[0037] When synthesizing the number of all selected target video clips and the number of extracted frame images in each target video clip, background music is added to obtain a synthesized video with background music;

[0038] Push the synthesized video with background music to the user or upload it to the cloud.

[0039] Optionally, the method further includes:

[0040] The object features in the object database are preset, and the object features include: human face features or living thing features.

[0041] Optionally, the similarity screening of the extracted frame pictures of each video clip to obtain the screened extracted frame pictures includes:

[0042] Perform dimensionality reduction and grayscale conversion on the frame-sampling images of each video clip to obtain the grayscale frame-sampling images;

[0043] Convert each position on the grayscaled frame-sampling image into a pixel matrix, where the colors in the pixel matrix are represented by 0 and 1;

[0044] Sequentially perform XOR operations on the same positions of the pixel matrices of the first and second adjacent extracted frame images after conversion to obtain a difference matrix;

[0045] Determine the similarity between the second frame-drawing picture and the first frame-drawing picture according to the number of 1s in the difference matrix;

[0046] When the similarity is greater than a set threshold, the second extracted frame image is removed;

[0047] When the similarity is not greater than the set threshold, the second frame-dropped picture is retained to obtain a filtered frame-dropped picture set.

[0048] According to a third aspect of an embodiment of the present invention, there is provided a video synthesis device, including:

[0049] The acquisition module is used to obtain the video clips recorded in the cloud and the target object features of the video to be synthesized;

[0050] A frame extraction module is used to extract frames from each video segment to obtain a frame extraction picture of the corresponding video segment;

[0051] A screening module is used to screen the frame pictures of each video clip for similarity to obtain screened frame pictures;

[0052] A feature extraction module is used to mark the objects on each of the filtered frame pictures and extract the object features of the marked objects on each frame picture;

[0053] A matching module is used to match the target object features with the object features extracted from each frame-by-frame image to obtain a successfully matched frame-by-frame image;

[0054] The synthesis module is used to synthesize all successfully matched frame images to obtain a synthesized video.

[0055] Optionally, the device further comprises:

[0056] A selection module, used for selecting the required number of target video segments and the required number of frame extraction pictures in each selected target video segment from all the frame extraction pictures that have successfully matched, according to a set length of the synthesized video, before the synthesis module synthesizes all the target frame extraction pictures that have successfully matched;

[0057] The synthesis module is also used to synthesize the number of all target video clips selected by the selection module and the number of frame-drawing pictures in each target video clip to obtain a synthesized video.

[0058] Optionally, the device further comprises:

[0059] An adding module, used for adding background music when the synthesis module synthesizes the number of all selected target video clips and the number of extracted frame pictures in each target video clip, so as to obtain a synthesized video with background music;

[0060] The calculation module is used to push the video after the background music is added by the adding module to the user or upload it to the cloud.

[0061] Optionally, the screening module includes:

[0062] A grayscale processing module is used to reduce the dimension of the frame-by-frame image of each video clip and grayscale it to obtain a grayscale frame-by-frame image;

[0063] A conversion module, used to convert each position on the grayscaled frame extraction image into a pixel matrix, where the colors in the pixel matrix are represented by 0 and 1;

[0064] An XOR module is used to sequentially perform XOR operations on the same positions of the pixel matrices after the adjacent first frame extraction picture and the second frame extraction picture are converted to obtain a difference matrix;

[0065] a similarity determination module, configured to determine the similarity between the second frame-drawing picture and the first frame-drawing picture according to the number of 1s in the difference matrix;

[0066] A removal module, configured to remove the second frame-drawing picture when the similarity determined by the similarity determination module is greater than a set threshold;

[0067] The retaining module is used to retain the second frame-drawing picture when the similarity determined by the similarity determination module is not greater than a set threshold, so as to obtain a filtered frame-drawing picture set.

[0068] According to a fourth aspect of an embodiment of the present invention, there is provided a video synthesis device, including:

[0069] The acquisition module is used to obtain the video clips recorded in the cloud;

[0070] A frame extraction module is used to extract frames from each video segment to obtain a frame extraction picture of the corresponding video segment;

[0071] A screening module is used to screen the frame pictures of each video clip for similarity to obtain screened frame pictures;

[0072] A first marking module is used to mark the objects on each of the screened extracted frame images to obtain the object labels on each of the extracted frame images;

[0073] An extraction module, used to extract object features of the area corresponding to each object label;

[0074] A matching module, used for matching the object features with the object features in the object database;

[0075] A second marking module is used to mark the extracted frame images of the object features that have not been successfully matched;

[0076] The synthesis module synthesizes the frame-by-frame images of all the marked object features that have not been successfully matched to obtain a synthesized video.

[0077] Optionally, the device further comprises:

[0078] A selection module is used to select the required number of target video segments and the number of frame pictures required for each selected target video segment from the frame pictures that mark all object features that have not been successfully matched according to the set length of the synthetic video;

[0079] The number of all selected target video clips and the number of extracted frame images in each target video clip are synthesized to obtain a synthesized video.

[0080] Optionally, the device further comprises:

[0081] An adding module is used to add background music when synthesizing the number of all selected target video clips and the number of extracted frame pictures in each target video clip to obtain a video with background music;

[0082] The push module is used to push the video after the background music is added by the adding module to the user or upload it to the cloud.

[0083] Optionally, the device further comprises:

[0084] The setting module is used to pre-set the object features in the object database, and the object features include: face features or living thing features.

[0085] Optionally, the screening module includes:

[0086] A grayscale processing module is used to reduce the dimension of the frame-by-frame image of each video clip and grayscale it to obtain a grayscale frame-by-frame image;

[0087] A conversion module, used to convert each position on the grayscaled frame extraction image into a pixel matrix, where the colors in the pixel matrix are represented by 0 and 1;

[0088] An XOR module is used to sequentially perform XOR operations on the same positions of the pixel matrices after the adjacent first frame extraction picture and the second frame extraction picture are converted to obtain a difference matrix;

[0089] a similarity determination module, which determines the similarity between the second frame-drawing picture and the first two frame-drawing pictures according to the number of 1s in the difference matrix;

[0090] A removal module, used for removing the second frame-drawing picture when the similarity is greater than a set threshold;

[0091] The retaining module is used to retain the second frame-drawing picture when the similarity determined by the similarity determination module is not greater than a set threshold, so as to obtain a filtered frame-drawing picture set.

[0092] According to a fifth aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0093] processor;

[0094] a memory for storing instructions executable by the processor;

[0095] The processor is configured to execute the instructions to implement the video synthesis method as described above.

[0096] According to a sixth aspect of an embodiment of the present invention, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video synthesis method as described above.

[0097] According to a seventh aspect of an embodiment of the present invention, there is provided a computer program product, comprising a computer program or instructions, which implement the video synthesis method as described above when executed by a processor of an electronic device.

[0098] The technical solution provided by the embodiments of the present invention brings at least the following beneficial effects:

[0099] In an embodiment of the present invention, a video clip recorded in the cloud and the target object features of the video to be synthesized are obtained; each video clip is framed to obtain a framed picture of the corresponding video clip; the framed pictures of each video clip are screened for similarity to obtain a screened framed picture; the objects on each screened framed picture are marked, and the object features of the marked objects on each framed picture are extracted; the target object features are matched with the object features on each extracted framed picture to obtain a successfully matched framed picture; all successfully matched framed pictures are synthesized to obtain a synthesized video. In other words, the embodiment of the present invention utilizes the feature of uniform resolution of the same video, and uses image processing technology to compare pixels at the same position, and calculates similarity based on the comparison results, removes duplicate images, completes filtering before matching, reduces computing resources, improves the efficiency of video synthesis, and provides users with a better experience.

[0100] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention, and do not constitute an improper limitation of the present invention. In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0102] Figure 1 It is a flow chart of a video synthesis method provided by an embodiment of the present invention.

[0103] Figure 2 It is a schematic diagram of similarity screening provided by an embodiment of the present invention.

[0104] Figure 3 It is a schematic diagram of selecting clips and synthesizing videos according to the total duration of a set synthesized video provided by an example of the present invention.

[0105] Figure 4 This is another flow chart of a video synthesis method provided by an embodiment of the present invention.

[0106] Figure 5 The present invention provides a block diagram of a video synthesis device.

[0107] Figure 6 It is another block diagram of a video synthesis device provided by an embodiment of the present invention.

[0108] Figure 7 This is another block diagram of a video synthesis device provided by an embodiment of the present invention.

[0109] Figure 8 It is a block diagram of an electronic device provided by an embodiment of the present invention.

[0110] Fig. 9 The invention is a block diagram of a device for video synthesis provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0111] In order to enable ordinary persons in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0112] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0113] In recent years, research on computer vision, deep learning, machine learning, image processing, image recognition and other technologies based on artificial intelligence has made important progress. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies and application systems for simulating and extending human intelligence. Artificial Intelligence is a comprehensive discipline involving many types of technologies such as chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, neural networks, etc. Computer vision, as an important branch of artificial intelligence, specifically allows machines to recognize the world. Computer vision technology usually includes face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, target detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, robot navigation and positioning and other technologies. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as security, urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, automatic driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, and smart temperature measurement.

[0114] Technical terms:

[0115] Video frame extraction: A video processing technique that involves extracting specific frames (i.e., static images) from a video sequence. Video is essentially a series of continuous static images that form a dynamic effect when played quickly. Frame extraction is to selectively extract some frames (static images) from this sequence.

[0116] Confidence: A concept in statistics that indicates the degree of credibility of statistical inference. In statistics, confidence is often used to describe the probability that an interval estimate (such as a confidence interval) contains a population parameter. In other words, it indicates how credible we are about the parameter estimate.

[0117] Feature matching: A key technology in computer vision, used to find similar feature points between different images and match them. This process includes three steps: feature extraction, feature description, and feature matching.

[0118] On the basis of understanding the above technical terms, please refer to the following embodiments.

[0119] See also Figure 1 , is a flow chart of a video synthesis method provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes the following steps:

[0120] Step 101: Obtain the video clips recorded in the cloud and the target object features of the video to be synthesized.

[0121] Step 102: extract frames from each video segment to obtain an extracted frame image corresponding to the video segment.

[0122] Step 103: Perform similarity screening on the extracted frame images of each video clip to obtain the filtered extracted frame images

[0123] Step 104: Mark the objects on each of the filtered extracted frame images, and extract object features of the marked objects on each of the extracted frame images.

[0124] Step 105: Match the target object features with the object features extracted from each frame-by-frame image to obtain a successfully matched frame-by-frame image.

[0125] Step 106: synthesize all successfully matched frame images to obtain a synthesized video.

[0126] The video synthesis method described in the present invention can be applied to terminals, servers, etc., without limitation herein. The terminal implementation device can be an electronic device such as a smart phone, a laptop computer, a tablet computer, a desktop computer, a personal digital assistant (PDA, Personal Digital Assistant) and a wearable device. The server can be an independent server or a server cluster, or a server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, intermediate services, domain name services, security services, content distribution networks, or a big data and artificial intelligence platform, etc., without limitation herein.

[0127] Combine the following Figure 1, the specific implementation steps of a video synthesis method provided by an embodiment of the present invention are described in detail.

[0128] In step 101, a video clip recorded in the cloud and features of a target object of a video to be synthesized are obtained.

[0129] In this step, each recorded video clip can be obtained from the cloud, and the video clip can be a video clip recorded by the cloud home watcher, or a surveillance video clip recorded, etc. Of course, this embodiment can also obtain video clips from other servers, applications, etc., and this embodiment does not limit it. The content of the video clip can include data of different scenes, user behavior data, etc.

[0130] The target object features of the video to be synthesized may be facial features of a user, or features of an animal, etc., which are not limited in this embodiment.

[0131] In step 102, each video segment is subjected to frame extraction to obtain a frame-extracted picture corresponding to the video segment.

[0132] In this step, each video segment is framed, and the information of each framed picture after the extraction and the time information of the framed picture in the video are recorded.

[0133] For example, a single video is about 1 minute long, and the video segment is framed at 2s per frame to obtain 30 framed pictures. After frame extraction, it is also necessary to record the image information of each framed picture and the time information of each framed picture in the video segment. The specific process of frame extraction for a video segment is well known to those skilled in the art and will not be described in detail here.

[0134] In step 103, the extracted frame pictures of each video segment are screened for similarity to obtain screened extracted frame pictures.

[0135] In this step, the extracted image of each video clip is reduced in dimension and grayed to obtain the grayed extracted image; each position on the grayed extracted image is converted into a pixel matrix, and the colors in the pixel matrix are represented by 0 and 1; XOR operations are performed on the same positions of the pixel matrices after the conversion of the adjacent first and second extracted images in turn to obtain a difference matrix; the similarity between the second extracted image and the first extracted image is determined according to the number of 1s in the difference matrix; when the similarity is greater than a set threshold, the second extracted image is removed; or when the similarity is not greater than the set threshold, the second extracted image is retained to obtain a filtered extracted image set.

[0136] Specifically, in this embodiment, in order to save computing resources and reduce the call of the algorithm, before marking the object on each frame-drawing picture, all the frame-drawing pictures are first scaled (such as 64*64, etc.), and then gray-scaled, and then each pixel position in the gray-scaled frame-drawing picture is mapped to each element in the two-dimensional matrix, and the element value is 0 or 1; by comparing each position on the two-dimensional matrix of two adjacent frame-drawing pictures, such as XOR operation, etc., the difference matrix between adjacent frame-drawing pictures is obtained. Among them, adjacent frame-drawing pictures can be represented by a1, a2...an, and the difference matrix of a1 and a2 is compared. The number of element values ​​1 in the difference matrix of a1 and a2 is represented by N, and the similarity is calculated based on N, that is, the similarity calculation formula is: N / (64*64)*100%.

[0137] When the similarity value is not greater than a set threshold (such as 80%, etc., the specific threshold can be set according to implementation needs and is not limited in this embodiment), retain a1 and a2; if the similarity value is greater than the threshold, discard a2 and retain a1; continue to compare a1 and a3, if the similarity value of a1 and a3 is greater than the set threshold, discard a3, otherwise, if the similarity value of a1 and a3 is not greater than the set threshold, retain a3; continue to compare a3 and a4, and so on, until the similarities of all the extracted frame images are processed and compared, and the target image set is screened.

[0138] Specific as Figure 2 As shown, Figure 2 2, this embodiment takes two cat frame pictures as an example, still represented by a1 and a2, wherein, but in specific applications, it is not limited thereto.

[0139] In this embodiment, a1 and a2 are adjacent original frame-drawing pictures, and the original frame-drawing pictures are usually color pictures. The similarity comparison process specifically includes:

[0140] 1) The original frame-sampling images a1 and a2 are respectively reduced in dimension (e.g., 64*64, etc.) and gray-scaled to obtain gray-scale frame-sampling images a11 and a21;

[0141] 2) Convert each position of the grayscale frame-sampling pictures a11 and a21 into a pixel matrix respectively, where the colors in the pixel matrix are represented by 0 and 1, i.e., black and white.

[0142] 3) Perform an XOR operation on the same positions of the converted pixel matrices of the two frame-sampling images a11 and a21 to obtain a difference matrix.

[0143] 4) The number of 1s in the difference matrix is ​​counted, represented by N, and the similarity is calculated based on N. The calculation formula is N / (64*64)*100%.

[0144] 5) Determine whether the similarity value is greater than a set threshold. If the similarity value is greater than the set threshold, execute 6); otherwise, execute 7).

[0145] In this embodiment, the threshold of similarity is set to 80% as an example, but in practical applications, it is not limited thereto.

[0146] 6) Discard a2 and keep a1.

[0147] In other words, the frame-drawing images whose similarity values ​​are greater than the set threshold will not be called by the algorithm and will be directly discarded, reducing algorithm scheduling and saving computing resources.

[0148] 7) Keep a1 and a2 to obtain a set of extracted frame images after similarity screening.

[0149] The above process is repeated until the similarities of all adjacent extracted frame images are processed, and a set of extracted frame images after similarity screening is obtained, that is, a target image set.

[0150] In this embodiment, some repeated frame-drawing pictures are filtered out in advance through similarity, which reduces subsequent algorithm calls and improves the difference of frame-drawing pictures in the same video.

[0151] In step 104, the objects on each of the filtered extracted frame images are marked, and the object features of the marked objects on each of the extracted frame images are extracted.

[0152] In this step, the detection algorithm service is called to mark the objects (such as faces, animals, etc.) on each filtered frame image to obtain the label corresponding to each image. Of course, it can also include confidence information corresponding to the label, etc. After that, the object features of the marked objects on each frame image are extracted.

[0153] It should be noted that, if it is a human face, face detection technology can be used to mark the object on each filtered frame image, and extract the object features of the marked object on each frame image.

[0154] If the object is an animal, the object detection algorithm can be used. Of course, this method is also applicable to objects such as faces. That is, CV large models are usually used to detect objects in pictures, such as animals, faces, etc. The training method based on computer vision CV large models adopts a general training method, including:

[0155] 1) Obtaining a data set and dividing the data set: In this embodiment, the labeled log data set is divided into a training set, a validation set, and a test set.

[0156] Among them, the training set is used for model training, the validation set is used to adjust model parameters and select the optimal model, and the test set is used to evaluate model performance.

[0157] 2) Model training:

[0158] The large model is trained using the training set, and the parameters of the large model are adjusted through an optimization algorithm (such as gradient descent) to minimize the loss function.

[0159] Among them, the gradient descent algorithm is an iterative method for solving the minimum value of a function in machine learning and optimization problems. It calculates the gradient of the function at the current point (that is, the direction in which the function changes at this point), and then updates the parameters in the opposite direction of the gradient (that is, the direction in which the function decreases fastest), so as to gradually approach the minimum point of the function.

[0160] In this embodiment, a variety of strategies can be used to improve the performance of large models during the training process, such as data enhancement, early stopping, learning rate adjustment, etc.

[0161] 3) Model evaluation and optimization

[0162] 31) Performance evaluation:

[0163] This example uses the test set to evaluate the trained large model, calculates accuracy, recall, F1 score and other indicators to measure model performance, and analyzes the advantages and disadvantages of the model based on the evaluation results.

[0164] 42) Model optimization:

[0165] In view of the shortcomings of the large model performance, optimization and improvement are carried out based on the validation set, such as adjusting the large model structure, increasing training data, improving the annotation strategy, etc. Of course, model fusion, ensemble learning and other methods can also be used to further improve the model performance.

[0166] It should be noted that marking objects in a picture and extracting features of the marked objects are well known to those skilled in the art and will not be described in detail here.

[0167] Among them, to extract the features of the marked object, an algorithm can be used to analyze the mark to generate corresponding label information; then, the face in the area corresponding to the label is framed and the object features on each frame are obtained.

[0168] In step 105, the target object features are matched with the object features on each extracted frame-by-frame picture to obtain a successfully matched frame-by-frame picture.

[0169] In this step, the target object features (such as facial features, etc.) obtained in step 101 and the object features on each frame extracted in this step are matched. If the match is successful, the frame image with a successful match is marked until all matches are completed and all frame images with a successful match are obtained.

[0170] It should be noted that the target object in this embodiment may be a pre-set target object or a target object in a frame-extracted image, which is not limited in this embodiment.

[0171] In step 106, all successfully matched frame images are synthesized to obtain a synthesized video.

[0172] In this step, all the successfully matched extracted frame images can be synthesized in time order to obtain a synthesized video. The video synthesis method can be synthesized using streaming media technology.

[0173] Furthermore, the synthesized video can be pushed to the user.

[0174] In an embodiment of the present invention, a video clip recorded in the cloud and the target object features of the video to be synthesized are obtained; each video clip is framed to obtain a framed picture of the corresponding video clip; the framed pictures of each video clip are screened for similarity to obtain a screened framed picture; the objects on each screened framed picture are marked, and the object features of the marked objects on each framed picture are extracted; the target object features are matched with the object features on each extracted framed picture to obtain a successfully matched framed picture; all successfully matched framed pictures are synthesized to obtain a synthesized video. In other words, the embodiment of the present invention utilizes the feature of uniform resolution of the same video, and uses image processing technology to compare pixels at the same position, and calculates similarity based on the comparison results, removes duplicate images, completes filtering before matching, reduces computing resources, improves the efficiency of video synthesis, and provides users with a better experience.

[0175] Optionally, in another embodiment, based on the above embodiment, before synthesizing all successfully matched frame-by-frame pictures, the method may also include: selecting the required number of target video clips and the required number of frame-by-frame pictures in each selected target video clip from all successfully matched frame-by-frame pictures according to the set miniature synthesized video length; synthesizing all the selected target video clips and the number of frame-by-frame pictures in each target video clip to obtain a synthesized video.

[0176] In this step, it is necessary to pre-set the total duration N of the miniature synthesized video, select the video duration M and the sampling interval number S, and then, based on the total duration N, select the video duration M and the sampling interval number S to calculate the total number of video clips required N / 2, and the number of clips required for each video clip interval N / 2 / S. The selected number of video frequency bands and the number of pictures in each video clip are synthesized in chronological order through streaming media technology to obtain the synthesized video.

[0177] like Figure 3 FIG. 1 is a schematic diagram of selecting segments and synthesizing videos according to the total duration of a set synthesized video provided by an embodiment of the present invention. As shown in the figure, it includes:

[0178] 1) According to the total synthesis time N, select the start and end time of the video, and query all the clips from the parsing results, that is, select the required number of target video clips and the required number of frame pictures in each selected target video clip from all the successfully matched frame pictures;

[0179] 2) Divide all segments according to the sampling interval S, and the length of each interval is M / S;

[0180] 3) In each interval, use a random algorithm to screen the number of N / 2 / S candidate segments and the number of pictures in each candidate segment;

[0181] 4) All candidate segments and the number of pictures in each candidate segment are sorted in chronological order, and the video is synthesized using multimedia technology.

[0182] In the embodiment of the present invention, the diversity of video clip sources is ensured through a multi-interval sampling method, and the richness of cross-video synthesis is improved. That is to say, this embodiment determines the required target number of clips according to the length of the synthesized video, and performs a segmentation and re-screening strategy for the target time period to complete the final clip selection, improve the difference between clips, ensure that the video clips come from multiple independent videos, and enrich the final synthesized video effect.

[0183] Optionally, in another embodiment, based on the above embodiment, the method may also include: when synthesizing all the selected target video clips and the number of frame-drawing pictures in each target video clip, adding background music to obtain a synthesized video with background music; and pushing the synthesized video with background music to the user.

[0184] In an embodiment of the present invention, when all the screened target clips are synthesized and background music is added, they can be uploaded to the cloud; or the synthesized video can be pushed to the user at a fixed time so that the user can play it online or download the video to share it on social software, etc.

[0185] The embodiment of the present invention provides a strategy for repeatedly filtering frame images before video synthesis, so as to avoid sending all frame images to algorithm detection, save computing resources, and increase the richness of the content retained after single video analysis; the embodiment of the present invention adopts a segmented sampling strategy during synthesis to ensure that the clips come from multiple original video segments, thereby improving the overall content richness of the synthesized video, and further adding background music to the synthesized video to provide users with the best experience.

[0186] The present invention uses a camera to collect raw data, and after extracting frames, uses a similarity screening algorithm to reduce algorithm calls and save computing resources; in the synthesis stage, a segmented sampling mode is used to increase video richness.

[0187] By adopting the technical solution described in the embodiment of the present invention, duplicate images can be removed in the parsing stage, and segmented extraction can be performed in the synthesis stage to avoid the situation where the source of the segments is single or there are too many similar segments, thereby achieving the best synthesis effect and providing the best experience for users.

[0188] See also Figure 4 , another flow chart of a video synthesis method provided by the present invention is implemented, and the method comprises:

[0189] Step 401: Obtain the video clip recorded in the cloud;

[0190] In this step, each recorded video clip can be obtained from the cloud. The video clip can be a video clip recorded by Cloud Home Guard or a recorded surveillance video clip, etc. Of course, this embodiment can also obtain video clips from other servers, applications, etc., which is not limited in this embodiment.

[0191] Step 402: extract frames from each video segment to obtain an extracted frame image corresponding to the video segment;

[0192] In this step, each video segment is framed, and the information of each framed picture after the extraction and the time information of the framed picture in the video are recorded.

[0193] For example, a single video is about 2 minutes long, and the video segment is framed at 2s per frame to obtain 60 framed pictures. After frame extraction, it is also necessary to record the image information of each framed picture and the time information of each framed picture in the video segment. The specific process of frame extraction for a video segment is well known to those skilled in the art and will not be described in detail here.

[0194] Step 403: performing similarity screening on the extracted frame images of each video segment to obtain screened extracted frame images;

[0195] In this step, the dimensionality of the frame-drawing picture of each video clip is reduced and grayed to obtain the grayed frame-drawing picture; each position on the grayed frame-drawing picture is converted into a pixel matrix, and the colors in the pixel matrix are represented by 0 and 1; the same positions of the pixel matrices after the conversion of the adjacent first frame-drawing picture and the second frame-drawing picture are XORed in turn to obtain a difference matrix; the similarity between the second frame-drawing picture and the first frame-drawing picture is determined according to the number of 1s in the difference matrix; when the similarity is greater than a set threshold, the second frame-drawing picture is removed; Or when the similarity is not greater than the set threshold, retain the second framed picture, continue to compare the similarity of the first framed picture and the third framed picture, if the similarity value of the first framed picture and the third framed picture is greater than the set threshold, discard the third framed picture, otherwise, if the similarity value of the first framed picture and the third framed picture is not greater than the set threshold, retain the third framed picture; continue to compare the third framed picture and the fourth framed picture, and so on, until the similarities of all the framed pictures are processed and compared, and the target picture set is screened.

[0196] Step 404: marking the objects on each of the filtered extracted frame images to obtain the object labels on each of the extracted frame images;

[0197] In this step, the detection algorithm service is called to mark the objects (such as faces, animals, etc.) on each filtered frame image to obtain the label corresponding to each image. Of course, it can also include confidence information corresponding to the label, etc. After that, the object features of the marked objects on each frame image are extracted.

[0198] It should be noted that, if it is a human face, face detection technology can be used to mark the object on each filtered frame image, and extract the object features of the marked object on each frame image.

[0199] If the object is an animal, the object detection algorithm can be used. Of course, this method is also applicable to objects such as human faces. That is, CV large models are usually used to detect objects in pictures, such as animals, human faces, etc., and the training method based on computer vision CV large models adopts a general training method. For details, please see above and will not be repeated here.

[0200] It should be noted that marking objects in a picture and extracting features of the marked objects are well known to those skilled in the art and will not be described in detail here.

[0201] Step 405: Obtain object features of the area corresponding to each object tag;

[0202] The object features of the area corresponding to each object label are extracted to obtain the object features of the object label on each frame.

[0203] The specific feature extraction process is already a well-known technique for those skilled in the art and will not be described in detail here.

[0204] Step 406: matching the object features with the object features in the object database, and marking the extracted frame images of the object features that are not successfully matched;

[0205] In this step, the extracted object features (such as facial features, etc.) are matched with the object features in the object database. If the match is unsuccessful, the frame-drawing pictures that failed to match are marked until all matches are completed and all unmatched frame-drawing pictures are obtained.

[0206] It should be noted that the object features of the object database in this embodiment are pre-set object features. For example, facial images of family members are obtained in advance, features are extracted from the facial images, facial features of the corresponding members are obtained, and the facial features of each family member are stored in the object database.

[0207] If the video obtained from the cloud is a home surveillance video, and you want to check whether a thief has entered the house, after going through the above series of steps, if the match is successful, it means that they are all family members and there is no thief; if the match is unsuccessful, it means that an outsider has come to the house, such as a thief.

[0208] Step 407: Synthesize the frame images of the marked object features that have not been successfully matched to obtain a synthesized video

[0209] In this step, all the extracted frame images that failed to match can be synthesized in time order to obtain a synthesized video. The video synthesis method can be synthesized using streaming media technology.

[0210] Furthermore, based on the above embodiment, the method may further include: pushing the synthesized video to the user.

[0211] In an embodiment of the present invention, a video clip recorded in the cloud is obtained; each video clip is framed to obtain a framed picture of the corresponding video clip; the framed pictures of each video clip are screened for similarity to obtain a screened framed picture; the objects on each screened framed picture are marked to obtain an object label on each framed picture; the object features of the area corresponding to each object label are obtained; the object features are matched with the object features in the object database, and the framed pictures of the object features that are not successfully matched are marked; the framed pictures of the marked object features that are not successfully matched are synthesized to obtain a synthesized video. In other words, the embodiment of the present invention utilizes the feature of uniform resolution of the same video, and uses image processing technology to compare pixels at the same position, and calculates similarity based on the comparison results, removes duplicate images, completes filtering before matching, reduces computing resources, improves the efficiency of video synthesis, and provides users with a better experience.

[0212] Optionally, in another embodiment, based on the above embodiment, the method may further include: according to the set length of the synthesized video, selecting the required number of target video clips and the required number of extracted frame pictures in each selected target video clip from the extracted frame pictures corresponding to the object features marked as unsuccessful matches; synthesizing all the selected number of target video clips and the number of extracted frame pictures in each target video clip to obtain a synthesized video.

[0213] In this step, the length of the synthesized video requires pre-setting the total length N of the miniature synthesized video, selecting the video length M and the sampling interval number S, and then, based on the total length N, selecting the video length M and the sampling interval number S, calculating the total number of video clips required N / 2, and the number of clips required for each video clip interval N / 2 / S, and synthesizing the selected video frequency bands and the number of pictures in each video clip in chronological order through streaming media technology to obtain the synthesized video.

[0214] It should be noted that the process of performing video synthesis according to the total duration of the synthesized video in this embodiment is described above in detail and will not be repeated here.

[0215] Optionally, in another embodiment, based on the above embodiment, the method may also include: when synthesizing all selected target video clips and the number of frame-drawing pictures in each target video clip, adding background music to obtain a synthesized video with background music; pushing the synthesized video with background music to the user or uploading it to the cloud.

[0216] In an embodiment of the present invention, when all the screened target clips are synthesized and background music is added, they can be uploaded to the cloud; or the synthesized video can be pushed to the user at a fixed time so that the user can play it online or download the video to share it on social software, etc.

[0217] Optionally, in another embodiment, based on the above embodiment, the method may further include: presetting object features in the object database, wherein the object features include: facial features or living thing features.

[0218] A specific application example includes:

[0219] 1) The user installs the cloud home camera, activates the time-lapse service capability, and initially enters the portrait library information.

[0220] 2) Obtain the video clips recorded by Cloud Security in the cloud storage. A single video clip is about 1 minute, and each device has about 60 video clips per day (the terminal side provides basic recognition capabilities, such as human detection and living object detection).

[0221] 3) The terminal's parsing module can be used to extract frames from each video clip (one frame every 2 seconds), and a single video can generate 30 pictures.

[0222] 4) The parsing module performs similarity screening on the extracted images and removes duplicate images.

[0223] 5) The analysis module performs algorithmic analysis and marking on the remaining images (i.e., the remaining images after removing duplicate images) to generate corresponding label information; and extracts the face frame, marks the face through facial feature extraction and comparison, and generates video-image frame-image feature analysis result data.

[0224] 6) The synthesis module of the terminal calculates the required number of segments N / 2 according to the set total duration N of the epitome synthesis, the selected video duration M and the sampling interval S. The number of segments required for each time period is N / 2 / S.

[0225] 7) The synthesis module randomly selects images that meet the required labels from each time segment and extracts the corresponding segments before and after 1 second in the video.

[0226] 8) The synthesis module synthesizes all the selected target clips, adds background music, and uploads them to the cloud address.

[0227] 9) The synthesized video is pushed to the user at a fixed time, and the user can play it online or download the video and share it on social software.

[0228] The embodiment of the present invention utilizes the camera to collect raw data for frame extraction, and after frame extraction, removes duplicate frame extraction images through a similarity screening algorithm, reduces algorithm calls, saves computing resources, and adds background music through a segmented sampling mode in the synthesis stage, thereby improving the richness of the video.

[0229] The possible application scenarios of the embodiments of the present invention in the future may be biased towards the indoor home market smart miniature mode, as well as the outdoor tob market, store miniature mode, etc.

[0230] The embodiment of the present invention can also use the video analysis results to tag the video, so as to facilitate video screening according to the tags (such as video query and positioning according to the tags).

[0231] It should be noted that, for the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that this implementation disclosure is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present invention.

[0232] Figure 5 1 is a block diagram of a video synthesis device provided by an embodiment of the present invention. The device comprises: an acquisition module 501, a frame extraction module 502, a screening module 503, a feature extraction module 504, a matching module 505 and a synthesis module 506, wherein:

[0233] The acquisition module 501 is used to acquire the video clips recorded in the cloud and the target object features to be synthesized into the video;

[0234] The frame extraction module 502 is used to extract frames for each video segment to obtain a frame extraction picture corresponding to the video segment;

[0235] A screening module 503 is used to screen the frame-sampling pictures of each video segment for similarity to obtain screened frame-sampling pictures;

[0236] A feature extraction module 504 is used to mark the objects on each of the filtered frame pictures and extract the object features of the marked objects on each of the frame pictures;

[0237] A matching module 505 is used to match the target object features with the object features extracted from each frame-by-frame picture to obtain a successfully matched frame-by-frame picture;

[0238] The synthesis module 506 is used to synthesize all the successfully matched extracted frame pictures to obtain a synthesized video.

[0239] Optionally, in another embodiment, based on the above embodiment, the device further includes: a selection module 601, whose structural block diagram is as follows: Figure 6 As shown,

[0240] The selection module 601 is used to select the required number of target video segments and the required number of frame extraction pictures in each selected target video segment from all the frame extraction pictures that have successfully matched, according to the set length of the synthesized video, before the synthesis module 506 synthesizes all the target frame extraction pictures that have successfully matched;

[0241] The synthesis module 506 is further used to synthesize the number of all target video segments selected by the selection module and the number of extracted frame images in each target video segment to obtain a synthesized video.

[0242] Optionally, in another embodiment, based on the above embodiment, the device further includes:

[0243] An adding module, used for adding background music when the synthesis module synthesizes the number of all selected target video clips and the number of extracted frame pictures in each target video clip, so as to obtain a synthesized video with background music;

[0244] The calculation module is used to push the video after the background music is added by the adding module to the user or upload it to the cloud.

[0245] Optionally, in another embodiment, based on the above embodiment, the screening module includes:

[0246] A grayscale processing module is used to reduce the dimension of the frame-by-frame image of each video clip and grayscale it to obtain a grayscale frame-by-frame image;

[0247] A conversion module, used to convert each position on the grayscaled frame extraction image into a pixel matrix, where the colors in the pixel matrix are represented by 0 and 1;

[0248] An XOR module is used to sequentially perform XOR operations on the same positions of the pixel matrices after the adjacent first frame extraction picture and the second frame extraction picture are converted to obtain a difference matrix;

[0249] a similarity determination module, configured to determine the similarity between the second frame-drawing picture and the first frame-drawing picture according to the number of 1s in the difference matrix;

[0250] A removal module, configured to remove the second frame-drawing picture when the similarity determined by the similarity determination module is greater than a set threshold;

[0251] The retaining module is used to retain the second frame-drawing picture when the similarity determined by the similarity determination module is not greater than a set threshold, so as to obtain a filtered frame-drawing picture set.

[0252] See also Figure 7 , is another structural block diagram of a video synthesis device provided by an embodiment of the present invention, the device includes: an acquisition module 701, a frame extraction module 702, a screening module 703, a first marking module 704, an extraction module 705, a matching module 706, a second marking module 707 and a synthesis module 708, wherein,

[0253] The acquisition module 701 is used to acquire the video clips recorded in the cloud;

[0254] The frame extraction module 702 is used to extract frames for each video segment to obtain a frame extraction picture corresponding to the video segment;

[0255] A screening module 703 is used to screen the frame-sampling pictures of each video segment for similarity to obtain screened frame-sampling pictures;

[0256] A first marking module 704 is used to mark the objects on each of the filtered extracted frame images to obtain the object labels on each of the extracted frame images;

[0257] Extraction module 705, used to extract object features of the area corresponding to each object tag;

[0258] A matching module 706, configured to match the object features with the object features in the object database;

[0259] The second marking module 707 is used to mark the extracted frame images of the object features that have not been successfully matched;

[0260] The synthesis module 708 synthesizes the frame-by-frame images of all the marked object features that have not been successfully matched to obtain a synthesized video.

[0261] Optionally, in another embodiment, based on the above embodiment, the device further includes:

[0262] A selection module is used to select the required number of target video segments and the number of frame pictures required for each selected target video segment from the frame pictures that mark all object features that have not been successfully matched according to the set length of the synthetic video;

[0263] The number of all selected target video clips and the number of extracted frame images in each target video clip are synthesized to obtain a synthesized video.

[0264] Optionally, in another embodiment, based on the above embodiment, the device further includes:

[0265] An adding module is used to add background music when synthesizing the number of all selected target video clips and the number of extracted frame pictures in each target video clip to obtain a video with background music;

[0266] The push module is used to push the video after the background music is added by the adding module to the user or upload it to the cloud.

[0267] Optionally, in another embodiment, based on the above embodiment, the device further includes:

[0268] The setting module is used to pre-set the object features in the object database, and the object features include: face features or living thing features.

[0269] Optionally, in another embodiment, based on the above embodiment, the screening module includes:

[0270] A grayscale processing module is used to reduce the dimension of the frame-by-frame image of each video clip and grayscale it to obtain a grayscale frame-by-frame image;

[0271] A conversion module, used to convert each position on the grayscaled frame extraction image into a pixel matrix, where the colors in the pixel matrix are represented by 0 and 1;

[0272] An XOR module is used to sequentially perform XOR operations on the same positions of the pixel matrices after the adjacent first frame extraction picture and the second frame extraction picture are converted to obtain a difference matrix;

[0273] a similarity determination module, which determines the similarity between the second frame-drawing picture and the first two frame-drawing pictures according to the number of 1s in the difference matrix;

[0274] A removal module, used for removing the second frame-drawing picture when the similarity is greater than a set threshold;

[0275] The retaining module is used to retain the second frame-drawing picture when the similarity determined by the similarity determination module is not greater than a set threshold, so as to obtain a filtered frame-drawing picture set.

[0276] Optionally, an embodiment of the present invention further provides an electronic device, including:

[0277] processor;

[0278] a memory for storing instructions executable by the processor;

[0279] The processor is configured to execute the instructions to implement the video synthesis method as described above.

[0280] Optionally, an embodiment of the present invention further provides a computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video synthesis method as described above.

[0281] Optionally, an embodiment of the present invention further provides a computer program product, including a computer program or instructions, and when the computer program or instructions are executed by a processor of an electronic device, the video synthesis method as described above is implemented.

[0282] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0283] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0284] Figure 8 800 is a block diagram of an electronic device 800 provided in an embodiment of the present invention. For example, the electronic device 800 may be a mobile terminal or a server, and the electronic device is a mobile terminal as an example for explanation in the embodiment of the present invention. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a message transceiver device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0285] Reference Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0286] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0287] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0288] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0289] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0290] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0291] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0292] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0293] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0294] In an embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the video synthesis method shown above.

[0295] In an embodiment, a computer-readable storage medium is also provided, and when the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device 800 can perform the video synthesis method shown above. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0296] In an embodiment, a computer program product is further provided, including a computer program or instructions. When the computer program or instructions are executed by the processor 820 of the electronic device 800, the electronic device 800 executes the above-mentioned video synthesis method.

[0297] Fig. 9 is a block diagram of a device 900 for video synthesis provided by an embodiment of the present invention. For example, the device 900 may be provided as a server. Fig. 9 , the apparatus 900 includes a processing component 922, which further includes one or more processors, and a memory resource represented by a memory 932 for storing instructions, such as an application, that can be executed by the processing component 922. The application stored in the memory 932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute the instructions to perform the above method.

[0298] The device 900 may also include a power supply component 926 configured to perform power management of the device 900, a wired or wireless network interface 950 configured to connect the device 900 to a network, and an input / output (I / O) interface 958. The device 900 may operate based on an operating system stored in the memory 932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0299] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0300] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A video synthesis method, characterized in that: include: Obtain the video clips recorded in the cloud and the features of the target object to be synthesized into the video; Extract frames from each video clip to obtain an extracted frame image of the corresponding video clip; Perform similarity screening on the extracted frame images of each video clip to obtain the screened extracted frame images; Mark the objects on each of the filtered frame images, and extract the object features of the marked objects on each frame image; Matching the target object features with the object features extracted from each frame-by-frame image to obtain a successfully matched frame-by-frame image; All successfully matched frame images are synthesized to obtain a synthesized video.

2. The video synthesis method according to claim 1, characterized in that: Before synthesizing all successfully matched frame extraction pictures, the method further includes: According to the set epitome synthesized video length, the required number of target video segments and the required number of frame-drawing pictures in each selected target video segment are selected from all successfully matched frame-drawing pictures; The number of all selected target video clips and the number of extracted frame images in each target video clip are synthesized to obtain a synthesized video.

3. The video synthesis method according to claim 2, characterized in that: The method further comprises: When synthesizing the number of all selected target video clips and the number of extracted frame images in each target video clip, background music is added to obtain a synthesized video with background music; Push the synthesized video with background music to the user or upload it to the cloud.

4. The video synthesis method according to any one of claims 1 to 3, characterized in that: The similarity screening of the frame-sampling pictures of each video segment to obtain the screened frame-sampling pictures includes: Perform dimensionality reduction and grayscale conversion on the frame-sampling images of each video clip to obtain the grayscale frame-sampling images; Convert each position on the grayscaled frame-sampling image into a pixel matrix, where the colors in the pixel matrix are represented by 0 and 1; Sequentially perform XOR operations on the same positions of the pixel matrices of the first and second adjacent extracted frame images after conversion to obtain a difference matrix; Determine the similarity between the second frame-drawing picture and the first frame-drawing picture according to the number of elements with a value of 1 in the difference matrix; When the similarity is greater than a set threshold, the second extracted frame image is removed; When the similarity is not greater than the set threshold, the second frame-dropped picture is retained to obtain a filtered frame-dropped picture set.

5. A video synthesis method, characterized in that: include: Get video clips recorded in the cloud; Extract frames from each video clip to obtain an extracted frame image of the corresponding video clip; Perform similarity screening on the extracted frame images of each video clip to obtain the screened extracted frame images; Mark the objects on each filtered frame image to obtain the object label on each frame image; Get the object features of the area corresponding to each object label; Matching the object features with the object features in the object database, and marking the extracted frames of the object features that are not successfully matched; The frame-by-frame images of the marked object features that have not been successfully matched are synthesized to obtain a synthesized video.

6. The video synthesis method according to claim 5, characterized in that: The method further comprises: According to the set length of the synthetic video, the required number of target video segments and the required number of frame pictures in each selected target video segment are selected from the frame pictures corresponding to the object features marked as unsuccessfully matched; The number of all selected target video clips and the number of extracted frame images in each target video clip are synthesized to obtain a synthesized video.

7. The video synthesis method according to claim 6, characterized in that: The method further comprises: When synthesizing the number of all selected target video clips and the number of extracted frame images in each target video clip, background music is added to obtain a synthesized video with background music; Push the synthesized video with background music to the user or upload it to the cloud.

8. A video synthesis device, characterized in that: include: The acquisition module is used to obtain the video clips recorded in the cloud and the target object features of the video to be synthesized; A frame extraction module is used to extract frames from each video segment to obtain a frame extraction picture of the corresponding video segment; A screening module is used to screen the frame pictures of each video clip for similarity to obtain screened frame pictures; A feature extraction module is used to mark the objects on each of the filtered frame pictures and extract the object features of the marked objects on each frame picture; A matching module is used to match the target object features with the object features extracted from each frame-by-frame image to obtain a successfully matched frame-by-frame image; The synthesis module is used to synthesize all successfully matched frame images to obtain a synthesized video.

9. A video synthesis device, characterized in that: include: The acquisition module is used to obtain the video clips recorded in the cloud; A frame extraction module is used to extract frames from each video segment to obtain a frame extraction picture of the corresponding video segment; A screening module is used to screen the frame pictures of each video clip for similarity to obtain screened frame pictures; A first marking module is used to mark the objects on each of the screened extracted frame images to obtain the object labels on each of the extracted frame images; An extraction module, used to extract object features of the area corresponding to each object label; A matching module, used for matching the object features with the object features in the object database; A second marking module is used to mark the extracted frame images of the object features that have not been successfully matched; The synthesis module synthesizes the frame-by-frame images of all the marked object features that have not been successfully matched to obtain a synthesized video.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the video synthesis method as described in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video synthesis method as claimed in any one of claims 1 to 7.

Images