Intelligent editing method, device, equipment and storage medium for wonderful videos of infants and young children
Through computer vision technology, the video clips of infants and young children are automatically segmented and scored to generate personalized video highlights, which solves the problem of time-consuming and labor-intensive video editing in the existing technology and cannot meet personalized needs, and improves the video quality and viewing.
Patent Information
- Application Number
- CN202311292883.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-28
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-09-28
AI Technical Summary
The existing wonderful video editing technology of infants and young children cannot intelligently process original video materials, cannot meet users' personalized needs, and manual editing is time-consuming and labor-intensive, and has low quality.
Computer vision technology is used to obtain real-time video sequences of infant and young children's capture scenes, extract feature information and video tags, automatically segment video clips, and synthesize them based on scores and user tags to generate personalized video highlights.
Save time and manpower, improve the viewing and quality of video highlights, ensure that the content is more attractive and meets users' personalized needs.
Smart Images

Figure CN117425043B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for intelligently editing wonderful videos of infants and young children. Background Art
[0002] Video editing refers to the process of splitting and merging the original video by shots, and adding background music, scene effects, scenes and other materials in post-processing to generate a new video with unique expressiveness.
[0003] Wonderful video editing of infants and young children refers to making an interesting, vivid and infectious video of infants and young children's growth moments through editing, special effects, background music and other means. Although the existing wonderful video editing technology for infants and young children is very mature, it still has some shortcomings. On the one hand, manual editing has limitations: due to the collection of original video materials, the segmentation and merging of shots, and post-synthesis, a lot of manpower and energy are required. Users need to install video editing tools in electronic devices in advance and have basic video editing capabilities. When the user's video editing ability is low, it is difficult to create a high-quality video collection based on the original video material editing. Therefore, for users who do not have basic video editing capabilities, the video collection obtained by editing is not good in viewing and quality, and does not meet the user's preferences. On the other hand, although the existing intelligent video editing technology can avoid some limitations of manual editing, it cannot really meet the personalized needs of users and needs continuous exploration and improvement.
[0004] The existing Chinese patent CN106899809A discloses a video editing method and device based on deep learning, the method comprising: (1) converting the speech in the preset first video into text data through a deep learning algorithm; (2) training the text data with text features and training the batch collected videos with video image features through a deep learning algorithm; (3) establishing a video editing model based on the text feature training results and the video image feature training results, and editing the batch collected second videos through the video editing model. (4) optimizing the video editing model by taking the viewing popularity and evaluation of the edited video of the second video as the optimization input factor of the deep learning algorithm. However, the deep learning model has high requirements for the quality of the input data. If the speech quality of the first video is poor or there is an error in the text conversion process, the subsequent training and editing process may be affected. The conversion of speech to text is a complex task, especially for different voice accents and background noise. If the conversion is inaccurate, the quality of the text data will decrease, which will affect the subsequent training and editing. At the same time, video editing usually involves subjective judgment, and different people may have different editing preferences for the same video. Automating editing using deep learning algorithms may not fully meet audience expectations.
[0005] Therefore, how to intelligently edit the original video materials to obtain wonderful video highlights that meet the personalized needs of users is an urgent problem to be solved. Summary of the invention
[0006] In view of this, the present invention provides a method, device, equipment and storage medium for intelligently editing wonderful videos of infants and young children, so as to solve the problem in the prior art that the original video materials cannot be intelligently edited to obtain a wonderful video collection that meets the personalized needs of users.
[0007] The technical solution adopted by the present invention is:
[0008] In a first aspect, the present invention provides a method for intelligently editing wonderful videos of infants and young children, the method comprising:
[0009] S1: Acquire real-time video sequences in infant capture scenes;
[0010] S2: extracting feature information from the real-time video sequence to obtain detection frame information and real-time video labels of a preset target;
[0011] S3: Segmenting the real-time video sequence into video segments corresponding to the target scenes one by one according to the detection frame information and the real-time video label;
[0012] S4: Score each of the video clips, and synthesize the high-scoring video clips after post-processing to obtain a video collection.
[0013] Preferably, S2 includes:
[0014] S21: inputting the real-time video sequence into a target detection model, obtaining detection frame information of a preset target, and marking the preset target in the image to generate marked image data;
[0015] S22: inputting the annotated image data into a pre-trained multimodal large model to obtain language and text information corresponding to the question information in the annotated image data;
[0016] S23: According to the preset phrase category, the language and text information is segmented and counted to obtain the real-time video tag.
[0017] Preferably, the S22 includes:
[0018] S221: Obtaining a question and answer instruction in the infant capture scene;
[0019] S222: Obtaining question information input by the user according to the question-answering instruction;
[0020] S223: Input the question information and the annotated image data into the multimodal large model to obtain the language and text information corresponding to the question information in the annotated image data.
[0021] Preferably, S3 includes:
[0022] S31: Acquire adjacent video frames in the real-time video sequence, wherein the adjacent video frames include a first video frame and a second video frame;
[0023] S32: matching the first video frame and the second video frame according to the detection frame information and the real-time video tag, and outputting the real-time similarity between the first video frame and the second video frame;
[0024] S33: Obtain a preset similarity threshold, and output the adjacent video frames to a video segment corresponding to the target scene according to the real-time similarity.
[0025] Preferably, the S33 includes:
[0026] S331: Acquire the first video frame and the second video frame, and output the first video frame to a first video segment, wherein the first video segment corresponds to a first scene in the target scene;
[0027] S332: if the real-time similarity is greater than the similarity threshold, outputting the second video frame to the first video segment;
[0028] S333: If the real-time similarity is not greater than the similarity threshold, outputting the second video frame to a second video segment, wherein the second video segment corresponds to a second scene different from the first scene.
[0029] Preferably, the S4 includes:
[0030] S41: Obtaining preset evaluation parameters and evaluation weights, wherein the evaluation parameters include infant human figure parameters, infant face parameters, and scene parameters;
[0031] S42: performing weighted scoring on each of the video clips according to the evaluation parameters and the evaluation weights to obtain a scoring result;
[0032] S43: Obtaining the user's account tag, and correcting the scoring result according to the account tag to obtain a high-scoring video clip;
[0033] S44: Obtain the switching time point of the background music, and synthesize the high-scoring video clips into a video highlight according to the switching time point.
[0034] Preferably, the S43 includes:
[0035] S431: Acquire the account tag, wherein the account tag includes at least one of the following tags: a historical music tag, a historical special effect tag, and a historical video tag;
[0036] S432: Acquire a real-time tag corresponding to the real-time video sequence, wherein the real-time tag includes at least one of the following tags: a real-time video tag, a real-time music tag, and a real-time special effect tag;
[0037] S433: Correct the scoring result based on the similarity between the account tag and the real-time tag to obtain a high-scoring video clip.
[0038] In a second aspect, the present invention provides a device for intelligently editing wonderful videos of infants and young children, the device comprising:
[0039] A video sequence acquisition module is used to acquire real-time video sequences in infant capture scenes;
[0040] A feature extraction module is used to extract feature information from the real-time video sequence to obtain detection frame information and real-time video labels of a preset target;
[0041] A video segmentation module, used to segment the real-time video sequence into video segments corresponding to the target scene one by one according to the detection frame information and the real-time video label;
[0042] The video synthesis module is used to score each of the video clips, and synthesize the high-scoring video clips after post-processing to obtain a video highlight.
[0043] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method of the first aspect in the above-mentioned embodiment.
[0044] In a fourth aspect, an embodiment of the present invention further provides a storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method of the first aspect in the above-mentioned embodiment is implemented.
[0045] In summary, the beneficial effects of the present invention are as follows:
[0046] The present invention provides a method, device, equipment and storage medium for intelligently editing wonderful videos of infants and young children, the method comprising: obtaining a real-time video sequence in a scene of infant capture; extracting feature information in the real-time video sequence to obtain detection frame information and real-time video tags of a preset target; dividing the real-time video sequence into video clips corresponding to the target scene one by one according to the detection frame information and the real-time video tags; scoring each of the video clips, and synthesizing the high-scoring video clips after post-processing to obtain a video collection. The present invention uses computer vision technology to analyze the real-time video sequence in the scene of infant capture, automatically captures scenes of infants, young children, adults and other objects, and extracts relevant feature information, such as detection frame information and real-time video tags. This automated process can save a lot of time and labor costs, avoid manual operation, and by analyzing the detection frame information and real-time video tags, the scheme intelligently divides the real-time video sequence into video clips corresponding to the target scene one by one, ensuring that the final video collection only contains key scenes related to infants and young children, reducing irrelevant content, and after scoring each video clip, the scheme can select high-quality video clips according to the score. This helps create a personalized video collection that includes content that viewers may be more interested in. This improves the viewing experience and ensures that the content of the collection is more engaging. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for use in the embodiment of the present invention will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.
[0048] Figure 1This is a schematic diagram of the overall working process of the method for intelligently editing wonderful videos of infants and young children in Example 1 of the present invention;
[0049] Figure 2 This is a schematic diagram of a process for determining characteristic information in Embodiment 1 of the present invention;
[0050] Figure 3 This is a schematic diagram of a process for determining language and text information in Embodiment 1 of the present invention;
[0051] Figure 4 Schematic diagram of the process of segmenting a real-time video sequence in Embodiment 1 of the present invention;
[0052] Figure 5 Schematic diagram of a process of outputting adjacent video frames to corresponding video segments in Embodiment 1 of the present invention;
[0053] Figure 6 Schematic diagram of the process of video synthesis in Embodiment 1 of the present invention;
[0054] Figure 7 This is a schematic diagram of a process for determining a high-scoring video segment in Embodiment 1 of the present invention;
[0055] Figure 8 A schematic diagram of a process for determining the similarity between a real-time tag and a historical tag in Embodiment 1 of the present invention;
[0056] Fig. 9 It is a flowchart of processing the video sequence corresponding to the real-time privacy tag after the privacy tag is introduced into the account tag in Embodiment 2 of the present invention;
[0057] Fig.10 This is a structural block diagram of a device for intelligently editing wonderful videos of infants and young children in Embodiment 3 of the present invention;
[0058] Fig.11 It is a schematic diagram of the structure of an electronic device in Embodiment 4 of the present invention. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described in conjunction with the drawings in the embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. In the description of the present invention, it should be understood that the orientation or position relationship indicated by the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. In the absence of further restrictions, the elements defined by the phrase "comprising..." do not exclude the existence of other identical elements in the process, method, article or device comprising the elements. If there is no conflict, the embodiments of the present invention and the various features in the embodiments can be combined with each other, all within the protection scope of the present invention.
[0060] Example 1
[0061] See also Figure 1 Embodiment 1 of the present invention discloses a method for intelligently editing wonderful videos of infants and young children, the method comprising:
[0062] S1: Acquire real-time video sequences in infant capture scenes;
[0063] Specifically, in the infant snapshot scenario, parents or caregivers usually need to keep an eye on the safety and actions of infants. To achieve this goal, a care camera can be used to obtain real-time video sequences. These systems usually include components such as high-definition cameras, network connection devices, storage devices, and monitoring software. Parents or caregivers can use monitoring software to watch the activities and behaviors of infants at any time, such as sleeping conditions, feeding time, diaper changes, etc. This can help parents or caregivers better manage and care for infants and ensure their safety and health.
[0064] S2: extracting feature information from the real-time video sequence to obtain detection frame information and real-time video labels of a preset target;
[0065] Specifically, in real-time video sequences, in order to automatically identify or monitor certain specific targets or behaviors, it is necessary to use computer vision technology to extract feature information in the video, and process and analyze it. The feature information includes aspects such as detection frame information and text descriptions of scenes corresponding to the video sequence. Based on the feature information, different machine learning algorithms can be used to build models and detect and classify targets in video sequences.
[0066] In one embodiment, see Figure 2 , said S2 comprises:
[0067] S21: inputting the real-time video sequence into a target detection model, obtaining detection frame information of a preset target, and marking the preset target in the image to generate marked image data;
[0068] Specifically, for a large number of training image datasets related to infant care collected, use labeling tools such as Labelme to mark the common subjects of interest (adults, infants or objects) in the current scene with rectangular box selections to obtain a labeled dataset, and train the labeled dataset with a target detection network such as YOLOv6 to obtain a target detection model for the scene. In the target detection task, by inputting each frame of the real-time video sequence into the target detection model for processing, the infant's facial information, the infant's human figure information and the position information of the objects in the scene are obtained, and the above information is marked with a rectangular box to obtain the detection box information.
[0069] S22: inputting the annotated image data into a pre-trained multimodal large model to obtain language and text information corresponding to the question information in the annotated image data;
[0070] Specifically, based on the annotated image data, combined with the BLIP2 algorithm model, language and text information is obtained, and the BLIP2 algorithm model can be used to automatically generate scene text descriptions, reducing the workload and time of manual annotation, and can more efficiently generate scene text descriptions. Since the scene text description generated by the BLIP2 algorithm model is trained based on a pre-trained model and a large amount of data, the generated scene text description has a certain degree of normalization and standardization, which can improve the accuracy and consistency of the scene text description.
[0071] In one embodiment, see Figure 3 , the S22 comprises:
[0072] S221: Obtaining a question and answer instruction in the infant capture scene;
[0073] S222: Obtaining question information input by the user according to the question-answering instruction;
[0074] Specifically, the user's input of focus questions for specific scenarios is obtained. For example, for a snapshot scene with infants and young children as the focus subject, the focus question may be: "Can you describe where the baby or other people are doing in the picture?" In different specific scenarios, users have different needs for video editing. By obtaining the user's input of focus questions for specific scenarios, the user's real needs in the current care scenario can be understood, thereby ensuring that the edited video collection meets the user's preferences.
[0075] S223: Input the question information and the annotated image data into the multimodal large model to obtain the language and text information.
[0076] Specifically, first, a large amount of training image data related to the scene to be described is collected, and a corresponding scene text description is provided for each frame of the training image. The BLIP2 model is pre-trained using the above scene text description and training image data, and the pre-trained BLIP2 model is fine-tuned to a specific scene, such as an infant sleeping scene, an infant dining scene, etc., to obtain better performance in a specific scene. The annotated image data and the user's questions are input into the fine-tuned model to generate the corresponding answer. In this process, the model will use the detection box information in the annotated image data as the first feature vector, and the question information input by the user as the second feature vector, perform feature fusion on the first feature vector and the second feature vector, obtain a fusion vector, and obtain a natural language text description describing the image content as the language text information based on the fusion vector. A variety of scene text descriptions are generated through the BLIP2 algorithm model, and the scene is described from different angles and dimensions, which increases the richness and diversity of the scene text description. The BLIP2 algorithm model is applied to different snapshot scenes, and has a wide range of applicability and application scenarios. At the same time, the BLIP2 algorithm model is based on deep learning algorithms and big data training, and has certain intelligence and self-learning capabilities, which can continuously improve the accuracy and quality of scene text descriptions.
[0077] S23: According to the preset phrase category, the language and text information is segmented and counted to obtain the real-time video tag.
[0078] Specifically, the language and text information is segmented and counted, for example, the segmented words are statistically coded according to the four groups of infant description words, adult description words, behavior description words, and scene description words, wherein the relevant description words for infants and young children include: baby, kid, child, children, etc. These words can be coded with numbers, 00 represents no baby detected, 01 represents baby, 02 represents kid, 03 represents child, 04 represents children, and after the segmentation processing, the obtained classification words are used as the real-time video labels. By counting a large amount of language and text information, the segmentation processing can reduce redundant information, retain only key words, thereby reducing the amount of data and improving data processing efficiency. After the segmentation statistics, the key information and trends are more clearly displayed, thereby presenting the data results more intuitively.
[0079] S3: Segmenting the real-time video sequence into video segments corresponding to the target scenes one by one according to the detection frame information and the real-time video label;
[0080] Specifically, each video frame in the real-time video sequence is segmented in real time based on the extracted feature information, wherein the feature information includes: detection frame information and real-time video tags, and the real-time video sequence is segmented into video clips, wherein each of the video clips corresponds to a preset scene one by one, for example, the first video clip corresponds to an infant sleeping scene, and the second video clip corresponds to an infant dining scene. By segmenting the real-time video sequence into video clips corresponding to the target scene one by one according to the video segmentation rules, it is convenient for users to quickly view the video information corresponding to different scenes, and it is convenient for users to select the video clips of the corresponding scenes for synthesis according to their actual needs to obtain a wonderful video collection.
[0081] In one embodiment, see Figure 4 , said S3 comprises:
[0082] S31: Acquire adjacent video frames in the real-time video sequence, wherein the adjacent video frames include a first video frame and a second video frame;
[0083] S32: matching the first video frame and the second video frame according to the detection frame information and the real-time video tag, and outputting the real-time similarity between the first video frame and the second video frame;
[0084] Specifically, all video frames in the real-time video sequence are arranged in sequence to obtain an arranged list, and adjacent video frames in the list, such as the first video frame V1 and the second video frame V2, are obtained, and the first feature information of V1 and the second feature information of V2 are extracted. According to the similarity between the first feature information and the second feature information, the real-time similarity between the first video frame and the second video frame is obtained: the first weight corresponding to the detection frame information and the second weight corresponding to the video label are set. For example, if the similarity between the first detection frame information in V1 and the second detection frame information in V2 is x, the first weight is P1, the similarity between the first video label in V1 and the second video label in V2 is y, and the second weight is P2, then according to the formula: z=x*P1+y*P2, where z represents the similarity between the first feature information and the second feature information, 0<x, y, z<1 and 0<P1, P2<1, the real-time similarity between adjacent video frames is calculated. From the two dimensions of detection frame information and video label, combined with the weights set by the user according to the actual needs in different scenarios, the similarity between video frames is evaluated, and the accuracy of the similarity evaluation is guaranteed.
[0085] S33: obtaining a preset similarity threshold, and outputting the adjacent video frames to a video segment corresponding to the target scene according to the real-time similarity;
[0086] Specifically, for example, a preset similarity threshold of 0.85 is obtained, and based on the real-time similarity calculated in step S32, adjacent video frames with high similarity are divided into the same video segment, while video segments with low similarity are divided into different video segments. Since video segmentation is performed based on the difference between adjacent video frames, segmentation anomalies during scene switching are avoided.
[0087] In one embodiment, see Figure 5 , the S33 comprises:
[0088] S331: Acquire the first video frame and the second video frame, and output the first video frame to a first video segment, wherein the first video segment corresponds to a first scene in the target scene;
[0089] Specifically, for example, an infant wakes up from sleep and starts to play, if the scene corresponding to the first video frame is the infant sleeping scene, and the scene corresponding to the second video frame is the infant playing, then the first video frame is classified into the first video segment corresponding to the infant sleeping scene.
[0090] S332: if the real-time similarity is greater than the similarity threshold, outputting the second video frame to the first video segment;
[0091] Specifically, the real-time similarity obtained is 0.9. At this time, the real-time similarity is greater than the similarity threshold of 0.85, then it is considered that the similarity between the first video frame and the second video frame is high, that is, the overlap of the detection frame information and the video label is high, and the second video frame is also classified into the first video segment corresponding to the infant sleep scene.
[0092] S333: If the real-time similarity is not greater than the similarity threshold, outputting the second video frame to a second video segment, wherein the second video segment corresponds to a second scene different from the first scene.
[0093] Specifically, the real-time similarity is 0.7, and if the real-time similarity is not greater than the similarity threshold of 0.85, it is considered that the similarity between the first video frame and the second video frame is low, that is, the overlap of the detection frame information and the video label is low. At this time, based on the second detection frame information and the second video label, the second video frame is divided into the second video segment corresponding to the infant playing scene. By dividing each video segment into a video segment under the corresponding scene according to the level of real-time similarity, the abnormal problem of video segmentation is avoided when the scene switches suddenly.
[0094] S4: Score each of the video clips, and synthesize the high-scoring video clips after post-processing to obtain a video collection.
[0095] Specifically, the video clips are scored according to the preset evaluation parameters and evaluation weights in the video clips, and the video clips with higher scores are extracted, and background music and special effects are added to synthesize them to obtain a wonderful video collection.
[0096] In one embodiment, see Figure 6 , said S4 comprises:
[0097] S41: Obtaining preset evaluation parameters and evaluation weights, wherein the evaluation parameters include infant human figure parameters, infant face parameters, and scene parameters;
[0098] Specifically, evaluation parameters are obtained, and the evaluation parameters include at least: infant figure parameters, i.e., the completeness of the infant figure in the picture, infant face parameters, i.e., the completeness of the infant face, and scene parameters, i.e., the similarity between the current scene and the focus scene preset by the user, and the evaluation weights corresponding to the above evaluation parameters are obtained. The video clips are comprehensively scored through the evaluation parameters and the evaluation weights, which improves the excitement of the video highlights.
[0099] S42: performing weighted scoring on each of the video clips according to the evaluation parameters and the evaluation weights to obtain a scoring result;
[0100] S43: Obtaining the user's account tag, and correcting the scoring result according to the account tag to obtain a high-scoring video clip;
[0101] Specifically, the user's account tag is obtained, and the scoring result is corrected based on the account tag to obtain high-scoring video clips; this can increase the user's trust in the recommended video and improve the accuracy of the recommendation.
[0102] In one embodiment, see Figure 7 , the S43 comprises:
[0103] S431: Acquire the account tag, wherein the account tag includes at least one of the following tags: a historical music tag, a historical special effect tag, and a historical video tag;
[0104] S432: Acquire a real-time tag corresponding to the real-time video sequence, wherein the real-time tag includes at least one of the following tags: a real-time video tag, a real-time music tag, and a real-time special effect tag;
[0105] S433: Correct the scoring result based on the similarity between the account tag and the real-time tag to obtain a high-scoring video clip.
[0106] Specifically, the user's account tag is obtained: at least one of the following tags is extracted from the user's historical behavior: historical music tags, historical special effects tags, and historical video tags. These tags can reflect the user's interests and preferences. Then the real-time tags are obtained: real-time video tags, real-time music tags, and real-time special effects tags are extracted from the real-time video. These tags can reflect the content and characteristics of the current video. Similarity is calculated: by comparing the similarity between the account tag and the real-time tag, the attractiveness of the current video to the user is evaluated. Similarity can be calculated in a variety of ways, such as cosine similarity, Euclidean distance, etc. The higher the calculated similarity value, the more the current video meets the user's interests and preferences. The scoring results are corrected, and the scores of high-similarity video clips are adjusted to higher scores. Through the above-mentioned scoring correction process, the user's trust in the recommended video is increased, and the recommendation accuracy of the system is improved. Recommend high-scoring video clips: recommend high-scoring video clips that have been corrected to users to help them discover video content that is more in line with their interests and preferences.
[0107] In one embodiment, see Figure 8 , the S433 includes:
[0108] S4331: If the user's account tag is one of the following tags: historical music tag, historical special effect tag, and historical video tag, then according to the category of the account tag, matching the real-time tag of the corresponding category with the account tag, and outputting the similarity;
[0109] Specifically, suppose that the user's account tag includes "historical music tags" and the user is watching a newly uploaded video, which has real-time tags including "real-time music tags", "real-time special effects tags" and "real-time video tags". In this case, for the user's "historical music tags", the real-time tag "real-time music tags" is matched with the account tag, and the similarity is calculated. For example, it is found that the user's historical music tags are highly matched with the real-time music tags because they both involve music. This match can lead to a high similarity score, thereby affecting the rating result, making this video more likely to be liked by the user.
[0110] S4332: If the user account tag includes at least two of the following tags: historical music tag, historical special effects tag and historical video tag, then according to the category of the account tag, the real-time tag of the corresponding category is matched with the account tag, and multiple target similarities are output. According to the preset weight, each target similarity is weighted and the similarity is output.
[0111] Specifically, suppose that the user's account tags include "historical music tags" and "historical special effects tags", and the user is watching a newly uploaded video, which has real-time tags including "real-time music tags", "real-time special effects tags" and "real-time video tags". In this case, the two account tags of the user will be matched separately and the similarity will be calculated. For example, it is found that the user's historical music tags have a high match with the real-time music tags, and the user's historical special effects tags also have a high match with the real-time special effects tags. Then, the preset weights are used to weight the two similarity scores. If the music tag is considered more important, then it can give the music tag a higher weight, thereby affecting the final similarity score.
[0112] The purpose of steps S4331 and S4332 is to customize the user experience based on the similarity between the user's account tags and real-time tags, so that the recommended content is more in line with the user's interests and historical preferences. By considering the matching degree and weight between different tags, the scoring results can be adjusted more accurately to provide more personalized suggestions or recommendations.
[0113] S44: Obtain the switching time point of the background music, and synthesize the high-scoring video clips into a video highlight according to the switching time point.
[0114] Specifically, obtain the background music switching time point: analyze the background music and find the time point suitable for switching. For example, the switching time point can be determined by analyzing the waveform and rhythm of the background music track. Splice high-scoring video clips: splice the video clips according to the duration and switching time point of the high-scoring video clips so that a smooth transition can be made when the background music switches. The video clips can be sorted according to similarity, and the first few are selected for splicing. Add transition effects: in order to make the video transition more natural, you can add transition effects at the switching time point. Transition effects can be fade in and out, flash white, blur, etc. Adjust the video duration: adjust the video duration according to the length of the background music so that the length of the video and music match. This can keep the rhythm of the video and music consistent and make the entire video look smoother. Add subtitles and special effects: in order to enhance the expressiveness and attractiveness of the video, subtitles and special effects can be added to the video. Subtitles can display information such as the title and description of the video, and special effects can be color filters, special effects, etc. Output composite video: Output the composited video as a complete video file, which can be saved to a local device or uploaded to cloud storage for users to watch and share.
[0115] Example 2
[0116] In Example 1, the account tag includes at least one of the following tags: a historical music tag, a historical special effects tag, and a historical video tag. However, in actual wonderful video clips of infants and young children, there are often situations where infants and young children interact with their parents, the elderly, and pets. These interactive scenes have many cute and interesting elements, and therefore attract many viewers. However, the privacy issues related to these interactive scenes also need to be properly handled. Therefore, further introducing privacy tags in the account tags requires that the privacy issues related to these interactive scenes also need to be properly handled.
[0117] See also Fig. 9 , the S433 further includes:
[0118] S4333: Acquire training video sequences of infants and young children of multiple age groups interacting with preset objects, wherein the preset objects at least include: parents, elderly people, and pets;
[0119] Specifically, obtain training video sequences of infants and young children of multiple age groups interacting with preset objects, wherein the preset objects include at least: parents, the elderly and pets, formulate a detailed data collection plan, including how many videos to collect, the length of each video, the interaction situations covered, etc., contact families, nurseries, nursing homes or other appropriate institutions to obtain participants of infants and young children of multiple age groups, parents, the elderly and pets, and ensure that they agree to appear in the video and interact with other objects, use high-definition cameras to record videos in the daily environment of the interactive objects, ensure that the quality and resolution of the recorded videos are high enough to support subsequent deep learning model training, and ensure that the videos cover infants and young children of different ages, as well as a variety of different interaction situations, such as infant-parent interaction, infant-elderly interaction, pet-infant interaction, etc.
[0120] S4334: using a labeling tool to label preset objects in the training video sequence, and outputting a labeled target video sequence;
[0121] Specifically, use annotation tools, including open source tools (such as LabelImg, VGG Image Annotator) and commercial tools (such as Labelbox, Supervisely). Make sure that the selected tool supports video annotation functions. Import the training video sequence into the annotation tool. Usually, the annotation tool supports multiple video formats. Create a project or task and assign a task to each video sequence. Annotate each video sequence frame by frame. According to the needs, you can annotate infants, parents, the elderly, and pets. Use the functions of the annotation tool, such as drawing bounding boxes, adding key points, or marking the identity of the object. For each annotated object, make sure to assign it the correct timestamp to indicate the time range of the object's appearance and disappearance. This is very important for subsequent video clips. The annotation tool usually allows the export of annotation data. Make sure to save the annotated video sequence and the corresponding label data in a suitable file format for future use. Check the accuracy and consistency of the annotation regularly. If there are multiple annotators, perform cross-validation to ensure consistency. Repair or modify the annotation data as needed. Use the export function of the annotation tool to output the annotated video sequence as a target video sequence. This sequence should contain the annotated objects and their corresponding timestamp information. The selection of annotation tools and the quality control of the annotation process have an important impact on the training and performance of subsequent deep learning models. Therefore, we must be careful during the annotation process and ensure the accuracy and completeness of the annotation data.
[0122] S4335: Inputting the target video sequence into a deep learning model for training, and outputting the trained model as an interactive object recognition model;
[0123] Specifically, the target video sequence annotated in the above steps is used as training data, and the training data is split into a training set, a validation set, and a test set. Usually, the training set is used for model training, the validation set is used to adjust model hyperparameters and monitor model performance, and the test set is used for final model evaluation. According to the requirements of the task and the complexity of the data, an appropriate deep learning model architecture is selected, such as a convolutional neural network (CNN), a recurrent neural network (RNN), or a more advanced model such as Transformer, etc., and the selected deep learning model is constructed, including an input layer, a hidden layer, an output layer, and corresponding activation functions, loss functions, etc. Consider using pre-trained models (such as transfer learning) to accelerate model training and improve performance.
[0124] Develop a data loader to effectively load and process training data, including at least reading image frames, processing labels, and applying data augmentation techniques. Data preprocessing includes normalization, data augmentation (such as random cropping, flipping, rotation), and encoding labels into appropriate formats. Use the training set to train the deep learning model. During the training process, monitor the performance of the model on the validation set so that hyperparameters can be adjusted in time to prevent problems such as overfitting, and select appropriate optimization algorithms (such as stochastic gradient descent) and learning rate strategies. Use the test set to evaluate the performance of the final trained model. The evaluation indicators include at least accuracy, precision, recall, F1 score, etc. The specific indicators depend on the task requirements. Deploy the trained model to practical applications for interactive object recognition tasks.
[0125] S4336: Inputting the real-time video sequence into the interactive object recognition model, and outputting the real-time interactive objects in the real-time video sequence;
[0126] Specifically, obtain an interactive object recognition model that has been successfully trained and saved, use a deep learning framework (such as TensorFlow, PyTorch, etc.) to load the trained model, obtain a real-time video sequence, divide the video into a series of frames, usually processed at frames per second (fps), apply the same preprocessing steps as the training model to each frame, including image normalization, data enhancement, etc., and send each frame to the loaded interactive object recognition model for inference. The model interprets the output of each frame as the identification or probability of a real-time interactive object. According to the output of the model, determine the interactive objects and their location information in each frame. After identifying the interactive objects in each frame, synchronize the identification of the interactive objects with the time of the video according to the timestamp or frame rate information so as to know the time range of the appearance and disappearance of the interactive objects, and output the real-time interactive objects: add the identified real-time interactive object identification or location information to the video to realize the visualization of the real-time interactive objects, display the real-time interactive objects by drawing bounding boxes, labels or other forms of identification on the video, and take different actions according to the identification results of the real-time interactive objects. This may include automatic editing, real-time alerts, data recording, etc., depending on the application requirements.
[0127] S4337: Acquire a real-time privacy tag corresponding to the real-time interactive object in the real-time video sequence according to the real-time interactive object;
[0128] Specifically, based on the identification of the real-time interactive object, the real-time privacy label corresponding to the real-time interactive object in the real-time video sequence is automatically obtained to enhance privacy protection and content management. For example, when the interactive object is identified as an elderly person, given that the elderly are usually reluctant to share the interactive process, the privacy label can be set to "Sensitive-Elderly Privacy", which will trigger privacy protection measures, such as blurring or automatic editing to exclude the interactive process. Similarly, when the interactive object is identified as a mother, if a private picture of the mother feeding appears, the privacy label can be set to "Sensitive-Mother Feeding", thereby enabling corresponding privacy measures, such as shielding or blurring these pictures to respect the mother's privacy preferences. Through the setting of these privacy labels, it intelligently adapts to different interactive objects and privacy needs, providing users with a safer and more protected experience.
[0129] S4338: Obtain the historical privacy tag in the account tag, match the real-time privacy tag with the historical privacy tag, and when they match, process the video sequence corresponding to the real-time privacy tag.
[0130] Specifically, historical privacy label data is obtained from the user's account or profile. These labels record the user's previous privacy preferences and processing methods. Identify real-time interactive objects and real-time privacy labels: Use the interactive object recognition model to identify real-time interactive objects and obtain real-time privacy labels according to the method described above. Compare and match the real-time privacy labels with the historical privacy labels in the user's account. If there is a match, it means that the user has expressed similar privacy preferences before. The real-time video sequence can be processed according to the processing method of the historical labels. According to the processing method of the historical privacy labels, the real-time video sequence is processed accordingly. The processing methods may include: blurring: blurring sensitive content to protect privacy; clipping: deleting or hiding video clips containing sensitive information; shielding: replacing sensitive information with blur or placeholders to prevent sensitive content from being displayed; notifying users: sending notifications to users based on historical privacy labels to ensure that they understand the processing methods. Monitor the processed video sequences to ensure that the privacy protection measures are effective, provide feedback to users to inform them how the real-time privacy labels affect their content, and record the processing process and user feedback to continuously improve and optimize the privacy protection system. Through this process, real-time video sequences are intelligently processed based on the user's historical privacy preferences to provide content that is more in line with the user's privacy needs. This helps improve user satisfaction while protecting their privacy.
[0131] Example 3
[0132] See also Fig.10 Embodiment 2 of the present invention further provides a device for intelligently editing wonderful videos of infants and young children, the device comprising:
[0133] A video sequence acquisition module is used to acquire real-time video sequences in infant capture scenes;
[0134] A feature extraction module is used to extract feature information from the real-time video sequence to obtain detection frame information and real-time video labels of a preset target;
[0135] A video segmentation module, used to segment the real-time video sequence into video segments corresponding to the target scene one by one according to the detection frame information and the real-time video label;
[0136] The video synthesis module is used to score each of the video clips, and synthesize the high-scoring video clips after post-processing to obtain a video highlight.
[0137] Specifically, the intelligent editing device for the wonderful videos of infants and young children in the embodiment of the present invention is adopted, and the device includes: a video sequence acquisition module, which is used to obtain a real-time video sequence in a scene of infant capture; a feature extraction module, which is used to extract feature information in the real-time video sequence to obtain detection frame information and real-time video labels of preset targets; a video segmentation module, which is used to segment the real-time video sequence into video clips corresponding to the target scenes one by one according to the detection frame information and the real-time video labels; a video synthesis module, which is used to score each of the video clips, and synthesize the high-scoring video clips after post-processing to obtain a video highlight. This device uses computer vision technology to analyze real-time video sequences in infant capture scenes, automatically captures scenes of infants, adults and other objects, and extracts relevant feature information, such as detection frame information and real-time video tags. This automated process can save a lot of time and labor costs and avoid manual operations. By analyzing the detection frame information and real-time video tags, the solution intelligently divides the real-time video sequence into video clips that correspond to the target scene one by one, ensuring that the final video collection only contains key scenes related to infants and reduces irrelevant content. After scoring each video clip, the solution can select high-quality video clips based on the score. This helps create a personalized video collection that contains content that the audience may be more interested in. This can improve the viewing experience and ensure that the content of the collection is more attractive.
[0138] Example 4
[0139] In addition, combined Figure 1 The method for intelligently editing wonderful videos of infants and young children described in Example 1 of the present invention can be implemented by an electronic device. Fig.11 A schematic diagram of the hardware structure of an electronic device provided in Embodiment 4 of the present invention is shown.
[0140] The electronic device may include a processor and a memory storing computer program instructions.
[0141] Specifically, the processor may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present invention.
[0142] The memory may include a large capacity memory for data or instructions. By way of example and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory may include a removable or non-removable (or fixed) medium. Where appropriate, the memory may be inside or outside a data processing device. In a particular embodiment, the memory is a non-volatile solid-state memory. In a particular embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.
[0143] The processor implements any one of the intelligent editing methods for wonderful videos of infants and young children in the above-mentioned embodiments by reading and executing computer program instructions stored in the memory.
[0144] In one example, the electronic device may further include a communication interface and a bus. As shown in 11, the processor, the memory, and the communication interface are connected via the bus and communicate with each other.
[0145] The communication interface is mainly used to implement communication between the modules, devices, units and / or equipment in the embodiments of the present invention.
[0146] The bus includes hardware, software or both, and the parts of the device are coupled to each other. For example, but not limitation, the bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In suitable cases, the bus may include one or more buses. Although the embodiment of the present invention describes and shows a specific bus, the present invention considers any suitable bus or interconnection.
[0147] Example 5
[0148] In addition, in combination with the method for intelligently editing wonderful videos of infants and young children in the above embodiment 1, embodiment 5 of the present invention can also provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the methods for intelligently editing wonderful videos of infants and young children in the above embodiments is implemented.
[0149] In summary, the embodiments of the present invention provide a method, device, equipment and storage medium for intelligently editing wonderful videos of infants and young children.
[0150] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.
[0151] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.
[0152] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.
[0153] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.
Claims
1. A method for intelligently editing wonderful videos of infants and young children, characterized in that: The method comprises: S1: Acquire real-time video sequences in infant capture scenes; S2: extracting feature information from the real-time video sequence to obtain detection frame information and real-time video labels of a preset target; S3: Segmenting the real-time video sequence into video segments corresponding to the target scenes one by one according to the detection frame information and the real-time video label; S4: scoring each of the video clips, and synthesizing the high-scoring video clips after post-processing to obtain a video collection; The S4 includes: S41: Obtaining preset evaluation parameters and evaluation weights, wherein the evaluation parameters include infant human figure parameters, infant face parameters, and scene parameters; S42: performing weighted scoring on each of the video clips according to the evaluation parameters and the evaluation weights to obtain a scoring result; S43: Obtaining the user's account tag, and correcting the scoring result according to the account tag to obtain a high-scoring video clip; S44: obtaining a switching time point of the background music, and synthesizing the high-scoring video clips into a video highlight according to the switching time point; The S43 includes: S431: Acquire the account tag, wherein the account tag includes at least one of the following tags: a historical music tag, a historical special effect tag, and a historical video tag; S432: Acquire a real-time tag corresponding to the real-time video sequence, wherein the real-time tag includes at least one of the following tags: a real-time video tag, a real-time music tag, and a real-time special effect tag; S433: Correct the scoring result based on the similarity between the account tag and the real-time tag to obtain a high-scoring video clip.
2. The method for intelligently editing wonderful videos of infants and young children according to claim 1, characterized in that: The S2 includes: S21: inputting the real-time video sequence into a target detection model, obtaining detection frame information of a preset target, and marking the preset target in the image to generate marked image data; S22: inputting the annotated image data into a pre-trained multimodal large model to obtain language and text information corresponding to the question information in the annotated image data; S23: According to the preset phrase category, the language and text information is segmented and counted to obtain the real-time video tag.
3. The method for intelligently editing wonderful videos of infants and young children according to claim 2, characterized in that: The S22 includes: S221: Obtaining a question and answer instruction in the infant capture scene; S222: Obtaining question information input by the user according to the question-answering instruction; S223: Input the question information and the annotated image data into the multimodal large model to obtain the language and text information.
4. The method for intelligently editing wonderful videos of infants and young children according to claim 1, characterized in that: The S3 includes: S31: Acquire adjacent video frames in the real-time video sequence, wherein the adjacent video frames include a first video frame and a second video frame; S32: matching the first video frame and the second video frame according to the detection frame information and the real-time video tag, and outputting the real-time similarity between the first video frame and the second video frame; S33: Obtain a preset similarity threshold, and output the adjacent video frames to a video segment corresponding to the target scene according to the real-time similarity.
5. The method for intelligently editing wonderful videos of infants and young children according to claim 4, characterized in that: The S33 includes: S331: Acquire the first video frame and the second video frame, and output the first video frame to a first video segment, wherein the first video segment corresponds to a first scene in the target scene; S332: if the real-time similarity is greater than the similarity threshold, outputting the second video frame to the first video segment; S333: If the real-time similarity is not greater than the similarity threshold, outputting the second video frame to a second video segment, wherein the second video segment corresponds to a second scene different from the first scene.
6. An intelligent editing device for wonderful videos of infants and young children, characterized in that: The device comprises: A video sequence acquisition module is used to acquire real-time video sequences in infant capture scenes; A feature extraction module is used to extract feature information from the real-time video sequence to obtain detection frame information and real-time video labels of a preset target; A video segmentation module, used to segment the real-time video sequence into video segments corresponding to the target scene one by one according to the detection frame information and the real-time video label; A video synthesis module is used to score each of the video clips, and synthesize the high-scoring video clips after post-processing to obtain a video collection; The video clips are scored, and high-scoring video clips are post-processed and synthesized to obtain a video collection including: Acquiring preset evaluation parameters and evaluation weights, wherein the evaluation parameters include infant human figure parameters, infant face parameters, and scene parameters; Performing a weighted score on each of the video clips according to the evaluation parameters and the evaluation weights to obtain a score result; Obtaining the user's account tag, and correcting the rating result based on the account tag to obtain a high-rated video clip; Obtaining the switching time point of the background music, and synthesizing the high-scoring video clips into a video highlight according to the switching time point; The step of obtaining the user's account tag and correcting the scoring result according to the account tag to obtain a high-scoring video clip includes: Acquire the account tag, wherein the account tag includes at least one of the following tags: a historical music tag, a historical special effect tag, and a historical video tag; Acquire a real-time tag corresponding to a real-time video sequence, wherein the real-time tag includes at least one of the following tags: a real-time video tag, a real-time music tag, and a real-time special effect tag; Based on the similarity between the account tags and the real-time tags, the scoring results are corrected to obtain high-scoring video clips.
7. An electronic device, characterized in that: include: At least one processor, at least one memory and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, implement the method according to any one of claims 1 to 5.
8. A storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.
