Image processing method, electronic device and storage medium

Abstract

Disclosed in the present application are an image processing method, an electronic device and a storage medium. The method comprises: acquiring image data generated by a camera device; performing attribute recognition on the image data, so as to obtain an attribute recognition result; and, on the basis of the attribute recognition result, generating instruction information used for human-computer interaction. In this way, before, during, and / or after video capturing by a camera device, a user can automatically acquire instruction information used for human-computer interaction, and perform subsequent operations on the camera device on the basis of the instruction information, thereby enhancing the real-time performance of interaction, facilitating timely elimination of device failures of camera devices and removal of unusable footage generated by camera devices, and further achieving the effects of saving storage space of camera devices and / or reducing capturing time.

Description

Image processing methods, electronic devices and storage media Technical Field

[0001] This application relates to the field of video shooting, and more particularly to an image processing method, electronic device, and storage medium. Background Technology

[0002] During video recording, due to limitations in storage space and recording time, there is a need to remove invalid or discarded segments (also known as unusable footage) that do not meet the final video production requirements. Related technologies often require filtering and removing unusable footage from a large number of video clips during post-production editing, which is cumbersome and time-consuming. Furthermore, it is difficult to meet the real-time requirements for unusable footage identification, thus failing to effectively save storage space and / or recording time on the recording equipment.

[0003] Summary of the Invention

[0004] In view of this, embodiments of this application provide an image processing method, an electronic device, and a storage medium, which aim to effectively identify unusable footage generated during video shooting and save storage space and / or shooting time of the shooting device.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] In a first aspect, embodiments of this application provide an image processing method, including:

[0007] Acquire image data generated by the imaging device;

[0008] The image data is subjected to attribute recognition to obtain the attribute recognition results;

[0009] Based on the attribute recognition results, instruction information for human-computer interaction is generated.

[0010] Secondly, embodiments of this application provide an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, is configured to:

[0011] Acquire image data generated by the imaging device;

[0012] The image data is subjected to attribute recognition to obtain the attribute recognition results;

[0013] Based on the attribute recognition results, instruction information for human-computer interaction is generated.

[0014] Thirdly, embodiments of this application provide a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect of embodiments of this application.

[0015] The technical solution provided in this application embodiment acquires image data generated by a shooting device; performs attribute recognition on the image data to obtain attribute recognition results; and generates instruction information for human-computer interaction based on the attribute recognition results. In this way, users can automatically acquire instruction information for human-computer interaction before, during, and / or after video shooting, and perform further operations on the shooting device based on this instruction information. This enhances the real-time nature of the interaction, facilitates timely troubleshooting of equipment malfunctions, and removes unusable footage generated by the shooting device, thereby saving storage space and / or shooting time. Attached Figure Description

[0016] Figure 1 is a schematic flowchart of the image processing method according to an embodiment of this application;

[0017] Figure 2 is a schematic diagram of the first prompt message in an application example of this application;

[0018] Figure 3 is a schematic diagram of the second prompt information in an application example of this application;

[0019] Figure 4 is a schematic diagram of the interactive options after video recording in an application example of this application;

[0020] Figure 5 is a flowchart illustrating the intelligent human-computer interaction of the shooting device in the application embodiment of this application;

[0021] Figure 6 is a flowchart illustrating a video editing method according to an embodiment of this application;

[0022] Figure 7 is a schematic diagram of the database construction stage in the application embodiment;

[0023] Figure 8 is a schematic diagram of the principle of the personalized editing stage in the application embodiment of this application;

[0024] Figure 9 is a flowchart illustrating a video editing method according to another embodiment of this application;

[0025] Figure 10 is a schematic diagram illustrating the principle of the video editing method in the application embodiment of this application;

[0026] Figure 11 is a schematic diagram of the two-stage video editing process in the application embodiment of this application;

[0027] Figure 12 is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0028] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0030] In related technologies, in order to remove unusable footage generated during the shooting process, all unusable footage in the video is usually removed uniformly through hardware background algorithms or cloud algorithms after the shooting process is completed. This results in the inability to remove unusable footage in real time during video shooting, i.e., a lack of real-time removal of unusable footage. In addition, it is impossible to intelligently identify problems such as lens pollution and obstruction before shooting, to promptly remind users that the current shooting content is unusable during shooting, and to provide users with an interactive option to stop shooting. Consequently, it is impossible to effectively save the storage space and / or shooting time of the shooting equipment.

[0031] Based on this, in various embodiments of this application, by performing attribute recognition on the image data generated by the shooting device, and generating instruction information for human-computer interaction based on the attribute recognition results, the shooting device can be controlled by human-computer interaction based on the instruction information, which enhances the real-time performance of waste film removal and can effectively save the storage space and / or shooting time of the shooting device.

[0032] This application provides an image processing method that can be applied to a shooting device, as shown in Figure 1. The method includes:

[0033] Step 101: Acquire image data generated by the shooting device.

[0034] Here, the acquired image data can be image data generated before, during and / or after the camera device shoots video, that is, image data corresponding to at least one of the above.

[0035] Step 102: Perform attribute recognition on the image data to obtain the attribute recognition result.

[0036] Here, the shooting device can be pre-configured with a recognition algorithm to perform attribute recognition on the image data, obtaining the attribute recognition result. This recognition algorithm can be an AI (artificial intelligence) algorithm that supports understanding and attribute recognition of image data, such as, but not limited to, image matching and recognition models and image-text multimodal models. Attribute recognition is determined based on image feature recognition of the image data. Different image features are pre-learned to correspond to attribute categories, and subsequent processing is determined based on these attribute categories. Attribute categories can include: occlusion categories, image blur categories, shakiness categories, dirt categories, and other attribute categories corresponding to image anomalies. In another embodiment, attribute categories can be further divided into: unusable image categories, highlight categories, etc. The specific division method can be customized as needed.

[0037] For example, the step of performing attribute recognition on the image data to obtain attribute recognition results includes:

[0038] The image content in the image data is identified to obtain image features;

[0039] The attribute recognition result is determined based on the correspondence between image features and attribute categories.

[0040] Understandably, the shooting device can perform feature extraction processing on the image content of the image data based on a preset recognition algorithm to obtain image features, and determine the attribute recognition result based on the correspondence between the image features and the predefined attribute categories, thereby supporting the classification of image data into attribute categories based on image content. For example, it can classify image anomalies, classify whether the video clip corresponding to the image data is a defective film, and classify whether the video clip corresponding to the image data is a highlight clip.

[0041] Step 103: Generate instruction information for human-computer interaction based on the attribute recognition results.

[0042] Here, the attribute recognition result can characterize the attribute category corresponding to the image data. In this way, the camera device can generate and output instruction information based on the attribute category, so that the user can perform the next operation of the shooting device based on the instruction information.

[0043] Understandably, users can automatically obtain instruction information for human-computer interaction before, during, and / or after video recording, and perform further operations on the recording device based on this instruction information. This enhances the real-time nature of the interaction, facilitates timely troubleshooting of equipment malfunctions, and removes unusable footage generated by the recording device, thereby saving storage space and / or recording time.

[0044] For example, generating instruction information for human-computer interaction based on the attribute recognition result includes:

[0045] If, based on the attribute recognition results, it is determined that there is an image anomaly in the image data, a first prompt message is generated indicating that the shooting device should be checked.

[0046] Here, the shooting device can perform attribute recognition on the image data generated before or during video shooting based on the recognition algorithm, and generate the first prompt message indicating that the shooting device needs to be checked based on the attribute recognition result. In this way, the user can be reminded to check the shooting device in a timely manner before or during the actual video shooting, so as to avoid the phenomenon of shooting unusable footage due to lens obstruction, blurry image, shaking, lens dirt, etc., thereby minimizing the invalid shooting time and saving the storage space of the shooting device.

[0047] Here, the attribute recognition results generated by the shooting device for the image data include at least one of the following attribute categories: occlusion category, image blur category, shake category, and dirt category. If, based on the attribute recognition results, it is determined that the first image data contains at least one of the following categories: occlusion category, image blur category, shake category, and dirt category, for example, if it is recognized that the image data is abnormal due to lens dirt, lens breakage, or lens blur, a first indication message can be generated to promptly remind the user to check and avoid affecting the shooting process. For example, the first indication message can include at least one of the following: a screen pop-up, voice reminder, or a specific color flashing indicator light. For instance, in one application example, as shown in Figure 2, the first indication message can be a screen pop-up displaying "The current lens is contaminated or obstructed; please check before shooting," which can remind the user to clean or replace the lens cover before video shooting to avoid affecting the shooting.

[0048] For example, the attribute recognition result includes an attribute category and a confidence level corresponding to the attribute category. Before generating the first prompt message indicating that the shooting device should be checked, the method further includes:

[0049] The confidence level of the attribute category corresponding to the image anomaly is determined to be greater than or equal to a set confidence threshold.

[0050] In one application example, the attribute recognition result includes the attribute category and the confidence level corresponding to the attribute category. When the shooting device determines that the image data has at least one of the categories of occlusion, blur, shaking, and dirt, it compares the confidence level of the corresponding attribute category with a set confidence level threshold. If the confidence level is determined to be greater than or equal to the set confidence level threshold, a first prompt message instructing the shooting device to be checked is generated. In this way, the probability of false alarms of the first prompt message can be reduced based on the confidence level threshold, thereby improving the accuracy of the first prompt message and enhancing the user experience.

[0051] It should be noted that the confidence threshold set above can be set based on the user's personal preferences or based on the shooting scene, and this application embodiment does not limit this.

[0052] For example, the image data is image data corresponding to video segments generated by the shooting device during video recording; the generation of instruction information for human-computer interaction based on the attribute recognition result includes:

[0053] If the video clip is determined to be invalid based on the attribute recognition result, the current video recording will be stopped and / or the video clip will be deleted, or a second prompt message will be generated indicating that the video recording will be stopped and / or the video clip will be deleted.

[0054] Here, the shooting device can perform attribute recognition on the image data generated during video recording. If the video clip is determined to be unusable based on the attribute recognition results, the shooting device can automatically stop the current video recording and / or delete the identified unusable clip, or generate a second prompt message instructing the user to stop video recording and / or delete the identified unusable clip, allowing the user to decide whether to stop video recording and / or delete the identified unusable clip. For example, in one application example, as shown in Figure 3, the second prompt message could be a pop-up window on the shooting device's screen displaying "The previous recorded clip may be unusable; please confirm whether to delete it." This can remind the user to delete unusable clips during video recording, thereby saving storage space on the shooting device and improving the efficiency of post-production editing.

[0055] Understandably, based on the above interactive control, the shooting device can identify unusable footage generated during the shooting process in real time, thereby saving storage space and facilitating the selection of materials for post-production video, reducing the time required for sorting and creation.

[0056] For example, the image data is the image data corresponding to each video segment after the shooting device completes video shooting; the generation of instruction information for human-computer interaction based on the attribute recognition result includes:

[0057] Based on the attribute recognition results corresponding to each video segment, the video segment is determined to be a discarded video segment, and a third prompt message indicating that the video segment is a discarded video segment is generated;

[0058] The method further includes:

[0059] Based on the third prompt information, a waste film deletion instruction is obtained, and at least a portion of the waste films in the set of waste films are deleted.

[0060] Here, after video recording is complete, the shooting equipment can acquire image data corresponding to each video segment, perform attribute recognition on each image data, obtain attribute recognition results for each video segment, and perform unified deletion of unusable footage based on these results. For example, users can delete all unusable footage with a single click based on a third-party prompt, or select specific unusable footage to delete, deleting only a portion of the selected footage. This allows for comprehensive processing after shooting, saving storage space on the shooting equipment and facilitating post-production material selection, thus saving time on organization and creation.

[0061] For example, generating instruction information for human-computer interaction based on the attribute recognition result includes:

[0062] Based on the attribute recognition results corresponding to each video segment, the highlight moments in the video segments are determined, and a fourth prompt message indicating the highlight segments is generated;

[0063] The method further includes:

[0064] Based on the fourth prompt information, obtain the instruction to extract the highlight segment, and extract the highlight segment corresponding to the highlight moment.

[0065] Here, after video recording is complete, the shooting device can acquire image data corresponding to each video segment, perform attribute recognition on each image data, obtain attribute recognition results for each video segment, and extract highlight segments uniformly based on the attribute recognition results of each video segment. For example, users can extract highlight segments based on the fourth prompt information. In this way, the overall processing of video segments after shooting can be achieved, saving time on organization and creation.

[0066] In one application example, the shooting device can initiate the aforementioned unified process of deleting unwanted footage and / or generating a set of highlight clips based on the user's human-computer interaction. For example, the user can initiate the attribute recognition action after video recording using methods such as button input, voice input, touch input, or text input. This triggers the shooting device to acquire image data of the recorded video clips; perform attribute recognition on each image data to obtain attribute recognition results; and perform unwanted footage deletion and / or generate a set of highlight clips based on the attribute recognition results corresponding to each video clip. For example, as shown in Figure 4, in one application example, the screen of the shooting device can display the options "One-click deletion of unwanted footage" and "One-click generation of a set of highlight clips" after video recording is completed. The user can select either option based on human-computer interaction, thereby triggering the unwanted footage deletion and / or generating a set of highlight clips.

[0067] Here, a highlight segment refers to a segment that has visual impact, emotional tension, or is a key plot point in the narrative, which can attract the audience's attention and leave a deep impression. The attribute recognition result includes the attribute label corresponding to the highlight segment.

[0068] For example, the step of performing waste clip deletion processing and / or generating a set of highlight clips based on the attribute recognition results corresponding to each of the video clips includes:

[0069] Based on the attribute recognition results, if the video segment is determined to be a discarded segment, a discarded segment identifier is marked, and a set of discarded segments corresponding to the discarded segment identifier is output. Based on the obtained discarded segment deletion instructions, at least a portion of the discarded segments in the discarded segment set are deleted; and / or,

[0070] Based on the attribute recognition results, if the video segment is determined to be a highlight segment, a highlight identifier is marked, and the set of highlight segments corresponding to the highlight identifier is output.

[0071] Here, the discard deletion command can be a first command input by the user to delete all discarded videos, and the shooting device can delete all video segments in the discarded video set based on this first command. Alternatively, the discard deletion command can be a second command to delete selected discarded videos from the discarded video set based on this second command. This allows the shooting device to manually delete discarded videos from the discarded video set based on human-computer interaction.

[0072] For example, generating instruction information for human-computer interaction based on the attribute recognition result includes:

[0073] Based on the attribute recognition results corresponding to each video segment, the time of the discarded segment in the video segment is determined, and a fifth prompt message indicating the discarded sub-segment is generated;

[0074] The method further includes:

[0075] Based on the fifth prompt, at least some of the invalid video segments are deleted from each of the aforementioned video segments.

[0076] It is understood that the attribute recognition results of this application embodiment may also include the invalid clip moments in the video segment. In this way, the user can delete the invalid clip moments in the video segment based on the fifth prompt information that prompts the invalid clip moments, thereby enabling invalid clip processing at a smaller granularity and improving the accuracy of invalid clip deletion.

[0077] For example, the attribute recognition result includes: the recognition result of at least one attribute category selected from occlusion attribute, jitter attribute, exposure attribute, and meaningless attribute; the step of determining the video clip as a defective piece based on the attribute recognition result includes:

[0078] If the identification result of at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is yes, then the corresponding video segment is determined to be a defective video.

[0079] Here, a "yes" result for the occlusion attribute indicates that the video clip is invalid due to screen occlusion; a "yes" result for the jitter attribute indicates that the video clip is invalid due to screen jitter; a "yes" result for the exposure attribute indicates that the video clip is invalid due to overexposure or underexposure; and a "yes" result for the meaninglessness attribute indicates that the video clip is invalid due to meaninglessness.

[0080] Understandably, the shooting device can identify at least one of the following attributes of image data: occlusion attribute, shaking attribute, exposure attribute, and meaningless attribute, and determine the video clip as a defective piece based on the identification result of at least one of them, thus achieving automatic identification of defective pieces.

[0081] For example, the attribute recognition result further includes the confidence level corresponding to the attribute category, and determining that the recognition result of at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is yes includes:

[0082] If the confidence level of at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is greater than or equal to the corresponding confidence threshold, then the identification result is determined to be yes.

[0083] Here, by introducing a confidence level and a corresponding confidence threshold, defective films with low confidence levels (i.e., below the confidence threshold) can be recalled, thereby effectively improving the accuracy of defective film identification. The confidence threshold can be set based on user needs or based on the shooting scenario of the shooting device; this embodiment does not limit this setting.

[0084] For example, the attribute recognition result further includes: the recognition result of scene attributes, and the method further includes:

[0085] Based on the recognition results of the scene attributes and the established mapping relationship, a confidence threshold for at least one attribute category among the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is determined; wherein, the established mapping relationship includes the correspondence between the confidence thresholds under each scene attribute.

[0086] Here, the shooting device can also identify scene attributes of image data based on recognition algorithms, and determine the confidence threshold of each of the aforementioned attributes based on the recognition results of scene attributes and the set mapping relationship. In this way, the confidence threshold of each attribute can be automatically updated based on the shooting scene of the shooting device, thereby improving the accuracy of rejecting images.

[0087] For example, scene attributes can include, but are not limited to: everyday scenes, first-person motion scenes, third-person motion scenes, underwater scenes, etc. The shooting device can automatically update the confidence threshold of each attribute based on the identified current scene attributes, thereby improving the accuracy of identifying unusable footage.

[0088] For example, the attribute recognition result further includes: the recognition result of the highlight attribute and the corresponding confidence level; the step of determining the video clip as a defective clip based on the attribute recognition result further includes:

[0089] If the identification result of at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is determined to be yes, and the identification result of the highlight attribute is determined to be yes, then based on the confidence level corresponding to the highlight attribute and the confidence level corresponding to at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute, it is determined whether the video clip is a defective clip.

[0090] Here, a positive result for the highlight attribute indicates that the video segment is the aforementioned highlight segment. In this embodiment, the identified unusable segments can also be recalled based on the highlight attribute. For example, if a video segment is determined to be unusable based on the recognition results of at least one of the aforementioned occlusion attribute, jitter attribute, exposure attribute, and meaningless attribute, and simultaneously determined to be a highlight segment based on the highlight attribute, then whether the video segment is unusable can be determined based on the confidence level corresponding to the highlight attribute and the confidence level corresponding to at least one of the occlusion attribute, jitter attribute, exposure attribute, and meaningless attribute. For example, if the confidence level corresponding to the unusable segment (i.e., the confidence level corresponding to at least one of the occlusion attribute, jitter attribute, exposure attribute, and meaningless attribute) is less than or equal to the confidence level corresponding to the highlight attribute, then unusable segment recall is performed; if the confidence level corresponding to the unusable segment is greater than the confidence level corresponding to the highlight attribute, then the video segment is determined to be unusable, thereby improving the accuracy of unusable segment identification.

[0091] Exemplarily, the method further includes:

[0092] The confidence threshold of at least one of the following attribute categories—occlusion attribute, jitter attribute, exposure attribute, and meaningless attribute—is configured based on the human-computer interaction operation.

[0093] It is understood that, in the embodiments of this application, the confidence threshold of at least one of the attribute categories of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute can be configured based on human-computer interaction operations, thereby meeting the personalized setting requirements for waste film identification.

[0094] For example, the step of performing attribute recognition on the image data to obtain attribute recognition results includes:

[0095] The image data is used to perform attribute recognition based on a pre-trained image-text multimodal model to obtain attribute recognition results;

[0096] The image-text multimodal model is configured with predefined attribute categories and corresponding text attribute definition information. The image-text multimodal model is used to encode the image data to obtain image encoding features, and output attribute recognition results based on the image encoding features and the text encoding features of the text attribute definition information.

[0097] It is understandable that the recognition algorithm pre-installed on the shooting device can be a pre-trained image-text multimodal model. Since the shooting device needs to perform attribute recognition on image data, this image-text multimodal model can achieve cross-modal semantic understanding. It can then perform cross-modal understanding based on the image encoding features of the image data and the text encoding features of the predefined attribute category corresponding to the text attribute definition information, thereby obtaining the attribute recognition result. Thus, the attribute category of the image data can be defined based on the text attribute definition information. Traditional image / video-image / video matching models often require matching against an image / video standard library (covering all image / video content in multiple cases for each type). During inference, they require the hardware to pre-store all image / video encoding content of the entire standard library. Based on this, the image-text multimodal model introduced in this application only requires pre-storing the unique text description encoding for each attribute type in the hardware, improving the convenience of hardware storage and retrieval. Furthermore, compared to previous classification models that extract features from videos and use classifiers for judgment, which fix the number of model categories and discrimination criteria after training, the image-text multimodal model introduced in this application can not only modify the current attribute discrimination criteria by directly modifying the text description encoding for discrimination, but also easily expand and add the discrimination ability of other attributes, thus having the characteristics of easy expansion and adjustment.

[0098] For example, the image data is multi-frame image data obtained by extracting frames from a video segment of a set duration, and the step of image encoding the image data to obtain image encoding features includes:

[0099] Based on the image encoder, the multi-frame image data is encoded as a whole to obtain the image coding features of the multi-frame image data, or...

[0100] The image encoder encodes each frame of the multi-frame image data separately to obtain the image coding features of each frame.

[0101] Here, each video segment can be divided based on a set duration, and the image data corresponding to each video segment can be multi-frame image data obtained by the shooting device after extracting frames from the video segment.

[0102] In one application example, the camera can segment captured video images in real time at fixed durations (e.g., three seconds). For each fixed-duration video segment, frames are extracted at a fixed ratio. The extracted set of images is used as input to the model data and fed into an image feature encoder for image feature extraction. Two extraction methods can be selected: Method 1, extracting unique image features from all extracted frames in the set, thus obtaining unique image coding features for multiple frames; Method 2, extracting image features from each frame of all extracted frames in the set, thus obtaining image coding features for each frame of the extracted images.

[0103] For example, the text encoding feature output attribute recognition result based on the image encoding features and the text attribute definition information includes:

[0104] Based on the similarity between the image coding features of the multi-frame image data or the image coding features of each frame image data and the text coding features of each attribute category, the attribute recognition result is output; or,

[0105] Based on the autoregressive generator, the image coding features of the multi-frame image data or the image coding features of each frame image data and the text coding features of each attribute category are processed to output the attribute recognition results.

[0106] Here, the discrimination methods for the image-text multimodal model include two types: similarity discrimination and autoregressive prediction.

[0107] The principle of similarity determination is as follows:

[0108] A pre-trained image-text multimodal model is used. This model includes an image encoder for generating image-coded features and a text encoder for generating text-coded features. Leveraging its matching ability between image and text modalities (for the same image and its corresponding text description, the feature values ​​of the image-coded features extracted by the image encoder and the text-coded features extracted by the text encoder should be consistent), the text features corresponding to the text encoder in the example above are pre-extracted and stored internally. Any video frame is input, and the image encoder extracts image-coded features (which can be unique image-coded features or image-coded features from each frame of image data). Using the pre-trained image-text multimodal model, the similarity between the pre-stored text features and the real-time input and extracted image features is calculated. The similarity score determines whether the current video segment possesses the text attribute, and the similarity score is converted into a confidence score (if it is a unique image-coded feature input, the result is directly output; if it is image-coded features from each frame of image data, the average of the matching results for each frame is used as the final output).

[0109] The principle of autoregressive prediction is as follows:

[0110] A pre-trained image-text multimodal autoregressive model (equipped with an image encoder, text encoder, and autoregressive generator) is used. The image encoder and text encoder are the same as described above; this section focuses on the autoregressive generator. The autoregressive generator receives image feature input and text feature input, where image features represent the query object and text features represent the query (or inquiry) content. It outputs specific attribute discriminations on the input image / video data through a unified input and output format, thereby performing attribute discrimination.

[0111] It should be noted that in the image-text multimodal model, the text attribute definition information corresponding to the attribute category needs to be predefined. The text attribute definition information corresponding to each attribute category is input into the text encoder, which extracts the text encoding features and uses them as a fixed evaluation standard for image data.

[0112] In one application example, the text attribute definition information for each attribute category is shown below:

[0113] [Occlusion Attributes]: Consider whether the occluded area exceeds the preset size of the image (e.g., one-third). Consider whether it affects the display of the main content. Does the obstruction protrude or affect the display of the segment? Occlusion is divided into lens obstruction, such as an object touching or resting on the camera lens, causing the shooting scene to be blocked. It also includes scenes where objects obstruct or affect the complete display of the subject during shooting, usually located between the lens and the subject. First-person handheld shooting, hugging and other interactive behaviors, as well as intentional tracking shots should not be considered occlusion.

[0114] [Shake Attributes]: Rapid vibration, slight vertical and horizontal rotation of the image, especially rapid shaking, and irregular camera flips should be considered video quality shake. Irregular shaking or shaking to a certain extent reduces the watchability of the video. Shots with definite movement should not be considered shake. Camera movement subjectively operated by the photographer should not be considered shake. Shots with definite actions or behaviors should not be considered shake.

[0115] [Exposure Attributes]: Determine if the video is overexposed (too bright), underexposed (too dark), and if there are colored horizontal stripes, colored vertical stripes, green stripes, pink screen, or purple screen.

[0116] [Meaningless Attributes]: This includes meaningless elements such as extending and retracting the selfie stick, adjusting the camera, looking at the camera, using the phone / computer, working, chatting, walking on the street, talking to the camera, and static scenes during the filming process. Meeting videos, narration, walking, and live streams without clear activities are all considered meaningless. Videos containing monotonous scenes such as the ground, sky, ceiling, pure darkness, snow, and walls are invalid and meaningless.

[0117] [Highlight Attributes]: Does the video clip include actions such as holding hands, hugging, kissing, cycling, fishing, playing basketball, skiing, playing games, skateboarding, swimming, etc., and are these actions particularly exciting? Capturing exciting moments of action and interaction is a highlight. Videos of playing on a phone, chatting, or talking to the camera should not be considered highlights.

[0118] In one application example, the format of the attribute recognition results output by the image-text multimodal model is defined as follows: [Occlusion]: Yes (0.75); [Jitter]: No; [Exposure]: No; [Meaningless]: Yes (0.8); [Highlight]: Yes (0.9); where each attribute is limited to either Yes or No; each attribute is separated by a semicolon, and the number after 'Yes' represents the confidence level of the result, ranging from 0 to 1. A score of 0 indicates that the 'Yes' result is completely unreliable, while a score of 1 indicates that it is absolutely reliable. If the answer is 'No', no score is required.

[0119] The image processing method of the application embodiment will be described by way of example below with reference to an application example.

[0120] In this application embodiment, the recognition algorithm built into the shooting device employs a pre-trained image-text multimodal model, such as CLIP (Contrastive Language-Image Pre-training), VICLIP (Video CLIP), or variations thereof. CLIP is a multimodal (text and image) pre-trained model developed by OpenAI. The CLIP model learns how to compare text and images, thereby achieving cross-modal understanding. This contrastive learning method allows CLIP to learn the semantic relationships between text and images without any supervised labels. Based on this image-text multimodal model, this embodiment can effectively understand the content of the captured image in the current frame, intelligently identify and filter video footage with occlusion, shakiness, blurriness, or meaninglessness, and provide corresponding confidence scores. This embodiment can perform overall recognition of video segments or extract frames from the video for recognition; no limitation is made here.

[0121] For example, the AI ​​algorithm's model analyzes and encodes the input frame-by-frame video, and directly returns the confidence score (i.e., confidence level) of the current attribute (occlusion, jitter, blur, meaningless content such as boredom, and highlights) through the model, giving the specific score in the form of a question and answer. The following is an example, that is, firstly, it is determined whether the input video has the problem. If it does, a flag "Yes" is given and the specific confidence score is returned. Otherwise, "No" is returned: "attribute_caption_useless":"[Occlusion]:No;[Shaking or Jittering]:Yes(0.8);[Exposure]:No;[Edited]:No;[Meaningless]:No;[Highlight]:No;", that is, the useless label attributes: "[Occlusion]:No;[Jitter]:Yes(0.8);[Exposure]:No;[Edited]:No;[Meaningless]:No;[Highlight]:No;".

[0122] It should be noted that in this application embodiment, a recall algorithm can be set to identify whether a video clip is a defective clip. Specifically, it can include the following two recall methods:

[0123] 1) Set the confidence threshold for discarded items. By default, a discarded item is considered discarded when it shows "Yes" (Score = 0.0). You can set the sensitivity of recognition, for example, only when the score is greater than 0.5 is it considered discarded. That is, only when the confidence score is greater than the set confidence threshold of 0.5 is it considered discarded, thereby recalling some discarded items with low confidence.

[0124] 2) Recall is performed by comprehensively comparing the scores of highlight attributes and defective attributes. When the score of highlight attributes is greater than or equal to the score of defective attributes, recall is performed.

[0125] In practical applications, the shooting device can automatically recall based on the recall algorithm, or remind the user to interact and recall through human-computer interaction information.

[0126] In this application embodiment, as shown in Figure 5, the intelligent human-computer interaction of the shooting device includes the following steps:

[0127] Step 501: Human-computer interaction before shooting begins.

[0128] The shooting device can acquire image data before the actual video recording after powering on. Based on the AI ​​algorithm model, it performs attribute recognition and judges the occlusion attribute. If the occlusion attribute is determined to be yes, or the confidence level of yes is greater than or equal to the confidence level threshold, the user is notified of the current camera occlusion through human-computer interaction (such as screen pop-up, voice reminder, flashing of a specific color indicator light, etc.). This makes it easier to remind the user to clean or replace the lens cover in time before shooting to avoid affecting the shooting.

[0129] Step 502: Capture the human-computer interaction in progress.

[0130] Here, by identifying whether a captured video clip is a "discarded clip," the smart hardware camera product (i.e., the shooting device) can interact with the user to remind them or stop shooting in advance. The interaction can be illustrated with the following example flow: immediately stopping video recording upon detecting a discarded clip; or, after detecting a discarded clip, notifying the user through product interaction (such as a screen pop-up, voice reminder, or a specific color flashing indicator light) that the current captured image has been identified as a meaningless segment by the algorithm, allowing the user to decide whether to stop shooting. Users can set the discarded clip recognition threshold according to their preferences, or set a discarded clip recognition threshold for specific scenarios, such as first-person motion scenes, to reduce the sensitivity of discarded clip recognition in those specific scenarios. This discarded clip recognition process not only saves camera storage space but also facilitates post-production material selection, saving time on organization and creation.

[0131] Step 503: Human-computer interaction after shooting.

[0132] Here, by uniformly identifying whether all captured video content is "unusual footage" and "highlights," the smart hardware camera product can interact with the user. An example of this interaction is as follows: users can use function buttons, voice, or text to instruct the smart hardware camera product to remove "unusual footage" and generate "highlights" with a single click. Users can interactively select from all video content provided by the hardware product, deleting unnecessary "unusual footage" and retaining the best "highlight" moments. This identification process not only saves camera storage space but also facilitates post-production material selection, saving time on organization and creation.

[0133] In an exemplary embodiment, this application also provides a video editing method, which allows an electronic device to perform video editing on the image data after it has been captured, thereby obtaining a personalized edited video.

[0134] As shown in Figure 6, this video editing method includes:

[0135] Step 601: Obtain the structured text data of the video clip to be edited.

[0136] Here, the video clip to be edited can be video data shot by the user, such as video data obtained by using the aforementioned image processing method, or video data downloaded from the Internet, or a combination of both. This application embodiment does not limit this. The video clip to be edited can be understood as a collection of multiple video clips, and each video clip can be understood as an image sequence composed of multiple frames.

[0137] In this embodiment, after receiving the video clip to be edited, the electronic device can convert the video data into corresponding structured text data using a video frame description model. Here, structured text data refers to structured data that describes the video using text. In one embodiment, the structured text data includes a text description; in another embodiment, in addition to the text description, the structured text data also includes attribute data (i.e., text attributes). For example, the electronic device can perform text conversion processing on the video clip to be edited based on a pre-built AI (artificial intelligence) algorithm to obtain structured text data. The AI ​​algorithm can employ a multimodal image recognition model or a GPT (Generative Pre-Trained Transformer) model, which can automatically identify the text description and text attributes of the video data. This embodiment does not limit this. The text description corresponds to the screen content of the video clip, and the text attributes correspond to various attribute descriptions of the video clip. These various attributes include, but are not limited to, a discard attribute indicating whether the video clip is unusable, a highlight attribute indicating whether the video clip is a highlight clip, and a theme attribute indicating the subject of the video clip.

[0138] Step 602: Obtain editing instructions that represent personalized editing needs.

[0139] Here, the electronic device can receive editing instructions input by the user. For example, obtaining the editing instructions that characterize personalized editing needs includes:

[0140] Acquire user voice or text data based on natural language input;

[0141] Convert the voice data or text data into editing instructions in a set format.

[0142] Understandably, users can generate editing instructions through voice or text input, which is different from the traditional operation of dragging and dropping the video timeline. This simplifies the user's interaction and supports accepting voice or text data based on natural language input, making the input of editing instructions highly scalable and facilitating the expansion of more interactive functions in the future.

[0143] In one embodiment, after obtaining a pre-edited video, the user can browse and watch the preliminary video. If the user is not satisfied with the preliminary video, they can adjust it through voice interaction. Adjustments include adding or deleting segments, or changing the order of segments. Adjusting video content through voice interaction greatly simplifies the user's operation and helps improve user satisfaction.

[0144] Step 603: Based on the editing instructions and the structured text data, adjust the video segment to be edited to obtain the personalized edited target video.

[0145] It is understood that, after obtaining the structured text data of the video segment to be edited, the embodiments of this application can obtain personalized editing instructions through human-computer interaction. Based on the editing instructions and the structured text data, the video segment to be edited is adjusted and processed to obtain the personalized edited target video. This simplifies the user's operation of adjusting the final video and can also be adjusted and processed based on the structured text data, which greatly saves computing overhead and improves video editing efficiency.

[0146] For example, the step of adjusting the video segment to be edited based on the editing instructions and the structured text data to obtain a personalized edited target video includes:

[0147] Based on the editing instructions and the structured text data, determine the adjustment position and adjustment action that need to be adjusted;

[0148] The video segment to be edited is adjusted based on the adjustment action and the adjustment position to obtain a personalized edited target video.

[0149] Understandably, electronic devices can determine the adjustment positions and actions required within the video segment to be edited based on the acquired editing instructions and structured text data of the video segment. Then, based on these adjustment positions and actions, the electronic device performs adjustments to the video segment to obtain a personalized edited target video. Here, the adjustment position can be the target video segment to be adjusted, and the adjustment action can be the corresponding adjustment action in the editing instructions.

[0150] For example, the editing instructions include one or more of the following: instructions to add video segments, instructions to delete video segments, instructions to adjust the order of segments, and instructions to undo.

[0151] Understandably, the "Add Video Segment" command allows users to add desired video segments to the video to be edited; the "Delete Video Segment" command allows users to delete desired video segments from the video to be edited; the "Adjust Segment Order" command allows users to adjust the narrative logic of the video to be edited; and the "Undo" command allows users to revert their editing interactions. This satisfies users' flexible and quick personalized editing needs.

[0152] For example, the structured text data of the video segment to be edited includes: multiple structured text segments, each structured text segment corresponding to a video segment number; determining the adjustment position and adjustment action based on the editing instructions and the structured text data; and adjusting the video segment to be edited based on the adjustment action and the adjustment position to obtain a personalized edited target video, including:

[0153] Based on the editing instructions, the target sequence number of the structured text fragment that needs to be adjusted is determined from the plurality of structured text fragments;

[0154] The arrangement sequence of the structured text fragments is redefined based on the adjustment actions indicated in the editing instructions and the target sequence number.

[0155] The arrangement sequence is mapped to the corresponding video segment to be edited, and the video segment to be edited is edited based on the arrangement sequence to obtain the personalized edited target video.

[0156] Understandably, the video clip to be edited includes multiple video clips. Correspondingly, the structured text data of the video clip to be edited includes multiple structured text fragments. Each structured text fragment corresponds to a video clip number, meaning there is a one-to-one correspondence between structured text fragments and video clips, and each video clip has a corresponding number. Thus, based on editing instructions, the target number of the structured text fragment that needs to be adjusted can be determined from multiple structured text fragments. That is, the adjustment position is determined by the target number, and then the arrangement sequence of the structured text fragments is re-determined by the adjustment action and the target number, resulting in the adjusted arrangement order of the video clips. Based on each number in this arrangement sequence, it can be mapped to the corresponding video clip, thereby obtaining the personalized edited target video.

[0157] For example, the video clips to be edited include at least one of: unthemed video clips and themed video clips.

[0158] Here, the video clip to be edited can be obtained by processing the initial video material data by an electronic device. For example, it can produce at least one of thematic and unthemed video clips. This can enrich the diversity of video editing and better meet the personalized editing needs of users.

[0159] Exemplarily, the method further includes:

[0160] Obtain the structured text fragments corresponding to each initial video segment;

[0161] The structured text segments corresponding to each initial video segment are arranged to generate unthemed video segments and / or themed video segments.

[0162] Here, the structured text fragments of each initial video segment include: attribute data of the video segment; wherein, the attribute data includes at least topic attributes, which include but are not limited to the following topic categories: scene, event, character, time, etc., wherein each topic category includes at least one topic attribute, so that video segments with different topic attributes can be divided based on the attribute data.

[0163] For example, based on the structured text segments corresponding to each initial video segment, the following steps are taken to generate themeless video segments and / or theme-based video segments:

[0164] Clustering is performed on the attribute data of each initial video segment to obtain at least one video segment with a theme.

[0165] Understandably, electronic devices can use clustering algorithms to cluster the attribute data of each initial video segment to obtain at least one themed video segment. This at least one themed video segment can serve as a subsequent editing option for the user. That is, the user can choose a video segment without a theme or any themed video segment as the video segment to be edited, thus better meeting the user's personalized editing needs.

[0166] For example, before obtaining the structured text data of the video segment to be edited, the method further includes:

[0167] The original video footage data is converted into multiple initial video segments based on a set duration;

[0168] Based on the pre-trained model, structured text segments of each initial video segment are generated, and the structured text segments of each initial video segment are stored in the database based on their serial numbers;

[0169] The process of obtaining structured text data for the video clip to be edited includes:

[0170] Based on the sequence number of the video segment to be edited, the structured text segment of the video segment to be edited is read from the database.

[0171] Here, the electronic device can first convert the acquired raw video footage data into corresponding structured text segments based on video fragments and store them in the database, thus achieving text-based storage of video data. For example, the raw video footage data can be converted into multiple initial video segments divided based on a set duration (e.g., 3 seconds). Based on a pre-trained model (i.e., the aforementioned video frame description model), structured text segments for each of the initial video segments are generated, and these structured text segments are stored in the database based on their sequence numbers. When the user subsequently edits the video, they only need to input the sequence number of the video segment to be edited. The electronic device can then read the structured text segment of the video segment to be edited from the database based on the sequence number. In this way, video editing can be performed using text data, saving data overhead during video editing.

[0172] For example, the pre-trained model is a text-image multimodal model, wherein the text-image multimodal model is configured to use the text-image multimodal matching model to encode the image data of the initial video segment to obtain image encoding features, and output structured text data based on the image encoding features.

[0173] For example, the editing instruction is an instruction to add a video segment, and the method further includes:

[0174] The first target video segment to be added is determined from the database based on the keywords in the instruction to add the video segment; wherein the database stores structured text segments of each video segment based on the video segment's sequence number.

[0175] The step of determining the target sequence number of the structured text segment that needs adjustment from the plurality of structured text segments based on the editing instructions includes:

[0176] Based on the sequence number of the first target video segment, the target sequence number of the structured text segment that needs to be adjusted is determined from the plurality of structured text segments.

[0177] Here, if the editing instruction is to add a video segment, the electronic device can search the database for the sequence number of the first target video segment to be added based on the keywords in the instruction. For example, if the user inputs the editing instruction "add a screen viewing segment", the electronic device can search the database for the structured text segment "screen viewing" as the first target video segment. Then, based on the sequence number of the first target video segment, the target sequence number of the structured text segment to be adjusted is determined from the sequence numbers of each structured text segment of the video segment to be edited. This results in a newly determined arrangement sequence of structured text segments. For example, the sequence numbers are deduplicated, and the first target video segment is added to the video segment to be edited based on the temporal relationship to obtain a new arrangement sequence of structured text segments. Then, the sequence numbers in the arrangement sequence are mapped to the corresponding video segments, and the video segments are merged to obtain the personalized edited target video.

[0178] For example, the editing instruction is an instruction to delete a video segment, and the step of determining the target sequence number of the structured text segment that needs to be adjusted from the plurality of structured text segments based on the editing instruction includes:

[0179] Based on the editing instructions, search for the second target video segment to be deleted from the plurality of structured text segments;

[0180] Based on the sequence number of the second target video segment, the target sequence number of the structured text segment that needs to be adjusted is determined from the plurality of structured text segments.

[0181] Here, if the editing instruction is to delete a video segment, the electronic device can search for the sequence number of the second target video segment to be deleted from multiple structured text segments based on the keywords in the instruction. For example, if the user inputs the editing instruction "add cable car segment", the electronic device can search for the structured text segment containing "cable car" as the second target video segment in the structured text segment of the video segment to be edited. Then, based on the sequence number of the second target video segment, the target sequence number of the structured text segment to be adjusted is determined from the sequence numbers of multiple structured text segments of the video segment to be edited. This results in a newly determined arrangement sequence of structured text segments, that is, the target sequence number is deleted from the original arrangement sequence of the video segment to be edited, resulting in a new arrangement sequence of structured text segments. The sequence number in the arrangement sequence is then mapped to the corresponding video segment, and the video segments are merged to obtain the personalized edited target video.

[0182] For example, the editing instruction is an instruction to adjust the order of segments, and the step of determining the target sequence number of the structured text segment to be adjusted from the plurality of structured text segments based on the editing instruction includes:

[0183] Based on the editing instructions, a third target video segment whose segment order needs to be adjusted is searched from the plurality of structured text segments;

[0184] Based on the sequence number of the third target video segment, the target sequence number of the structured text segment that needs to be adjusted is determined from the plurality of structured text segments.

[0185] Here, if the editing instruction is to adjust the order of segments, the electronic device can search for the sequence number of the third target video segment whose order needs to be adjusted from multiple structured text segments based on the keywords in the instruction. For example, if the user inputs the editing instruction "put the cable car in front", the electronic device can search for the structured text segment containing "cable car" as the third target video segment from the structured text segments of the video segment to be edited. Then, based on the sequence number of the third target video segment, the target sequence number of the structured text segment to be adjusted is determined from the sequence numbers of multiple structured text segments of the video segment to be edited. This results in a newly determined arrangement sequence of structured text segments. That is, the sequence number of the third target video segment is moved to the beginning of the sequence from the original arrangement sequence of the video segment to be edited, resulting in a new arrangement sequence of structured text segments. The sequence number in the arrangement sequence is then mapped to the corresponding video segment, and the video segments are merged to obtain the personalized edited target video.

[0186] Exemplarily, the method further includes:

[0187] The arrangement sequence corresponding to the editing instructions stored in history;

[0188] Obtain a retraction instruction that indicates the cancellation of the previous editing instruction, and based on the retraction instruction, return the edited video corresponding to the arrangement sequence before the execution of the previous editing instruction.

[0189] Here, the electronic device can also store the arrangement sequence corresponding to the editing instructions that the user has performed. If the user is not satisfied with the video corresponding to the current editing instruction, he / she can enter a reversal instruction. The electronic device can then return to the editing video corresponding to the previous arrangement sequence based on the stored arrangement sequence. In this way, the user can choose to revert to the editing video corresponding to any editing instruction, which helps to improve editing efficiency and enhance the user's editing experience.

[0190] The video editing method of this application embodiment will be described in further detail below with reference to an application example.

[0191] This application embodiment provides a personalized video editing method. Referring to Figures 7 and 8, the method includes a database construction stage and a personalized editing stage. Figure 7 corresponds to the database construction stage, and Figure 8 corresponds to the personalized editing stage.

[0192] Referring to Figure 7, the database construction stage includes converting the original video into structured text data, obtaining preliminary cuts and multiple theme candidates based on the structured text data, and storing the corresponding structured text data in a database, waiting for personalized editing to be called.

[0193] For example, cutting-edge AI algorithms, such as those based on multimodal large models like GPT, can be used to understand the content of multiple video segments in the original video. This allows the video segments to be converted into structured text fragments that include structured text descriptions and attribute data. For instance, a 654-second video source file shot by a user can be converted into 218 video fragments, each with a duration of 3 seconds. The corresponding structured text fragments include: fragment number (clip_id), fragment start time (start_time), fragment end time (end_time), fragment description (raw_caption), attribute description for topic clustering (attribute_caption_useful), and attribute description for removing unusable segments (attribute_caption_useless).

[0194] For example, a story can be arranged based on structured text fragments of each video segment to obtain a preliminary cut, and multiple corresponding topic candidates can be obtained based on text clustering. These text structure data are stored in a database for further use. For example, the segment number (i.e., the arrangement sequence) corresponding to the preliminary cut can be represented as [2, 11, 12, 3, 26, 27, 43, 52, 54, 55, 56, 57, 60, 71, 72, 76, 77, 123, 137, 140, 141, 143, 144, 161, 178, 185, 189, 195]. The candidate fragment numbers for the beach theme are [2,12,26,54,55,56,71,72,76,123,137,140,141,143]; the candidate fragment numbers for the coffee brewing theme are [86,90,99]; and the candidate fragment numbers for the cable car theme are [178,180,182,185,186,189,190,192,195,196].

[0195] Referring to Figure 8, the personalized editing stage includes the following steps:

[0196] 1) Natural language input;

[0197] In this application embodiment, instructions for adjusting the final video are given in the form of natural language (e.g., voice or text input). The input editing instructions can be processed by a central control agent model (e.g., GPT), which determines the specific task corresponding to the input instruction. For example, if the text contains "add a clip," the central control agent model determines the task as retrieval and inserts the retrieved clip into the appropriate position in the original video. Similarly, candidate tasks could include deleting clips, adjusting the narrative order, etc. The design structure of this application embodiment allows for arbitrary text input, thus possessing good scalability and adapting to more interactive needs.

[0198] 2) Perform the task;

[0199] After the central control agent model decides to execute a task, the executor (GPT) retrieves available information from the database maintained in Figure 2 to adjust the final content and output a new orchestration structure. Interaction variables are stored in the user's individual interaction history cache file.

[0200] 3) Interactive video production;

[0201] The electronic device maps the arrangement structure back to video segments and renders them, then outputs the finished video to the user.

[0202] It should be noted that the electronic device can perform multiple rounds of iterations based on steps 1) to 3) above. That is, the user can input new interactive commands based on the returned video results until a satisfactory output is obtained. If the user is not satisfied with the result of a certain round of interaction, they can also choose to revert to any previous interaction (based on the interaction history cache file).

[0203] 4) Output the final product.

[0204] Understandably, when the user stops interacting with the input, the electronic device outputs the final video.

[0205] For example, during any interval between multiple rounds of user command interaction, users can also choose to manually add or delete segments to meet their needs for manually adjusting the final video.

[0206] In one application example, assuming the video clip to be edited is the preliminary cut shown in Figure 7, and the user's natural language input command is "delete the cable car clip," then the search for the clip corresponding to the cable car theme in the preliminary cut's clip numbers yields clip numbers [178, 189, 195]. Deleting the cable car theme clip numbers from the preliminary cut's clip numbers results in the following arrangement sequence: [2, 11, 12, 3, 26, 27, 43, 52, 54, 55, 56, 57, 60, 71, 72, 76, 77, 123, 137, 140, 141, 143, 144, 161, 185]. Mapping this arrangement sequence to the corresponding video clips yields the target video for editing. Thus, in automatic editing, a video editing algorithm based on natural language interaction simplifies the user's operation of adjusting the final cut, increases the controllability of the final cut adjustments, and improves the user's efficiency in organizing and creating content.

[0207] In an exemplary embodiment, this application also provides a video editing method, in which an electronic device can perform video editing on the image data after capturing the image data to obtain an edited video that conforms to the personalized story arrangement logic.

[0208] As shown in Figure 9, this video editing method includes:

[0209] Step 901: Obtain the video footage data to be edited.

[0210] Here, the acquired video footage data to be edited can be video data shot by the user, such as video data obtained by using the aforementioned image processing method, or video data downloaded from the internet, or a combination of both. This application embodiment does not limit this. The video footage data can be understood as a collection of multiple video segments, and each video segment can be understood as an image sequence composed of multiple frames.

[0211] Step 902: Determine the story arrangement logic.

[0212] Here, story arrangement logic can be understood as the structural framework of video editing. It is used to combine with the textual content of the video footage data to construct the editing script, thereby obtaining a video with natural shot transitions. Story arrangement logic includes one or more of the following: story arrangement, beginning and ending, story focus, shooting intention, camera movement, transitions, special effects, theme, and events. Among them, [story arrangement] refers to the overall story logic and rhythm arrangement of the video, such as what the beginning, development, and ending of the video present, and the overall focus of the video; [beginning and ending] indicates whether the video has a special beginning and ending, such as a quick review of the overall content at the beginning, or a beginning and end echoing each other, or a landscape shot with blank space at the end, etc. [story focus] indicates the video style, such as showing a scene of sports, a family outing, or beautiful scenery, etc. [shooting intention] indicates what the shooting intention of the video segment is, such as wanting to shoot a video of sports. [camera movement] indicates what kind of shooting method the video segment uses, such as first-person or third-person, following or fixed shooting, etc. [transitions and special effects] indicates the transition effects in the video segment; [theme] indicates the theme around which the video segment revolves, etc.

[0213] In one embodiment, the story arrangement logic can be determined based on a template. For example, users can upload their favorite video clips as templates, and the logic can be determined based on learning from these templates. Alternatively, multiple story arrangement logics can be preset for users to choose from, or user-input story arrangement logic can be directly accepted.

[0214] Step 903: Based on the story arrangement logic, the video material data is edited to obtain an edited video.

[0215] It is understood that the embodiments of this application can edit video material data based on a determined story arrangement logic, which can make the logical continuity and natural connection of video segments in the edited video, that is, the shot connection of the edited video is natural, thereby effectively improving the video quality of the edited video.

[0216] For example, the determination of story arrangement logic includes:

[0217] Obtain reference video clips uploaded by users and use the reference video clips as editing reference templates;

[0218] The editing reference template is analyzed, and the story arrangement logic is determined based on the analysis results.

[0219] Here, the electronic device can acquire reference video clips uploaded by the user as editing templates. The device analyzes these templates and determines the story arrangement logic required by the user based on the analysis results. Furthermore, it can determine the story arrangement logic for the current edit based on the user's interactive operations, thus meeting the user's personalized editing needs. Specifically, learning is performed based on the user-uploaded reference video clips. The algorithm divides the template video into segments of preset duration (e.g., 5 seconds), then extracts frames from all the segments and inputs them into a pre-trained attribute recognition model to obtain all attributes of the segment, including [event, shooting intention, camera movement, theme], etc. Alternatively, the entire user-uploaded template video can be input into a holistic attribute recognition model to generate overall video attributes, including [story arrangement, beginning and ending, story focus]. All of these attributes are output as text by the algorithm.

[0220] For example, the step of analyzing the clip reference template and determining the story arrangement logic based on the analysis results includes:

[0221] Perform overall video attribute recognition on the editing reference template to obtain the overall video attribute recognition result;

[0222] The story arrangement logic is determined based on the overall attribute recognition results of the video.

[0223] Here, the electronic device can perform overall video attribute recognition on the editing reference template based on a pre-set AI (artificial intelligence) algorithm, obtain the overall video attribute recognition result, and determine the story arrangement logic based on the overall video attribute recognition result. The AI ​​algorithm can employ a multimodal image-text recognition model or a GPT (Generative Pre-Trained Transformer) model, which can automatically recognize the overall video attribute recognition result of the video data; this embodiment does not limit this approach.

[0224] For example, the overall video attribute recognition result includes one or more of the following: a first descriptive text representing the story arrangement, a second descriptive text representing the story beginning and / or the story ending, and a third descriptive text representing the story's focus. In one application example, the overall video attribute recognition result includes the aforementioned first descriptive text, second descriptive text, and third descriptive text, thus determining the story arrangement logic, including the story arrangement, the story beginning and / or the story ending, and the story's focus.

[0225] For example, story arrangement can represent the overall narrative logic and pacing of the video, such as what is presented at the beginning, development, and end, and the overall focus of the video. The story beginning and / or ending can indicate whether the video has a special opening and / or closing, such as a quick overview of the overall content at the beginning, a beginning-and-end echoing each other, or a scenic shot with minimal background at the end. Story focus can represent the video's style, such as a significant portion of the video needing to highlight action scenes, family outings, or beautiful scenery.

[0226] In some embodiments, determining the story arrangement logic includes:

[0227] Receive story arrangement constraint information input by the user, and determine story arrangement logic based on the story arrangement constraint information; or...

[0228] The story arrangement logic is determined based on the preset arrangement logic.

[0229] In one application example, an electronic device can determine story arrangement logic based on story arrangement constraints input by the user. Here, the story arrangement constraints input by the user can be the content of voice or text input. The story arrangement constraints can include at least one of the aforementioned story arrangement, story beginning and / or story ending, and story emphasis. In other words, the electronic device can determine one or more of the aforementioned story arrangement, story beginning and / or story ending, and story emphasis based on the content of the user's voice or text input, and then generate story arrangement logic.

[0230] In another application example, if the electronic device does not obtain the story arrangement constraint information input by the user, nor does it obtain the reference video clip uploaded by the user, the electronic device can also determine the story arrangement logic of the current video clip based on the preset story arrangement logic. For example, the story arrangement logic of the previous video clip can be saved as the default story arrangement logic. In the case that the user has not input personalized story arrangement logic information, the video clip can be edited based on the default story arrangement logic. In this way, the video clipping is highly intelligent and can take into account the editing needs of different users. For example, it can be compatible with the editing needs of users who have set the default story arrangement logic.

[0231] For example, the step of editing the video material data based on the story arrangement logic to obtain an edited video includes:

[0232] Obtain the structured text data of the video material;

[0233] The editing script is determined based on the story arrangement logic and the structured text data;

[0234] The video footage data is edited based on the editing script to obtain an edited video.

[0235] Here, electronic devices can convert video footage data into structured text data based on algorithms that recognize video content. For example, they can use image-text multimodal recognition models or GPT models to identify the text content corresponding to the video footage data, obtaining predefined structured text data. Based on this, the electronic devices can determine an editing script based on the current story arrangement logic and the structured text data, and then edit the video footage data based on the editing script to obtain an edited video. The editing script can be understood as a description file obtained by processing the structured text data according to the story arrangement logic based on a text story arrangement algorithm. This editing script can provide text descriptions of each shot in the edited video. Thus, based on the editing script, video segments can be selected and sequentially connected, making the logical continuity and natural transitions of the video segments in the edited video possible.

[0236] For example, the editing script includes: text descriptions of multiple shots, and the arrangement order of the multiple shots; the editing process of the video material data based on the editing script includes:

[0237] Matching the text description and structured text data of each shot to determine the target structured text fragment that matches the text description of each shot.

[0238] Obtain the target video segment corresponding to each target structured text segment;

[0239] The target video segments are synthesized according to the arrangement order to obtain an edited video.

[0240] Understandably, electronic devices can match the text descriptions of each shot in the editing script with structured text data to obtain target structured text fragments that match the text descriptions of each shot in the editing script. Then, the target structured text fragments are mapped to the corresponding target video fragments, and the video fragments are synthesized according to the arrangement order to obtain the edited video.

[0241] For example, the step of matching the text description and structured text data of each shot to determine the target structured text fragment that matches the text description of each shot includes:

[0242] Semantic matching is performed between the text description of each shot and the text description in the structured text data to obtain the target structured text fragment that matches the text description of each shot. The timestamp information corresponding to the target structured text fragment is determined. The timestamp information refers to the start time and end time of the target structured text fragment in the structured text data.

[0243] The step of obtaining the target video segment corresponding to each target structured text segment includes:

[0244] Based on the timestamp information, the target video segment corresponding to each target structured text segment is extracted from the video material data.

[0245] Understandably, the electronic device performs semantic matching between the text description of each shot and the text description in the structured text data. After obtaining the target structured text fragment that matches the text description of each shot, it can also extract the timestamp information corresponding to the target structured text fragment. Thus, the electronic device can extract the target video fragment corresponding to each target structured text fragment from the video footage data based on this timestamp information. It should be noted that the structured text data of the video footage data includes multiple structured text fragments, where each structured text fragment corresponds to a video fragment, and each video fragment has corresponding timestamp information.

[0246] For example, determining the edited script based on the story arrangement logic and the structured text data includes:

[0247] The story arrangement logic and the structured text data are used as inputs to a pre-trained script arrangement model to obtain the edited script output by the script arrangement model.

[0248] Here, the story arrangement algorithm can be a pre-trained script arrangement model. The electronic device will use the story arrangement logic and the structured text data as input to the pre-trained script arrangement model to obtain the edited script output by the script arrangement model. The pre-trained script arrangement model can be trained based on a pre-constructed training sample set, which includes the story arrangement logic, structured text data, and corresponding edited videos.

[0249] Exemplarily, the method further includes:

[0250] The video material data is converted into multiple video segments based on a set duration;

[0251] The multiple video segments are subjected to segment filtering processing to obtain at least one video segment library;

[0252] The process of editing the video material data based on the story arrangement logic to obtain an edited video includes:

[0253] Based on the story arrangement logic, the at least one video clip library is subjected to video synthesis processing to obtain an edited video.

[0254] Here, before editing the video footage, the electronic device can also filter video segments within the video footage data. This not only improves editing efficiency but also better meets the user's personalized editing needs.

[0255] For example, the step of performing segment filtering processing on the plurality of video segments to obtain at least one video segment library includes:

[0256] Perform segment attribute recognition on the multiple video segments to obtain the segment attribute recognition results for each video segment;

[0257] Based on the segment attribute recognition results of each video segment, waste segment filtering and / or highlight segment extraction are performed on the multiple video segments to obtain at least one video segment library.

[0258] Here, after the electronic device divides the video footage data to be edited into multiple video segments, it can perform segment attribute recognition on each video segment and, based on the segment attribute recognition results, filter out unusable segments and / or extract highlight segments, thereby obtaining a video segment library composed of video segments that more closely match the editing needs. This reduces the amount of subsequent data processing and improves editing efficiency. This segment attribute recognition is based on image feature recognition of image data. Different image features are pre-learned to correspond to attribute categories, and subsequent processing is determined based on these attribute categories. Attribute categories can include: occlusion categories, image blur categories, jitter categories, dirt categories, etc. In another embodiment, attribute categories can be further divided into: unusable segment categories, highlight categories, theme categories, etc. The specific division method can be customized as needed.

[0259] For example, the step of performing segment attribute recognition on the plurality of video segments to obtain segment attribute recognition results for each video segment includes:

[0260] The image data of each video segment in the multiple video segments are used to identify attributes based on a pre-trained image-text multimodal model to obtain the segment attribute identification results of each video segment.

[0261] The image-text multimodal model is configured with predefined attribute categories and corresponding text attribute definition information. The image-text multimodal model is used to encode the image data to obtain image encoding features, and output fragment attribute recognition results based on the image encoding features and the text encoding features of the text attribute definition information.

[0262] It is understandable that since electronic devices need to perform attribute recognition on image data, the use of a text-image multimodal model can achieve cross-modal semantic understanding. This allows for cross-modal understanding based on the image encoding features of image data and the text encoding features of predefined attribute categories, thereby obtaining attribute recognition results. In this way, the attribute categories of image data can be defined based on text attribute definition information. Traditional image / video-image / video matching models often require matching against an image / video standard library (covering all image / video content in multiple cases for each type). During inference, the hardware needs to pre-store all image / video encoding content of the entire standard library. Based on this, the text-image multimodal model introduced in this application only needs to pre-store the unique text description encoding for each attribute type in the hardware, improving the convenience of hardware storage and retrieval. Furthermore, compared to previous classification models that extract features from videos and use classifiers for judgment, which fix the number of model categories and discrimination criteria after training, the image-text multimodal model introduced in this application can not only modify the current attribute discrimination criteria by directly modifying the text description encoding for discrimination, but also easily expand and add the discrimination ability of other attributes, thus having the characteristics of easy expansion and adjustment.

[0263] For example, the image data is multi-frame image data obtained by extracting frames from a video segment of a set duration, and the step of image encoding the image data to obtain image encoding features includes:

[0264] Based on the image encoder, the multi-frame image data is encoded as a whole to obtain the image coding features of the multi-frame image data, or...

[0265] The image encoder encodes each frame of the multi-frame image data separately to obtain the image coding features of each frame.

[0266] Here, each video segment can be divided based on a set duration, and the image data corresponding to each video segment can be multi-frame image data obtained after frame extraction of the video segment.

[0267] It should be noted that, in order to meet the diverse needs of users for video editing, a collection of video clips with multiple themes can be extracted from the video material data to be edited, thereby satisfying diverse editing requirements. Based on this, in some embodiments, a step of clustering the video clips based on theme attributes can also be introduced; that is, clip selection can also include clustering processing.

[0268] For example, prior to performing waste filtering and / or highlight extraction on the plurality of video clips, the method further includes:

[0269] Clustering processing is performed on the multiple video segments;

[0270] The step of filtering out unusable video clips and / or extracting highlight clips from the plurality of video clips to obtain at least one video clip library includes:

[0271] The clustered video clip sets are then subjected to waste clip filtering and / or highlight clip extraction to obtain at least one video clip library.

[0272] Understandably, in this application example, the video footage data to be edited is first clustered into video segments, and then the fragment sets of each cluster are filtered for unusable segments and / or highlight segments are extracted to obtain at least one video segment library.

[0273] For example, after performing waste filtering and / or highlight extraction on the plurality of video clips, the method further includes:

[0274] After filtering out the waste footage and / or extracting the highlight segments, the video segments are clustered to obtain at least one video segment library.

[0275] Understandably, in this application example, video clips in the video material data to be edited can first be filtered for defective clips and / or have highlights extracted, and then the processed video clips can be clustered to obtain at least one video clip library.

[0276] For example, the fragment attribute recognition result includes: a first recognition result characterizing the topic attribute; the clustering process includes:

[0277] Based on the first identification result, each video segment is clustered to obtain a set of video segments with at least one clustered theme and a set of video segments without a theme.

[0278] Here, the theme attribute represents the theme surrounding the video segment, such as including but not limited to the following theme categories: scene, event, person, time, etc. Each theme category includes at least one theme attribute. Thus, video segments with different theme attributes can be divided based on the first recognition result.

[0279] For example, if the electronic device does not collect parameters indicating the thematic tendency of the video clips, it can perform clustering processing on the video segments based on the first recognition result to obtain a set of video segments without a theme and a set of video segments with at least one clustered theme. The number and thematic tendency of the at least one clustered theme can be determined using preset parameters. In this way, multiple video segment libraries can be output for users to subsequently select target video segment libraries for editing, thereby enriching the diversity of video editing and better meeting users' personalized editing needs.

[0280] For example, the fragment attribute recognition result includes: a first recognition result characterizing the topic attribute; the method further includes:

[0281] Get the first parameter that indicates the thematic tendency of the video clip;

[0282] The target topic tendency is determined based on the first parameter;

[0283] The clustering process includes:

[0284] Based on the first identification result, a set of target video segments related to the target theme is extracted.

[0285] Understandably, if an electronic device collects a first parameter indicating the thematic tendency of a video clip—that is, if the user inputs a first parameter indicating the thematic tendency of the video clip—the electronic device can determine the target thematic tendency based on this first parameter, and then perform clustering processing on the video segments based on this first recognition result to extract a set of target video segments related to the target thematic tendency, thus obtaining a video segment library corresponding to the target thematic tendency. In this way, a video clip corresponding to the thematic tendency desired by the user can be directly obtained.

[0286] For example, the step of performing video synthesis processing on the at least one video clip library based on the story arrangement logic to obtain an edited video includes:

[0287] Select a target video clip library from at least one of the video clip libraries;

[0288] Obtain the structured text data of the target video clip library;

[0289] The editing script is determined based on the story arrangement logic and the structured text data;

[0290] Based on the editing script, the target video clip library is edited to obtain an edited video.

[0291] Understandably, if the user has already input a first parameter indicating the thematic tendency of the video clip, the electronic device can directly obtain the target video clip library based on clip filtering. If the electronic device has not acquired this first parameter, it can output indication information for multiple generated video clip libraries and receive interactive operations input by the user based on this indication information, selecting the target video clip library based on the interactive operations. The electronic device then acquires the structured text data of the target video clip library, determines the editing script based on the story arrangement logic and this structured cultural data, and edits the target video clip library based on the editing script to obtain the edited video. The specific process of determining the editing script and editing the target video clip library can be referred to the aforementioned description of editing video material data based on story arrangement logic, and will not be repeated here.

[0292] For example, the segment attribute recognition result of a video clip may further include: a second recognition result characterizing the attributes of discarded clips and a third recognition result characterizing the text description. The aforementioned discarded clip filtering and / or highlight segment extraction based on the segment attribute recognition result includes:

[0293] For multiple video clips, waste clip filtering and / or highlight clip extraction are performed based on at least one of the first recognition result, the second recognition result, and the third recognition result.

[0294] Here, the electronic device can identify defective video clips from multiple video segments based on the second recognition result, filter the identified defective video clips, and then extract highlight segments from the filtered video segments based on the first and / or third recognition results. For example, the electronic device first performs clustering processing on multiple video clips in the video material data to be edited, obtaining a set of clips corresponding to multiple theme categories, and then performs defective clip filtering and / or highlight segment extraction on each set of clips to obtain a video clip library corresponding to each theme category.

[0295] The video editing method of this application will be further described in detail below with reference to application examples.

[0296] This application embodiment provides a personalized video editing method. Referring to Figure 10, the method includes an editing preparation stage and an editing execution stage. The editing preparation stage includes receiving a user-uploaded video material library (i.e., video material data to be edited) and determining the story arrangement logic. The editing execution stage includes processing the video material data based on the story arrangement logic to obtain the edited video.

[0297] In this application embodiment, after receiving the video material library uploaded by the user, the electronic device can perform segment attribute recognition on each video segment in the video material library (e.g., dividing the video segment into segments every 3 seconds) to obtain a first recognition result, a second recognition result, and a third recognition result for each video segment. The first recognition result corresponds to a theme attribute, the second recognition result corresponds to a discarded segment attribute, and the third recognition result corresponds to a text description. Based on the second recognition result, the electronic device automatically determines the discarded segment attribute to obtain a usable segment material library. For example, based on the second recognition result, video segments with occlusion, jitter, or blurriness are filtered out as discarded segments from the initial video segments to obtain a usable segment material library.

[0298] It's important to note that during the editing preparation stage, the electronic device can also receive input from the user via touch, text, or voice interaction, including: the desired final cut time, and suggestions for a thematic cut (e.g., wanting to edit around a specific scene or event, providing a script, specifying scenes or content to be removed, or requiring the addition of a scene or content). Furthermore, it can receive pre-edited clips uploaded by the user for style reference. This involves receiving reference video clips and obtaining style reference information based on them. This style reference information includes the following: story structure, story beginnings and / or endings, and story emphasis. Understandably, the electronic device can determine the story's logical structure based on this style reference information.

[0299] It should be noted that the electronic device can replace all available segments in the available selection material library with algorithmically generated text descriptions for subsequent steps. Furthermore, all personalized user input will also be converted into text instructions for the next stage, including converting interactive commands into text and uploaded videos into text descriptions of the video's narrative, development, and specific content. This application example does not limit the text conversion; it can use any video captioning algorithm or speech-to-text algorithm.

[0300] In this application embodiment, if the user inputs the aforementioned preference prompt: then a topic search is first performed on all video clips, and video clips related to all topics are retrieved before video editing. If the user does not input the aforementioned preference prompt: then the topic attributes of all video clips are first clustered to generate a set of video clips without a topic and a set of video clips with at least one clustered topic, and then topic-free video editing and topic-related video editing are performed respectively.

[0301] In this application embodiment, video editing includes segment selection and story arrangement of the selected video segment library. The following example illustrates this process using the scenario where the user has not entered a theme preference:

[0302] Referring to Figure 11, the user-uploaded video files are first segmented into multiple video clips. Then, attribute recognition is performed on each video clip based on its attributes, yielding a first recognition result (corresponding to "useful" in Figure 11), a second recognition result (corresponding to "useless" in Figure 11), and a third recognition result (corresponding to "caption" in Figure 11). Based on the first recognition result, attribute clustering is performed to obtain a set of video clips without a specific theme and a set of video clips with multiple themes. Next, a two-stage story arrangement is performed on each set of video clips, as detailed below:

[0303] Stage 1: Based on the first, second, and third recognition results, each video clip set is filtered for invalid clips and highlights are extracted, forming the video clip library for Stage 2. Here, if the user uploads a template video (i.e., a reference video clip), the clip attributes of the template video are used as the basis for highlight selection. If no template video is available, default highlight selection criteria are used (e.g., including but not limited to: focusing on moving scenes rather than still images, focusing on interactions between people, including hugs, kisses, entertainment, etc.).

[0304] Stage 2: Using a story arrangement algorithm, the video material text selected in Stage 1 is arranged into a complete story editing script according to the order of the beginning, development and ending of the complete story. Then, the segments are selected in sequence and the selected segments are video synthesized to obtain the edited video.

[0305] Specifically, in Phase 1, the video clips within each cluster are first arranged chronologically. For example, ten video clips are grouped together, resulting in n groups. The algorithm then filters out unwanted clips and selects highlight clips for each group. Finally, the selected video clip sets from all groups are merged. If the total number of clips in the set does not meet the condition at this point, and it exceeds a preset maximum value, it means that there are too many usable video clips. Phase 1 needs to be run multiple times until it falls within the preset maximum value range (this strategy is to prevent excessive user uploads or a large number of duplicate highlight clips from being selected, leading to too many clips being input for Phase 2 editing and making it impossible to consider all clips). If it is less than a preset minimum value, it means that there are too few usable video clips. In this case, Phase 1 is restarted, without filtering out unwanted clips, only highlight selection is performed, and then the result is directly output to Phase 2 (this strategy is to prevent the unwanted clip attribute from blocking a large number of videos, resulting in a very small number of editable video clips).

[0306] Understandably, the first stage described above can be performed multiple times, thereby effectively reducing the number of segments in the candidate segment set and thus effectively controlling the length of the final video. The second stage, based on the video segments output from the first stage, can be edited according to the story arrangement logic, making the logical continuity and natural transitions of the edited video segments, thereby effectively improving the video quality of the edited video.

[0307] It should be noted that during video editing, users can input personalized editing instructions using touch, text, or voice interaction. For example, as shown in Figure 10, users can input editing instructions for various video clip sets (with or without a theme) through human-computer interaction, such as deleting or adding video clips. Based on this, editing can be performed according to the story's arrangement logic to obtain the edited video. For instance, a user can select the output video's negative and then delete unsatisfactory clips, or add unselected clips through content retrieval, such as adding a video clip about a girl skiing. This allows users to search for or manually select desired video clips from all automatically edited videos to add to the final edit, or search for video clips from all available clips to add to the final product. Users can also input voice or text to have the algorithm automatically recommend the top 5 most suitable video clips for addition, from which they can then select the clip to add. All interactive content is converted into text by the algorithm. Through analysis of user interaction text, the algorithm transforms all personalized inputs into basic operations such as adding, deleting, modifying, and querying, calling the corresponding operation programs to complete the personalized interactive service. The editing algorithm takes into account the user-selected negatives, deleted segments, and retrieved added segments, considering the logic and rationality of the video. Following the chronological order and the sequence of events, it rationally plans and outputs the final interactive, automatically edited video without changing the content of the negatives.

[0308] It is understood that the method in this application embodiment, which performs video editing based on story arrangement logic, can incorporate the story-related arrangement logic into the editing process, compared to sequentially splicing together the highlight segments detected by the algorithm as the final edited video output. This results in logical continuity and natural transitions between video segments in the edited video. Furthermore, it supports outputting clips without a theme or multiple sets of clips with different themes for users to choose from, making the themes more diverse and better meeting users' editing needs. Moreover, by introducing human-computer interaction before and / or during editing, the editing style and content of the automatic editing can be personalized for users, better meeting the specific needs of a broad user base.

[0309] To implement the methods of the embodiments of this application, an electronic device is also provided. FIG12 only shows an exemplary structure of the electronic device and not the entire structure; some or all of the structures shown in FIG12 can be implemented as needed.

[0310] As shown in Figure 12, the electronic device 1200 provided in this embodiment includes at least one processor 1201, a memory 1202, a user interface 1203, and at least one network interface 1204. The various components in the electronic device 1200 are coupled together via a bus system 1205. It can be understood that the bus system 1205 is used to implement communication between these components. In addition to a data bus, the bus system 1205 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 1205 in Figure 12.

[0311] The user interface 1203 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0312] It is understood that the user interface 1203 is a human-computer interaction unit used for human-computer interaction operation of electronic devices. The electronic device 1200 also includes an image capturing unit for capturing images, and can thus be used as a capturing device.

[0313] The memory 1202 in this embodiment is used to store various types of data to support the operation of the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0314] The image processing method and / or video editing method disclosed in the embodiments of this application can be applied to, or implemented by, processor 1201. Processor 1201 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the image processing method can be completed by the integrated logic circuitry in the hardware of processor 1201 or by instructions in software form. The processor 1201 described above can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 1201 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory 1202. The processor 1201 reads the information in memory 1202 and, in conjunction with its hardware, completes the steps of the image processing method and / or video editing method provided in the embodiments of this application.

[0315] For example, when the processor is used to run a computer program, it is configured to:

[0316] Acquire image data generated by the imaging device;

[0317] The image data is subjected to attribute recognition to obtain the attribute recognition results;

[0318] Based on the attribute recognition results, instruction information for human-computer interaction is generated.

[0319] For example, the processor is configured as follows:

[0320] The image content in the image data is identified to obtain image features;

[0321] The attribute recognition result is determined based on the correspondence between image features and attribute categories.

[0322] For example, the processor is configured to:

[0323] If, based on the attribute recognition results, it is determined that there is an image anomaly in the image data, a first prompt message is generated indicating that the shooting device should be checked.

[0324] For example, the attribute recognition result includes an attribute category and a confidence level corresponding to the attribute category. Before generating the first prompt information indicating that the shooting device should be checked, the processor is configured to:

[0325] The confidence level of the attribute category corresponding to the image anomaly is determined to be greater than or equal to a set confidence threshold.

[0326] For example, the image data is image data corresponding to video segments generated by the shooting device during video recording; the processor is configured to:

[0327] If the video clip is determined to be invalid based on the attribute recognition result, the current video recording will be stopped and / or the video clip will be deleted, or a second prompt message will be generated indicating that the video recording will be stopped and / or the video clip will be deleted.

[0328] For example, the image data is the image data corresponding to each video segment after the shooting device has completed video shooting; the processor is configured to:

[0329] Based on the attribute recognition results corresponding to each video segment, the video segment is determined to be a discarded video segment, and a third prompt message indicating that the video segment is a discarded video segment is generated;

[0330] Based on the third prompt information, a waste film deletion instruction is obtained, and at least a portion of the waste films in the set of waste films are deleted.

[0331] For example, the processor is configured to:

[0332] Based on the attribute recognition results corresponding to each video segment, the highlight moments in the video segments are determined, and a fourth prompt message indicating the highlight segments is generated;

[0333] Based on the fourth prompt information, obtain the instruction to extract the highlight segment, and extract the highlight segment corresponding to the highlight moment.

[0334] For example, the processor is configured to:

[0335] Based on the attribute recognition results corresponding to each video segment, the time of the discarded segment in the video segment is determined, and a fifth prompt message indicating the discarded sub-segment is generated;

[0336] Based on the fifth prompt, at least some of the invalid video segments are deleted from each of the aforementioned video segments.

[0337] For example, the attribute recognition result includes the recognition result of at least one attribute category selected from occlusion attribute, jitter attribute, exposure attribute, and meaningless attribute, and the processor is configured to:

[0338] If the identification result of at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is yes, then the corresponding video segment is determined to be a defective video.

[0339] For example, the attribute recognition result also includes the confidence level corresponding to the attribute category, and the processor is configured to:

[0340] If the confidence level of at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is greater than or equal to the corresponding confidence threshold, then the identification result is determined to be yes.

[0341] For example, the attribute recognition result further includes: the recognition result of scene attributes, and the processor is configured to:

[0342] Based on the recognition results of the scene attributes and the established mapping relationship, a confidence threshold for at least one attribute category among the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is determined; wherein, the established mapping relationship includes the correspondence between the confidence thresholds under each scene attribute.

[0343] For example, the attribute recognition result further includes: the recognition result of the highlight attribute and the corresponding confidence level, and the processor is configured to:

[0344] If the identification result of at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is determined to be yes, and the identification result of the highlight attribute is determined to be yes, then based on the confidence level corresponding to the highlight attribute and the confidence level corresponding to at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute, it is determined whether the video clip is a defective clip.

[0345] For example, the processor is configured to:

[0346] The confidence threshold of at least one of the following attribute categories—occlusion attribute, jitter attribute, exposure attribute, and meaningless attribute—is configured based on the human-computer interaction operation.

[0347] For example, the processor is configured to:

[0348] The image data is used to perform attribute recognition based on a pre-trained image-text multimodal model to obtain attribute recognition results;

[0349] The image-text multimodal model is configured with predefined attribute categories and corresponding text attribute definition information. The image-text multimodal model is used to encode the image data to obtain image encoding features, and output attribute recognition results based on the image encoding features and the text encoding features of the text attribute definition information.

[0350] For example, the image data is multi-frame image data obtained by extracting frames from a video segment of a set duration, and the processor is configured to:

[0351] Based on the image encoder, the multi-frame image data is encoded as a whole to obtain the image coding features of the multi-frame image data, or...

[0352] The image encoder encodes each frame of the multi-frame image data separately to obtain the image coding features of each frame.

[0353] For example, the processor is configured to:

[0354] Based on the similarity between the image coding features of the multi-frame image data or the image coding features of each frame image data and the text coding features of each attribute category, the attribute recognition result is output; or,

[0355] Based on the autoregressive generator, the image coding features of the multi-frame image data or the image coding features of each frame image data and the text coding features of each attribute category are processed to output the attribute recognition results.

[0356] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0357] It is understood that memory 1202 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Sync Link Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0358] In an exemplary embodiment, this application also provides a computer storage medium, specifically a computer-readable storage medium, such as a memory 1202 storing a computer program. This computer program can be executed by the processor 1201 of the electronic device 1200 to complete the steps described in the method of this application embodiment. The computer-readable storage medium can be a ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM, etc.

[0359] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a processor 1201 of an electronic device 1200 to perform the steps described in the method of this application embodiment.

[0360] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0361] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0362] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, comprising: Acquire image data generated by the imaging device; The image data is subjected to attribute recognition to obtain the attribute recognition results; Instructions for human-computer interaction are generated based on the attribute recognition results.

2. The method according to claim 1, wherein, The step of performing attribute recognition on the image data to obtain attribute recognition results includes: The image content in the image data is identified to obtain image features; The attribute recognition result is determined based on the correspondence between image features and attribute categories.

3. The method according to claim 1, wherein, The generation of human-computer interaction instruction information based on the attribute recognition results includes: If, based on the attribute recognition results, it is determined that there is an image anomaly in the image data, a first prompt message is generated indicating that the shooting device should be checked.

4. The method according to claim 3, wherein, The attribute recognition result includes the attribute category and the confidence level corresponding to the attribute category. Before generating the first prompt information indicating that the shooting device should be checked, the method further includes: The confidence level of the attribute category corresponding to the image anomaly is determined to be greater than or equal to a set confidence threshold.

5. The method according to claim 1, wherein, The image data refers to the image data corresponding to video segments generated by the shooting device during video recording; the generation of instruction information for human-computer interaction based on the attribute recognition result includes: If the video clip is determined to be invalid based on the attribute recognition result, the current video recording will be stopped and / or the video clip will be deleted, or a second prompt message will be generated indicating that the video recording will be stopped and / or the video clip will be deleted.

6. The method according to claim 1, wherein, The image data refers to the image data corresponding to each video segment after the shooting device has completed video shooting; Based on the attribute recognition results, instruction information for human-computer interaction is generated, including: Based on the attribute recognition results corresponding to each video segment, the video segment is determined to be a discarded video segment, and a third prompt message indicating that the video segment is a discarded video segment is generated; The method further includes: Based on the third prompt information, a waste film deletion instruction is obtained, and at least a portion of the waste films in the set of waste films are deleted.

7. The method according to claim 6, wherein, Based on the attribute recognition results, instruction information for human-computer interaction is generated, including: Based on the attribute recognition results corresponding to each video segment, the highlight moments in the video segments are determined, and a fourth prompt message indicating the highlight segments is generated; The method further includes: Based on the fourth prompt information, obtain the instruction to extract the highlight segment, and extract the highlight segment corresponding to the highlight moment.

8. The method according to claim 6, wherein, Based on the attribute recognition results, instruction information for human-computer interaction is generated, including: Based on the attribute recognition results corresponding to each video segment, the time of the discarded segment in the video segment is determined, and a fifth prompt message indicating the discarded sub-segment is generated; The method further includes: Based on the fifth prompt, at least some of the invalid video segments are deleted from each of the aforementioned video segments.

9. The method according to claim 5, wherein, The attribute recognition results include: recognition results for at least one attribute category selected from occlusion attribute, jitter attribute, exposure attribute, and meaningless attribute. Determining the video clip as unusable based on the attribute recognition results includes: If the identification result of at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is yes, then the corresponding video segment is determined to be a defective video.

10. The method according to claim 9, wherein, The attribute recognition result also includes the confidence level corresponding to the attribute category. Determining that the recognition result of at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is "yes" includes: If the confidence level of at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is greater than or equal to the corresponding confidence threshold, then the identification result is determined to be yes.

11. The method according to claim 10, wherein, The attribute recognition result also includes: the recognition result of scene attributes, and the method further includes: Based on the recognition results of the scene attributes and the established mapping relationship, the confidence threshold of at least one attribute category among the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is determined; wherein, the established mapping relationship includes the correspondence between the confidence thresholds under each scene attribute.

12. The method according to claim 11, wherein, The attribute recognition result further includes: highlight attribute and corresponding confidence level. The step of determining the video clip as unusable based on the attribute recognition result further includes: If the identification result of at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is determined to be yes, and the identification result of the highlight attribute is determined to be yes, then based on the confidence level corresponding to the highlight attribute and the confidence level corresponding to at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute, it is determined whether the video clip is a defective clip.

13. The method according to claim 10, wherein, The method further includes: The confidence threshold of at least one of the following attribute categories—occlusion attribute, jitter attribute, exposure attribute, and meaningless attribute—is configured based on the human-computer interaction operation.

14. The method according to claim 1, wherein, The step of performing attribute recognition on the image data to obtain attribute recognition results includes: The image data is used to perform attribute recognition based on a pre-trained image-text multimodal model to obtain attribute recognition results; The image-text multimodal model is configured with predefined attribute categories and corresponding text attribute definition information. The image-text multimodal model is used to encode the image data to obtain image encoding features, and output attribute recognition results based on the image encoding features and the text encoding features of the text attribute definition information.

15. The method according to claim 14, wherein, The image data is obtained by extracting frames from a video segment of a set duration to obtain multiple frames of image data. The step of encoding the image data to obtain image encoding features includes: Based on the image encoder, the multi-frame image data is encoded as a whole to obtain the image coding features of the multi-frame image data, or... The image encoder encodes each frame of the multi-frame image data separately to obtain the image coding features of each frame.

16. The method according to claim 15, wherein, The text encoding feature output attribute recognition result based on the image encoding features and the text attribute definition information includes: Based on the similarity between the image coding features of the multi-frame image data or the image coding features of each frame image data and the text coding features of each attribute category, the attribute recognition result is output; or, Based on the autoregressive generator, the image coding features of the multi-frame image data or the image coding features of each frame image data and the text coding features of each attribute category are processed to output the attribute recognition results.

17. An electronic device comprising: A processor and memory for storing computer programs that can run on the processor, wherein, The processor, when used to run computer programs, is configured as follows: Acquire image data generated by the imaging device; The image data is subjected to attribute recognition to obtain the attribute recognition results; Instructions for human-computer interaction are generated based on the attribute recognition results.

18. The electronic device according to claim 17, wherein, The processor is configured to: The image content in the image data is identified to obtain image features; The attribute recognition result is determined based on the correspondence between image features and attribute categories.

19. The electronic device according to claim 17, wherein, The processor is configured to: If, based on the attribute recognition results, it is determined that there is an image anomaly in the image data, a first prompt message is generated indicating that the shooting device should be checked.

20. The electronic device according to claim 19, wherein, The attribute recognition result includes the attribute category and the confidence level corresponding to the attribute category. Before generating the first prompt information for checking the shooting device, the processor is configured to: The confidence level of the attribute category corresponding to the image anomaly is determined to be greater than or equal to a set confidence threshold.

21. The electronic device according to claim 17, wherein, The image data is the image data corresponding to the video clips generated by the shooting device during video recording; the processor is configured to: If the video clip is determined to be invalid based on the attribute recognition result, the current video recording will be stopped and / or the video clip will be deleted, or a second prompt message will be generated indicating that the video recording will be stopped and / or the video clip will be deleted.

22. The electronic device according to claim 17, wherein, The image data refers to the image data corresponding to each video segment after the capturing device completes video capture; the processor is configured to: Based on the attribute recognition results corresponding to each video segment, the video segment is determined to be a discarded video segment, and a third prompt message indicating that the video segment is a discarded video segment is generated; Based on the third prompt information, a waste film deletion instruction is obtained, and at least a portion of the waste films in the set of waste films are deleted.

23. The electronic device according to claim 22, wherein, The processor is configured to: Based on the attribute recognition results corresponding to each video segment, the highlight moments in the video segments are determined, and a fourth prompt message indicating the highlight segments is generated; Based on the fourth prompt information, obtain the instruction to extract the highlight segment, and extract the highlight segment corresponding to the highlight moment.

24. The electronic device according to claim 22, wherein, The processor is configured to: Based on the attribute recognition results corresponding to each video segment, the time of the discarded segment in the video segment is determined, and a fifth prompt message indicating the discarded sub-segment is generated; Based on the fifth prompt, at least some of the invalid video segments are deleted from each of the aforementioned video segments.

25. The electronic device according to claim 21, wherein, The attribute recognition results include: recognition results for at least one attribute category selected from occlusion attributes, jitter attributes, exposure attributes, and meaningless attributes; the processor is configured to: If the identification result of at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is yes, then the corresponding video segment is determined to be a defective video.

26. The electronic device according to claim 25, wherein, The attribute recognition result also includes the confidence score corresponding to the attribute category, and the processor is configured as follows: If the confidence level of at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is greater than or equal to the corresponding confidence threshold, then the identification result is determined to be yes.

27. The electronic device according to claim 26, wherein, The attribute recognition result also includes: the recognition result of scene attributes, and the processor is configured to: Based on the recognition results of the scene attributes and the established mapping relationship, the confidence threshold of at least one attribute category among the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is determined; wherein, the established mapping relationship includes the correspondence between the confidence thresholds under each scene attribute.

28. The electronic device according to claim 27, wherein, The attribute recognition result also includes: a highlight attribute and its corresponding confidence level, and the processor is configured to: If the identification result of at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute is determined to be yes, and the identification result of the highlight attribute is determined to be yes, then based on the confidence level corresponding to the highlight attribute and the confidence level corresponding to at least one of the occlusion attribute, the jitter attribute, the exposure attribute, and the meaningless attribute, it is determined whether the video clip is a defective clip.

29. The electronic device according to claim 26, wherein, The processor is configured to: The confidence threshold of at least one of the following attribute categories—occlusion attribute, jitter attribute, exposure attribute, and meaningless attribute—is configured based on the human-computer interaction operation.

30. The electronic device according to claim 17, wherein, The processor is configured to: The image data is used to perform attribute recognition based on a pre-trained image-text multimodal model to obtain attribute recognition results; The image-text multimodal model is configured with predefined attribute categories and corresponding text attribute definition information. The image-text multimodal model is used to encode the image data to obtain image encoding features, and output attribute recognition results based on the image encoding features and the text encoding features of the text attribute definition information.

31. The electronic device according to claim 30, wherein, The image data is obtained by extracting frames from a video segment of a set duration to obtain multiple frames of image data, and the processor is configured to: Based on the image encoder, the multi-frame image data is encoded as a whole to obtain the image coding features of the multi-frame image data, or... The image encoder encodes each frame of the multi-frame image data separately to obtain the image coding features of each frame.

32. The electronic device according to claim 31, wherein, The processor is configured to: Based on the similarity between the image coding features of the multi-frame image data or the image coding features of each frame image data and the text coding features of each attribute category, the attribute recognition result is output; or, Based on the autoregressive generator, the image coding features of the multi-frame image data or the image coding features of each frame image data and the text coding features of each attribute category are processed to output the attribute recognition results.

33. A computer storage medium storing a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 16.

Images