Video processing method and related equipment

Abstract

The invention provides a video processing method and related equipment. The method comprises the following steps: acquiring a to-be-processed video and a target duration; determining video text description of the to-be-processed video based on the image content of the to-be-processed video; determining semantic correlation between video frames in the to-be-processed video and the corresponding video text description; and determining the continuous video frame with the maximum total score in the target duration in the to-be-processed video as a target video clip, wherein the total score is obtained based on the semantic correlation and the image quality corresponding to the continuous video frame.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video processing method, device, equipment, medium and program product. Background Art

[0002] Sometimes video editing is required before sharing the video. However, manual video editing is cumbersome and the edited video clips are not effective. This results in low efficiency and poor effect of video editing. Summary of the invention

[0003] The present disclosure provides a video processing method, device, equipment, storage medium and program product to solve the technical problems of low efficiency and poor effect of video editing to a certain extent.

[0004] In a first aspect, the present disclosure provides a video processing method, comprising:

[0005] Get the video to be processed and the target duration;

[0006] Determining a video text description of the video to be processed based on the image content of the video to be processed;

[0007] Determining the semantic relevance between the video frames in the video to be processed and the corresponding video text descriptions;

[0008] The continuous video frames with the largest total score within the target duration in the video to be processed are determined as target video segments, and the total score is obtained based on the semantic relevance and image quality corresponding to the continuous video frames.

[0009] In a second aspect of the present disclosure, a video processing device is provided, comprising:

[0010] The acquisition module is used to obtain the video to be processed and the target duration;

[0011] A text description module, used to determine a video text description of the video to be processed based on the image content of the video to be processed;

[0012] A picture-text matching module, used to determine the semantic relevance between the video frame in the video to be processed and the corresponding video text description;

[0013] The target segment module is used to determine the continuous video frames with the largest total score within the target duration in the video to be processed as the target video segment, wherein the total score is obtained based on the semantic relevance and image quality corresponding to the continuous video frames.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, characterized in that it includes one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the program includes instructions for executing the method described in the first aspect or the second aspect.

[0015] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors execute the method described in the first aspect or the second aspect.

[0016] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method described in the first aspect.

[0017] From the above, it can be seen that the video processing method and related equipment provided by the present disclosure select target video segments with target duration in the video based on aspects such as image quality and relevance to video content, and can automatically extract target video segments with high image quality and high relevance to video content from the video to be processed without the need for manual editing operations, thereby improving the effect and efficiency of video editing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of a video processing architecture according to an embodiment of the present disclosure.

[0020] Figure 2 The figure is a schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure.

[0021] Figure 3 The figure is a schematic flow chart of a video processing method according to an embodiment of the present disclosure.

[0022] Figure 4 Schematic diagram of a graph-text matching network according to an embodiment of the present disclosure.

[0023] Figure 5 Schematic diagram of a video processing method according to an embodiment of the present disclosure.

[0024] Figure 6 Schematic diagram of a video processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0026] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0027] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0028] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0029] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0030] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0031] Figure 1 FIG. 1 is a schematic diagram showing a video processing architecture of an embodiment of the present disclosure. Figure 1The video processing architecture 100 may include a server 110, a terminal 120, and a network 130 that provides a communication link. The server 110 and the terminal 120 may be connected via a wired or wireless network 130. The server 110 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, security services, and CDN.

[0032] The terminal 120 may be implemented in hardware or software. For example, when the terminal 120 is implemented in hardware, it may be various electronic devices having a display screen and supporting page display, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, and desktop computers, etc. When the terminal 120 device is implemented in software, it may be installed in the electronic devices listed above; it may be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.

[0033] It should be noted that the video processing method provided in the embodiment of the present application can be executed by the terminal 120 or by the server 110. It should be understood that Figure 1 The number of terminals, networks and servers in the embodiment is only for illustration and is not intended to limit the number of terminals, networks and servers. Any number of terminals, networks and servers may be provided as required.

[0034] Figure 2 FIG. 2 shows a schematic diagram of the hardware structure of an exemplary electronic device 200 provided in an embodiment of the present disclosure. Figure 2 As shown, the electronic device 200 may include: a processor 202, a memory 204, a network module 206, a peripheral interface 208 and a bus 210. The processor 202, the memory 204, the network module 206 and the peripheral interface 208 are connected to each other in communication within the electronic device 200 through the bus 210.

[0035] Processor 202 may be a central processing unit (CPU), a video processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. Processor 202 may be used to perform functions related to the technology described in this disclosure. In some embodiments, processor 202 may also include multiple processors integrated into a single logical component. For example, Figure 2 As shown, processor 202 may include a plurality of processors 202a, 202b, and 202c.

[0036] The memory 204 may be configured to store data (eg, instructions, computer code, etc.). Figure 2 As shown, the data stored in the memory 204 may include program instructions (e.g., program instructions for implementing the video processing method of the embodiment of the present disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). The processor 202 may also access the program instructions and data stored in the memory 204, and execute the program instructions to operate on the data to be processed. The memory 204 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 204 may include a random access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0037] The network module 206 can be configured to provide communication with other external devices to the electronic device 200 via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC) etc.), a cellular network, the Internet or a combination thereof. It is understood that the type of network is not limited to the above specific examples. In some embodiments, the network module 306 can include any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, etc., in any combination.

[0038] The peripheral interface 208 can be configured to connect the electronic device 200 to one or more peripheral devices to achieve information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touch pad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

[0039] The bus 210 can be configured to transmit information between various components of the electronic device 200 (e.g., the processor 202, the memory 204, the network module 206, and the peripheral interface 208), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0040] It should be noted that, although the architecture of the electronic device 200 only shows the processor 202, the memory 204, the network module 206, the peripheral interface 208 and the bus 210, in the specific implementation process, the architecture of the electronic device 200 may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the architecture of the electronic device 200 may also only include the components necessary for implementing the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0041] With the rapid development of video sharing platforms, video shooting and editing have become important tasks before video sharing. Video editors combine background music, video materials, special effects, subtitles, transitions and other content to form a video that can be uploaded to the sharing platform. However, video editing and creation often require certain professional skills. For example, users usually want to edit creative pictures and content combinations, but in the actual conversion process, the feedback effect is often lower than expected, resulting in poor video editing results. Moreover, manual editing operations are cumbersome, making video editing inefficient. Therefore, how to improve the effect and efficiency of video editing at the same time has become a technical problem that needs to be solved urgently.

[0042] In view of this, the embodiments of the present disclosure provide a video processing method and related equipment. Based on aspects such as image quality and relevance to video content, a target video segment with a target duration in a video is selected, and a target video segment with high image quality and high relevance to video content can be automatically extracted from the video to be processed, without manual editing operations, thereby improving the effect and efficiency of video editing.

[0043] Specifically, for the video to be processed video, a video text description text_video reflecting its main content can be determined. And the semantic relevance R_frame_video between the video frame frame in the video to be processed and the video text description text_video is determined, which can reflect the relevance of the video frame frame and the video content of the video to be processed video. The higher the semantic relevance R_frame_video is, the more relevant the video frame frame is to the main content of the video to be processed video. The frames in the video can be scored based on the semantic relevance R_fame_video of the video frame frame and the image quality Q_frame to obtain a total score S_total. The continuous video frames with the maximum total score ∑S_total and the target duration in the video to be processed video are selected as the target video clip of the video to be processed. The target video clip can be the wonderful part of the video to be processed, that is, the video highlight. It can be seen that the target video clip can not only reflect the main content of the video to be processed, including representative video pictures, but also has a good visual experience, such as clear images, brilliant colors, prominent subjects, balanced picture composition, etc. At the same time, there is no need for users to perform manual operations, which reduces the requirements for user operations and improves the efficiency of video editing.

[0044] See also Figure 3 , Figure 3 A schematic flowchart of a video processing method according to an embodiment of the present disclosure is shown. Figure 3 In the example, the video processing method 300 may further include the following steps.

[0045] In step S310, the video to be processed and the target duration are obtained.

[0046] The video to be processed may be uploaded locally or obtained through the network. The target duration may be a duration set by the user based on the interactive interface. For example, one or more preset target durations may be presented to the user based on the interactive interface, and the user may select the final target duration from them based on a user operation (e.g., a click); or the target duration may be directly output based on an input box in the interactive interface.

[0047] In some embodiments, the method 300 may further include: performing frame extraction processing on the video to be processed at a preset sampling frequency to obtain a sequence of video frames to be processed. The sampling frequency may be represented by a frame rate FPS (frame per second), and the frame rate FPS represents the number of images contained in the video data with a duration of 1 second. For example, if 30 images are included in the video data within a duration of 1 second, the FPS of the video data is 30. Generally, the frame rate of the video data is between 30 and 60. When the sampling frequency is represented by the sampling frame rate FPS, it may represent the number of images sampled within 1 second. Specifically, the sampling frequency may be a frame rate FPS=a, and the video to be processed may be frame extracted, and a frames may be extracted every second. For example. For video data with a frame rate of 30, 30 frames of images are included in the duration of each second. If sampling is performed at a sampling frequency of FPS=2, 2 frames of images may be uniformly used from every 30 frames of images. Since the frame rate of video data is generally high, it takes time to process the video. The video data may be sampled based on downsampling to reduce computational complexity and improve video processing efficiency and response speed.

[0048] In some embodiments, the to-be-processed video frame sequence can be used to replace the to-be-processed video for subsequent processing, thereby reducing the amount of video processing data, saving computing costs, and improving the efficiency of video processing. Specifically, the to-be-processed video with a duration of b seconds can be framed based on a frame rate a, that is, a video frame is extracted per second in chronological order to obtain a video frame sequence X_a_b including a*b frames. In the subsequent processing of the embodiment of the present disclosure, the video frame sequence can be used as a data basis to replace the to-be-processed video for selecting the target video segment.

[0049] In step S320, a video text description of the video to be processed is determined based on the image content of the video to be processed.

[0050] The video text description may refer to a representative text description used to describe the main image content of the video to be processed, such as a video summary.

[0051] In some embodiments, determining a video text description of the video to be processed based on image content of a video frame in the video to be processed includes:

[0052] Determining a video frame text description corresponding to the video frame based on image content of the video frame;

[0053] Extracting features of the video frame text description to obtain frame text features;

[0054] Calculating the similarity between the frame text features;

[0055] The video frame text description corresponding to the frame text feature with the greatest similarity is determined as the video text description of the video to be processed.

[0056] Among them, based on the video frame text description corresponding to each video frame in the video to be processed, the video frame text description with the highest similarity to other video frame text descriptions can be selected from the video frame text descriptions as the video summary of the video to be processed. Specifically, the video to be processed or the sequence of video frames to be processed is input into the image description network, which can be represented as h, and the output image text description is t, then t=h(x), x is the input image. For example, if the input image x contains a dog and grass, the image text description t can be "the dog is playing on the lawn".

[0057] The image description network can be used to describe the video frames in the video to be processed or the video frame sequence to be processed, and output the video frame text description corresponding to each video frame. For example, if the video to be processed or the video frame sequence to be processed includes multiple video frames Fi, i=1, 2, ... and is a positive integer, the frame text description text_Fi corresponding to the video frame Fi can be output based on the image description network M_image to text. The frame text description text_Fi is extracted based on the text feature network (the input text information can be converted into a text feature of a fixed length, for example, the input text texta can be converted into a feature vector of length 512, that is, a floating point vector) to obtain the frame text feature Feature_text_Fi, and the similarity (for example, cosine similarity) between the frame text feature Feature_text_Fi and other frame text features Feature_text_F1, ..., Feature_text_Fi-1, Feature_text_Fi+1, ... is calculated respectively, and the frame text description corresponding to the frame text feature Feature_text_Fmax with the largest similarity is determined as the video summary of the video to be processed or the video frame sequence to be processed, that is, the video text description text of the video to be processed.

[0058] In step S330, the semantic correlation between the video frames in the to-be-processed video and the corresponding video text description is determined.

[0059] Among them, semantic relevance can refer to the similarity between image features and text features, that is, cosine similarity. The semantic relevance between the video frame and the video text description can reflect the relevance between the video frame and the main content of the entire video to be processed, which is conducive to the accurate extraction of the video highlight segment, that is, the target video segment.

[0060] In some embodiments, determining the semantic relevance between the video frame and the corresponding video text description includes:

[0061] Extracting features from the video frame to obtain image features, and extracting features from the video text description to obtain video text features;

[0062] The semantic correlation between the image feature and the video text feature is calculated to obtain the semantic correlation between the video frame and the video text description.

[0063] Specifically, the semantic relevance between the video frame and the video text description can be determined based on the image-text matching network. Figure 4 , Figure 4 A schematic diagram of a graph-text matching network according to an embodiment of the present disclosure is shown. Figure 4 In the figure, the image-text matching network 400 may include an image feature network 410 (which may convert the input image information into an image feature of a fixed length, for example, the input image may be converted into a feature vector of length 512, i.e., a floating point vector) and a text feature network 420. Among them, the image feature network 410 may perform feature extraction on the video frame Fi to obtain the image feature Feature_Fi, and the text feature network 420 may perform feature extraction on the video text description text to obtain the video text feature Feature_text. The image-text matching network 400 may also calculate the cosine similarity between the image feature Feature_Fi and the video text feature Feature_text as the semantic correlation between the video frame and the video text description. It should be understood that the text feature network 420 may be used to perform feature extraction on the video frame text description in step S320 to obtain the frame text feature.

[0064] In some embodiments, the initial neural network can be trained based on the image-text matching training data to obtain a trained image-text matching network 400. Specifically, the image-text matching training data can include corresponding m images image_j, j=1, 2, ..., m is a positive integer and m texts text_j, then the image-text similarity between the m images image_j and the m texts text_j can be calculated to obtain an m×m symmetric matrix, in which the values ​​on the diagonal of the symmetric matrix represent the similarity f1 between each image image_j and the corresponding text description text_j; the data outside the diagonal represent the similarity f2 between each image image_j and the text text_l (l≠j) that does not match it. Based on the symmetric matrix, the weight of the initial neural network is adjusted to obtain a trained image-text matching network, so that the value f1 on the diagonal of the symmetric matrix is ​​close to 1, and the value f2 on the non-diagonal is close to 0. It should be understood that the above-mentioned image-text matching network can actually also calculate the similarity between texts and the similarity between images.

[0065] In step S340, the continuous video frames with the maximum total score within the target duration in the video to be processed are determined as target video segments, and the total score is obtained based on the semantic relevance and image quality corresponding to the continuous video frames.

[0066] The image quality may be evaluated based on multiple image quality dimensions, for example, the multiple image quality dimensions may include image clarity, subject clarity, color richness, etc. It should be understood that the above image quality dimensions are only examples and are not intended to be limiting. The image quality dimensions may include fewer or more dimensions, which may be set according to actual conditions and are not limited here.

[0067] In some embodiments, the total score is obtained based on the semantic relevance and image quality of the consecutive video frames, including:

[0068] Scoring the image quality of the video frame based on at least one image quality dimension to obtain an image quality score of the video frame; and

[0069] Scoring the video frame based on the semantic relevance to obtain a semantic relevance score for the video frame;

[0070] The total score is obtained based on the sum of the image quality score and the semantic relevance score.

[0071] Among them, the higher the image quality score, the better the visual photosensitivity experience of the image and the better the presentation effect. The higher the semantic relevance score, the closer the image content is to the video to be processed. Combining the image quality score and the semantic relevance score to get the total score, the final video highlight segment can be selected from the two aspects of visual effect and performance content, which is conducive to improving the accuracy and presentation effect of the video highlight segment.

[0072] In some embodiments, scoring the image quality of the video frame based on at least one image quality dimension to obtain an image quality score of the video frame includes:

[0073] The video frame is input into an image quality network to obtain the image quality score; wherein the image quality network is obtained by training an initial neural network based on training data, wherein the training data includes training image samples and true image quality scores of the training image samples determined based on the at least one image quality dimension.

[0074] The image quality of the input image can be scored by a trained image quality network to obtain a corresponding image quality score. The trained image quality network can be obtained by training an initial neural network based on training data including training image samples and corresponding true scores.

[0075] In some embodiments, the training data includes a first training image sample having a first true image quality score and a second training image sample having a second true image quality score;

[0076] The image quality network is obtained by training an initial neural network based on training data, including:

[0077] Based on the initial neural network, image quality scores are respectively performed on the first training image sample and the second training image sample to obtain a first estimated score and a second estimated score;

[0078] Based on a first difference between the first estimated score and the second estimated score, and a second difference between the first image quality true score and the second image quality true score, obtaining a product of the first difference and the second difference;

[0079] The maximum value between the preset value and the product is used to obtain the loss function;

[0080] The image quality network is obtained by adjusting the weight parameters of the initial neural network based on the loss function to minimize the loss function.

[0081] Specifically, the image quality network can be recorded as S=f(x), where x is the input image, f is the image quality network, and S is the image quality score. During the training process, the true score of each training image sample can be obtained, for example, 0 to 1 points, 0 points represents poor image quality, unclear, unremarkable portrait, unremarkable subject, dull color, unbalanced composition, etc., 1 point represents high image quality, clear, rich colors, reasonable design, strong visual sense, etc. The loss function of training is pair-wise loss. Specifically, for two training image samples: the first training image sample x1 and the second training image sample x2, corresponding to the first image quality true score y1 and the second image quality true score y2, the predicted image quality scores are S1=f(x1) and S2=f(x2). The training goal may include that the relative order of y1 and y2 is maintained between S1 and S2. For example, if y1>y2, then S1>S2, and vice versa. Therefore, the loss function may include:

[0082]

[0083] It can be seen that as long as the first difference (S1-S2) and the second difference (y1-y2) have the same sign, the loss is equal to 0; if the two have different signs, the loss function is greater than 0. Therefore, the training goal can be to minimize the above loss function For any two images, the relative image quality scores given by the image quality network are consistent with the relative image quality scores of the real images.

[0084] In some embodiments, determining the continuous video frames with the largest total score within the target duration in the video to be processed as the target video segment includes:

[0085] Determining a target number of the continuous video frames based on the target duration;

[0086] Sliding a sliding window having the target number of frames in the video to be processed, calculating a total score of continuous video frames in the sliding window;

[0087] The continuous video frames in the sliding window with the largest total score are determined as the target video segment.

[0088] Specifically, the user can specify to obtain the video highlights of the target length, such as K seconds. Since the preset frame rate is a frames / s, for example, a=3, the target frame number can be 3K. Then, a sliding window with a length of 3K can be set based on the target frame number, and the sum of the total scores of all continuous video frames in the sliding window ∑Score=∑(Score1+Score2) is calculated, where Score1 is the image quality score of the video frame, and Score2 is the semantic relevance score. The sliding window can slide from the starting position of the video to be processed or the sequence of video frames to be processed to the end position, and the largest continuous video frame with a length of 3K among all ∑Scores during this period is used as the target video segment. The target video segment is a video highlight segment that meets the user's requirements, has high relevance in content and good visual effects.

[0089] In some embodiments, see Figure 5 , Figure 5 A schematic diagram of a video processing method according to an embodiment of the present disclosure is shown. Figure 5 In the embodiment of the present disclosure, the video processing method can be executed based on the video processing model 500. Specifically, the video to be processed is input into the video processing model 500, and the video processing model can perform video frame extraction processing on the input video to be processed, for example, based on the preset frame rate 3FPS, to obtain N video frames. The N video frames are input into the image quality network 510 to obtain N image quality scores Score1. And the N video frames are input into the text description network 520 to obtain N video frame text descriptions, and the N video frame text descriptions are input into the text feature network 530 for feature extraction to obtain N video frame text features, and then the similarity between the video frame text features is calculated, and the video frame text description corresponding to the video frame text feature with the highest similarity is selected as the video text description of the video to be processed. The N video frames are input into the image feature network 540 to obtain the image features of the N video frames, and the cosine similarity between the N image features and the video text description is calculated to obtain N semantic relevance scores Score2. The target number of frames is obtained according to the target duration, and the total score sum ∑Score of all continuous video frames in the sliding window including the target number of frames is calculated. The continuous video frame with the largest total score sum ∑Score during the sliding window is taken as the target video segment.

[0090] It can be seen that the video processing method according to the embodiment of the present disclosure can not only return the most representative content and the most ornamental pictures in the entire video to be processed, but also can adaptively adjust according to the target duration specified by the user, thereby improving the effect and efficiency of video editing processing.

[0091] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0092] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0093] Based on the same technical concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a video processing device, see Figure 6 , the video processing device comprises:

[0094] The acquisition module is used to obtain the video to be processed and the target duration;

[0095] A text description module, used to determine a video text description of the video to be processed based on the image content of the video to be processed;

[0096] A picture-text matching module, used to determine the semantic relevance between the video frame in the video to be processed and the corresponding video text description;

[0097] The target segment module is used to determine the continuous video frames with the largest total score within the target duration in the video to be processed as the target video segment, wherein the total score is obtained based on the semantic relevance and image quality corresponding to the continuous video frames.

[0098] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0099] The device of the above embodiment is used to implement the corresponding video processing method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0100] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the video processing method described in any of the above embodiments.

[0101] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0102] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the video processing method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0103] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0104] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0105] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0106] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A video processing method, the method comprising: Get the video to be processed and the target duration; Determining a video text description of the video to be processed based on the image content of the video to be processed; Determining the semantic relevance between the video frames in the video to be processed and the corresponding video text descriptions; The continuous video frames with the largest total score within the target duration in the video to be processed are determined as target video segments, and the total score is obtained based on the semantic relevance and image quality corresponding to the continuous video frames.

2. The method according to claim 1, wherein Determining a video text description of the video to be processed based on image content of a video frame in the video to be processed includes: Determining a video frame text description corresponding to the video frame based on image content of the video frame; Extracting features of the video frame text description to obtain frame text features; Calculating the similarity between the frame text features; The video frame text description corresponding to the frame text feature with the greatest similarity is determined as the video text description of the video to be processed.

3. The method according to claim 1, wherein Determining the semantic relevance between the video frame and the corresponding video text description includes: Extracting features from the video frame to obtain image features, and extracting features from the video text description to obtain video text features; The semantic correlation between the image feature and the video text feature is calculated to obtain the semantic correlation between the video frame and the video text description.

4. The method according to claim 1, wherein The total score is obtained based on the semantic relevance and image quality of the consecutive video frames, including: Scoring the image quality of the video frame based on at least one image quality dimension to obtain an image quality score of the video frame; and Scoring the video frame based on the semantic relevance to obtain a semantic relevance score for the video frame; The total score is obtained based on the sum of the image quality score and the semantic relevance score.

5. The method according to claim 4, wherein: Scoring the image quality of the video frame based on at least one image quality dimension to obtain an image quality score of the video frame includes: The video frame is input into an image quality network to obtain the image quality score; wherein the image quality network is obtained by training an initial neural network based on training data, wherein the training data includes training image samples and true image quality scores of the training image samples determined based on the at least one image quality dimension.

6. The method according to claim 5, wherein: The training data includes a first training image sample having a first true image quality score and a second training image sample having a second true image quality score; The image quality network is obtained by training an initial neural network based on training data, including: Based on the initial neural network, image quality scores are respectively performed on the first training image sample and the second training image sample to obtain a first estimated score and a second estimated score; Based on a first difference between the first estimated score and the second estimated score, and a second difference between the first image quality true score and the second image quality true score, obtaining a product of the first difference and the second difference; The maximum value between the preset value and the product is used to obtain the loss function; The image quality network is obtained by adjusting the weight parameters of the initial neural network based on the loss function to minimize the loss function.

7. The method according to claim 1, determining the continuous video frames with the maximum total score within the target duration in the video to be processed as the target video segment, comprising: Determining a target number of the continuous video frames based on the target duration; Sliding a sliding window having the target number of frames in the video to be processed, calculating a total score of continuous video frames in the sliding window; The continuous video frames in the sliding window with the largest total score are determined as the target video segment.

8. A video processing device, comprising: The acquisition module is used to obtain the video to be processed and the target duration; A text description module, used to determine a video text description of the video to be processed based on the image content of the video to be processed; A picture-text matching module, used to determine the semantic relevance between the video frame in the video to be processed and the corresponding video text description; The target segment module is used to determine the continuous video frames with the largest total score within the target duration in the video to be processed as the target video segment, wherein the total score is obtained based on the semantic relevance and image quality corresponding to the continuous video frames.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program. 10 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to claim 1 .

Images