Video processing method and related device

Abstract

The present disclosure provides a video processing method and a related device. The method comprises: acquiring a video to be processed and a target duration; on the basis of the image content of said video, determining video text descriptions of said video; determining the semantic relatedness between video frames in said video and the corresponding video text descriptions; and determining consecutive video frames having the maximum total score within the target duration in said video as a target video clip, the total score being obtained on the basis of the semantic relatedness and the image quality corresponding to the consecutive video frames.

Description

Video processing method and related equipment

[0001] This application claims priority to the Chinese invention patent application entitled “Video Processing Method and Related Equipment” and application number 2023110221387, filed on August 14, 2023, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of computer technology, and in particular to a video processing method, apparatus, device, medium, and program product. Background Art

[0003] Sometimes video editing is required before sharing a video. However, manual video editing is cumbersome and the edited video clips are not effective, resulting in low efficiency and poor effect of video editing.

[0004] Summary of the Invention

[0005] The present disclosure provides a video processing method, apparatus, device, storage medium and program product to solve the technical problems of low efficiency and poor effect of video editing to a certain extent.

[0006] In a first aspect of the present disclosure, a video processing method is provided, comprising: obtaining a video to be processed and a target duration; determining a video text description of the video to be processed based on the image content of the video to be processed; determining the semantic relevance between video frames in the video to be processed and the corresponding video text descriptions; and determining the continuous video frames with the largest total score within the target duration in the video to be processed as target video segments, wherein the total score is obtained based on the semantic relevance and image quality corresponding to the continuous video frames.

[0007] In a second aspect of the present disclosure, a video processing device is provided, comprising: an acquisition module for acquiring a video to be processed and a target duration; a text description module for determining a video text description of the video to be processed based on the image content of the video to be processed; a picture-text matching module for determining the semantic relevance between video frames in the video to be processed and the corresponding video text description; and a target segment module for determining the continuous video frames with the largest total score within the target duration in the video to be processed as a target video segment, wherein the total score is obtained based on the semantic relevance and image quality corresponding to the continuous video frames.

[0008] In a third aspect of the present disclosure, an electronic device is provided, characterized in that it includes one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to the first aspect or the second aspect.

[0009] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors execute the method described in the first aspect or the second aspect.

[0010] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] FIG1 is a schematic diagram of a video processing architecture according to an embodiment of the present disclosure.

[0013] FIG2 is a schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure.

[0014] FIG3 is a schematic flowchart of a video processing method according to an embodiment of the present disclosure.

[0015] FIG4 is a schematic diagram of an image-text matching network according to an embodiment of the present disclosure.

[0016] FIG5 is a schematic diagram of a video processing method according to an embodiment of the present disclosure.

[0017] FIG6 is a schematic diagram of a video processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0019] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.

[0020] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0021] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0022] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0023] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0024] FIG1 shows a schematic diagram of a video processing architecture according to an embodiment of the present disclosure. Referring to FIG1 , the video processing architecture 100 may include a server 110, a terminal 120, and a network 130 providing a communication link. The server 110 and the terminal 120 may be connected via a wired or wireless network 130. The server 110 may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, security services, and CDN.

[0025] Terminal 120 can be implemented in hardware or software. For example, when implemented in hardware, terminal 120 can be any electronic device with a display screen that supports page display, including but not limited to smartphones, tablet computers, e-book readers, laptop computers, and desktop computers. When terminal 120 is implemented in software, it can be installed in the electronic devices listed above; it can be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or it can be implemented as a single software or software module, and no specific limitations are given here.

[0026] It should be noted that the video processing method provided in the embodiments of the present application can be executed by the terminal 120 or by the server 110. It should be understood that the number of terminals, networks, and servers in FIG1 is for illustration only and is not intended to limit the number thereof. Any number of terminals, networks, and servers can be used as required.

[0027] Figure 2 shows a schematic diagram of the hardware structure of an exemplary electronic device 200 provided by an embodiment of the present disclosure. As shown in Figure 2, electronic device 200 may include: a processor 202, a memory 204, a network module 206, a peripheral interface 208, and a bus 210. The processor 202, memory 204, network module 206, and peripheral interface 208 are communicatively connected to each other within electronic device 200 via bus 210.

[0028] The processor 202 may be a central processing unit (CPU), a video processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device (PLD), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or one or more integrated circuits. The processor 202 may be configured to perform functions related to the technology described in this disclosure. In some embodiments, the processor 202 may also include multiple processors integrated into a single logical component. For example, as shown in FIG2 , the processor 202 may include multiple processors 202 a, 202 b, and 202 c.

[0029] The memory 204 can be configured to store data (e.g., instructions, computer codes, etc.). As shown in Figure 2, the data stored in the memory 204 can include program instructions (e.g., program instructions for implementing the video processing method of the embodiment of the present disclosure) and data to be processed (e.g., the memory can store configuration files of other modules, etc.). The processor 202 can also access the program instructions and data stored in the memory 204, and execute the program instructions to operate on the data to be processed. The memory 204 can include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 204 can include a random access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0030] The network module 206 can be configured to provide the electronic device 200 with communication with other external devices via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC)), a cellular network, the Internet, or a combination thereof. It will be appreciated that the type of network is not limited to the specific examples above. In some embodiments, the network module 306 can include any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, and the like.

[0031] The peripheral interface 208 can be configured to connect the electronic device 200 to one or more peripheral devices to implement information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touchpad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

[0032] The bus 210 can be configured to transmit information between the various components of the electronic device 200 (e.g., the processor 202, the memory 204, the network module 206, and the peripheral interface 208), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0033] It should be noted that although the architecture of the electronic device 200 shown above only includes the processor 202, the memory 204, the network module 206, the peripheral interface 208, and the bus 210, in a specific implementation, the architecture of the electronic device 200 may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the architecture of the electronic device 200 may also include only the components necessary to implement the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0034] With the rapid development of video sharing platforms, video shooting and editing have become essential tasks before sharing videos. Video editors combine background music, video footage, special effects, subtitles, transitions, and other content to create a video that can be uploaded to sharing platforms. However, video editing and creation often require certain professional skills. For example, users often hope to edit a creative combination of images and content, but during the actual conversion process, the feedback is often lower than expected, resulting in poor video editing results. Moreover, manual editing operations are cumbersome, making video editing inefficient. Therefore, how to simultaneously improve the effectiveness and efficiency of video editing has become a technical problem that needs to be solved urgently.

[0035] In view of this, embodiments of the present disclosure provide a video processing method and related equipment. This method selects target video segments of a target duration from a video based on aspects such as image quality and relevance to the video content. This method automatically extracts target video segments with high image quality and high relevance to the video content from the video to be processed, eliminating the need for manual editing and improving the effectiveness and efficiency of video editing.

[0036] Specifically, for the video to be processed, a text description (text_video) reflecting its main content can be determined. The semantic relevance (R_frame_video) between the video frame in the video to be processed and the text description (text_video) can also be determined. This relevance reflects the relevance of the video frame to the main content of the video to be processed. The higher the semantic relevance (R_frame_video), the more relevant the video frame is to the main content of the video to be processed. Based on the semantic relevance (R_fame_video) and the image quality (Q_frame), the frames in the video can be scored to obtain a total score (S_total). The continuous video frames with the maximum total score (∑S_total) and a target duration in the video to be processed are selected as the target video segment of the video to be processed. This target video segment can be the highlight of the video to be processed. Therefore, the target video segment not only reflects the main content of the video to be processed, including representative video footage, but also provides a good visual experience, such as clear images, vivid colors, prominent subjects, and balanced composition. Furthermore, no manual user operation is required, reducing user experience requirements and improving video editing efficiency.

[0037] Referring to Figure 3, Figure 3 shows a schematic flow chart of a video processing method according to an embodiment of the present disclosure. In Figure 3, the video processing method 300 may further include the following steps.

[0038] In step S310, the video to be processed and the target duration are obtained. The video to be processed can be uploaded locally or obtained via the network. The target duration can be a duration set by the user based on an interactive interface. For example, one or more preset target durations can be presented to the user based on the interactive interface, and the user can select the final target duration based on a user operation (e.g., a click). Alternatively, the target duration can be directly input through an input box in the interactive interface.

[0039] In some embodiments, method 300 may further include: performing frame extraction on the video to be processed at a preset sampling frequency to obtain a sequence of video frames to be processed. The sampling frequency can be expressed as a frame rate (FPS), which represents the number of images contained in one second of video data. For example, if 30 images are contained in one second of video data, the FPS of the video data is 30. Typically, the frame rate of video data is between 30 and 60. The sampling frequency, when expressed as a frame rate (FPS), can represent the number of images sampled in one second. Specifically, the sampling frequency can be FPS = a, which allows the video to be extracted, extracting a frames per second. For example, if video data has a frame rate of 30, each second contains 30 frames. Sampling at a sampling frequency of FPS = 2 allows an even distribution of 2 frames from every 30 frames. Because video data generally has a high frame rate, which makes video processing time consuming, downsampling can be used to reduce computational complexity and improve video processing efficiency and response speed.

[0040] In some embodiments, a sequence of video frames to be processed can be used in place of the video to be processed for subsequent processing, thereby reducing the amount of data for video processing, saving computing costs, and improving the efficiency of video processing. Specifically, a video to be processed with a duration of b seconds can be frame-extracted based on a frame rate a, that is, a video frame is extracted per second in chronological order to obtain a video frame sequence X_a_b including a*b frames. In the subsequent processing of the embodiments of the present disclosure, the video frame sequence can be used as the data basis to replace the video to be processed for selecting the target video segment.

[0041] In step S320 , a video text description of the video to be processed is determined based on the image content of the video to be processed.

[0042] The video text description may refer to a representative text description used to describe the main image content of the video to be processed, such as a video summary.

[0043] In some embodiments, determining a video text description of the video to be processed based on the image content of the video frame in the video to be processed includes: determining a video frame text description corresponding to the video frame based on the image content of the video frame; performing feature extraction on the video frame text description to obtain frame text features; calculating the similarity between the frame text features; and determining the video frame text description corresponding to the frame text feature with the greatest similarity as the video text description of the video to be processed.

[0044] Based on the video frame text description corresponding to each video frame in the video to be processed, the video frame text description with the highest similarity to other video frame text descriptions can be selected from the video frame text descriptions as the video summary of the video to be processed. Specifically, the video to be processed or the sequence of video frames to be processed is input into the image description network, which can be represented as h. The output image text description is t, where t = h(x), where x is the input image. For example, if the input image x contains a dog and grass, the image text description t can be "dog playing on the lawn."

[0045] The image description network can be used to describe the video frames in the video to be processed or the video frame sequence to be processed, and output the video frame text description corresponding to each video frame. For example, if the video to be processed or the video frame sequence to be processed includes multiple video frames Fi, i = 1, 2, ... and is a positive integer, then the frame text description text_Fi corresponding to the video frame Fi can be output based on the image description network M_image to text. The frame text description text_Fi is extracted based on the text feature network (which can convert the input text information into a text feature of fixed length, for example, the input text texta can be converted into a feature vector of length 512, i.e., a floating point vector) to obtain the frame text feature Feature_text_Fi. The similarity (e.g., cosine similarity) between the frame text feature Feature_text_Fi and other frame text features Feature_text_F1, ..., Feature_text_Fi-1, Feature_text_Fi+1, ...) is calculated respectively. The frame text description corresponding to the frame text feature Feature_text_Fmax with the largest similarity is determined as the video summary of the video to be processed or the video frame sequence to be processed, i.e., the video text description text of the video to be processed.

[0046] In step S330, the semantic relevance between the video frame in the video to be processed and the corresponding video text description is determined. Semantic relevance can refer to the degree of similarity between image features and text features, i.e., cosine similarity. The semantic relevance between the video frame and the video text description can reflect the relevance between the video frame and the main content of the entire video to be processed, facilitating the accurate extraction of video highlights, i.e., target video segments.

[0047] In some embodiments, determining the semantic correlation between the video frame and the corresponding video text description includes: performing feature extraction on the video frame to obtain image features, and performing feature extraction on the video text description to obtain video text features; calculating the semantic correlation between the image features and the video text features to obtain the semantic correlation between the video frame and the video text description.

[0048] Specifically, the semantic relevance between the video frame and the video text description can be determined based on the image-text matching network. Referring to Figure 4, Figure 4 shows a schematic diagram of the image-text matching network according to an embodiment of the present disclosure. In Figure 4, the image-text matching network 400 may include an image feature network 410 (which can convert the input image information into image features of a fixed length, for example, the input image can be converted into a feature vector of length 512, i.e., a floating-point vector) and a text feature network 420. Among them, the image feature network 410 can perform feature extraction on the video frame Fi to obtain the image feature Feature_Fi, and the text feature network 420 can perform feature extraction on the video text description text to obtain the video text feature Feature_text. The image-text matching network 400 can also calculate the cosine similarity between the image feature Feature_Fi and the video text feature Feature_text as the semantic relevance between the video frame and the video text description. It should be understood that the text feature network 420 can be used to perform feature extraction on the video frame text description in step S320 to obtain the frame text feature.

[0049] In some embodiments, the initial neural network can be trained based on the image-text matching training data to obtain a trained image-text matching network 400. Specifically, the image-text matching training data can include corresponding m images image_j, j = 1, 2, ..., m is a positive integer and m texts text_j. Then, the image-text similarity between the m images image_j and the m texts text_j can be calculated to obtain an m×m symmetric matrix, in which the values ​​on the diagonal of the symmetric matrix represent the similarity f1 between each image image_j and the corresponding text description text_j; the data outside the diagonal represent the similarity f2 between each image image_j and the text text_1 (1≠j) that does not match it. Based on the symmetric matrix, the weight of the initial neural network is adjusted to obtain a trained image-text matching network, so that the values ​​f1 on the diagonal of the symmetric matrix are close to 1 and the values ​​f2 on the off-diagonal are close to 0. It should be understood that the above-mentioned image-text matching network can actually also calculate the similarity between texts and the similarity between images.

[0050] In step S340, consecutive video frames with the largest total score within the target duration in the video to be processed are determined as target video segments, where the total score is obtained based on the semantic relevance and image quality corresponding to the consecutive video frames.

[0051] Image quality can be evaluated based on multiple image quality dimensions, such as image clarity, subject definition, and color richness. It should be understood that the above image quality dimensions are merely illustrative and not intended to be limiting. Fewer or more image quality dimensions may be included, and can be set based on actual needs, without limitation here.

[0052] In some embodiments, the total score is obtained based on the semantic relevance and image quality of the continuous video frames, including: scoring the image quality of the video frames based on at least one image quality dimension to obtain the image quality score of the video frames; and scoring the video frames based on the semantic relevance to obtain the semantic relevance score of the video frames; and obtaining the total score based on the sum of the image quality score and the semantic relevance score.

[0053] A higher image quality score indicates a better visual experience and presentation quality. A higher semantic relevance score indicates a closer match between the image content and the video being processed. Combining the image quality and semantic relevance scores to create a total score allows for the selection of the final video highlight segments based on both visual quality and content, improving the accuracy and presentation quality of the video highlight segments.

[0054] In some embodiments, scoring the image quality of the video frame based on at least one image quality dimension to obtain the image quality score of the video frame includes: inputting the video frame into an image quality network to obtain the image quality score; wherein the image quality network is obtained by training an initial neural network based on training data, and the training data includes training image samples and the true image quality scores of the training image samples determined based on the at least one image quality dimension.

[0055] The image quality of the input image can be scored using a trained image quality network to obtain a corresponding image quality score. The trained image quality network can be obtained by training an initial neural network based on training data including training image samples and corresponding true scores.

[0056] In some embodiments, the training data includes a first training image sample having a first true image quality score and a second training image sample having a second true image quality score; the image quality network is obtained by training an initial neural network based on the training data, including: performing image quality scoring on the first training image sample and the second training image sample based on the initial neural network to obtain a first estimated score and a second estimated score; based on a first difference between the first estimated score and the second estimated score, and a second difference between the first image quality true score and the second image quality true score, obtaining the product of the first difference and the second difference; obtaining a loss function by taking the maximum value between a preset value and the product; and adjusting the weight parameters of the initial neural network based on the loss function to obtain the image quality network so as to minimize the loss function.

[0057] Specifically, the image quality network can be recorded as S = f(x), where x is the input image, f is the image quality network, and S is the image quality score. During the training process, the true score of each training image sample can be obtained, for example, 0 to 1 points, where 0 represents poor image quality, unclear, inconspicuous portraits, inconspicuous subjects, dull colors, unbalanced composition, etc., and 1 represents high image quality, clarity, rich colors, reasonable design, strong visual perception, etc. The training loss function is pair-wise loss. Specifically, for two training image samples: the first training image sample x1 and the second training image sample x2, corresponding to the first image quality true score y1 and the second image quality true score y2, respectively, the predicted image quality scores are S1 = f(x1) and S2 = f(x2). The training goal may include maintaining the relative order of y1 and y2 between S1 and S2. For example, if y1>y2, then S1>S2, and vice versa. Therefore, the loss function may include:

[0058] It can be seen that as long as the first difference (S1-S2) and the second difference (y1-y2) have the same sign, the loss is equal to 0; if the two have different signs, the loss function is greater than 0. Therefore, the training goal can be to minimize the above loss function For any two images, the relative image quality scores given by the image quality network are consistent with the relative quality scores of the true images.

[0059] In some embodiments, the continuous video frames with the largest total score within the target duration in the video to be processed are determined as the target video segment, including: determining the target number of frames of the continuous video frames based on the target duration; sliding a sliding window with the target number of frames in the video to be processed, and calculating the total score of the continuous video frames within the sliding window; and determining the continuous video frames within the sliding window with the largest total score as the target video segment.

[0060] Specifically, the user can specify to obtain the video highlights of the target length, such as K seconds. Since the preset frame rate is a frame / s, for example, a=3, the target number of frames can be 3K. Then, a sliding window with a length of 3K can be set based on the target number of frames, and the sum of the total scores of all continuous video frames in the sliding window ∑Score=∑(Score1+Score2) is calculated. Score1 is the image quality score of the video frame, and Score2 is the semantic relevance score. The sliding window can slide from the starting position of the video to be processed or the sequence of video frames to be processed to the end position, and the continuous video frame with the largest length of 3K among all ∑Scores during this period is used as the target video segment. The target video segment is a video highlight segment that meets the user's requirements, has high relevance in content and good visual effects.

[0061] In some embodiments, reference may be made to FIG5 , which shows a schematic diagram of a video processing method according to an embodiment of the present disclosure. In FIG5 , the video processing method according to an embodiment of the present disclosure may be executed based on a video processing model 500 . Specifically, the video to be processed is input into the video processing model 500 , and the video processing model may perform video frame extraction on the input video to be processed, for example, based on a preset frame rate of 3 FPS, to obtain N video frames. The N video frames are input into the image quality network 510 to obtain N image quality scores Score1 . The N video frames are input into the text description network 520 to obtain N video frame text descriptions, and the N video frame text descriptions are input into the text feature network 530 for feature extraction to obtain N video frame text features. The similarity between each pair of video frame text features is then calculated, and the video frame text description corresponding to the video frame text feature with the highest similarity is selected as the video text description of the video to be processed. The N video frames are input into the image feature network 540 to obtain image features of the N video frames, and the cosine similarity between the N image features and the video text description is calculated to obtain N semantic relevance scores Score2 . The target number of frames is obtained based on the target duration, and the total score sum ∑Score of all consecutive video frames in the sliding window including the target number of frames is calculated. The consecutive video frames with the largest total score sum ∑Score during the sliding window are used as the target video clip.

[0062] It can be seen that the video processing method according to the embodiment of the present disclosure can not only return the most representative content and the most ornamental pictures in the entire video to be processed, but also can adaptively adjust according to the target duration specified by the user, thereby improving the effect and efficiency of video editing processing.

[0063] It should be noted that the method of the embodiments of the present disclosure can be performed by a single device, such as a computer or server. The method of the embodiments of the present disclosure can also be applied in a distributed scenario, where multiple devices cooperate to perform the method. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiments of the present disclosure, and the multiple devices will interact with each other to complete the method.

[0064] It should be noted that the above description is limited to some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0065] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a video processing device, see Figure 6, the video processing device includes: an acquisition module, used to acquire the video to be processed and the target duration; a text description module, used to determine the video text description of the video to be processed based on the image content of the video to be processed; a picture-text matching module, used to determine the semantic correlation between the video frames in the video to be processed and the corresponding video text description; a target segment module, used to determine the continuous video frames with the largest total score within the target duration in the video to be processed as the target video segment, and the total score is obtained based on the semantic correlation and image quality corresponding to the continuous video frames.

[0066] For the convenience of description, the above devices are described as being functionally divided into various modules. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0067] The apparatus of the above embodiment is used to implement the corresponding video processing method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0068] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the video processing method described in any of the above embodiments.

[0069] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0070] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the video processing method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0071] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0072] In addition, to simplify the description and discussion, and so as not to obscure the embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0073] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0074] The embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A video processing method, the method comprising: Get the video to be processed and the target duration; Determining a video text description of the video to be processed based on the image content of the video to be processed; Determining the semantic relevance between the video frames in the video to be processed and the corresponding video text descriptions; The continuous video frames with the largest total score within the target duration in the video to be processed are determined as target video segments, and the total score is obtained based on the semantic relevance and image quality corresponding to the continuous video frames.

2. The method according to claim 1, wherein Determining a video text description of the video to be processed based on image content of a video frame in the video to be processed includes: Determining a video frame text description corresponding to the video frame based on image content of the video frame; Extracting features of the video frame text description to obtain frame text features; Calculating the similarity between the frame text features; The video frame text description corresponding to the frame text feature with the greatest similarity is determined as the video text description of the video to be processed.

3. The method according to claim 1, wherein Determining the semantic relevance between the video frame and the corresponding video text description includes: Extracting features from the video frame to obtain image features, and extracting features from the video text description to obtain video text features; The semantic correlation between the image feature and the video text feature is calculated to obtain the semantic correlation between the video frame and the video text description.

4. The method according to claim 1, wherein The total score is obtained based on the semantic relevance and image quality of the consecutive video frames, including: Scoring the image quality of the video frame based on at least one image quality dimension to obtain an image quality score of the video frame; and Scoring the video frame based on the semantic relevance to obtain a semantic relevance score for the video frame; The total score is obtained based on the sum of the image quality score and the semantic relevance score.

5. The method according to claim 4, wherein: Scoring the image quality of the video frame based on at least one image quality dimension to obtain an image quality score of the video frame includes: The video frame is input into an image quality network to obtain the image quality score; wherein the image quality network is obtained by training an initial neural network based on training data, wherein the training data includes training image samples and true image quality scores of the training image samples determined based on the at least one image quality dimension.

6. The method according to claim 5, wherein: The training data includes a first training image sample having a first true image quality score and a second training image sample having a second true image quality score; The image quality network is obtained by training an initial neural network based on training data, including: Based on the initial neural network, image quality scores are respectively performed on the first training image sample and the second training image sample to obtain a first estimated score and a second estimated score; Based on a first difference between the first estimated score and the second estimated score, and a second difference between the first image quality true score and the second image quality true score, obtaining a product of the first difference and the second difference; The maximum value between the preset value and the product is used to obtain the loss function; The image quality network is obtained by adjusting the weight parameters of the initial neural network based on the loss function to minimize the loss function.

7. The method according to claim 1, determining the continuous video frames with the maximum total score within the target duration in the video to be processed as the target video segment, comprising: Determining a target number of the continuous video frames based on the target duration; Sliding a sliding window having the target number of frames in the video to be processed, calculating a total score of continuous video frames in the sliding window; The continuous video frames in the sliding window with the largest total score are determined as the target video segment.

8. A video processing device, comprising: The acquisition module is used to obtain the video to be processed and the target duration; A text description module, used to determine a video text description of the video to be processed based on the image content of the video to be processed; A picture-text matching module, used to determine the semantic relevance between the video frame in the video to be processed and the corresponding video text description; The target segment module is used to determine the continuous video frames with the largest total score within the target duration in the video to be processed as the target video segment, wherein the total score is based on the language corresponding to the continuous video frames. The semantic correlation and image quality are obtained.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program. 10 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to claim 1 .

11. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 7.

Images