Video processing method and related device

Abstract

The present disclosure provides a video processing method and a related device. The method comprises: acquiring a video to be processed; on the basis of scene switching positions, dividing said video into a plurality of scene video clips; determining a text description corresponding to each scene video clip; and on the basis of semantic relevance between the text descriptions, removing the scene video clips having repeated semantics in said video, to obtain a target video.

Description

Video processing method and related equipment

[0001] This application claims priority to the Chinese invention patent application entitled “Video Processing Method and Related Equipment” and application number 2023110221283, filed on August 14, 2023, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of computer technology, and in particular to a video processing method, apparatus, device, medium, and program product. Background Art

[0003] In video creation, high-quality and long video materials often take up a large storage space. Importing these materials into video editing software often requires more computing resources, which not only poses a challenge to the storage space and computing power of the editing equipment, but also increases the workload of users in editing videos and reduces the efficiency of video processing.

[0004] Summary of the Invention

[0005] The present disclosure provides a video processing method, apparatus, device, storage medium and program product to solve the technical problem of low efficiency of video editing to a certain extent.

[0006] In a first aspect of the present disclosure, a video processing method is provided, comprising: obtaining a video to be processed; dividing the video to be processed into a plurality of scene video segments based on scene switching positions; determining a text description corresponding to each of the scene video segments; and removing the scene video segments with repeated semantics in the video to be processed based on the semantic relevance between the text descriptions to obtain a target video.

[0007] In a second aspect of the present disclosure, a video processing device is provided, comprising: an acquisition module for acquiring a video to be processed; a segmentation module for dividing the video to be processed into a plurality of scene video segments based on scene switching positions; a text module for determining a text description corresponding to each of the scene video segments; and a deduplication module for removing scene video segments with repeated semantics in the video to be processed based on the semantic relevance between the text descriptions to obtain a target video.

[0008] In a third aspect of the present disclosure, an electronic device is provided, characterized in that it includes one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to the first aspect or the second aspect.

[0009] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors execute the method described in the first aspect or the second aspect.

[0010] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] FIG1 is a schematic diagram of a video processing architecture according to an embodiment of the present disclosure.

[0013] FIG2 is a schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure.

[0014] FIG3 is a schematic flowchart of a video processing method according to an embodiment of the present disclosure.

[0015] FIG4 is a schematic diagram of a video processing method according to an embodiment of the present disclosure.

[0016] FIG5 is a schematic diagram of a video processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0018] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.

[0019] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0020] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0021] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0023] FIG1 shows a schematic diagram of a video processing architecture according to an embodiment of the present disclosure. Referring to FIG1 , the video processing architecture 100 may include a server 110, a terminal 120, and a network 130 providing a communication link. The server 110 and the terminal 120 may be connected via a wired or wireless network 130. The server 110 may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, security services, and CDN.

[0024] Terminal 120 can be implemented in hardware or software. For example, when implemented in hardware, terminal 120 can be any electronic device with a display screen that supports page display, including but not limited to smartphones, tablet computers, e-book readers, laptop computers, and desktop computers. When terminal 120 is implemented in software, it can be installed in the electronic devices listed above; it can be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or it can be implemented as a single software or software module, and no specific limitations are given here.

[0025] It should be noted that the video processing method provided in the embodiments of the present application can be executed by the terminal 120 or by the server 110. It should be understood that the number of terminals, networks, and servers in FIG1 is for illustration only and is not intended to limit the number thereof. Any number of terminals, networks, and servers can be used as required.

[0026] Figure 2 shows a schematic diagram of the hardware structure of an exemplary electronic device 200 provided by an embodiment of the present disclosure. As shown in Figure 2, electronic device 200 may include: a processor 202, a memory 204, a network module 206, a peripheral interface 208, and a bus 210. The processor 202, memory 204, network module 206, and peripheral interface 208 are communicatively connected to each other within electronic device 200 via bus 210.

[0027] The processor 202 may be a central processing unit (CPU), a video processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device (PLD), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or one or more integrated circuits. The processor 202 may be configured to perform functions related to the technology described in this disclosure. In some embodiments, the processor 202 may also include multiple processors integrated into a single logical component. For example, as shown in FIG2 , the processor 202 may include multiple processors 202 a, 202 b, and 202 c.

[0028] The memory 204 can be configured to store data (e.g., instructions, computer codes, etc.). As shown in Figure 2, the data stored in the memory 204 can include program instructions (e.g., program instructions for implementing the video processing method of the embodiment of the present disclosure) and data to be processed (e.g., the memory can store configuration files of other modules, etc.). The processor 202 can also access the program instructions and data stored in the memory 204, and execute the program instructions to operate on the data to be processed. The memory 204 can include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 204 can include a random access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0029] The network module 206 can be configured to provide the electronic device 200 with communication with other external devices via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC)), a cellular network, the Internet, or a combination thereof. It will be appreciated that the type of network is not limited to the specific examples above. In some embodiments, the network module 306 can include any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, and the like.

[0030] The peripheral interface 208 can be configured to connect the electronic device 200 to one or more peripheral devices to implement information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touchpad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

[0031] The bus 210 can be configured to transmit information between the various components of the electronic device 200 (e.g., the processor 202, the memory 204, the network module 206, and the peripheral interface 208), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0032] It should be noted that although the architecture of the electronic device 200 shown above only includes the processor 202, the memory 204, the network module 206, the peripheral interface 208, and the bus 210, in a specific implementation, the architecture of the electronic device 200 may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the architecture of the electronic device 200 may also include only the components necessary to implement the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0033] Advances in filming equipment allow people to capture video footage anytime, anywhere, and the development of high-capacity storage devices allows for increasingly longer footage. These high-quality, storage-intensive, and lengthy videos undoubtedly present significant obstacles and difficulties for video creators when editing. Importing these videos into video editing software also requires increased computing resources, placing higher demands and challenges on editing equipment. Therefore, how to conserve the storage and computing resources involved in video editing, improve editing efficiency, and maintain optimal presentation quality have become pressing technical challenges.

[0034] In view of this, embodiments of the present disclosure provide a video processing method and related equipment. By segmenting a video to be processed into multiple scene video segments based on scenes, and removing scene video segments with repeated semantics based on the semantic relevance between the scene video segments, the method can reduce the occurrence of repeated images or content while ensuring that the complete content of the video to be processed is reflected. This not only saves storage and computing resources, improves the efficiency of video editing, but also takes into account the presentation effect of video editing.

[0035] Specifically, the transition position of the video to be processed can be monitored to determine one or more scene switching positions, and based on the one or more scene switching positions, the video to be processed can be cut into multiple different scene video segments video_scene_i, i=1, 2,..., n, where n is a positive integer. For each scene video segment video_scene_i, its text description text_scene_i (for example, the summary corresponding to the scene video segment) can be determined, and the text description text_scene_i can be feature extracted to obtain the corresponding text feature feature_scene_i. Based on the semantic relevance between the text features feature_scene_i (that is, the semantic relevance between the corresponding scene video segments), the scene video segments with repeated semantics can be removed from the video to be processed to obtain the final target video. It can reduce the occurrence of repeated images or repeated content, save storage and computing resources, and improve the efficiency of video editing.

[0036] Referring to Figure 3, Figure 3 shows a schematic flow chart of a video processing method according to an embodiment of the present disclosure. In Figure 3, the video processing method 300 may further include the following steps.

[0037] In step S310, a video to be processed is obtained.

[0038] The video to be processed may include video frames of different scenes, and different scenes may include transition images connecting the previous and next scene videos, such as fade-in and fade-out, hard cut, rotation transformation, stretch transformation, etc. The video to be processed can be uploaded locally or obtained via the network.

[0039] In some embodiments, method 300 may further include: performing frame extraction on the video to be processed at a preset sampling frequency to obtain a sequence of video frames to be processed. The sampling frequency can be expressed as a frame rate (FPS), which represents the number of images contained in one second of video data. For example, if 30 images are contained in one second of video data, the FPS of the video data is 30. Typically, the frame rate of video data is between 30 and 60. The sampling frequency, when expressed as a frame rate (FPS), can represent the number of images sampled in one second. Specifically, the sampling frequency can be FPS = a, which allows the video to be extracted, extracting a frames per second. For example, if video data has a frame rate of 30, each second contains 30 frames. Sampling at a sampling frequency of FPS = 2 allows an even distribution of 2 frames from every 30 frames. Because video data generally has a high frame rate, which makes video processing time consuming, downsampling can be used to reduce computational complexity and improve video processing efficiency and response speed.

[0040] In some embodiments, a sequence of video frames to be processed can be used in place of the video to be processed for subsequent processing, thereby reducing the amount of data for video processing, saving computing costs, and improving the efficiency of video processing. Specifically, a video to be processed with a duration of b seconds can be frame-extracted based on a frame rate a, that is, a video frame is extracted per second in chronological order to obtain a video frame sequence X_a_b including a*b frames. In the subsequent processing process of the disclosed embodiment, the video frame sequence can be used as the data basis to replace the video to be processed to remove repeated semantic segments.

[0041] In step S320, the video to be processed is divided into a plurality of scene video segments based on scene switching positions.

[0042] The scene switching position may refer to a transition position in a video. Each scene video clip corresponds to the same scene, and the image in the scene video clip has no jitter and the video content remains substantially unchanged.

[0043] In some embodiments, the video to be processed is divided into multiple scene video segments based on the scene switching position, including: determining the scene switching probability of the video frame in the video to be processed; determining the video frame whose scene switching probability is greater than or equal to the switching threshold as the scene switching position; and cropping the video to be processed at the scene switching position to obtain the multiple scene video segments.

[0044] Specifically, the video to be processed can be input into a transition detection network to determine the scene switching positions, and the video can be cut at these scene switching positions to obtain multiple scene video segments. For example, referring to FIG4 , FIG4 shows a schematic diagram of a video processing method according to an embodiment of the present disclosure. In FIG4 , the video to be processed 410 is subjected to video frame extraction to obtain a sequence of video frames to be processed sequence, and m (m is a positive integer) video frames in the sequence of video frames to be processed sequence are sequentially input into a transition detection network 420 to obtain n scene video segments video_scene_1, ..., video_scene_n. Among them, the transition detection network 420 can perform transition detection on the m video frames, for example, the transition score sk = f(xk), where xk is the kth video frame input and f is the transition scoring function. Based on this, the transition score S = [s1, s2, s3, ..., sm]) of the m video frames can be obtained to represent the probability that each video frame is a transition. A switching threshold threshold 1 can be set. In the transition score S = [s1, s2, s3, ..., sm]), the video frame with a transition score greater than or equal to threshold 1 is the scene switching location. For example, if s20 is greater than or equal to threshold 1, the video frame frame 20 corresponding to s20 is the scene switching location. The sequence of video frames to be processed, sequence, can then be truncated at the scene switching location to obtain multiple scene video segments video_scene_1, ..., video_scene_n.

[0045] In some embodiments, the transition detection network can be obtained by training an initial neural network based on transition detection training data. Furthermore, the transition detection training data may include training images and corresponding transition true scores, and the higher the transition true score, the greater the probability that the training image is a transition. Specifically, the training image is used as input layer data and the corresponding transition true score is used as output layer data to train the initial neural network. Based on the transition training network, the transition estimation score of the training image is obtained, and the difference (or variance) between the transition estimation score and the transition true score is calculated to obtain a loss function. Based on the loss function, the weight of the initial neural network is adjusted to minimize the loss function to obtain a trained transition detection network.

[0046] In step S330, a text description corresponding to each of the scene video clips is determined.

[0047] The text description corresponding to the scene video clip may refer to a representative text description used to describe the main image content of the scene video clip, such as a scene video summary.

[0048] In some embodiments, determining the text description corresponding to each of the scene video clips includes: determining multiple video frame text descriptions corresponding to the video frame based on the image content of each video frame in the scene video clip; performing feature extraction on the multiple video frame text descriptions to obtain multiple frame text features; for each frame text feature, calculating the frame text similarity between the frame text feature and other frame text features; and determining the video frame text description corresponding to the frame text feature with the largest frame text similarity as the text description of the scene video clip.

[0049] For each scene video clip, the video frame text description corresponding to each video frame in the scene video clip can be used to select the video frame text description with the highest similarity to other video frame text descriptions from the video frame text descriptions as the video summary of the scene video clip. Specifically, the scene video clip is input into the image description network, which can be represented as h. The output text description is t, where t = h(x), where x is the input image. For example, if the input image x contains a dog and grass, the text description t can be "dog playing on the lawn."

[0050] The image description network 430 can be used to perform image description on the video frames in each scene video clip, and output the video frame text description corresponding to each video frame. Specifically, as shown in FIG4 , text descriptions text_scene_1, ..., text_scene_n corresponding to n scene video clips video_scene_1, ..., video_scene_n can be obtained. For example, the scene video clip video_scene_1 includes K1 video frames, K1 is a positive integer, and then K1 video frame text descriptions corresponding to the K1 video frames can be output respectively based on the image description network 430. The K1 video frame text descriptions are subjected to feature extraction based on a text feature network (which can convert the input text information into a text feature of a fixed length, for example, the input text information can be converted into a feature vector of a length of 512, i.e., a floating point vector) to obtain K1 frame text features. For each frame text feature, calculate the similarity between that frame text feature and other inter-frame features, resulting in K1-1 similarities (e.g., cosine similarity). The frame text description corresponding to the frame text feature with the greatest similarity is determined as the text description text_scene_1 for the scene video segment video_scene_1. Similarly, text descriptions for other scene video segments can be obtained, which will not be repeated here.

[0051] In step S340, the scene video segments with repeated semantics in the video to be processed are removed based on the semantic relevance between the text descriptions to obtain a target video.

[0052] Semantic relevance refers to the degree of similarity between image features and text features, i.e., cosine similarity. The semantic relevance between the text descriptions of scene video segments can reflect the overall relevance between the scene video segments and the main content of the entire video being processed, as well as the segment relevance between scene video segments.

[0053] In some embodiments, the scene video segments with repeated semantics in the video to be processed are removed based on the semantic correlation between the text descriptions to obtain a target video, including: determining the overall correlation between the scene video segments and the video to be processed and the segment correlation between the scene video segments based on the semantic correlation between the text descriptions; determining the retained segments and the to-be-removed segments in the scene video segments based on the overall correlation and the segment correlation, the to-be-removed segments having repeated semantics with at least one of the retained segments; and removing the to-be-removed segments in the video to be processed to obtain the target video.

[0054] Segment correlation accurately determines whether there is semantic duplication between scene video segments, while overall correlation ensures the content integrity of the entire video being processed. Combining these two methods allows for the identification of semantically duplicated segments to be removed while ensuring the presentation of complete content. Removing these segments from the video to produce the target video is then done. This reduces the occurrence of duplicate images or content, conserving storage and computing resources, and improving video editing efficiency.

[0055] In some embodiments, the retained segments and to-be-removed segments in the scene video segments are determined based on the overall correlation and the segment correlation, and the to-be-removed segments have repeated semantics with at least one of the retained segments, including: recording the scene video segment with the smallest overall correlation as a retained segment in the retained segment set, and recording the scene video segment with not the smallest overall correlation as a to-be-removed segment in the to-be-removed segment set; repeating the following steps until a preset condition is met: calculating the semantic similarity between each to-be-removed segment in the to-be-removed segment set and the retained segment set based on the segment correlation; when the minimum value in the semantic similarity is less than a similarity threshold, moving the to-be-removed segment corresponding to the minimum value to the retained segment set; wherein the preset condition includes: the to-be-removed segment set is empty, and / or, the semantic similarity of all to-be-removed segments in the to-be-removed segment set with the retained segment set is less than the similarity threshold.

[0056] In some embodiments, the overall correlation between the scene video segments and the video to be processed and the segment correlation between the scene video segments are determined based on the semantic correlation between the text descriptions, including: extracting features from each of the text descriptions to obtain corresponding text features; calculating multiple text similarities between the text feature and other text features for each text feature to obtain the segment correlation between the scene video segments; and obtaining the overall correlation between the scene video segment corresponding to each text feature and the video to be processed based on the average value of the multiple text similarities.

[0057] In some embodiments, calculating the semantic similarity between each to-be-removed segment in the to-be-removed segment set and the retained segment set based on the segment relevance includes:

[0058] The maximum value of the segment relevance between the segment to be removed and each of the retained segments in the retained segment set is determined as the semantic similarity between the segment to be removed and the retained segment set.

[0059] Specifically, as shown in FIG4 , semantic deduplication can be performed on n scene video segments video_scene_1, ..., video_scene_n based on an iterative deduplication strategy. For example, the text descriptions text_scene_1, ..., text_scene_n of the multiple scene video segments output by the image description network 430 can be input into the text feature network 440. The text feature network 440 can perform feature extraction on each text description text_scene_1, ..., text_scene_n to obtain the corresponding n text features feature_1, ..., feature_n.

[0060] For each text feature feature_i (i=1, ..., n), calculate the text similarity between the text feature feature_i and other text features feature_j (j≠i), and you can get n-1 text similarities k1, ..., kn-1. Find the average value of these n-1 text similarities Σ(k1+ ... +kn-1)k / (n-1) to get the overall correlation similarity_i between the scene video segment video_scene_i and the video frame sequence to be processed (or the video to be processed). Since there are n text features, you can get n overall correlations. On this basis, the scene video segment video_scene_p corresponding to the minimum value similarity_p = min(similarity_i) among the n overall correlations is used as the starting segment, that is, segment K1, and is placed in the retained segment set K = [K1]. The other scene video segments are recorded as R2, ..., Rn-1 and placed in the set to be removed R = [R2, ..., Rn-1].

[0061] The final retained segments and segments to be removed can be determined based on the following iterative deduplication strategy: for each element Ry in the set to be removed R, y = 1, ..., n-1, calculate the text similarity_text between each element Ry and each element in the set of retained segments K, and determine the maximum value of the text similarity_text as the semantic similarity_Ry_K between the element Ry and the set of retained segments K. In this way, the n-1 semantic similarities of all n-1 elements in the removal set R and the set of retained segments K can be obtained. The minimum value of the semantic similarity_Ry_K is compared with the preset similarity threshold threshold2. If the minimum value is less than the similarity threshold threshold2 (for example, threshold2 = 0.8), the element corresponding to the minimum value is moved to the set of retained segments K as the retained segment. Repeat the calculation of each element in the set to be removed R based on the ascending iterative deduplication strategy until the preset conditions are met.

[0062] Specifically, the preset condition may include that the retained segment set K may include n segments, that is, the set to be removed is empty. The preset condition may also include that the semantic similarity between all the to-be-removed segments in the to-be-removed segment set and the retained segment set is less than the similarity threshold threshold2, that is, at this time, the similarity of the elements in the retained segment set K and the to-be-removed segment set R are both greater than the above-mentioned similarity threshold threshold2, and there are no scene video segments with low similarity. It can be seen that after the preset condition is met, the to-be-removed segment in the to-be-removed segment set is a scene video segment that has repeated semantics with at least one retained segment in the retained segment set. Furthermore, other video processing steps can be performed on the target video obtained after removing the scene video segments with repeated semantics, such as extracting highlight scenes with a specified duration on the target video, so that a video result with uniform duration for each scene can be obtained.

[0063] It can be seen that the video processing method according to the embodiment of the present disclosure can reduce the occurrence of repeated images or content while ensuring that the complete content of the video to be processed is reflected. It not only saves storage and computing resources, improves the efficiency of video editing, but also takes into account the presentation effect of video editing.

[0064] It should be noted that the method of the embodiments of the present disclosure can be performed by a single device, such as a computer or server. The method of the embodiments of the present disclosure can also be applied in a distributed scenario, where multiple devices cooperate to perform the method. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiments of the present disclosure, and the multiple devices will interact with each other to complete the method.

[0065] It should be noted that the above description is limited to some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0066] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a video processing device, see Figure 5, the video processing device includes: an acquisition module, used to acquire the video to be processed and the target duration; an acquisition module, used to acquire the video to be processed; a segmentation module, used to divide the video to be processed into multiple scene video segments based on the scene switching position; a text module, used to determine the text description corresponding to each of the scene video segments; a deduplication module, used to remove the scene video segments with repeated semantics in the video to be processed based on the semantic correlation between the text descriptions, to obtain the target video.

[0067] For the convenience of description, the above devices are described as being functionally divided into various modules. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0068] The apparatus of the above embodiment is used to implement the corresponding video processing method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0069] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the video processing method described in any of the above embodiments.

[0070] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0071] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the video processing method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0072] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0073] In addition, to simplify the description and discussion, and so as not to obscure the embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0074] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0075] The embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A video processing method, the method comprising: Get the video to be processed; Dividing the video to be processed into a plurality of scene video segments based on scene switching positions; Determining a text description corresponding to each of the scene video clips; Based on the semantic relevance between the text descriptions, the scene video segments with repeated semantics in the video to be processed are removed to obtain a target video.

2. The method according to claim 1, wherein The process of removing the scene video segments with repeated semantics from the video to be processed based on the semantic relevance between the text descriptions to obtain a target video includes: Determine the overall relevance between the scene video segments and the video to be processed and the segment relevance between the scene video segments based on the semantic relevance between the text descriptions; Determine a retained segment and a to-be-removed segment in the scene video segment based on the overall relevance and the segment relevance, wherein the to-be-removed segment has repeated semantics with at least one of the retained segments; The to-be-removed segment is removed from the to-be-processed video to obtain the target video.

3. The method according to claim 2, wherein: Determining a retained segment and a to-be-removed segment in the scene video segment based on the overall relevance and the segment relevance, wherein the to-be-removed segment has duplicate semantics with at least one of the retained segments, includes: Recording the scene video segment with the smallest overall correlation as a retained segment in the retained segment set, and recording the scene video segment with not the smallest overall correlation as a to-be-removed segment in the to-be-removed segment set; Repeat the following steps until the preset conditions are met: Calculating the semantic similarity between each to-be-removed segment in the to-be-removed segment set and the retained segment set based on the segment relevance; Moving the to-be-removed segments whose semantic similarity is less than a similarity threshold to the reserved segment set; The preset conditions include: the set of segments to be removed is empty, and / or the semantic similarity between all the segments to be removed in the set of segments to be removed and the set of retained segments is less than the similarity threshold.

4. The method according to claim 3, wherein: Determining the overall relevance between the scene video segments and the video to be processed and the segment relevance between the scene video segments based on the semantic relevance between the text descriptions includes: Performing feature extraction on each of the text descriptions to obtain corresponding text features; For each text feature, calculating multiple text similarities between the text feature and other text features to obtain segment correlations between the scene video segments; The overall relevance between the scene video segment corresponding to each of the text features and the video to be processed is obtained based on an average value of the multiple text similarities.

5. The method according to claim 4, wherein: Calculating the semantic similarity between each to-be-removed segment in the to-be-removed segment set and the retained segment set based on the segment relevance includes: The maximum value of the segment relevance between the segment to be removed and each of the retained segments in the retained segment set is determined as the semantic similarity between the segment to be removed and the retained segment set.

6. The method according to claim 1, wherein: The video to be processed is divided into a plurality of scene video segments based on the scene switching position, including: Determining a scene switching probability of a video frame in the video to be processed; Determine the video frame whose scene switching probability is greater than or equal to the switching threshold as the scene switching position; The video to be processed is cropped at the scene switching position to obtain a plurality of the scene video clips.

7. According to the method of claim 1, determining the text description corresponding to each of the scene video clips comprises: Determining a plurality of video frame text descriptions corresponding to each video frame in the scene video clip based on image content of the video frame; Extracting features from the plurality of video frame text descriptions to obtain a plurality of frame text features; For each frame text feature, calculating the frame text similarity between the frame text feature and other frame text features; The video frame text description corresponding to the frame text feature with the largest frame text similarity is determined as the text description of the scene video segment.

8. A video processing device, comprising: An acquisition module is used to acquire the video to be processed; A segmentation module, used for dividing the video to be processed into a plurality of scene video segments based on scene switching positions; A text module, used to determine a text description corresponding to each of the scene video clips; The deduplication module is used to remove scene video segments with repeated semantics in the video to be processed based on the semantic relevance between the text descriptions to obtain a target video.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program. 10 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to claim 1 .

11. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 7.

Images