Multi-semantic video processing method, device, equipment, medium and program product

Abstract

The embodiment of the present invention provides a multi-semantic video processing method, device, equipment, medium and program product, the method comprising: firstly extracting features from a spatial perspective to obtain the spatial features of each video frame in the video to be processed, and then extracting features from a temporal perspective to obtain the spatiotemporal features of each video frame. Then, the spatiotemporal features of the entire video to be processed are obtained according to the spatiotemporal features of each video frame. Finally, the semantics of the video can be analyzed according to the spatiotemporal features of the entire video to be processed to locate the semantic boundary points in the video, that is, the target video frames with semantic changes. Among them, the spatiotemporal features reflect the spatial information, color information and posture change information of the object in the video frame. Therefore, the spatiotemporal features of the entire video can more completely and accurately reflect the posture changes of the object in the video, so that the semantic analysis of the video can be performed in a more fine-grained manner, and the positioning accuracy of the semantic boundary points in the video can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-semantic video processing method, device, equipment, medium and program product. Background Art

[0002] A large amount of video and picture materials can be stored in electronic devices, such as mobile phones and tablet computers. Some users hope to automatically extract the highlights from the video materials and use them as a video collection, or use the highlights to make a video blog (Vlog) or other purposes.

[0003] In practice, it is necessary to understand the semantics of the video, and based on the semantic understanding results, extract valid segments with complete semantics from the video, and then extract highlights from the valid video segments. And it is easy to understand that a video can include at least one shot, a shot can include at least one scene, and a scene can include at least one event. Therefore, the valid segments and highlights extracted from a video can be at the shot level, scene level or event level.

[0004] Based on the above description, in order to ensure the extraction effect of effective segments, how to refine the granularity of video semantic recognition becomes an urgent problem to be solved. Summary of the invention

[0005] In view of this, embodiments of the present invention provide a multi-semantic video processing method, apparatus, device, medium and program product to refine the granularity of semantic recognition.

[0006] In a first aspect, an embodiment of the present invention provides a multi-semantic video processing method, including:

[0007] Extracting spatial features of the video to be processed to obtain spatial features of each video frame in the video to be processed;

[0008] Determining the spatiotemporal features of the video frames according to the spatial features of the video frames;

[0009] Determining the spatiotemporal features of the video to be processed according to the spatiotemporal features of the respective video frames;

[0010] According to the spatiotemporal features, a target video frame having a semantic change in the video to be processed is determined.

[0011] In a second aspect, an embodiment of the present invention provides a multi-semantic video processing device, including:

[0012] A spatial feature determination module is used to extract spatial features from the video to be processed to obtain spatial features of each video frame in the video to be processed;

[0013] A first spatiotemporal feature determination module, configured to determine the spatiotemporal features of each of the video frames according to the spatial features of each of the video frames;

[0014] A second spatiotemporal feature determination module, configured to determine the spatiotemporal features of the video to be processed according to the spatiotemporal features of each of the video frames;

[0015] The semantic boundary point determination module is used to determine the target video frame with semantic changes in the video to be processed according to the spatiotemporal features.

[0016] In a third aspect, an embodiment of the present invention provides a terminal device, comprising a memory and a processor; wherein the memory is used to store one or more computer instructions, and when the one or more computer instructions are executed by the processor, the multi-semantic video processing method in the first aspect is implemented. The terminal device may also include a communication interface for communicating with other devices or communication systems.

[0017] In a fourth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of a non-temporary machine, the processor can at least implement the multi-semantic video processing method as described in the first aspect.

[0018] In a fifth aspect, an embodiment of the present invention provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the processor is enabled to implement the multi-semantic video processing method as described in the first aspect.

[0019] In the multi-semantic video processing method provided by the embodiment of the present invention, for each video frame in the video to be processed, feature extraction is first performed from a spatial perspective to obtain the spatial features of each video frame, and then feature extraction is performed from a temporal perspective to obtain the spatiotemporal features of each video frame. Furthermore, the spatiotemporal features of the entire video to be processed are obtained based on the spatiotemporal features of each video frame. Finally, the semantics of the video can be analyzed based on the spatiotemporal features of the entire video to be processed to locate the target video frame whose semantics has changed in the video, and the position of the target video frame in the video to be processed is also the semantic boundary point of the video to be processed. Among them, the spatiotemporal features can reflect the change information of the posture of the object in the video on the basis of reflecting the spatial information, color information, and texture information of the object in the video frame.

[0020] The above method uses the spatiotemporal features of the entire video, which can more completely reflect the changing information of the object posture in the video, so that the semantics of the video can be analyzed in a more fine-grained manner, and the semantic boundary points at the event level in the video can be obtained, which is to improve the positioning accuracy of the semantic boundary points in the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A flowchart of a multi-semantic video processing method provided by an embodiment of the present invention;

[0023] Figure 2 A conceptual framework diagram of a multi-semantic video processing method provided by an embodiment of the present invention;

[0024] Figure 3 A structural diagram of a semantic analysis model provided by an embodiment of the present invention;

[0025] Figure 4 A flow chart of a spatiotemporal feature extraction method provided by an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of a semantic analysis model working process provided by an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of the working process of another semantic analysis model provided by an embodiment of the present invention;

[0028] Figure 7 A schematic diagram of a working process of a semantic calibration model provided by an embodiment of the present invention;

[0029] Figure 8a A schematic diagram of an interface of a terminal device provided by an embodiment of the present invention;

[0030] Figure 8b A schematic diagram of an interface of another terminal device provided by an embodiment of the present invention;

[0031] Figure 8c A schematic diagram of an interface of another terminal device provided by an embodiment of the present invention;

[0032] Fig. 9 A schematic diagram of a software architecture of a terminal device provided by an embodiment of the present invention;

[0033] Fig.10 A schematic diagram of the hardware structure of another terminal device provided in an embodiment of the present invention;

[0034] Fig.11 A schematic diagram of the structure of a multi-semantic video processing device provided by an embodiment of the present invention;

[0035] Fig.12 A schematic diagram of the structure of a terminal device provided by an embodiment of the present invention;

[0036] Fig.13 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two, but does not exclude the inclusion of at least one.

[0039] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0040] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to identifying", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is identified" may be interpreted as "when determining" or "in response to determining" or "when identifying (stated condition or event)" or "in response to identifying (stated condition or event)", depending on the context.

[0041] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0042] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such a product or system. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the product or system including the elements.

[0043] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features and steps in the embodiments can be combined with each other. The step sequence in the following method embodiments is only an example and not a strict limitation.

[0044] Before describing the contents of the following embodiments of the present invention in detail, the application scenarios of the present invention may also be described first:

[0045] As mentioned in the background technology, users have a need to reprocess video materials, where the reprocessing can specifically be to extract highlights from the video and further generate a Vlog or video summary.

[0046] The video captured by the camera during the period from being turned on once to being turned off once includes a shot. It is easy to understand that different shots may have different semantics. The camera may be installed on any electronic device with a shooting capability.

[0047] Furthermore, for a video containing one shot, if it is shot in a scene, such as a room, an office, etc., then one shot includes one scene. If it is shot from one scene to another scene, such as from outdoor to indoor room, or from theater stage to theater auditorium, etc., then one shot includes multiple scenes. It is easy to understand from the above examples that different scenes can have different semantics.

[0048] Furthermore, a scene in a video may also include at least one event. For example, in a theater stage scene, different postures of an actor may be captured, such as the actor's dance movements and dialogue performance. The different postures of the actor, namely the dance movements and dialogue performance, may be considered as two events occurring in the scene. As can be easily understood from the above examples, different events may also have different semantics.

[0049] Based on the above introduction to shots, scenes and events, it is easy to understand that the highlights extracted from the video can be at the shot level, scene level or event level, and the highlights are further extracted from the effective segments. Therefore, in order to ensure the accuracy of the highlights extraction, the multi-semantic video processing method provided by the following embodiments of the present invention can be used to achieve event-level semantic recognition capabilities to ensure the accuracy of effective segment extraction. Based on accurate effective segments, the quality of the highlights can be further guaranteed.

[0050] In addition, the multi-semantic video processing method provided in the following embodiments of the present invention can be executed by any first device with video processing capability, which may include but is not limited to mobile phones, laptops, tablet computers, etc. The video to be processed in the methods provided in the embodiments of the present invention can be any second device with video shooting capability, which may include but is not limited to mobile phones, tablet computers, smart screens, etc. And the first device and the second device can be the same device or different devices. When the first device and the second device are the same electronic device, the electronic device can provide video shooting function and video processing function at the same time, and can be commonly used as terminal devices such as mobile phones and tablet computers.

[0051] Based on the above description, various embodiments of the present invention may be described in detail below. Figure 1 The flowchart of a method for processing multi-semantic video provided by an embodiment of the present invention. The method provided by an embodiment of the present invention can be executed by any electronic device with video processing capability. Figure 1 As shown, the method may include the following steps:

[0052] S101, performing spatial feature extraction on the video to be processed to obtain spatial features of each video frame in the video to be processed.

[0053] Optionally, the electronic device may provide an operation interface corresponding to the material library, and the user may trigger a selection operation on the candidate video in the material library on the operation interface. Then, the electronic device obtains the video to be processed in response to the selection operation. In one optional acquisition method, the electronic device may directly obtain the video to be processed stored locally. In another optional acquisition method, the electronic device may send a video acquisition request to the cloud server to obtain the video to be processed sent by the cloud server. Optionally, the operation interface may be as follows: Figure 8a shown.

[0054] Optionally, the electronic device may acquire a long video stored locally or on a cloud server. In order to reduce the data processing pressure of the electronic device, the video to be processed may be at least one video segment of the long video.

[0055] Furthermore, the electronic device may perform spatial feature extraction on the acquired video to be processed to obtain spatial features of different video frames in the video to be processed.

[0056] In order to ensure the accuracy of spatial feature extraction, the electronic device can optionally perform pre-processing such as decoding and frame extraction on the video to be processed to obtain a video frame sequence corresponding to the video to be processed. Then, the spatial features of each video frame in the video frame sequence are extracted in turn. Among them, any video frame in the video frame sequence can be considered as an image, and the spatial features of any video frame can also be considered as image features, which can specifically include color features, texture features, shape features, spatial structure features, etc. of the video frame. And the spatial features can describe the global features of the video frame.

[0057] Although electronic devices can also recognize the semantics of any video frame by using spatial features, unlike a single image, at least one video frame adjacent to any video frame in the video to be processed can also affect the semantics of the video frame. In order to improve the accuracy of semantic recognition, the electronic device can also perform the following spatiotemporal feature extraction steps based on obtaining the spatial features of the video frame.

[0058] S102, determining the spatiotemporal features of the respective video frames according to the spatial features of the respective video frames.

[0059] The electronic device may further extract the temporal and spatial features of each video frame. Optionally, the electronic device may fuse the spatial features of adjacent video frames of any video frame with the spatial features of the any video frame, and the fusion result may be used as the temporal and spatial features of the any video frame.

[0060] As for the method of determining adjacent video frames, optionally, the adjacent video frames of any video frame can be determined according to a preset window length. For example, assuming that any video frame is the Mth frame in the video to be processed, and the window length is 3, then the adjacent video frames are the M-1th frame and the M+1th frame in the video to be processed. Then the fusion result of the spatial features of the M-1th frame, the Mth frame, and the M+1th frame is the spatiotemporal features of the Mth frame video.

[0061] Since the spatiotemporal features of any video frame are actually obtained by fusing the spatial features of multiple adjacent frames in the video to be processed, the spatiotemporal features can reflect the temporal changes in the number of objects in any video frame, and can also reflect the temporal changes in the posture of an object in any video. And according to the above introduction to shots, scenes, and events, the change in the number of objects means that there are multiple scenes in the video to be processed, and the change in the posture of the same object means that there are multiple events in a scene in the video to be processed. Therefore, the granularity of semantic recognition can be refined according to the spatial features of the video frame, thereby improving the accuracy of semantic recognition.

[0062] S103, determining the spatiotemporal features of the video to be processed according to the spatiotemporal features of the respective video frames.

[0063] S104, determining a target video frame with semantic changes in the video to be processed according to the spatiotemporal features of the video to be processed.

[0064] Furthermore, the electronic device can also obtain the spatiotemporal features of the entire video to be processed based on the spatiotemporal features of the respective video frames. Optionally, the spatiotemporal features of the respective video frames are directly spliced ​​to obtain the spatiotemporal features of the entire video to be processed. Finally, the electronic device can determine the target video frame with semantic changes in the video to be processed based on the spatiotemporal features of the entire video to be processed.

[0065] It should be noted that, since the target video frame is a video frame whose semantics has changed in the video to be processed, the position of the target video frame in the video to be processed can also be called a semantic boundary point.

[0066] Compared with the spatiotemporal features of each video frame, the spatiotemporal features of the entire video to be processed can more completely reflect the temporal changes in the posture of an object in the video. Therefore, the electronic device can more accurately identify semantic changes at the event level.

[0067] In this embodiment, for each video frame in the video to be processed, feature extraction is first performed from a spatial perspective to obtain the spatial features of each video frame, and then feature extraction is performed from a temporal perspective to obtain the spatiotemporal features of each video frame. Furthermore, the spatiotemporal features of the entire video to be processed are obtained based on the spatiotemporal features of each video frame. Finally, the semantics of the video can be analyzed based on the spatiotemporal features of the entire video to be processed to locate the target video frame whose semantics has changed in the video, that is, to determine the semantic boundary points of the video to be processed. Among them, the spatiotemporal features can reflect the change information of the posture of the object in the video on the basis of reflecting the spatial information, color information, and texture information of the object in the video frame.

[0068] The above method uses the spatiotemporal features of the entire video, which can more completely reflect the changing information of the object posture in the video, so that the semantics of the video can be analyzed in a more fine-grained manner, and the semantic boundary points at the event level in the video can be obtained, which is to improve the positioning accuracy of the semantic boundary points in the video.

[0069] Optionally, any video frame contained in the video to be processed may include multiple objects, and in practice, the main object among the multiple objects often affects the semantics of any video frame. Before performing spatial feature extraction, the electronic device may also identify the main objects contained in each video frame in the video to be processed. Afterwards, the electronic device may extract spatial features from the area where the main object is located in the video frame. For the specific extraction process, please refer to Figure 1 The relevant description of step S101 in the illustrated embodiment.

[0070] Optionally, the electronic device can use the image segmentation model configured by itself to identify the main object in the video frame. Optionally, the main object can be the largest object in the video frame or an object located at a specific position in the video frame, etc.

[0071] In this embodiment, the recognition of the main object in the video frame can improve the interference caused by the non-main object in the video frame on the semantic recognition of the video frame, thereby ensuring the accuracy of the semantic recognition of the video frame.

[0072] Furthermore, in order to improve the accuracy of semantic boundary point recognition, Figure 1 The semantic boundary points are located after the embodiment shown. Optionally, the electronic device can also calibrate the semantic boundary points.

[0073] Furthermore, after completing the semantic boundary point recognition or semantic boundary point calibration, optionally, a semantically complete valid segment can be extracted from the video to be processed according to the semantic boundary points. Specifically, in order to ensure the quality of the valid segment, the electronic device can first divide the video frame to be processed into multiple video segments according to the position of the target video frame in the video to be processed. Afterwards, the multiple video segments can be edited according to the similarity between the multiple video segments, and the editing result is a high-quality valid segment extracted from the video to be processed. Among them, the specific editing process can be referred to the description in the following related embodiments.

[0074] In summary, the overall concept of the present invention may include multiple parts: the first part is to use the temporal and spatial characteristics of the entire video to be processed to determine the semantic boundary points of the video to be processed. The second part is to calibrate the semantic boundary points. The third part is to extract video clips from the video to be processed, and then use the calibrated semantic boundary points to edit the video clips to obtain semantically coherent and complete valid clips. The above concept can also be seen in Figure 2The framework diagram shown.

[0075] As for the first part of the overall inventive concept, the process of extracting the spatial features of each video frame in the video to be processed, that is, an optional implementation process of step S101, may be described in detail below.

[0076] Optionally, a semantic analysis model may be deployed in the electronic device, and the spatial features of each video frame in the video to be processed may be performed by the semantic analysis model. And the structure of the semantic analysis model may be as follows: Figure 3 Specifically, the video frame sequence obtained by extracting frames from the video to be processed can be input into the semantic analysis model, so that the spatial features of each video frame can be extracted by the first feature extraction network in the model. Optionally, the first feature extraction network can specifically be a backbone network commonly used in deep learning.

[0077] Optionally, there may be multiple videos to be processed that require semantic boundary recognition, and the sizes of the multiple videos may also be different. In this case, the sizes of the multiple videos to be processed may be unified first, and then the video frame sequence with unified sizes may be input into the semantic analysis model.

[0078] Optionally, in order to extract more abundant information contained in the video frame, the first feature extraction network may first use a convolution kernel of a preset size to extract the original spatial features of each video frame, and then use at least one first sampling rate to downsample the original spatial features of each video frame to obtain at least one scale of target spatial features of each video frame. Since the first sampling rate can downsample the spatial features of a certain video frame, using the first sampling rate is actually still a dimensionality reduction process of the original spatial features in the spatial dimension.

[0079] Assuming that there are T frames in the video to be processed, the scale of the original spatial feature is H*W, the number of first sampling rates is 4, and the first sampling rates increase successively, then optionally, the scales of the target spatial features obtained by downsampling using the first sampling rate can be respectively: , , , .

[0080] in, .

[0081] In order to reduce the subsequent data processing pressure of the semantic analysis model, the target space features of different scales can also be pooled to convert the target space features into one-dimensional feature vectors. Continuing with the above example, the length of the target space features after pooling can be expressed as: , , , .

[0082] Since the first sampling rate increases sequentially, the multi-scale spatial features of any video frame can form a feature pyramid regardless of whether it is processed by pooling. Spatial features of different scales can describe the information contained in any video frame at different granularities.

[0083] In this embodiment, the electronic device can extract the spatial features of the video frame by means of the semantic analysis model deployed by itself, and the use of multiple sampling rates can also enrich the information contained in the video frame.

[0084] The following also describes in detail the process of extracting the spatiotemporal features of each of the video frames in the to-be-processed video frames, wherein the spatiotemporal features can be extracted by the second feature extraction network in the semantic analysis model.

[0085] Figure 4 Flow chart of a spatiotemporal feature extraction method provided by an embodiment of the present invention. Optionally, this embodiment is also a specific implementation of step S102. Figure 4 As shown, the following steps may be included:

[0086] S201, splicing the spatial features of the respective video frames to obtain an original spatial feature sequence.

[0087] According to the description in the above embodiment, the first feature extraction network in the semantic analysis model can output the spatial features of each video frame in the video to be processed. The spatial features can be the target spatial features mentioned in the above embodiment, and the spatial features can also be used as the input of the second feature extraction network, so that the second feature extraction network can splice the spatial features of each video frame to obtain the original spatial feature sequence, and the number of features in the original feature sequence is equal to the total number of video frames in the video to be processed. Among them, the original feature sequence can be used , m is a positive integer.

[0088] When the first sampling rate is not used, the second feature extraction network can obtain an original spatial feature sequence. When M first sampling rates are used, the second feature extraction network can obtain M original spatial feature sequences.

[0089] S202: downsample the original spatial feature sequences using at least one second sampling rate to obtain candidate spatial feature sequences of the video to be processed.

[0090] Then, the second feature extraction network can further use the second sampling rate to downsample the original spatial feature sequence to obtain an alternative spatial feature sequence. And it should be noted that, since the original spatial feature sequence contains the spatial features of different video frames in the video to be processed, and different video frames correspond to different playback times, therefore, unlike the first sampling rate, the second sampling rate actually samples the original spatial feature sequence from the time dimension.

[0091] When M first sampling rates and N second sampling rates are used, the second feature extraction network can obtain M*N candidate spatial feature sequences. The candidate spatial feature sequences can be expressed as , m and n are positive integers. Any feature in the candidate spatial feature sequence can be expressed as , p is a positive integer, and T is the total number of features contained in the candidate space feature sequence.

[0092] S203, generating, according to the original spatial feature sequence and the candidate spatial feature sequence, the temporal and spatial features of the intermediate video frames in the video to be processed at different first sampling rates, wherein the intermediate video frames are any video frames in the video to be processed except the last frame.

[0093] Then, the second feature extraction network can also extract the original spatial feature sequence and alternative spatial feature sequences Generate the temporal and spatial features of the intermediate video frames in the video to be processed at different first sampling rates. The intermediate video frames can be video frames other than the last frame in the video to be processed. The temporal and spatial features of the qth frame in the video to be processed at the mth first sampling rate can be expressed as , .

[0094] S204, using a filter with a preset window size to perform convolution processing on the candidate spatial feature sequence to obtain the spatiotemporal features of the last frame in the video to be processed at different first sampling rates.

[0095] At the same time, the second feature extraction network can use a filter with a preset window size to perform convolution processing on the candidate spatial sequence features to obtain the spatiotemporal features of the last frame at different first sampling rates. The spatiotemporal features of the last frame in the video to be processed at the mth first sampling rate can be expressed as , , is a convolution operator with a kernel dimension of 1.

[0096] So far, the spatiotemporal features of each frame in the to-be-processed video frame at different first sampling rates have been obtained.

[0097] In this embodiment, after downsampling the spatial feature sequences at different first sampling rates at different second sampling rates (i.e., at different time dimensions), a feature sequence including time information can be obtained, and the spatiotemporal features of the video frames at different first sampling rates are further calculated.

[0098] In addition, the contents not described in detail in this embodiment and the technical effects that can be achieved can all be found in the relevant contents of the above embodiments, and will not be repeated here.

[0099] exist Figure 4 On the basis of the illustrated embodiment, the process of extracting the spatiotemporal features of the entire video to be processed may be described in detail below, which is an optional implementation process of step S103. Optionally, the spatiotemporal features of the entire video to be processed may also be executed by a semantic analysis model deployed in an electronic device. Specifically, the second feature extraction network in the semantic analysis model may also splice the spatiotemporal features of each video frame in the video to be processed at the same first sampling rate. The splicing result is the overall spatiotemporal features of the video to be processed at a certain first sampling rate. The spatiotemporal features may be expressed as .in, , The kernel dimension is 1 and contains The convolution operator has kernels, and the length of the concatenated result is .

[0100] Finally, the splicing results corresponding to the different first sampling rates are concatenated to obtain the spatiotemporal characteristics of the video to be processed. The spatial characteristics of the video to be processed can be expressed as .in, .

[0101] In this embodiment, with the help of the second feature extraction network in the semantic analysis model, the spatiotemporal features of the entire video to be processed can be obtained through convolution. The spatiotemporal features can more completely reflect the change information of the object posture in the video, so that the semantics of the video can be analyzed in a more fine-grained manner, and the semantic boundary points of the event level in the video can be obtained, that is, the positioning accuracy of the semantic boundary points in the video can be improved.

[0102] The process of extracting the spatiotemporal features of the entire video to be processed, which is an optional implementation process of step S104, may be described in detail below.

[0103] Optionally, this process can be performed by a decoding network in a semantic analysis model, that is, the spatiotemporal features of each video frame in the video to be processed are input into the decoding network in the semantic analysis model, so that the decoding network determines the correlation between the spatiotemporal features of each video frame in the video to be processed, and then outputs the position of the target video frame with semantic changes in the video to be processed and the confidence of the position according to the spatiotemporal features and correlation of the video to be processed. Optionally, the correlation between spatiotemporal features can be reflected by the similarity between spatiotemporal features.

[0104] Optionally, the decoding network may specifically include two one-dimensional convolutional layers.

[0105] Optionally, in order to reduce the influence of noise in the features on the output of semantic boundary points, the decoding network may also have a convolution layer with a Gaussian kernel, and this convolution layer may be connected after the aforementioned two convolution layers.

[0106] In this embodiment, the decoding network in the semantic analysis model is used to realize the positioning of semantic boundary points. In the positioning process, not only the spatiotemporal characteristics that reflect the spatial and temporal characteristics of the video frame are taken into account, but also the correlation between different video frames is taken into account, so the accuracy of semantic boundary point positioning can be improved.

[0107] Figure 3 The working process of extracting the above-mentioned spatial features and spatiotemporal features by the semantic analysis model shown is also the process of the first part of the overall inventive conception, which can also be combined with Figure 5 as well as Figure 6 Understand. Figure 6 In the example, N first sampling rates are used to sample to obtain lengths of … The feature vector of .

[0108] For the second part of the overall inventive concept, the calibration process of the semantic boundary points can be described in detail below. The semantic analysis model can determine the target video frames whose semantics have changed in the video to be processed in the manner provided in the above embodiment. Optionally, the electronic device can also verify the video frames output by the semantic analysis model according to the spatiotemporal characteristics of the target video frames in the video to be processed to obtain the verified video frames. Optionally, a semantic calibration model can be deployed in the electronic device, so that the model can perform calibration of the semantic boundary points.

[0109] In order to quickly split the video to be processed into multiple video segments according to the semantic boundary points later, the position information can also be embedded into the spatiotemporal features of each video frame in the video to be processed to obtain the processed spatiotemporal features of the video to be processed, wherein the embedded position information can reflect the position of the video frame corresponding to the spatiotemporal features in the video to be processed. Furthermore, the processed spatiotemporal features can be input into the semantic calibration model, so that the semantic calibration model outputs the position of the calibrated video frame in the video to be processed, and the confidence that the calibrated video frame is a video frame with semantic changes in the video to be processed.

[0110] Optionally, before adding the position information, the temporal and spatial features of the respective video frames in the video to be processed may be sorted according to the target video frames output by the semantic analysis model and the confidence of the target video frames, and the position information may be added according to the sorting results.

[0111] Alternatively, the structure of the semantic calibration model can be as follows Figure 7 Understand. Figure 7 As shown, the semantic calibration model can be a model of an encoder-decoder structure.

[0112] Based on this semantic calibration model, the processed spatiotemporal features are input into the encoder with a cross-attention mechanism in the semantic calibration model, so that the encoder performs dimensionality reduction processing on the processed spatiotemporal features to obtain reduced-dimensional spatiotemporal features. Furthermore, the reduced-dimensional spatiotemporal features can be input into the first decoder with a cross-attention mechanism in the semantic calibration model, so that the decoder performs knowledge distillation on the reduced-dimensional spatiotemporal features to obtain distilled spatiotemporal features. Afterwards, the distilled spatiotemporal features are input into the second decoder in the semantic calibration model, so that the second decoder outputs the position of the calibrated target video frame and the confidence corresponding to the position.

[0113] In this embodiment, the semantic calibration model deployed in the electronic device can be used to calibrate the target video frame output by the semantic analysis model. The dimensionality reduction processing of the semantic calibration model can reduce the computational pressure of the semantic calibration model, and the knowledge distillation process of the semantic calibration model can make the semantic model output semantic boundary points more accurately.

[0114] Optionally, the semantic calibration model can be trained in a supervised manner. Specifically, a training video and reference positions of video frames with semantic changes in the training video can be obtained first. Then, the training video is used as a training sample and the reference positions are used as supervisory information to train the semantic calibration model. Optionally, the reference positions can be obtained by manual annotation.

[0115] In this embodiment, the semantic calibration model can be trained in a supervised manner so that the model can accurately calibrate the semantic boundary points output by the semantic analysis model and ultimately ensure the extraction quality of the effective fragments.

[0116] As for the third part of the overall inventive concept, the following may describe in detail how to further edit a plurality of video segments after dividing the video to be processed according to semantic boundary points to ensure the semantic integrity of the video segments.

[0117] In an optional video clip editing method, if the similarity between the spatial features of the first and last frames of any video clip is greater than a first preset threshold, indicating that there is a drastic change in the picture content of any video clip, any video clip can be deleted.

[0118] Optionally, the first preset threshold value can be set to 0.1. Optionally, the similarity between video frames can be measured by Euclidean distance, Manhattan distance, etc. And the smaller the distance, the greater the similarity between spatial features. Taking Euclidean distance as an example, the Euclidean distance between video frames can be calculated by the following formula: .

[0119] in, For any video clip The characteristic vector of The last frame of any video clip The characteristic vector of With end frame The difference in the Euclidean distance between them.

[0120] Another optional video clip editing method is to merge adjacent video clips if the duration of each adjacent video clip in the plurality of video clips is less than a preset duration and the adjacent video clips meet a first preset requirement. The first preset requirement includes that the similarity between the last frame of the preceding video clip and the first frame of the succeeding video clip in the adjacent video clips is less than a second preset threshold. Optionally, the preset duration can be set to 2 seconds. The second preset threshold can be set to 0.6. Then the merged adjacent video clips are both highly similar and fragmented video clips.

[0121] In another optional video clip editing method, if the duration of any video clip is less than a preset duration, it indicates that the video clip is too fragmented, and the video clip is deleted.

[0122] In another optional video clip editing method, if adjacent video clips in the plurality of video clips meet a second preset requirement, the adjacent video clips are deleted. The second preset requirement includes that the similarity between the last frame of the preceding video clip and the first frame of the succeeding video clip in the adjacent video clips is less than a third preset threshold. Optionally, the third preset threshold can be set to 0.05. The deleted adjacent video clips have a very high similarity, and the contents of the adjacent video clips are in a static state and do not contain much valuable information, so they can be deleted.

[0123] Optionally, at least one of the above-mentioned multiple video segment editing methods can be selected for use, and finally the video segment obtained after deletion and / or merging can be determined as a semantically complete valid segment.

[0124] In this embodiment, for the multiple video segments obtained by dividing the video to be processed according to the semantic boundary points, the video segments can be further integrated according to the semantic coherence and the video length to obtain semantically coherent valid segments.

[0125] In practice, the electronic device for performing semantic boundary point recognition, calibration and effective segment extraction can be a terminal device used by a user. And the effective segment extraction process can also provide a basis for the "one-click blockbuster" function provided by the terminal device to the user.

[0126] The specific process of users enabling the "One-click Blockbuster" function can also be described in detail below: After the user enters the material library interface, the interface can display a prompt message indicating whether to enable the "One-click Blockbuster" function. The user can trigger the "One-click Blockbuster" enabling operation in the material library interface, and further trigger the video selection operation in the material library page. In response to the selection operation, the terminal device determines the video to be processed selected by the user from the material library, and begins to identify the target video frames with semantic changes in the video to be processed and completes the calibration, and finally obtains the semantic boundary points of the video to be processed. Furthermore, the terminal device can obtain a valid segment with complete semantics in the video to be processed based on the located semantic boundary points. Furthermore, the terminal device can also filter out wonderful segments from the valid segments.

[0127] Among them, the user can trigger a video shooting operation on the terminal device. In response to the operation, the controller in the terminal device can control the camera on the terminal device to start and shoot a video, and the shot video can be stored in the material library of the terminal device.

[0128] The activation process of the above-mentioned "one-click blockbuster" function can also be combined with the Figure 8a~Figure 8c The operation interface shown is understood. Figure 8a~Figure 8c As shown in the interface, the terminal device can also display the current progress of the "one-click blockbuster" function in real time on the interface while responding to video selection operations, semantic boundary point recognition, and effective segment extraction.

[0129] For terminal devices that provide the "one-click blockbuster" function, the software system architecture of the device can also be described in detail below. Fig. 9 A schematic diagram of the software architecture of a terminal device provided in an embodiment of the present invention.

[0130] like Fig. 9As shown, the software system of the terminal device provided in the embodiment of the present invention can adopt a layered architecture. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. Fig. 9 As shown in FIG. 1 , the software system of the terminal device can be divided into four layers, namely, the application layer, the application framework layer, the operating system layer, and the hardware driver layer from top to bottom. The main processor of the terminal device performs data calculation based on this software architecture.

[0131] The application layer can include a series of third-party application packages that come with the terminal device or are downloaded by the user. Fig. 9 As shown, the application package may include applications such as gallery, calendar, map, WLAN, Bluetooth, music, material library, instant messaging, etc. Moreover, the multi-semantic video processing method provided by the above embodiments of the present invention may also be used as a nested function in the application package of the material library.

[0132] The application framework layer can provide an application programming interface (Application Programming Interface, referred to as API) and a programming framework for the application programs in the application layer. The application framework layer includes some predefined functions.

[0133] like Fig. 9 As shown, the application framework layer may also include a view system and its interface, a graphics system and its interface, and other system services. Among them, the system services may include: a window manager, a content provider, a view system, a resource manager, a notification manager, and the like.

[0134] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0135] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, browsing history, and bookmarks.

[0136] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications.

[0137] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0138] The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of an icon or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, and an indicator light flashes, etc.

[0139] The application layer and the application framework layer can run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0140] The operating system layer can include multiple functional modules, such as: Surface Manager, Surface Flinger, Media Libraries, 3D graphics processing library (such as OpenGLES), 2D graphics engine (such as SGL), etc.

[0141] The surface manager is used to manage the display composition system and provides fusion of 2D and 3D layers for multiple applications.

[0142] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0143] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing, etc.

[0144] A 2D graphics engine is a drawing engine for 2D drawings.

[0145] The hardware driver layer can be a layer between hardware and software. The hardware driver layer at least includes display driver, camera driver, audio driver, sensor driver, etc.

[0146] The hardware structure of the terminal device may also be described in detail below. Fig.10 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present invention.

[0147] The terminal device may include a main processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, etc.

[0148] The main processor 110 in the terminal device is used to execute the above Figure 1 to Figure 7 The illustrated embodiment is used to implement the recognition and calibration of semantic boundary points and the extraction of valid segments.

[0149] The role of the main processor 110 used in the semantic boundary recognition process may be first introduced below.

[0150] The main processor 110 may include one or more processing units, for example, the main processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and the DSP may specifically include an ADSP. A baseband processor and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0151] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0152] The main processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the main processor 110 is a cache memory. The memory may store instructions or data that the main processor 110 has just used or cyclically used. If the main processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the main processor 110, and thus improves the efficiency of the system.

[0153] In some embodiments, the main processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, and / or a universal serial bus (USB) interface, etc.

[0154] The functions of other components in the terminal device can be further introduced below.

[0155] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device. The external memory card communicates with the main processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.

[0156] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 may include a program storage area and a data storage area. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The main processor 110 executes various functional applications and data processing of the terminal device by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0157] The USB interface 130 is an interface that complies with the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to transmit data between the terminal device and the peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other terminal devices, such as AR devices, etc.

[0158] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the terminal device. In other embodiments of the present invention, the terminal device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0159] The charging management module 140 is used to receive charging input from a charger, where the charger can be a wireless charger or a wired charger.

[0160] The power management module 141 is used to connect the battery 142, the charging management module 140 and the main processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the main processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the main processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0161] The wireless communication function of the terminal device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0162] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0163] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to terminal devices. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, and filter, amplify and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the main processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the main processor 110.

[0164] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the main processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0165] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) and the like applied to the terminal device. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering processing, and sends the processed signal to the main processor 110. The wireless communication module 160 can also receive the signal to be sent from the main processor 110, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0166] In some embodiments, antenna 1 of the terminal device is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that the terminal device can communicate with the network and other devices through wireless communication technology.

[0167] The terminal device can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor.

[0168] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the main processor 110, or some functional modules of the audio module 170 can be arranged in the main processor 110.

[0169] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device can listen to music or listen to a hands-free call through the speaker 170A.

[0170] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device receives a call or voice message, the voice can be heard by placing the receiver 170B close to the ear.

[0171] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 170C with his mouth to input the sound signal into the microphone 170C. The terminal device can be provided with at least one microphone 170C, which can not only collect sound signals, but also realize noise reduction, identify the sound source to realize directional recording function, etc.

[0172] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm Open Mobile Terminal Platform (OMTP) standard interface or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

[0173] The terminal device realizes display function through GPU, display screen 194, and application processor.

[0174] The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The main processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0175] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the terminal device may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0176] The terminal device can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0177] The ISP is used to process the data fed back by the camera 193. For example, when shooting a video, light is transmitted to the camera photosensitive element through the lens, and the light signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the image. In some embodiments, the ISP can be set in the camera 193.

[0178] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0179] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the terminal device is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0180] Video codecs are used to compress or decompress digital videos. Terminal devices can support one or more video codecs. In this way, terminal devices can play or record videos in multiple encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0181] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and continuously self-learn. NPU can realize applications such as intelligent cognition of terminal devices, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0182] The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

[0183] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194 .

[0184] The gyro sensor 180B may be used to determine the motion posture of the terminal device 100 .

[0185] The air pressure sensor 180C is used to measure air pressure.

[0186] The magnetic sensor 180D includes a Hall sensor, and the terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case.

[0187] The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal device in all directions (generally three axes). When the terminal device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the terminal device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0188] The distance sensor 180F is used to measure the distance. The terminal device can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal device can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0189] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode.

[0190] The terminal device emits infrared light through a light-emitting diode. The terminal device uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device. When insufficient reflected light is detected, the terminal device can determine that there is no object near the terminal device.

[0191] The ambient light sensor 180L is used to sense the brightness of the ambient light. The terminal device can adaptively adjust the brightness of the display screen 194 according to the sensed brightness of the ambient light.

[0192] The fingerprint sensor 180H is used to collect fingerprints. The terminal device can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, etc.

[0193] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal device reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection.

[0194] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal device, which is different from the position of the display screen 194.

[0195] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0196] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal device may receive key input and generate key signal input related to user settings and function control of the terminal device.

[0197] Motor 191 can generate vibration prompts. Motor 191 can be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0198] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.

[0199] In addition, any device other than the terminal device Fig. 9 An electronic device or a device with the software system architecture shown Fig.10 An electronic device in which the processor has the same function can also execute the methods provided in the above embodiments of the present invention to realize the recognition, calibration and extraction of semantic boundary points and valid segments.

[0200] The following will describe in detail the multi-semantic video processing device of one or more embodiments of the present invention. Those skilled in the art will appreciate that these speaker control devices can be configured using commercially available hardware components through the steps taught in this solution.

[0201] Fig.11 A schematic diagram of the structure of a multi-semantic video processing device provided by an embodiment of the present invention is shown in FIG. Fig.11 As shown, the device comprises:

[0202] The spatial feature determination module 11 is used to extract spatial features from the video to be processed to obtain spatial features of each video frame in the video to be processed.

[0203] The first spatiotemporal feature determination module 12 is used to determine the spatiotemporal features of each of the video frames according to the spatial features of each of the video frames.

[0204] The second spatiotemporal feature determination module 13 is used to determine the spatiotemporal features of the video to be processed according to the spatiotemporal features of each of the video frames.

[0205] The semantic boundary point identification module 14 is used to determine the target video frame with semantic changes in the video to be processed according to the spatiotemporal features.

[0206] Optionally, the spatial feature determination module 11 is used to input the video frame sequence corresponding to the video to be processed into a semantic analysis model, so that the first feature extraction network in the semantic analysis model outputs the spatial features of each of the video frames.

[0207] Among them, the first feature extraction network is used to extract features from the video frame sequence to obtain the original spatial features of each video frame in the video to be processed; and downsample the original spatial features of each video frame using at least one first sampling rate to obtain target spatial features of at least one scale for each video frame.

[0208] Optionally, the first spatiotemporal feature determination module 12 is used to splice the spatial features of the respective video frames to obtain an original spatial feature sequence;

[0209] Downsampling the original spatial feature sequences respectively using at least one second sampling rate to obtain candidate spatial feature sequences of the video to be processed;

[0210] Generate, according to the original spatial feature sequence and the candidate spatial feature sequence, the spatiotemporal features of the intermediate video frames in the video to be processed at different first sampling rates, wherein the intermediate video frames are the video frames in the video to be processed except the last frame;

[0211] The candidate spatial feature sequence is convolved using a filter with a preset window size to obtain the spatiotemporal features of the last frame in the video to be processed at the different second sampling rates.

[0212] Optionally, the second spatiotemporal feature determination module 13 is used to splice the spatiotemporal features of each video frame in the video to be processed at the same first sampling rate;

[0213] The splicing results corresponding to the different first sampling rates are respectively concatenated to obtain the spatiotemporal characteristics of the video to be processed.

[0214] Optionally, the first spatiotemporal feature determination module 12 is used to input the video frame sequence corresponding to the video to be processed into a semantic analysis model, so that the second feature extraction network in the semantic analysis model extracts the spatiotemporal features of the video frames in the video to be processed.

[0215] The second spatiotemporal feature determination module 13 is used to input the video frame sequence corresponding to the video to be processed into the semantic analysis model so that the second feature extraction network in the semantic analysis model can extract the spatiotemporal features of the video to be processed.

[0216] Optionally, the semantic boundary point identification module 14 is used to input the spatiotemporal features of each video frame in the video to be processed into a decoding network in a semantic analysis model, so that the decoding network can determine the correlation between the spatiotemporal features of each video frame in the video to be processed, and then output the position of the target video frame with semantic changes in the video to be processed and the confidence of the position based on the spatiotemporal features of the video to be processed and the correlation.

[0217] Optionally, the device further comprises an identification module 15, which is used to identify a main object contained in each video frame in the video to be processed.

[0218] The spatial feature determination module 11 is used to extract spatial features from the area where the main object is located in the video frame, so as to use the extraction results as the spatial features of each video frame in the video to be processed.

[0219] Optionally, the device further comprises a calibration module 16, configured to calibrate the target video frame according to the spatiotemporal characteristics of respective video frames in the video to be processed.

[0220] Optionally, the calibration module 16 is used to embed position information for the spatiotemporal features of each of the video frames to obtain processed spatiotemporal features of the video to be processed, wherein the position information reflects the position of the video frame corresponding to the spatiotemporal features in the video to be processed;

[0221] The processed spatiotemporal features are input into a semantic calibration model so that the semantic calibration model outputs the position of the calibrated video frame in the video to be processed and the confidence that the calibrated video frame is a video frame with semantic changes in the video to be processed.

[0222] Optionally, the calibration module 16 is used to input the processed spatiotemporal features into an encoder with a cross-attention mechanism in the semantic calibration model, so that the encoder performs dimensionality reduction processing on the processed spatiotemporal features to obtain dimensionality-reduced spatiotemporal features;

[0223] Inputting the dimension-reduced spatiotemporal features into a first decoder with a cross-attention mechanism in the semantic calibration model, so that the decoder performs knowledge distillation on the dimension-reduced spatiotemporal features to distill the spatiotemporal features;

[0224] The distilled spatiotemporal features are input into a second decoder in the semantic calibration model, so that the second decoder outputs the position and the confidence.

[0225] Optionally, the device further comprises a training module 17 for obtaining a training video and reference positions of video frames with semantic changes in the training video; using the training video as a training sample and the reference positions as supervision information to train the semantic calibration model.

[0226] Optionally, the device also includes an editing module 18, which is used to divide the video to be processed into multiple video segments according to the position of the target video frame in the video to be processed; and edit the multiple video segments according to the similarity between the multiple video segments to obtain valid segments with complete semantics in the video to be processed.

[0227] Optionally, the editing module 18 is configured to delete any video segment if the similarity between the spatial features of the first and last frames of any video segment is greater than a first preset threshold;

[0228] If the duration of any video segment is less than a preset duration, deleting any video segment;

[0229] The remaining video segments are determined as the valid segments.

[0230] Optionally, the editing module 18 is used to merge the adjacent video segments among the multiple video segments if the respective durations of the adjacent video segments are less than a preset duration and the adjacent video segments meet a first preset requirement; wherein the first preset requirement includes that the similarity between the last frame of the preceding video segment and the first frame of the succeeding video segment among the adjacent video segments is less than a second preset threshold.

[0231] Optionally, the editing module 18 is used to delete the adjacent video segments among the multiple video segments if the adjacent video segments meet a second preset requirement; wherein the second preset requirement includes that the similarity between the last frame of the previous video segment and the first frame of the subsequent video segment among the adjacent video segments is less than a third preset threshold.

[0232] Fig.11 The device shown can perform Figures 1 to 7 For the method of the embodiment shown in the figure, the part not described in detail in this embodiment can be referred to Figures 1 to 7 The implementation process and technical effects of this technical solution refer to Figures 1 to 7 The description in the illustrated embodiment will not be repeated here.

[0233] In a possible design, the multi-semantic video processing method provided in the above embodiments may be applied in an electronic device, such as Fig.12 As shown, the electronic device may include: a first processor 21 and a first memory 22, wherein the first memory 22 is used to store the electronic device to perform the above Figures 1 to 7 The program of the multi-semantic video processing method provided in the illustrated embodiment, the first processor 21 is configured to execute the program stored in the first memory 22 .

[0234] The program includes one or more computer instructions, wherein when the one or more computer instructions are executed by the first processor 21, the following steps can be implemented:

[0235] Extracting spatial features of the video to be processed to obtain spatial features of each video frame in the video to be processed;

[0236] Determining the spatiotemporal features of the video frames according to the spatial features of the video frames;

[0237] Determining the spatiotemporal features of the video to be processed according to the spatiotemporal features of the respective video frames;

[0238] According to the spatiotemporal features, a target video frame having a semantic change in the video to be processed is determined.

[0239] Optionally, the first processor 21 is further configured to execute the aforementioned Figures 1 to 6 All or part of the steps in the illustrated embodiments.

[0240] The structure of the electronic device may further include a first communication interface 23 for the electronic device to communicate with other devices or communication systems.

[0241] In a possible design, the multi-semantic video processing method provided in the above embodiments may be applied in a terminal device, such as Fig.13 As shown, the terminal device may include: a second processor 31 and a second memory 32. The second memory 32 is used to store the data that supports the terminal device to execute the above Figures 1 to 7 The program of the multi-semantic video processing method provided in the illustrated embodiment, the second processor 31 and the second memory 32 are configured to execute the program stored in the second memory 32 .

[0242] The program includes one or more computer instructions, wherein when the one or more computer instructions are executed by the second processor 31, the following steps can be implemented:

[0243] Extracting spatial features of the video to be processed to obtain spatial features of each video frame in the video to be processed;

[0244] Determining the spatiotemporal features of the video frames according to the spatial features of the video frames;

[0245] Determining the spatiotemporal features of the video to be processed according to the spatiotemporal features of the respective video frames;

[0246] According to the spatiotemporal features, a target video frame having a semantic change in the video to be processed is determined.

[0247] Optionally, the second processor 31 is further configured to execute the aforementioned Figures 1 to 7 All or part of the steps in the illustrated embodiments.

[0248] The structure of the terminal device may further include a second communication interface 33 for the electronic device to communicate with other devices or communication systems.

[0249] In addition, an embodiment of the present invention provides a non-transitory machine storage medium for storing computer software instructions used by the non-transitory machine, which includes instructions for executing the above-mentioned Figure 1 to Figure 7 The procedures involved in the multi-semantic video processing method shown.

[0250] In addition, an embodiment of the present invention provides a computer program product. The computer program product includes a computer program or an instruction. When the computer program or the instruction is executed by a processor, the processor is enabled to implement the above Figure 1 to Figure 7 The steps or functions of the multi-semantic video processing method shown.

[0251] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing multi-semantic video, characterized in that: include: Extracting spatial features of the video to be processed to obtain spatial features of each video frame in the video to be processed; Generate, according to the splicing results of the spatial features of the respective video frames and the downsampling results of the splicing results, the splicing results of the intermediate video frames in the video to be processed and the spatiotemporal features at different first sampling rates, wherein the intermediate video frames are the video frames in the video to be processed except the last frame; Performing convolution processing on the down-sampling result to obtain the spatiotemporal features of the last frame in the video to be processed at the different first sampling rates; Determining the spatiotemporal features of the video to be processed according to the spatiotemporal features of each of the video frames at the different first sampling rates; According to the spatiotemporal features, a target video frame having a semantic change in the video to be processed is determined.

2. The method according to claim 1, characterized in that The extracting spatial features of the video to be processed to obtain spatial features of each video frame in the video to be processed includes: The video frame sequence corresponding to the video to be processed is input into the semantic analysis model, so that the first feature extraction network in the semantic analysis model outputs the spatial features of each of the video frames.

3. The method according to claim 2, characterized in that The first feature extraction network is used to extract features from the video frame sequence to obtain original spatial features of each video frame in the video to be processed; The original spatial features of the video frames are respectively downsampled using at least one first sampling rate to obtain target spatial features of at least one scale of the video frames.

4. The method according to claim 1, characterized in that: The step of generating the splicing results of the spatial features of the respective video frames and the downsampling results of the splicing results to generate the splicing and temporal and spatial features of the intermediate video frames in the video to be processed at different first sampling rates comprises: splicing the spatial features of the respective video frames to obtain an original spatial feature sequence as the splicing result; Downsampling the original spatial feature sequences respectively using at least one second sampling rate to obtain candidate spatial feature sequences as the downsampling results; Generate spatiotemporal features of intermediate video frames in the video to be processed at different first sampling rates according to the original spatial feature sequence and the candidate spatial feature sequence; The performing convolution processing on the down-sampling result includes: performing convolution processing on the candidate spatial feature sequence using a filter of a preset window size.

5. The method according to claim 4, characterized in that The step of determining the spatiotemporal features of the video to be processed according to the spatiotemporal features of each of the video frames at the different first sampling rates includes: splicing the spatiotemporal features of each video frame in the video to be processed at the same first sampling rate; The splicing results corresponding to the different first sampling rates are respectively concatenated to obtain the spatiotemporal characteristics of the video to be processed.

6. The method according to claim 4 or 5, characterized in that: The method further comprises: The video frame sequence corresponding to the video to be processed is input into the semantic analysis model, so that the second feature extraction network in the semantic analysis model extracts the spatiotemporal features of each video frame in the video to be processed and the spatiotemporal features of the video to be processed.

7. The method according to claim 1, characterized in that The step of determining, according to the spatiotemporal features, a target video frame with a semantic change in the video to be processed comprises: The spatiotemporal features of each video frame in the video to be processed are input into the decoding network in the semantic analysis model, so that the decoding network determines the correlation between the spatiotemporal features of each video frame in the video to be processed, and then outputs the position of the target video frame with semantic changes in the video to be processed and the confidence of the position based on the spatiotemporal features of the video to be processed and the correlation.

8. The method according to claim 1, characterized in that Before extracting spatial features from the video to be processed, the method further includes: Identifying the subject objects contained in each of the video frames in the video to be processed; The extracting spatial features of the video to be processed to obtain spatial features of each video frame in the video to be processed includes: A spatial feature extraction is performed on the area where the main object is located in the video frame, so that the extraction result is used as the spatial feature of each video frame in the video to be processed.

9. The method according to claim 1, characterized in that: After determining the target video frame in the video to be processed whose semantics has changed according to the spatiotemporal features, the method further includes: The target video frame is verified according to the temporal and spatial characteristics of each video frame in the video to be processed.

10. The method according to claim 9, characterized in that The step of verifying the target video frame according to the spatiotemporal features of the respective video frames in the video to be processed includes: Embedding position information for the spatiotemporal features of each of the video frames to obtain processed spatiotemporal features of the video to be processed, wherein the position information reflects the position of the video frame corresponding to the spatiotemporal features in the video to be processed; The processed spatiotemporal features are input into a semantic calibration model so that the semantic calibration model outputs the position of the calibrated video frame in the video to be processed and the confidence that the calibrated video frame is a video frame with semantic changes in the video to be processed.

11. The method according to claim 10, characterized in that The step of inputting the processed spatiotemporal features into a semantic calibration model so that the semantic calibration model outputs a position of the target video frame in the video to be processed and a confidence level that the target video frame is a video frame with semantic changes in the video to be processed, comprising: Inputting the processed spatiotemporal features into an encoder with a cross-attention mechanism in the semantic calibration model, so that the encoder performs dimensionality reduction processing on the processed spatiotemporal features to obtain dimensionality-reduced spatiotemporal features; Inputting the dimension-reduced spatiotemporal features into a first decoder with a cross-attention mechanism in the semantic calibration model, so that the decoder performs knowledge distillation on the dimension-reduced spatiotemporal features to distill the spatiotemporal features; The distilled spatiotemporal features are input into a second decoder in the semantic calibration model, so that the second decoder outputs the position and the confidence.

12. The method according to claim 10, characterized in that The method further comprises: Obtaining a training video and reference positions of video frames in the training video where semantics have changed; The semantic calibration model is trained by using the training video as a training sample and the reference position as supervision information.

13. The method according to claim 1, characterized in that After determining the target video frame in the video to be processed whose semantics has changed according to the spatiotemporal features, the method further includes: Dividing the video to be processed into a plurality of video segments according to the position of the target video frame in the video to be processed; The multiple video segments are edited according to the similarities between the multiple video segments to obtain valid segments with complete semantics in the video to be processed.

14. The method according to claim 13, characterized in that The editing of the multiple video segments according to the similarities between the multiple video segments to obtain valid segments with complete semantics in the video to be processed includes: If the similarity between the spatial features of the first and last frames of any video segment is greater than a first preset threshold, deleting the video segment; If the duration of any video segment is less than a preset duration, deleting any video segment; The remaining video segments are determined as the valid segments.

15. The method according to claim 13, characterized in that The editing of the multiple video clips according to the similarities between the multiple video clips includes: If the duration of each adjacent video segment in the multiple video segments is less than the preset duration, and the adjacent video segments meet the first preset requirement, merging the adjacent video segments; The first preset requirement includes that the similarity between the last frame of the preceding video segment and the first frame of the succeeding video segment in the adjacent video segments is less than a second preset threshold.

16. The method according to claim 13, characterized in that The editing of the multiple video clips according to the similarities between the multiple video clips includes: If adjacent video segments among the multiple video segments meet a second preset requirement, deleting the adjacent video segments; The second preset requirement includes that the similarity between the last frame of the preceding video segment and the first frame of the succeeding video segment in the adjacent video segments is less than a third preset threshold.

17. A multi-semantic video processing device, characterized in that: include: A spatial feature determination module is used to extract spatial features from the video to be processed to obtain spatial features of each video frame in the video to be processed; A first spatiotemporal feature determination module is used to generate spatiotemporal features of intermediate video frames in the video to be processed at different first sampling rates according to the splicing results of the spatial features of the respective video frames and the downsampling results of the splicing results, wherein the intermediate video frames are video frames other than the last frame in the video to be processed; and perform convolution processing on the downsampling results to obtain the spatiotemporal features of the last frame in the video to be processed at the different first sampling rates; A second spatiotemporal feature determination module, configured to determine the spatiotemporal features of the video to be processed according to the spatiotemporal features of each of the video frames at the different first sampling rates; The semantic boundary point determination module is used to determine the target video frame with semantic changes in the video to be processed according to the spatiotemporal features.

18. A terminal device, characterized in that: include: A memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the multi-semantic video processing method as described in any one of claims 1 to 16.

19. A non-transitory machine-readable storage medium, characterized in that The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes the multi-semantic video processing method according to any one of claims 1 to 16.

20. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, enables the processor to implement the steps in the multi-semantic video processing method according to any one of claims 1 to 16.

Images