Model generation method, video processing method and device

Abstract

Embodiments of the present application provide a model generation method, a video processing method and equipment. In the method, a plurality of image frames are extracted from a video to be processed; the plurality of image frames are input into a video processing model; the video processing model comprises a feature extractor and at least one calculation module; the feature extractor comprises a plurality of feature extraction modules; the plurality of feature extraction modules are respectively extracted from a plurality of image processing models; based on frame features extracted from the plurality of image frames by the plurality of feature extraction modules respectively, a first video feature corresponding to the video to be processed is obtained; and the first video feature is processed by the at least one calculation module respectively to obtain at least one processing result. The technical solution provided by the embodiments of the present application improves the accuracy of the video processing result.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data processing, and in particular to a model generation method, a video processing method and equipment. BACKGROUND

[0002] In the field of data processing, processing of video data is often involved, such as video classification, feature calculation, etc., so that corresponding operations can be performed based on the processing result of the video data. At present, machine learning models such as neural network models are usually used to process video data, and the model structure, training method, etc. will affect the accuracy of the video processing result. How to provide a video processing model with more accurate video processing result has become a technical problem to be solved by those skilled in the art. SUMMARY

[0003] Embodiments of the present application provide a model generation method, a video processing method and equipment to improve the accuracy of the video processing result.

[0004] In a first aspect, a video processing method is provided in embodiments of the present application, comprising:

[0005] extracting a plurality of image frames from a to-be-processed video;

[0006] inputting the plurality of image frames into a video processing model; wherein the video processing model comprises a feature extractor and at least one calculation module; the feature extractor comprises a plurality of feature extraction modules; the plurality of feature extraction modules are respectively extracted from a plurality of image processing models;

[0007] obtaining a first video feature corresponding to the to-be-processed video based on frame features extracted by the plurality of feature extraction modules from the plurality of image frames;

[0008] processing the first video feature by using the at least one calculation module to obtain at least one processing result. In a second aspect, a model generation method is provided in embodiments of the present application, comprising:

[0009] determining a plurality of image processing models;

[0010] extracting a feature extraction module from the plurality of image processing models to obtain a plurality of feature extraction modules;

[0011] connecting the plurality of feature extraction modules in parallel to construct a feature extractor;

[0012] connecting an output of the feature extractor to a calculation module of at least one video task to construct a video processing model;

[0013] The model parameters of the feature extractor are kept unchanged, and the video processing model is trained by using the training samples of the at least one video task; and the video processing model is used to process a to-be-processed video to obtain processing results corresponding to the at least one video task respectively.

[0014] In a third aspect, a model generation method is provided in the embodiments of the present application, including:

[0015] determining a plurality of image processing models;

[0016] extracting a feature extraction module from each of the plurality of image processing models to obtain a plurality of feature extraction modules;

[0017] connecting outputs of the plurality of feature extraction modules to a feature fusion module respectively to construct a feature extractor;

[0018] The feature extractor is used to extract a video feature of a to-be-processed video; the plurality of feature extraction modules respectively extract frame features of a plurality of image frames corresponding to the to-be-processed video based on the plurality of image frames; and the feature fusion module is used to fuse the plurality of frame features extracted by the plurality of feature extraction modules to obtain the video feature.

[0019] In a fourth aspect, a computing device is provided in the embodiments of the present application, including a storage component and a processing component. The storage component stores one or more computer instructions. The one or more computer instructions are used to be called and executed by the processing component to implement the video processing method in the first aspect, or to implement the model generation method in the second aspect, or to implement the model generation method in the third aspect.

[0020] In the embodiments of the present application, the feature extraction modules are extracted from the plurality of image processing models, the feature extractor is constructed by the feature extraction modules in the plurality of image processing models, the video processing model is constructed by connecting the feature extractor and the computing module of the at least one video task, the model parameters of the feature extractor are kept unchanged during the model training, the model parameters of the computing module are obtained by training, and therefore the video processing model can process the to-be-processed model to obtain the processing results corresponding to the at least one video task. The feature extractor is constructed by the feature extraction modules in the plurality of image processing models, and is used to extract the video feature. The video feature is composed of the features extracted by the feature extraction modules in the plurality of image processing models, which can significantly improve the accuracy of the video feature extraction, and therefore the accuracy of the video processing result can be improved.

[0021] These aspects and other aspects of the present application will be more apparent in the following description of the embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0023] Figure 1 A structural schematic diagram of an embodiment of a data processing system provided by the present application is shown;

[0024] Figure 2 A flow chart of an embodiment of a model generation method provided by the present application is shown;

[0025] Figure 3a A structural schematic diagram of a feature extractor in an actual application of an embodiment of the present application is shown;

[0026] Figure 3b A structural schematic diagram of a video processing model in an actual application of an embodiment of the present application is shown;

[0027] Figure 3c A structural schematic diagram of a video processing model in another actual application of an embodiment of the present application is shown;

[0028] Figure 3d A structural schematic diagram of a video processing model in another actual application of an embodiment of the present application is shown;

[0029] Figure 3e A structural schematic diagram of a merging model in an actual application of an embodiment of the present application is shown;

[0030] Figure 3f A structural schematic diagram of a comprehensive processing model in an actual application of an embodiment of the present application is shown;

[0031] Figure 4 A flow chart of another embodiment of a model generation method provided by the present application is shown;

[0032] Figure 5 A flow chart of an embodiment of a video processing method provided by the present application is shown;

[0033] Figure 6a A flow chart of another embodiment of a video processing method provided by the present application is shown;

[0034] Figure 6b A scene interaction schematic diagram in an actual application of an embodiment of the present application is shown;

[0035] Figure 7A structural diagram of one embodiment of a model generation apparatus provided by the present application is shown;

[0036] Figure 8 A structural diagram of one embodiment of a computing device provided by the present application is shown;

[0037] Figure 9 A structural diagram of one embodiment of a video processing apparatus provided by the present application is shown;

[0038] Figure 10 A structural diagram of one embodiment of a computing device provided by the present application is shown. DETAILED DESCRIPTION

[0039] In order to enable persons skilled in the art to better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0040] In some processes described in the specification and claims of the present application and the above-described drawings, a plurality of operations appearing in a specific order are included, but it should be clearly understood that these operations can be executed not in the order in which they appear in the present text or in parallel. The serial numbers of the operations, such as 101, 102, etc., are merely used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel. It should be noted that the descriptions of “first”, “second”, etc. in the present text are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence. In addition, “first” and “second” are not of different types.

[0041] The technical scheme of the embodiments of the present application can be applied to the video data processing scene. With the development of artificial intelligence technology and Internet technology, video data processing is also becoming more and more.

[0042] As described in the background, currently, machine learning models are usually used for video processing, such as neural network models. The inventors have found that when a video processing model processes a video, it first needs to extract video features, and then processes the video based on the video features. Therefore, the video features are a key factor affecting whether the video processing result is accurate. Therefore, the inventors have thought that if the accuracy of video feature extraction can be improved, the accuracy of the video processing result can also be improved. The inventors have further found that if the accuracy of video feature extraction is to be improved, the feature extraction module in the video processing model can be improved. For example, in a neural network model, the feature extraction is usually performed by multiple network layers in the neural network model. The accuracy of video feature extraction can be ensured by increasing the number of network layers. However, the inventors have found that in actual applications, multiple video tasks often need to be processed. For each video task, a model needs to be trained, and the accuracy of processing needs to be ensured. The structure of each model will become very complex, resulting in a large amount of computation, which will affect the processing efficiency.

[0043] In view of the above findings, the inventors have creatively proposed the technical solutions of the present application. In the embodiments of the present application, a feature extractor is constructed by a plurality of feature extraction modules in a plurality of image processing models, to extract video features. The features extracted by the plurality of feature extraction modules in the plurality of image processing models constitute the video features, which can significantly improve the accuracy of video feature extraction, thereby improving the accuracy of the video processing result. In the case of multiple video tasks, the plurality of video tasks can share the feature extractor. Compared with the model training mode for a single video task, the model training complexity can be reduced. Since the feature extractor can ensure the accuracy of the video processing result, the light-weight structure of the computing module can still ensure the accuracy of the video processing result. Therefore, on the premise of ensuring the accuracy of the video processing result, the amount of computation can be reduced, the operation speed can be improved, and the impact on the performance of the device running the model can be reduced. In the actual application scenario of using the processing result of the video processing model to perform a corresponding operation, such as the scenario of recommending audio data for a to-be-processed video, the operation accuracy can be ensured, and the accuracy of audio data recommendation can be improved.

[0044] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person skilled in the art without creative labor are within the scope of protection of the present application.

[0045] The technical solutions of the embodiments of the present application can be applied to, for example, Figure 1The data processing system can include, for example, a first server 101, a second server 102, and a client 103. The first server 101 can construct and train a video processing model. The trained video processing model can be deployed on the second server 102 or the client 103.

[0046] When the video processing model is deployed on the second server 102, the second server 102 can process a video to be processed based on the video processing model according to a video processing request sent by the client 103.

[0047] When the video processing model is deployed on the client 103, the client 103 can directly process the video to be processed based on the video processing model.

[0048] In addition, in actual applications, the first server 101 and the second server 102 can be the same server. That is, the same server can complete the training of the video processing model and the processing of the video to be processed.

[0049] In addition, video processing using the video processing model can generally be performed by the second server 102, and can also be performed by the client 103. The present application does not limit this.

[0050] The above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. When the server is software, it can be implemented as multiple software modules or as a single software module.

[0051] The client 103 can generally be understood as an application deployed in an electronic device. The electronic device can be one or more of a smartphone, a tablet computer, and a portable computer, and can also be a desktop computer, and the like. For ease of understanding, Figure 1 The device is mainly used to represent the client. Various other applications such as search applications and instant messaging applications can also be configured in the electronic device. Of course, the client 103 can also refer to a browser, a web application such as an H5 (HyperText Markup Language 5, HyperText Markup Language version 5) application, or a light application (also known as a small program, a lightweight application), and the like. The present application does not specifically limit this.

[0052] The implementation details of the technical solutions of the embodiments of the present application are described in detail below.

[0053] Figure 2A flowchart of an embodiment of a model generation method provided by the present application, the technical solution of the present embodiment can be executed by, for example Figure 1 the first server or the second server in the system shown in the figure, and the method can include the following steps:

[0054] 201: Determine a plurality of image processing models.

[0055] The plurality of image processing models can correspond to a plurality of image tasks.

[0056] The plurality of image tasks can include one or more of an image classification task, an object detection task, and an image segmentation task. The image classification task may, for example, include a background classification task, such as identifying the background of a mountain, river, city, etc. in an image, or a target object classification task, such as identifying a cat, dog, person, car, etc. in an image. The object detection task may, for example, include a face detection task, such as outputting a region image where the face position is located. The image segmentation task may, for example, be to predict the object category to which each pixel point belongs, or to segment to obtain a specific region image, etc.

[0057] Optionally, the plurality of image processing models can correspond to different image tasks respectively, and can be used to implement different image processing, etc.

[0058] The plurality of image processing models can use existing models, and in addition, can be trained to obtain. Therefore, optionally, determining a plurality of image processing models includes:

[0059] Training a plurality of image processing models using training samples corresponding to a plurality of image tasks, wherein the plurality of image tasks includes one or more of an image classification task, an object detection task, and an image segmentation task.

[0060] That is, using training samples corresponding to each image task, and training an image processing model corresponding to the image task, for a plurality of image tasks, a plurality of image processing models can be trained to obtain.

[0061] The training samples corresponding to each image task can be composed of a sample image and label data corresponding to the sample image and belonging to the image task, for example, for an image classification task, the label data can be a specific category corresponding to the sample image, etc.

[0062] 202: Extract a feature extraction module from each of the plurality of image processing models to obtain a plurality of feature extraction modules.

[0063] 203: Connect the plurality of feature extraction modules in parallel to construct a feature extractor.

[0064] 204: Connect the output of the feature extractor to the computing module of at least one video task to construct a video processing model.

[0065] As the image processing model performs image processing, for the input image, image features are extracted, and image processing is performed based on the image features. The image features are usually in the form of vectors that can represent the image content. When using the model to process the video, frame features need to be extracted from the image frames of the video data, and video processing is performed based on the frame features. The frame features are also the image features. Therefore, the inventors think that in the embodiments of the present application, the feature extraction modules in each image processing model can be extracted, and the feature extraction modules in the multiple image processing models can be connected in parallel to construct a feature extractor. Then, the feature extractor is connected to the computing modules of at least one video task, and a video processing model can be constructed. Thus, each feature extraction module can be used to extract frame features of image frames in the video data, and the feature extractor can fuse the frame features extracted by the multiple feature extraction modules, so that the corresponding video features representing the video content can be obtained.

[0066] 205: The model parameters of the feature extractor are kept unchanged, and the video processing model is trained using the training samples of at least one video task.

[0067] The video processing model can be used to process the to-be-processed video to obtain processing results corresponding to at least one video task, respectively. The feature extractor can be used to extract first video features of the to-be-processed video. The feature extraction model can be used to extract frame features corresponding to multiple image frames in the to-be-processed video, respectively. The first video features can be obtained by fusing the multiple frame features extracted by each feature extraction module, and the like.

[0068] The video processing model provided in the embodiments of the present application can connect the computing modules corresponding to at least one video task, which can include the case of connecting the computing module of one video task, so that the video processing operation corresponding to the one video task can be implemented for the to-be-processed video. It can also include the case of connecting the computing modules corresponding to multiple video tasks, so that the video processing operation corresponding to the multiple video tasks can be implemented. In the case of including the computing modules corresponding to multiple video tasks, the multiple video tasks use a common feature extractor, which can reduce the computational amount, improve the model response speed and video processing efficiency while ensuring the video processing accuracy.

[0069] Since the feature extractor is constructed based on multiple feature extraction modules, the multiple feature extraction modules are extracted from multiple image processing models, the image tasks of each image processing model can be different, so the features extracted by the feature extractor are equivalent to integrating different types of features, making the obtained features more accurate. And multiple video tasks can share a feature extractor. In order to reduce the training complexity and training cost, when training the video processing model, the model parameters of the feature extractor can be ensured to be unchanged, and only the model parameters of the calculation module of each video task are trained. Compared with the way of training a separate model for a single video task, not only is the model accuracy higher, but also the accuracy of the video processing result is higher, and the operation amount is reduced and the video processing efficiency is improved.

[0070] When the image processing model is a neural network model, it is usually composed of an input layer, at least one intermediate layer, and an output layer. Each intermediate layer is used to generate corresponding features of the image. The closer the intermediate layer is to the output layer, the lower the accuracy of the features generated by the intermediate layer in expressing the content of the image, and the closer the features are to the image processing result. Therefore, the multiple feature extraction modules can be obtained by removing the corresponding number of network layers from the end of each image processing model, that is, starting from the output layer, removing a certain number of network layers at the end of each image processing model, and constructing the feature extraction module from the remaining network layers.

[0071] Optionally, extracting the feature extraction module from the multiple image processing models to obtain the multiple feature extraction modules can include: starting from the end of the multiple image processing models, removing the corresponding number of network layers to obtain the multiple feature extraction modules according to the task types corresponding to the multiple image processing models, respectively.

[0072] That is, the network layers that need to be removed or the network layers that need to be retained can be selected according to different task types. For example, for image classification tasks, only the output can be removed. For image segmentation tasks, since the output result of the image processing model is an image, the output features of the intermediate layers closer to the output layer have lower purity and larger data volume. In order to avoid data redundancy, the network layers between the output layer and the intermediate layer located in the middle position can be removed starting from the output layer, and only the input layer and a certain number of network layers can be retained.

[0073] In addition, in some embodiments, connecting the multiple feature extraction modules in parallel to construct the feature extractor can include:

[0074] The outputs of the multiple feature extraction modules are connected to a feature fusion module respectively to obtain multiple sub-extractors; wherein the feature fusion module is used to fuse the multiple frame features output by the connected feature extraction modules to obtain fused features;

[0075] The outputs of the multiple sub-extractors are connected to a feature connection module to obtain a feature extractor; wherein the feature connection model is used to fuse the multiple fusion features output by the multiple sub-extractors to obtain the first video feature.

[0076] When processing video data, each feature extraction module extracts frame features from multiple image frames corresponding to the video data. Therefore, a feature extraction module outputs multiple frame features. To reduce computational complexity, a feature fusion module can be used to fuse the multiple frame features output by the feature extraction modules to obtain a single fused feature.

[0077] Among them, the fusion processing method can be achieved by calculating the time domain mean, for example. Since the feature is usually a vector, the time domain mean of multiple feature vectors corresponding to each dimension can be calculated to obtain a multi-dimensional fusion feature. For ease of understanding, assume that the input feature extraction module is f image frames, so that f frame features can be obtained, expressed as Each frame feature can be a d-dimensional vector, and each frame feature can be expressed as The specific calculation formula for the fusion features corresponding to each dimension can be described as follows:

[0078]

[0079] Of course, in addition to calculating the time domain mean, other methods can also be used for fusion, such as weighted average, sum, etc., which are not specifically limited in this application.

[0080] Since there are multiple sub-extractors, each sub-extractor will obtain a fusion feature. In order to further reduce the computational complexity, multiple sub-extractors can be connected to a feature connection module. The feature connection module can further fuse the multiple fusion features corresponding to the multiple sub-extractors, thereby finally obtaining the first video feature that can characterize the video content.

[0081] Among them, the feature connection module fuses multiple fusion features, which can be splicing multiple fusion features. For example, each fusion feature is a D-dimensional vector. If there are 3 fusion features, they can be spliced ​​to obtain a 3*D-dimensional first video feature.

[0082] To further understand, Figure 3a In the structural diagram of the feature extractor shown, the feature extractor may include a feature extraction module 301, a feature fusion module 302, and a feature connection module 303. Multiple image frames may be input into a feature extraction module 301 to obtain multiple frame features. The multiple frame features are processed by the feature fusion module 302 to obtain fused features. The multiple fused features are then processed by the feature connection module 303 to obtain the first video feature.

[0083] In one practical scenario, the at least one video task can include multiple video tasks, and thus the step 204 can be connecting the output of the feature extractor to multiple computing modules of the multiple video tasks respectively to construct the video processing model.

[0084] Thus, the video processing model can be specifically used to process the to-be-processed video to obtain processing results respectively corresponding to the multiple video tasks.

[0085] In the training of the video processing model, the model parameters of the feature extractor can be kept unchanged, and the video processing model can be trained by using the training samples of the at least one video task.

[0086] To further improve the accuracy of the video processing result, as an optional implementation manner, the computing module can include a feature adapter and a video processor; in addition, the computing module for some specific video task can further include at least one auxiliary processor.

[0087] Thus, keeping the model parameters of the feature extractor unchanged, and training the video processing model by using the training samples of the at least one video task can include:

[0088] Keeping the model parameters of the feature extractor unchanged, inputting the training samples of each video task into the feature extractor to obtain first video features;

[0089] Inputting the first video features into the feature adaptation module corresponding to the video task to obtain second video features;

[0090] If the computing module corresponding to the video task includes at least one auxiliary processor, inputting the second video features into the video processor and the at least one auxiliary processor corresponding to the video task respectively to obtain a video processing result and at least one auxiliary processing result;

[0091] Based on the label data respectively corresponding to the video processor and the at least one auxiliary processor in the training samples, and the video processing result and the at least one auxiliary processing result, adjusting the model parameters of the video processor, the at least one auxiliary processor and the feature adapter;

[0092] If the computing module corresponding to the video task does not include at least one auxiliary processor, inputting the second video features into the video processor corresponding to the video task to obtain a video processing result;

[0093] Based on the label data corresponding to the video processor in the training samples, and the video processing result, adjusting the model parameters of the video processor and the feature adapter;

[0094] After the training is completed, the at least one auxiliary processor is pruned from the video processing model to obtain a trained video processing model.

[0095] The processing type of the at least one auxiliary processor can be different from the processing type of the video processor. For example, the video processor is used to calculate video-specific features, and the at least one auxiliary processor can be used to implement video classification, etc.

[0096] Taking a plurality of video tasks including a video category identification task and a video signature feature extraction task as an example, the video processing model can be obtained as shown in FIG. 1. Figure 3b In the video processing model, the calculation module of the video category identification task connected to the output of the feature extractor 300 can include a classification feature adaptation module 31 and a video classifier 32; and the calculation module of the video signature feature extraction task connected to the output of the feature extractor 300 can include a signature feature adaptation module 33, a signature vector calculator 34, and at least one auxiliary classifier 35, which can include a multi-classifier for identifying a specific video category or a binary classifier for identifying whether a specific object exists in the video, etc.

[0097] By adding the auxiliary processor, the video processor can combine the auxiliary processor to reasonably adjust the parameters of the feature adapter, avoid the model from being trained poorly, and avoid the situation of being unable to process or having low processing accuracy. After the training is completed, the auxiliary processor can be pruned, that is, a trained video processing model as shown in FIG. 1 can be obtained. Figure 3c

[0098] In an actual application, the technical solution of the embodiment of the present application can be applied to an audio data recommendation scene by using the video category and the video signature feature obtained by the video processing model. The similarity between the video signature feature and the audio signature feature can be calculated, for example, the similarity can be represented by using the Euclidean distance, and the audio data matched with the video to be processed can be determined according to the similarity. When the video processing model is trained, the model input in the training sample of the video signature feature extraction task can be the sample video data, and the model output, that is, the label data corresponding to the signature vector calculator, can be the distance range between the signature feature of the sample video data and the audio data matched with the sample video data. In the training process, the video processing result calculated by the signature vector calculator can be used to calculate the Euclidean distance with the audio signature feature of the audio data. The model parameters are adjusted according to the distance range in the label data as a constraint condition, so that the Euclidean distance between the video processing result of the signature vector calculator and the audio signature feature of the audio data is in the distance range, that is, it is considered that the training requirement is met, and the training is completed.

[0099] ​In another practical scenario, the at least one video task may include one video task, and step 204 may be to connect the output of the feature extractor to a computing module of the video task to construct a video processing model.

[0100] The video processing model can process the video to be processed to obtain processing results corresponding to the video tasks.

[0101] The video processing model can be obtained by training based on training samples of the video task.

[0102] In this implementation scenario, the video processing model can be used to process one video task. Since there may be processing requirements for multiple video tasks, in order to improve processing efficiency and reduce the amount of computation, as another embodiment, after training the video processing model using training samples corresponding to at least one video task while keeping the model parameters of the feature extractor unchanged, the method may further include:

[0103] Determining a plurality of video processing models constructed by a feature extractor and a computing module of a video task, respectively; the plurality of video processing models corresponding to different video tasks;

[0104] Multiple video processing models are integrated to obtain a comprehensive processing model composed of a feature extractor and computing modules corresponding to multiple video tasks connected to the output of the feature extractor; the comprehensive processing model is used to process the video to be processed to obtain computing results corresponding to multiple video tasks.

[0105] Since multiple video processors are built based on feature extractors, multiple video processing models built based on feature extractors can be trimmed and merged to obtain a common feature extractor, and a comprehensive processing model for multiple video task processing can be implemented.

[0106] The comprehensive processing model and Figure 3b or Figure 3c The difference between the video processing models is that, Figure 3b or Figure 3c The video processing model shown here is constructed using a shared feature extractor, integrating computational modules from multiple video tasks, and then training using training samples from these tasks. This integrated processing model can be built for a specific video task by first building and training a video processing model based on a feature extractor, then trimming and merging multiple video processing models. The integrated processing model can be used immediately without further training.

[0107] The video task computing module can include a feature adaptation module and a video processor. In addition, for a specific video task, the computing module can also include at least one auxiliary processor. The specific video task can refer to any video task, and can also be determined according to the type of the video task.

[0108] In some embodiments, the model parameters of the feature extractor are kept unchanged, and the video processing model is trained using the training samples corresponding to the at least one video task, which can include:

[0109] The model parameters of the feature extractor are kept unchanged, and the training samples are input into the feature extractor to obtain first video features.

[0110] The first video features are input into the feature adaptation module to obtain second video features.

[0111] If the computing module includes at least one auxiliary processor, the second video features are input into the video processor and the at least one auxiliary processor respectively to obtain video processing results and at least one auxiliary processing result.

[0112] Based on the label data in the training samples corresponding to the video processor and the at least one auxiliary processor respectively, and the video processing results and the at least one auxiliary processing result, the model parameters of the video processor, the at least one auxiliary processor and the feature adapter are adjusted until the training requirements are met.

[0113] After the training is completed, the at least one auxiliary processor is cut from the video processing model.

[0114] If the computing module does not include at least one auxiliary processor, the second video features are input into the video processor to obtain video processing results.

[0115] Based on the label data in the training samples and the video processing results, the model parameters of the video processor and the feature adapter are adjusted until the training requirements are met.

[0116] The plurality of video processing models are integrated to obtain a comprehensive processing model composed of the feature extractor and a plurality of computing modules corresponding to a plurality of video tasks connected to the output of the feature extractor.

[0117] The plurality of video processing models are integrated to obtain a comprehensive processing model composed of the feature extractor and a plurality of computing modules corresponding to a plurality of video tasks connected to the output of the feature extractor.

[0118] That is, the plurality of video processing modules are cut and combined to obtain a comprehensive processing model composed of the feature extractor and a plurality of feature adaptation modules and video processing modules corresponding to a plurality of video tasks connected to the output of the feature extractor.

[0119] For the purpose of clearly understanding the generation process of the comprehensive processing model, the following still takes multiple video tasks including the video category identification task and the video signature feature extraction task as an example for illustration. Figure 3d For the video processing model corresponding to the video category identification task, it can be known that it includes a feature extractor 300, a feature adaptation module 36 connected with the feature extractor 300, and a video classifier 37 connected with the feature adaptation module; Figure 3e For the video processing model corresponding to the video signature feature extraction task, it can be known that it includes a feature extractor 300, a feature adaptation module 38 connected with the feature extractor 300, and a signature vector calculator 39 and at least one auxiliary classifier 40 connected with the feature adaptation module respectively. Figure 3f For the purpose of clearly understanding the generation process of the comprehensive processing model, the following still takes multiple video tasks including the video category identification task and the video signature feature extraction task as an example for illustration. Figure 3e After the video processing model of Figure 3f is trained and the auxiliary classifier is cut off, the video processing model obtained is Figure 3d The comprehensive processing model can be obtained by merging the video processing model of Figure 3c , and the structure of the comprehensive processing model is the same as that of the video processing model shown in

[0120] In an actual application, the video category and the video signature feature obtained by using the video processing model can be applied to an audio data recommendation scene according to the technical solution of the embodiments of the present application, the similarity between the video signature feature and the audio signature feature can be calculated, such as using the Euclidean distance to represent the similarity, and the audio data matched with the video to be processed can be determined according to the similarity. When training the video processing model, the model input in the training sample of the video signature feature extraction task can be the sample video data, and the model output, that is, the label data corresponding to the signature vector calculator, can be the distance range between the signature feature of the sample video data and the audio data matched therewith. In the training process, the video processing result calculated by the signature vector calculator can be used to calculate the Euclidean distance with the audio signature feature of the audio data, and the model parameters can be adjusted according to the distance range in the label data as a constraint condition, so that the Euclidean distance between the video processing result of the signature vector calculator and the audio signature feature of the audio data is within the distance range, that is, it is considered to meet the training requirements, and the training is completed.

[0121] In addition, the embodiments of the present application also provide a model generation method, as described in Figure 4 , which can include the following steps:

[0122] 401: Determine multiple image processing models.

[0123] 402: Extract a feature extraction module from each of the multiple image processing models to obtain multiple feature extraction modules.

[0124] 403: Connect multiple feature extraction modules in parallel to construct a feature extractor.

[0125] Among them, the feature extractor can be used to extract video features of the video to be processed.

[0126] The operations of steps 401 to 403 can be found in the previous text. Figure 2 The operations of steps 201 to 203 in the illustrated embodiment will not be repeated here. In this embodiment, the feature extractor constructed by connecting the feature extraction modules in multiple image processing models can be used as an independent feature extraction model to extract video features of the video to be processed. Since the feature extractor is constructed based on multiple feature extraction modules, and the multiple feature extraction modules are extracted from multiple image processing models, the image tasks of each image processing model can be different. Therefore, the features extracted by the feature extractor are equivalent to integrating different types of features, so that the features obtained will be more accurate.

[0127] Based on the video processing model obtained by training in the above corresponding embodiments, video processing can be performed. Figure 5 As shown in FIG, a flowchart of an embodiment of a video processing method provided by an embodiment of the present application, the method may include the following steps:

[0128] 501: Extract multiple image frames from the video to be processed.

[0129] One method for extracting the multiple image frames is to extract one image frame every predetermined number of image frames from the image frames of the video to be processed to obtain the multiple image frames. Another method is to extract key frames from the processed video. Of course, in theory, it can also refer to all video frames that make up the video to be processed.

[0130] 502: Input multiple image frames into the video processing model.

[0131] The video processing model includes a feature extractor and at least one calculation module; the feature extractor includes multiple feature extraction modules; and the multiple feature extraction modules are extracted from multiple image processing models respectively.

[0132] The video processing model is trained using training samples corresponding to at least one video task, and the model parameters of the feature extractor are kept unchanged during the training process to ensure the accuracy of the video processing model.

[0133] Among them, the specific construction and training methods of the video processing model can be found in the corresponding embodiments described above and will not be repeated here.

[0134] 503: Obtain the first video feature corresponding to the video to be processed based on the frame features extracted from the multiple image frames by the multiple feature extraction modules respectively.

[0135] The multiple image frames are input into the video processing model, and each feature extraction module is input respectively, and each feature extraction module extracts frame features from the multiple image frames respectively, and each feature extraction module can extract multiple frame features corresponding to the multiple image frames.

[0136] Based on the multiple frame features output by the multiple feature extraction modules respectively, the first video feature of the video to be processed can be obtained.

[0137] Optionally, the feature extractor specifically includes multiple sub-extractors connected by the outputs of the multiple feature extraction modules to the feature fusion modules respectively, and a feature connection module connected to the outputs of the multiple sub-extractors, and the obtaining of the first video feature corresponding to the video to be processed based on the frame features extracted from the multiple image frames by the multiple feature extraction modules respectively includes:

[0138] The frame features of the multiple image frames are extracted by the multiple feature extraction modules respectively, and the multiple frame features are fused by the respective corresponding feature fusion modules to obtain fused features.

[0139] The multiple fused features output by the multiple sub-extractors are fused by the feature connection module to obtain the first video feature.

[0140] 504: Obtain at least one processing result by processing the first video feature by at least one calculation module respectively.

[0141] Each calculation module can include a feature adaptation module and a video processor, and the first video feature can be output to the at least one calculation module, and the feature adaptation module in each calculation module processes the first video feature to obtain a second video feature, and the video processor processes the second video feature to obtain a corresponding processing result.

[0142] The specific implementation and use of the feature extraction module, the feature fusion module, the feature connection module, the feature video module and the video processor are described in detail in the related embodiments of the model generation method, which will not be repeated here.

[0143] As can be known from the foregoing examples, the video processing model provided by the embodiments of the present application can be used in the audio data recommendation scene for video data, such as in a video sharing system or a video processing system. In the video sharing system or the video processing system, a user can upload video data shot by the user, and the system can recommend matching audio data based on the uploaded video data of the user, and combine the audio data with the video data to improve the attention of the video data. In order to improve the accuracy of the audio data matching the video data, the inventors think that the video category and the video signature feature can be combined for matching, wherein the video category may, for example, refer to a video style type, such as classical, traditional, jazz, etc., and the video signature feature is a specific type of video feature, which may, for example, refer to a multidimensional real number vector for uniquely representing the video content. The technical solutions of the embodiments of the present application can be used to construct and train a video processing model with more accurate processing results, so as to improve the accuracy of the video category and the video signature feature, and then the accurate recommendation of the audio data can be realized. The technical solutions of the embodiments of the present application are described below by taking the audio data recommendation as an example. Referring to FIG. 3, it is a flowchart of another embodiment of a video processing method provided by the embodiments of the present application, and the technical solutions of the embodiment can be executed by a second server deploying a video processing model, and of course, the technical solutions of the embodiment can also be executed by a client deploying the video processing model. The method can include the following steps. Figure 6a

[0144] 601: receiving a to-be-processed video uploaded by a user.

[0145] When the technical solutions of the embodiment are executed by the second server, the to-be-processed video can be uploaded to the second server by the client based on a user control operation, for example, the client can send a video processing request to the second server based on the to-be-processed video uploaded by the user, so that the second server can determine the to-be-processed video from the video processing request. The to-be-processed video can be obtained by the client based on a user control operation or read from a local system of the client based on a user instruction operation.

[0146] 602: extracting a plurality of image frames from the to-be-processed video.

[0147] 603: inputting the plurality of image frames into a video processing model to obtain a video category and a video signature feature.

[0148] The video processing model can determine the video category and the video signature feature, that is, the video processing model can be a video processing model as shown in FIG. 4. Figure 3c

[0149] Of course, as another embodiment, the plurality of image frames can also be input into a comprehensive processing model to obtain the video category and the video signature feature.

[0150] ​​Of course, as another embodiment, a plurality of image frames can also be input into a video processing model as shown in Figure 3d to obtain video categories, a plurality of image frames can be input into a video processing model as shown in Figure 3f to obtain video signature features, etc.

[0151] Wherein, in the case where the video processing model is composed of a feature extractor and at least one computing module; the feature extractor can include a plurality of feature extraction modules; the plurality of feature extraction modules respectively extract from a plurality of image processing models.

[0152] After inputting a plurality of image frames into the video processing model, the first video features corresponding to the video to be processed can be obtained based on the frame features extracted from the plurality of image frames by the plurality of feature extraction modules respectively; and the video categories and the video signature features can be obtained by respectively processing the first video features using the at least one computing module.

[0153] In addition, in the case where the feature extractor specifically includes a plurality of sub-extractors composed of the outputs of the plurality of feature extraction modules connected to the feature fusion modules respectively, and a feature connection module connected to the outputs of the plurality of sub-extractors, optionally, the frame features of the plurality of image frames can be extracted by the plurality of feature extraction modules respectively, and the plurality of frame features can be fused to obtain fusion features through the respective corresponding feature fusion modules; the plurality of fusion features output by the plurality of sub-extractors can be fused to obtain the first video features by using the feature connection module.

[0154] Wherein, the computing module performing the video category identification task can include a feature adaptation module and a video classifier, and the computing module performing the video signature feature extraction task can include a feature adaptation module and a signature vector calculator, so that the second video features can be obtained based on the first video features by the feature adaptation module in each computing module, the corresponding video categories can be obtained by classifying the first video features using the video classifier, and the video signature features can be obtained by signature vector calculation on the first video features using the signature vector calculator.

[0155] 604: Based on the video categories and the video signature features, at least one target audio data matched with the video to be processed is screened from an audio database.

[0156] 605: At least one target audio data is recommended to the user.

[0157] When the technical solution of the embodiment is executed by the second server, step 505 can specifically be sending recommendation prompt information of the at least one target audio data to the client, outputting the recommendation prompt information by the client, and achieving the purpose of recommending to the user.

[0158] The embodiment can screen at least one target audio data matching the to-be-processed video from the audio database based on the video category and the video signature feature. Since the accuracy of the video category and the video signature feature can be ensured, the accuracy of the audio data recommendation can be improved.

[0159] As an optional mode, screening at least one target audio data matching the to-be-processed video from the audio database based on the video category and the video signature feature can include:

[0160] screening at least one audio data matching the video category from the audio database;

[0161] calculating the similarity between the audio signature feature of the at least one audio data and the video signature feature respectively;

[0162] determining at least one target audio data whose similarity meets a similarity requirement;

[0163] recommending the at least one target audio data to the user.

[0164] The similarity may, for example, refer to the feature distance between the audio signature feature and the video signature feature, such as the Euclidean distance, the cosine distance, or the Mahalanobis distance.

[0165] The at least one target audio data whose similarity meets the similarity requirement may, for example, refer to at least one target audio data whose similarity is greater than a predetermined value or target audio data corresponding to the first predetermined number of similarities in a descending order of similarity.

[0166] As another optional mode, screening at least one target audio data matching the to-be-processed video from the audio database based on the video category and the video signature feature can include:

[0167] screening at least one first audio data matching the video category from the audio database;

[0168] calculating the similarity between the video signature feature and the audio signature feature of the audio data in the audio database respectively, and screening at least one second audio data whose similarity meets a similarity requirement;

[0169] determining at least one third audio data existing in the at least one audio data and the at least one second audio data;

[0170] selecting at least one target audio data from the at least one third audio data in a descending order of similarity;

[0171] recommending the at least one target audio data to the user.

[0172] The at least one target audio data may be, for example, audio data with the maximum similarity in the at least one third audio data, or a predetermined number of audio data, or the at least one third audio data.

[0173] The at least one third audio data is obtained by taking an intersection of the at least one audio data and the at least one second audio data.

[0174] Figure 6b A possible scene interaction schematic diagram of the technical scheme of the embodiment of the present application in an audio data recommendation scene is shown. The client 61 receives a user-uploaded video to be processed, and sends the video to be processed to the second server 62.

[0175] The second server 62 deploys a video processing model. The video processing model may be trained in the first server 63, or may be trained by the second server 62.

[0176] The second server 62 can identify the video category and video signature feature of the video to be processed, and determine at least one target audio data matched with the video to be processed based on the video category and the video signature feature.

[0177] The second server 62 sends recommendation prompt information of the at least one target audio data to the client 61, and the client 61 outputs the recommendation prompt information.

[0178] Figure 7 An embodiment of a structure schematic diagram of a model generation device provided by the embodiment of the present application is shown. The device may include:

[0179] A first determination module 701 is configured to determine a plurality of image processing models.

[0180] A first extraction module 702 is configured to extract a feature extraction module from each of the plurality of image processing models, to obtain a plurality of feature extraction modules.

[0181] A first construction module 703 is configured to connect outputs of the plurality of feature extraction modules to a feature fusion module, respectively, to construct a feature extractor.

[0182] The feature extractor can be configured to extract a video feature of a video to be processed.

[0183] In some embodiments, the device may further include:

[0184] A second construction module 704 is configured to connect an output of the feature extractor to a calculation module of at least one video task to construct a video processing model.

[0185] The first training module 705 is configured to keep the model parameters of the feature extractor unchanged, and train the video processing model by using the training samples of the at least one video task; the video processing model is configured to process the to-be-processed video to obtain processing results corresponding to the at least one video task respectively.

[0186] In some embodiments, the first building module can be specifically configured to connect outputs of the plurality of feature extraction modules to a feature fusion module respectively to obtain a plurality of sub-extractors; the feature fusion module is configured to fuse the frame features output by the feature extraction modules to obtain fused features; and the outputs of the plurality of sub-extractors are connected to a feature connection module to obtain the feature extractor; the feature connection module is configured to fuse the fused features output by the plurality of sub-extractors to obtain the first video feature.

[0187] In some embodiments, the second building module can be specifically configured to connect the output of the feature extractor to the calculation module of one video task to build the video processing model.

[0188] The apparatus can further include:

[0189] The second determining module is configured to determine a plurality of video processing models respectively built by the feature extractor and the calculation module of one video task; the plurality of video processing models correspond to different video tasks.

[0190] The model integration module is configured to integrate the plurality of video processing models to obtain a comprehensive processing model composed of the feature extractor and the calculation modules corresponding to the plurality of video tasks respectively and connected to the output of the feature extractor; the comprehensive processing model is configured to process the to-be-processed video to obtain calculation results corresponding to the plurality of video tasks respectively.

[0191] In some embodiments, the calculation module includes the feature adaptation module and the video processor, or the calculation module includes the feature adaptation module, the video processor and at least one auxiliary processor.

[0192] The first training module can be specifically configured to keep the model parameters of the feature extractor unchanged, input the training sample into the feature extractor to obtain first video features, input the first video features into the feature adaptation module to obtain second video features, if the computing module includes at least one auxiliary processor, input the second video features into the video processor and the at least one auxiliary processor respectively to obtain a video processing result and at least one auxiliary processing result, adjust the model parameters of the video processor, the at least one auxiliary processor and the feature adapter based on the label data corresponding to the video processor and the at least one auxiliary processor in the training sample and the video processing result and the at least one auxiliary processing result, if the computing module does not include at least one auxiliary processor, input the second video features into the video processor to obtain a video processing result, and adjust the model parameters of the video processor and the feature adapter based on the label data in the training sample and the video processing result.

[0193] The model integration module can be specifically configured to merge the plurality of video processing models to obtain a merged model composed of the computing modules corresponding to the plurality of video tasks respectively and connected by the feature extractor and the output of the feature extractor, and cut the auxiliary processor from the merged model to obtain a comprehensive processing model.

[0194] In some embodiments, the computing module includes the feature adapter and the video processor or the computing module includes the feature adapter, the video processor and at least one auxiliary processor.

[0195] The first training module can be specifically configured to keep the model parameters of the feature extractor unchanged, input the training sample into the feature extractor to obtain first video features, input the first video features into the feature adaptation module to obtain second video features, if the computing module includes at least one auxiliary processor, input the second video features into the video processor and the at least one auxiliary processor respectively to obtain a video processing result and at least one auxiliary processing result, adjust the model parameters of the video processor, the at least one auxiliary processor and the feature adapter based on the label data corresponding to the video processor and the at least one auxiliary processor in the training sample and the video processing result and the at least one auxiliary processing result, if the computing module does not include at least one auxiliary processor, input the second video features into the video processor to obtain a video processing result, and adjust the model parameters of the video processor and the feature adapter based on the label data in the training sample and the video processing result. After the training is completed, the auxiliary processor is cut from the video processing model to obtain a trained video processing model.

[0196] In some embodiments, the first determining module can be specifically configured to train the plurality of image processing models by using training samples corresponding to the plurality of image tasks respectively; the plurality of image tasks include one or more of an image classification task, an object detection task, and an image segmentation task.

[0197] In some embodiments, the first extracting module can be specifically configured to remove network layers of the corresponding number of layers from the end of the plurality of image processing models according to the types of tasks corresponding to the plurality of image processing models respectively, to obtain the plurality of feature extraction modules.

[0198] Figure 7 The model generation apparatus can perform Figure 2 The model generation method of the embodiments described above has the same implementation principles and technical effects. The specific operation modes of each module and unit in the model generation apparatus in the above embodiments have been described in detail in the embodiments of the method, and will not be described in detail here.

[0199] In one possible design, Figure 7 The model generation apparatus of the embodiments described above can be implemented as a computing device, which can be used as, for example, Figure 1 The first server or the second server in the embodiments described above, such as Figure 8 The computing device can include a storage component 801 and a processing component 802, as shown.

[0200] The storage component 801 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 802 to implement the model generation method as shown in Figure 2 or as shown in Figure 4 The model generation method.

[0201] Of course, the computing device can also include other components, such as an input / output interface, a communication component, and the like.

[0202] The input / output interface provides an interface between the processing component and a peripheral interface module, which can be an output device, an input device, and the like. The communication component is configured to facilitate wired or wireless communication between the computing device and other devices.

[0203] The computing device can be a physical device or an elastic computing host provided by a cloud computing platform, and the computing device can be a cloud server. The processing component, the storage component, and the like can be basic server resources rented or purchased from the cloud computing platform.

[0204] The embodiments of the present application also provide a computer readable storage medium storing a computer program, wherein the computer program is executed by a computer to implement the model generation method as described above. Figure 2 orFigure 4 The model generation method of the illustrated embodiment.

[0205] Figure 9 An embodiment of a video processing device provided by the present application has a structure diagram as shown in the figure. The device can include:

[0206] The data extraction module 901 is configured to extract a plurality of image frames from a to-be-processed video.

[0207] The data processing module 902 is configured to input the plurality of image frames into a video processing model, obtain a first video feature corresponding to the to-be-processed video based on frame features extracted from the plurality of image frames by the plurality of feature extraction modules, and process the first video feature by using the at least one computing module to obtain at least one processing result. The video processing model includes a feature extractor and at least one computing module. The feature extractor includes a plurality of feature extraction modules. The plurality of feature extraction modules are extracted from a plurality of image processing models.

[0208] In some embodiments, the feature extractor specifically includes a plurality of sub-extractors connected by outputs of the plurality of feature extraction modules to the feature fusion modules, respectively, and a feature connection module connected to outputs of the plurality of sub-extractors.

[0209] The data processing module can specifically include extracting frame features of the plurality of image frames by using the plurality of feature extraction modules, respectively, and fusing the plurality of frame features by using the respective corresponding feature fusion modules to obtain fused features. The first video feature is obtained by fusing the plurality of fused features output by the plurality of sub-extractors by using the feature connection module.

[0210] In some embodiments, the at least one processing result includes a video category and a video signature feature.

[0211] The device can further include:

[0212] The receiving module is configured to receive a to-be-processed video uploaded by a user.

[0213] As an optional way, the data processing module can specifically be configured to filter at least one audio data matching the video category from an audio database, calculate a similarity between an audio signature feature of the at least one audio data and the video signature feature, respectively, determine at least one target audio data satisfying a similarity requirement, and recommend the at least one target audio data to the user.

[0214] As another alternative, the data processing module specifically screens at least one first audio data from the audio database that matches the video category; respectively calculates the similarity between the video signature feature and the audio signature feature of the audio data in the audio database, screens at least one second audio data that satisfies the similarity requirement; determines at least one third audio data that exists in both the at least one audio data and the at least one second audio data; selects at least one target audio data from the at least one third audio data in descending order of similarity; and recommends the at least one target audio data to the user.

[0215] In one possible design, Figure 9 The video processing apparatus of the illustrated embodiment can be implemented as a computing device, which can be used as, for example, Figure 1 The second server in the illustrated embodiment can be implemented as a computing device, which can be used as, for example, Figure 10 As shown, the computing device can include a storage component 1001 and a processing component 1002.

[0216] The storage component 1001 stores one or more computer instructions, where the one or more computer instructions are called by the processing component 1002 for execution to implement the video processing method as shown in Figure 5

[0217] Of course, the computing device can also include other components, for example, an input / output interface, a communication component, and the like.

[0218] The input / output interface provides an interface between the processing component and a peripheral interface module, which can be an output device, an input device, and the like. The communication component is configured to facilitate wired or wireless communication between the computing device and other devices.

[0219] The computing device can be a physical device or an elastic computing host provided by a cloud computing platform, and in this case, the computing device can refer to a cloud server, and the processing component, the storage component, and the like can be basic server resources rented or purchased from the cloud computing platform.

[0220] The embodiments of the present application also provide a computer readable storage medium storing a computer program, and the computer program can implement the video processing method of the embodiments of 5 shown above when executed by a computer.

[0221] ​The processing component(s) can be a single processor or a plurality of processors, which are either co-located or distributed. The processing component(s) can be configured to execute computer-readable instructions to complete all or part of the methods described above. Of course, the processing component(s) can also be implemented as or with one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic devices, to perform the processing tasks described above.

[0222] The storage component is configured to store various types of data to support the operation of the terminal. The storage component can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0223] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0224] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0225] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0226] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video processing method, characterized in that: include: extracting multiple image frames from the video to be processed; Inputting the multiple image frames into a video processing model; wherein the video processing model includes a feature extractor and at least one computing module; the feature extractor includes multiple feature extraction modules; the multiple feature extraction modules are respectively extracted from multiple image processing models; the multiple image processing models correspond to multiple image tasks, the multiple image tasks including one or more of image classification tasks, object detection tasks, and image segmentation tasks, and the multiple image processing models are trained using training samples corresponding to the multiple image tasks; Obtaining a first video feature corresponding to the video to be processed based on the frame features extracted from the multiple image frames by the multiple feature extraction modules respectively; The first video features are processed respectively using the at least one computing module to obtain at least one processing result.

2. The method according to claim 1, characterized in that The feature extractor specifically includes a plurality of sub-extractors formed by connecting the outputs of the plurality of feature extraction modules to a feature fusion module, and a feature connection module connected to the outputs of the plurality of sub-extractors; The obtaining of the first video feature corresponding to the video to be processed based on the frame features extracted from the multiple image frames by the multiple feature extraction modules includes: Utilizing the plurality of feature extraction modules to respectively extract frame features of the plurality of image frames, and fusing the plurality of frame features through the corresponding feature fusion modules to obtain fused features; The feature connection module is used to fuse the multiple fusion features output by the multiple sub-extractors to obtain a first video feature.

3. The method according to claim 1, characterized in that The at least one processing result includes a video type and a video signature feature; Before extracting a plurality of image frames from the video to be processed, the method further includes: Receive videos uploaded by users for processing; After inputting the multiple image frames into the video processing model and obtaining at least one processing result, the method further includes: Filtering at least one audio data matching the video category from an audio database; respectively calculating similarities between the audio signature feature of the at least one audio data and the video signature feature; determining at least one target audio data whose similarity satisfies a similarity requirement; The at least one target audio data is recommended to the user.

4. The method according to claim 1, wherein The at least one processing result includes a video type and a video signature feature; Before extracting a plurality of image frames from the video to be processed, the method further includes: Receive videos uploaded by users for processing; After inputting the multiple image frames into the video processing model and obtaining at least one processing result, the method further includes: Filtering at least one first audio data matching the video category from an audio database; Calculating similarities between the video signature features and the audio signature features of the audio data in the audio database, and selecting at least one second audio data whose similarity meets the similarity requirement; determining at least one third audio data present in both the at least one first audio data and the at least one second audio data; selecting at least one target audio data from the at least one third audio data in descending order of similarity; The at least one target audio data is recommended to the user.

5. A model generation method, characterized in that: include: Determine multiple image processing models; Extracting feature extraction modules respectively from the multiple image processing models to obtain multiple feature extraction modules; Connecting the plurality of feature extraction modules in parallel to construct a feature extractor; Connecting the output of the feature extractor to a computing module of at least one video task to construct a video processing model; Keeping the model parameters of the feature extractor unchanged, the video processing model is trained using the training samples of the at least one video task; the video processing model is used to process the video to be processed to obtain processing results corresponding to the at least one video task; Determining a plurality of image processing models comprises: Multiple image processing models are trained using training samples corresponding to multiple image tasks, wherein the multiple image tasks include one or more of image classification tasks, target detection tasks, and image segmentation tasks.

6. The method according to claim 5, characterized in that Said connecting the plurality of feature extraction modules in parallel to construct a feature extractor comprises: The outputs of the plurality of feature extraction modules are respectively connected to a feature fusion module to obtain a plurality of sub-extractors; the feature fusion module is used to fuse the plurality of frame features output by the feature extraction module to obtain a fusion feature; The outputs of the multiple sub-extractors are connected to a feature connection module to obtain a feature extractor; the feature connection model is used to fuse the multiple fusion features output by the multiple sub-extractors to obtain a first video feature.

7. The method according to claim 5, characterized in that Connecting the output of the feature extractor to a computing module of at least one video task to construct a video processing model comprises: Connecting the output of the feature extractor to a computing module of a video task to construct a video processing model; After the model parameters of the feature extractor are kept unchanged and the video processing model is trained using the training samples corresponding to the at least one video task, The method further comprises: Determining a plurality of video processing models respectively constructed by the feature extractor and a computing module of a video task; the plurality of video processing models corresponding to different video tasks; The multiple video processing models are integrated to obtain a comprehensive processing model composed of the feature extractor and the calculation modules corresponding to the multiple video tasks connected to the output of the feature extractor; the comprehensive processing model is used to process the video to be processed to obtain calculation results corresponding to the multiple video tasks.

8. The method according to claim 7, characterized in that The computing module includes a feature adaptation module and a video processor or the computing module includes a feature adaptation module, a video processor and at least one auxiliary processor; The step of maintaining the model parameters of the feature extractor unchanged and training the video processing model using the training samples corresponding to the at least one video task comprises: Keeping the model parameters of the feature extractor unchanged, inputting the training sample into the feature extractor to obtain the first video feature; Inputting the first video feature into a feature adaptation module to obtain a second video feature; If the computing module includes at least one auxiliary processor, the second video feature is input into the video processor and the at least one auxiliary processor respectively to obtain a video processing result and at least one auxiliary processing result; Adjusting model parameters of the video processor, the at least one auxiliary processor, and the feature adapter based on label data corresponding to the video processor and the at least one auxiliary processor in the training samples, as well as the video processing result and the at least one auxiliary processing result; After the training is completed, pruning the at least one auxiliary processor from the video processing model; If the computing module does not include at least one auxiliary processor, inputting the second video feature into the video processor to obtain a video processing result; Adjusting model parameters of the video processor and the feature adapter based on the label data in the training sample and the video processing result; The step of integrating the multiple video processing models to obtain a comprehensive processing model composed of the feature extractor and computing modules corresponding to multiple video tasks connected to the feature extractor includes: The multiple video processing models are merged to obtain a merged model consisting of the feature extractor and the computing modules corresponding to the multiple video tasks connected to the output of the feature extractor.

9. The method according to claim 5, characterized in that The computing module includes a feature adapter and a video processor or the computing module includes a feature adapter, a video processor and at least one auxiliary processor; The step of keeping the model parameters of the feature extractor unchanged and training the video processing model using the training samples of the at least one video task comprises: Keeping the model parameters of the feature extractor unchanged, for each video task, inputting the training sample of the video task into the feature extractor to obtain a first video feature; Inputting the first video feature into a feature adaptation module corresponding to the video task to obtain a second video feature; If the computing module corresponding to the video task includes at least one auxiliary processor, inputting the second video feature into the video processor corresponding to the video task and the at least one auxiliary processor respectively to obtain a video processing result and at least one auxiliary processing result; Adjusting model parameters of the video processor, the at least one auxiliary processor, and the feature adapter based on label data corresponding to the video processor and the at least one auxiliary processor in the training samples, as well as the video processing result and the at least one auxiliary processing result; If the computing module corresponding to the video task does not include at least one auxiliary processor, inputting the second video feature into the video processor corresponding to the video task to obtain a video processing result; Adjusting model parameters of the feature adapter of the video processor based on the label data corresponding to the video processor in the training sample and the video processing result; After the training is completed, the auxiliary processor is trimmed from the video processing model to obtain a trained video processing model.

10. A model generation method, characterized in that: include: Determine multiple image processing models; Extracting feature extraction modules respectively from the multiple image processing models to obtain multiple feature extraction modules; Connecting the outputs of the multiple feature extraction modules to the feature fusion module respectively to construct a feature extractor; The feature extractor is used to extract video features of the video to be processed; the multiple feature extraction modules are respectively based on multiple image frames corresponding to the video to be processed, respectively extracting frame features of the multiple image frames; the feature fusion module is used to fuse the multiple frame features respectively extracted by the multiple feature extraction modules to obtain the video features; Determining a plurality of image processing models comprises: Multiple image processing models are trained using training samples corresponding to multiple image tasks, wherein the multiple image tasks include one or more of image classification tasks, target detection tasks, and image segmentation tasks.

11. A computing device, characterized in that including storage components and processing components; The storage component stores one or more computer instructions; the one or more computer instructions are used for the processing component to call and execute to implement the model generation method as described in claim 5 or 10 or to implement the video processing method as described in claim 1.

Images