Training of a content recognition model, content recognition method and apparatus

Abstract

The application relates to the technical field of data processing, in particular to a content recognition model training method and device, the method comprising: obtaining an initial content recognition model to be trained, wherein the initial content recognition model comprises a main task network and a first auxiliary task network sharing a first subnetwork; performing at least one round of iterative training on the initial content recognition model based on a sample video resource set; and establishing a target content recognition model based on the main task network output in the last round of iterative training. Thus, the training of the initial content recognition model takes into account the influence of video categories on target events and the influence of video resource content on target events, so that the trained content recognition model can meet the recognition of video resources of various video categories, can reduce the processing difficulty of video resources of different video categories in the actual processing process, and can provide the accuracy of content recognition.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a content recognition model training method and apparatus. Background Technology

[0002] In order to effectively perceive the content of a target video and not miss its exciting segments, information recognition methods can usually be used to identify the exciting segments that meet the requirements from a continuous video resource.

[0003] In related technologies, since the definition of highlights varies across different categories of video resources, it is usually necessary to train corresponding information recognition models for different categories of video resources in order to identify highlights in a targeted manner. For example, recognition model 1 is trained for video resources of football matches, recognition model 2 is trained for video resources of basketball matches, and recognition model 3 is trained for video resources of table tennis matches, and so on.

[0004] However, using the above method requires training corresponding information recognition models for different categories of video resources. Therefore, when there are many video categories corresponding to the video resources to be processed, a large number of information recognition models need to be trained. This not only requires a lot of storage space, but also reduces the portability of the information recognition models. Furthermore, for video resources with multiple video categories, since a single information recognition model can only process one type of video resource, multiple information recognition models corresponding to different video categories are needed to complete the recognition of video content, which further reduces the recognition efficiency of video resources. Summary of the Invention

[0005] This application provides a content recognition model training method and apparatus to solve the problem of low recognition efficiency when performing content recognition on different types of videos in the prior art.

[0006] Firstly, a training method for a content recognition model is proposed, including:

[0007] An initial content recognition model to be trained is obtained, wherein the initial content recognition model includes a main task network and a first auxiliary task network sharing a first sub-network;

[0008] The initial content recognition model is trained iteratively for at least one round based on the sample video resource set, wherein the following operations are performed during one round of iteration:

[0009] The main task network determines the predicted time range of the target event based on the image features of the sample video resources, and the first auxiliary task network determines the first predicted video category corresponding to the sample video resources based on the image features of the sample video resources, and adjusts at least the network parameters in the first sub-network based on the predicted time range and the first predicted video category.

[0010] Based on the main task network output from the last round of iteration training, a target content recognition model is established.

[0011] Secondly, a content recognition method is proposed, including:

[0012] Obtain the video resource to be identified;

[0013] The video resource is input into a trained target content recognition model to obtain the time range information of the preset target event occurring in the video resource. The target content recognition model is obtained through a content recognition model training method.

[0014] Thirdly, a training device for a content recognition model is proposed, comprising:

[0015] The obtaining unit is used to obtain an initial content recognition model to be trained, wherein the initial content recognition model includes a main task network and a first auxiliary task network sharing a first sub-network;

[0016] The training unit is used to perform at least one round of iterative training on the initial content recognition model based on the sample video resource set, wherein, during one round of iteration, the following operations are performed:

[0017] The main task network determines the predicted time range of the target event based on the image features of the sample video resources, and the first auxiliary task network determines the first predicted video category corresponding to the sample video resources based on the image features of the sample video resources, and adjusts at least the network parameters in the first sub-network based on the predicted time range and the first predicted video category.

[0018] A unit is established to build a target content recognition model based on the main task network output from the last round of iteration training.

[0019] Optionally, if the initial content recognition model further includes a second auxiliary task network that shares a second sub-network with the main task network, then during one round of iterative training, the training unit is further used for:

[0020] The main task network aggregates the contextual features of pixels in the sample video resources to obtain the pixel features of the sample video resources, wherein the second sub-network is included in the first sub-network;

[0021] The second auxiliary task network determines the second predicted video category corresponding to the sample video resource based on the pixel features.

[0022] Based on the predicted time range and the second predicted video category, at least the network parameters in the second sub-network are adjusted.

[0023] Optionally, when determining the predicted time range of the target event based on image features of sample video resources through the main task network, the training unit is used to:

[0024] A sample video resource is acquired and the sample video resource is divided into at least one sample video resource segment, wherein one sample video resource segment corresponds to one time range;

[0025] Through the main task network, a sample keyframe is extracted from each of the at least one sample video resource segments, and the context features of each pixel contained in each sample keyframe are aggregated to obtain the pixel feature matrix of each sample keyframe.

[0026] Through the main task network, the contextual influence between the key frames of each sample is aggregated in each pixel feature matrix to obtain the image feature matrix representing the image features of each key frame of each sample.

[0027] Based on the feature matrices of each image, the predicted time range for the occurrence of the target event is determined.

[0028] Optionally, when aggregating the context features of each pixel in each sample keyframe to obtain the pixel feature matrix of each sample keyframe, the training unit is used to:

[0029] For each pixel in each sample keyframe, perform the following operations:

[0030] Identify all other pixels within a specified range centered on a single pixel, and aggregate the context features of these other pixels to obtain the pixel features of the single pixel.

[0031] Based on the pixel features of each pixel in each processed keyframe, a pixel feature matrix corresponding to each keyframe is generated.

[0032] Optionally, when determining the predicted time range for the occurrence of the target event, the training unit is used to:

[0033] Using the main task network, based on the image feature matrix of each sample keyframe, the probability prediction result of the target event occurring within the time range corresponding to each sample keyframe is predicted respectively.

[0034] The target probability prediction results that are higher than a set threshold are selected from the various probability prediction results, and the time range corresponding to the target probability prediction results is used as the prediction time range.

[0035] Optionally, when adjusting the network parameters in the first sub-network based on the predicted time range and the first predicted video category, the training unit is used to:

[0036] A first loss function is used to determine a first loss value based on the prediction difference between the prediction time range and the corresponding sample time range, and the network parameters of the main task network, including the first sub-network, are adjusted based on the first loss value.

[0037] A second loss function is used to determine a second loss value based on the prediction difference between the first predicted video category and the corresponding sample video category. The network parameters of the first auxiliary task network, which includes the first sub-network, are then adjusted based on the second loss value.

[0038] Optionally, when establishing the target content recognition model based on the main task network output from the last round of iteration training, the establishment unit is used for:

[0039] When the preset model convergence condition is met, in the initial content recognition model output by the last iteration training, the network structure in the first auxiliary task network except for the first sub-network is deleted, and the target content recognition model is established based on the retained main task network.

[0040] Fourthly, a content recognition device is proposed, comprising:

[0041] The acquisition unit is used to acquire the video resources to be identified;

[0042] The input unit is used to input the video resource into a trained target content recognition model to obtain time range information of a preset target event occurring in the video resource, wherein the target content recognition model is obtained by the training method of the content recognition model described in any one of the first aspects.

[0043] Optionally, after acquiring the video resource to be identified, and before inputting the video resource into the trained target content recognition model, the acquisition unit is further configured to:

[0044] When it is determined that the duration of the video resource exceeds a preset processing duration threshold, a preset frame-sampling algorithm is used to perform frame-sampling processing on the video resource so that the duration of the processed video resource does not exceed the processing duration threshold.

[0045] Fifthly, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the training method of the content recognition model described in any one of the first aspects or the content recognition method described in any one of the second aspects.

[0046] In a sixth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the training method of the content recognition model described in any one of the first aspects or the content recognition method described in any one of the second aspects.

[0047] In a seventh aspect, a computer program product is proposed, comprising a computer program that, when executed by a processor, implements the training method for the content recognition model described in any one of the first aspects or the content recognition method described in any one of the second aspects.

[0048] The beneficial effects of this application are as follows:

[0049] This application proposes a content recognition model training method and apparatus. When training the content recognition model, the processing device constructs a content recognition model using a main task network and a first auxiliary task network sharing a first sub-task network. This allows the main task network to learn and identify the time periods of exciting segments in sample video resources, and the first auxiliary task network to learn and identify the video categories corresponding to the sample video resources. Information complementarity is achieved during the initial training of the content recognition model. By adjusting the network parameters, including those in the shared first sub-network, based on the learning results of the first auxiliary task network regarding video categories, consideration of the video category of the identified video resources is incorporated into the process of identifying the time range of the target event in the video resources. This effectively balances the influence of video category on the target event and the influence of video content on the target event. Consequently, the trained content recognition model is no longer limited to a single video category but can recognize video resources of multiple categories, reducing the processing difficulty for different video categories in actual processing and improving the accuracy of content recognition.

[0050] Furthermore, the content recognition model trained using this application can identify target events occurring in video resources of different video categories. In other words, a target content recognition model trained using this application can identify the time range of target events occurring in video resources of various video categories. This greatly reduces the storage space occupied by the target content recognition model for identifying video resources of various video categories, improves the portability of the target content recognition model, and reduces the difficulty of model deployment. At the same time, when using the same model to identify data of different video categories, the data can be processed uniformly according to a unified input information processing method, which improves the processing speed of video resources to be identified, thereby reducing the difficulty of content recognition of video resources and improving the recognition efficiency of video resources. Attached Figure Description

[0051] Figure 1 This is a schematic diagram illustrating one possible application scenario in the embodiments of this application;

[0052] Figure 2a This is a schematic diagram of the resource sharing process in an embodiment of this application;

[0053] Figure 2b This is a schematic diagram of the structure of the initial content recognition model in the embodiments of this application;

[0054] Figure 2c This is a schematic diagram of the structure of the initial content recognition model including the second auxiliary task network in the embodiments of this application;

[0055] Figure 2d This is a schematic diagram illustrating the content format of the training samples generated in the embodiments of this application;

[0056] Figure 2e This is a flowchart illustrating the training process in an embodiment of this application;

[0057] Figure 2f This is a flowchart illustrating the process of determining the predicted time range of a target event occurring in a sample video resource in an embodiment of this application.

[0058] Figure 2g This is a schematic diagram of the fully connected layer and the unified layer in the embodiments of this application.

[0059] Figure 3 This is a schematic diagram of the content recognition process in an embodiment of this application;

[0060] Figure 4 This is a schematic diagram of the logical structure of the training device for the content recognition model in the embodiments of this application;

[0061] Figure 5This is a schematic diagram of the logical structure of the content recognition device in the embodiments of this application;

[0062] Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application;

[0063] Figure 7 This is a schematic diagram of the structure of a computing device in an embodiment of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0065] The terms "first," "second," etc., used in this application's specification and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0066] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0067] LSTM (Long Short-Term Memory) is a type of temporal recurrent neural network suitable for processing and predicting important events with relatively long intervals and delays in time series. It can effectively solve the long path dependency problem of traditional recurrent neural networks.

[0068] Bidirectional Long Short-Term Memory (BI-LSTM) network: It can fully consider the relationship between the context of the time sequence and make full use of bidirectional information. In the embodiments of this application, a BI-LSTM network can be used to bidirectionally aggregate the mutual contextual influence between key frames based on each key frame with a time sequence.

[0069] Bi-Gated Recurrent Unit (Bi-GRU) network: It can extract hidden information in time series and can perform dimensionality increase processing on input time series features, increasing the dimension of the features and extracting more hidden information. In the embodiments of this application, a Bi-GRU network can be used to aggregate the mutual influence between various key frames in the time series.

[0070] Convolutional Neural Network (CNN): It has powerful feature extraction capabilities and can extract deep-level data features. In the embodiments of this application, it can aggregate the contextual features of pixels in keyframes.

[0071] Target event: refers to the event of interest that occurs in the video resource. In this application embodiment, the events corresponding to the highlights in various video resources are defined as target events. For example, for video resources in the field of sports, in basketball sports video resources, events such as spiking and alley-oops are defined as target events, and in football sports video resources, events such as penalty kicks and dribbling are defined as target events.

[0072] The design concept of the embodiments of this application is briefly introduced below:

[0073] In order to effectively perceive the content included in various video resources and not miss the wonderful segments in various video resources, it is usually necessary to train corresponding information recognition models for each type of video to identify the wonderful content included in different categories of video resources.

[0074] For example, when identifying highlights in sports video resources, an information recognition model is usually trained for the corresponding category of video resources. This allows the model to directly extract visual features from the video resources and identify the highlights based on the extracted visual features.

[0075] Thus, considering that the definition of highlights differs across different video data resources, it is necessary to create corresponding information recognition models for each video data resource. In the case of multiple video categories, if it is desired to effectively identify highlights in video resources, multiple information recognition models need to be configured accordingly. Therefore, to ensure the effective implementation of video resources, a significant amount of storage space is required, which reduces the portability of information recognition models and the recognition efficiency of video resources.

[0076] In view of this, this application proposes a content recognition model training method and apparatus. When training the content recognition model, the processing device constructs a content recognition model using a main task network and a first auxiliary task network sharing a first sub-task network. This allows the main task network to learn and identify the time periods of exciting segments in sample video resources, and the first auxiliary task network to learn and identify the video categories corresponding to the sample video resources. Information complementarity is achieved during the initial training of the content recognition model. By adjusting the network parameters, including those in the shared first sub-network, based on the learning results of the first auxiliary task network regarding video categories, consideration of the video category of the identified video resources is incorporated into the process of identifying the time range of the target event in the video resources. This effectively balances the influence of the video category on the target event and the influence of the video resource content on the target event. Consequently, the trained content recognition model is no longer limited to a single video category but can recognize video resources of multiple video categories, reducing the processing difficulty for different video categories in actual processing.

[0077] Furthermore, the content recognition model trained using this application can identify target events occurring in video resources of different video categories. In other words, a target content recognition model trained using this application can identify the time range of target events occurring in video resources of various video categories. This greatly reduces the storage space occupied by the target content recognition model for identifying video resources of various video categories, improves the portability of the target content recognition model, and reduces the difficulty of model deployment. At the same time, when using the same model to identify data of different video categories, the data can be processed uniformly according to a unified input information processing method, which improves the processing speed of video resources to be identified, thereby reducing the difficulty of content recognition of video resources and improving the recognition efficiency of video resources.

[0078] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0079] See Figure 1The diagram illustrates a possible application scenario in this application embodiment. This application scenario includes a server device 110, a server device 120, and a terminal device 130. Server device 110 and server device 120 may be servers corresponding to the same application or servers corresponding to different applications. Server device 110 and server device 120 can communicate via a wired or wireless network. Communication between server device 110 and terminal device 130, and between terminal device 130 and server device 120, can also be achieved via wired or wireless networks.

[0080] In this embodiment, server device 110 and server device 120 can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0081] Terminal device 130 can be a computer device with certain computing capabilities, such as a personal computer, mobile phone, tablet computer, laptop, e-book reader, smart home device, or vehicle terminal; or it can be a video resource acquisition device such as a camera or webcam.

[0082] In possible application scenarios of this application, the server device 110 can provide video resources to be identified to other devices, or it can receive video resources to be identified sent by other devices and perform content recognition operations on the video resources.

[0083] In some possible application scenarios, server-side device 110 can be a server providing content recognition services for video resources, and server-side device 120 can be a server providing the video resources to be recognized. This application can be applied to live streaming scenarios to identify the time period in which a target event occurs in a live video stream. For example, it can receive live video streams sent by a live streaming server in real time, selectively segment the live video stream according to video duration, and then perform processing using a trained target content recognition model. This application can also be applied to scenarios where video resources obtained from various channels are identified, such as content recognition of video resources published on public accounts, mini-programs, web pages, and public media platforms.

[0084] It should be noted that, in this embodiment of the application, the domain of the identified content resources is not specifically limited. For the content recognition of video resources in different domains, it is only necessary to create corresponding training samples in a targeted manner during the training process to meet the content recognition requirements. The domain of the video resources can be sports events, film and television, game live streaming, etc.

[0085] In this embodiment of the application, any one or all of the server device 110, server device 120, and terminal device 130 may be configured with a trained target content recognition model, so that the trained target content recognition model can be used to perform content recognition on the video resource to be recognized, or to obtain the time range of the target event obtained by other devices for the video resource to be recognized.

[0086] It should be noted that, in the embodiments of this application, the content recognition model used on the device may be trained by itself or by other devices.

[0087] Taking the example of server device 110 using a trained content recognition model to perform content recognition operations on the video resource to be recognized, the content recognition model used by server device 110 can be trained by itself, or it can be a content recognition model trained by other devices and then directly sent to server device 110.

[0088] The following description of this application will illustratively illustrate the training and application processes of the content recognition model from the perspective of a processing device. Depending on the different application scenarios of this application, the processing device may specifically correspond to… Figure 1 The application does not impose any specific restrictions on any one of the server device 110, server device 120, and terminal device 130 shown in the diagram.

[0089] See Figure 2a As shown, this is a schematic diagram of the training process of the content recognition model in this application embodiment. The following is a detailed explanation in conjunction with the attached diagram. Figure 2a The process of training the content recognition model on the processing device in the embodiments of this application is described below:

[0090] Step 201: The processing device obtains the initial content recognition model to be trained.

[0091] In possible embodiments of this application, the processing device can obtain the model structure of the initial content recognition model by self-configuration or by direct acquisition, according to actual processing needs. The initial content recognition model includes a main task network for learning the time range of target events occurring in video resources, and an auxiliary task network for identifying the video category corresponding to the video resources. The auxiliary task network provides complementary information to the learning of the main task network, so that when identifying the time range of target events occurring in video resources, the video category of the video resources themselves can be taken into account, thereby enabling more targeted identification of exciting segments for different types of video resources.

[0092] In this application embodiment, the initial content recognition model, which includes a main task network and a secondary task network, has the following two possible forms.

[0093] Form 1: The initial content recognition model includes a main task network and a secondary task network.

[0094] Specifically, the initial content recognition model obtained by the processing device includes a main task network and a first auxiliary task network that share a first sub-network, wherein the first sub-network is used to extract image features from video resources.

[0095] See Figure 2b As shown, this is a structural diagram of the initial content recognition model in this embodiment of the application. The currently constructed initial content recognition model includes a main task network and a first auxiliary task network. Figure 2b As shown, the main task network consists of a keyframe extraction network, a pixel feature aggregation network, an image feature aggregation network, and a fully connected layer and a normalization layer for the main task. The first auxiliary task network also consists of a keyframe extraction network, a pixel feature aggregation network, an image feature aggregation network, and a fully connected layer and a normalization layer for the first auxiliary task. The first sub-network shared by the main task network and the auxiliary task network includes the keyframe extraction network, the pixel feature aggregation network, and the image feature aggregation network.

[0096] Form 2: The initial content recognition model includes a main task network and multiple auxiliary task networks.

[0097] In a possible embodiment of this application, the initial content recognition model to be trained obtained by the processing device includes a second auxiliary task network in addition to the main task network and the first auxiliary task network. The second sub-network is included in the first sub-network and is used to extract pixel features.

[0098] For details, please refer to Figure 2c As shown, this is a schematic diagram of the structure of the initial content recognition model including the second auxiliary task network in an embodiment of this application. Figure 2c exist Figure 2b Building upon the first auxiliary task network, a second auxiliary task network is further added. This second auxiliary task network includes a keyframe extraction network, a pixel feature aggregation network, and a fully connected layer and a normalization layer for the second auxiliary task. The second auxiliary task network shares a second sub-network with the first auxiliary task network, which corresponds to the keyframe extraction network and the pixel feature extraction network.

[0099] It should be noted that in the structures of the initial content recognition models described in Forms 1 and 2 of this application, the keyframe extraction network can specifically be a network structure of the Deep Residual (RESNEP) series, such as RESNEP50, RESNEP101, etc.; the pixel feature aggregation network can specifically be a CNN network, or a network similar to a CNN network that can aggregate pixel features in image frames; the image feature aggregation network can specifically be a BI-LSTM network, or a BI-GRU network, or a network that can aggregate the contextual influence between video frames that exist in a temporal sequence. This application will not make specific limitations here.

[0100] Thus, this application transforms the problem of identifying highlights in video resources into determining the time range within which a target event occurs in the video resource. Since the time range corresponds to a segment of the video resource, it can be determined whether a target event occurs within that segment. Simultaneously, the first auxiliary task network constructed using Form 1 assists the main task network in learning to identify the category of the video resource while it learns to identify the time range of the target event. This allows the initial content recognition model to learn to identify target events under different video resource categories during training. The auxiliary task network constructed using Form 2 establishes auxiliary tasks for identifying the video resource category before and after image feature aggregation. This enables the classification of video resource categories based on features of different granularities, better adjusting the relevant parameters in the main task network to help it accurately learn to identify the time range of the target event in different video resource categories.

[0101] Step 202: The processing device performs at least one round of iterative training on the initial content recognition model based on the sample video resource set until the preset model convergence condition is met.

[0102] In this embodiment of the application, before training the obtained initial content recognition model, it is first necessary to construct a corresponding set of sample video resources.

[0103] Specifically, when constructing a sample video resource set, the processing device can, according to actual processing needs, acquire corresponding video resources for various categories of video resources to be identified, as well as for each custom target event in each category of video resources, to generate sample video resources. For each sample video resource, the device can label the corresponding video category and the time range of the target time in the sample video resource, and then generate a sample video resource set based on the constructed sample video resources.

[0104] For example, see Figure 2d As shown, this is a schematic diagram of the content format included in the training samples generated in this application embodiment. Taking sports videos as an example, sports videos may specifically include video resources of football, basketball, badminton, etc. When generating sample video resources, while marking the video resource category corresponding to the video resource, the target event in the video resource, i.e., the time range in which the exciting segment occurs, is also marked. Specifically, it takes the form of... Figure 2d The content shown is for a video resource in the current football category, and the time range marked is: 15S-26S, 1min30S-1min56S, and 2min27S-2min59S. That is to say, a target event occurs within these three time ranges.

[0105] It should be noted that the content recognition model proposed in this application is intended to identify the time range in which target events occur in video resources. Therefore, it may be necessary to pre-generate corresponding sample video resources for all categories of video resources that are to be identified and the defined target events. The target events can be custom events or events that have a positive impact on the outcome expressed by the video resources.

[0106] For example, in the field of sports videos, video resources for different sports are classified into different categories. Events that have a positive impact on the outcome of sports competitions can be used as target events. For example, in basketball sports videos, events such as slam dunks and alley-oops can be defined as target events.

[0107] For example, in the case of game videos, different types of games will produce different types of game videos. Events that have a positive impact on the final result of the game can be used as target events. For example, in tower defense games, events such as "triple kill", "quadruple kill" and "quintuple kill" can be defined as target events.

[0108] Furthermore, after obtaining the sample video resource set, the processing device uses the sample video resource set to iteratively train the initial content recognition model until the preset model convergence condition is met. The following description will only take the operation performed in one round of iterative training as an example to illustrate the training process of the initial content recognition model in this application:

[0109] It should be noted that, in this embodiment of the application, in order to ensure the recognition effect of the content recognition model, the processing device can selectively perform periodic training on the initial content recognition model, and during the training process in each period, with the help of newly added training samples, update the target events that the trained target content recognition model can recognize, and can add recognition functions for video resources of other video categories according to actual recognition needs.

[0110] In addition, considering that the constructed initial content recognition model may have two forms, different forms of initial content recognition models will be processed differently when processing based on the auxiliary task network. Therefore, the training methods of the two forms of initial content recognition models will be explained separately below when using the auxiliary task network for sample video resource recognition.

[0111] See Figure 2e As shown, it is a flowchart illustrating the training process in an embodiment of this application. The following is a description of the process in conjunction with the attached diagram. Figure 2e The operations performed during the first round of training of the initial content recognition model are explained.

[0112] Step 2021: The processing device determines the predicted time range of the target event based on the image features of the sample video resources through the main task network.

[0113] Specifically, during step 2021, the processing device performs the following operations. See also... Figure 2f As shown, this is a flowchart illustrating the process of determining the predicted time range of a target event occurring in a sample video resource in an embodiment of this application. The following is a detailed explanation in conjunction with the attached diagram. Figure 2f The specific operations to be performed will be explained in detail:

[0114] Step 2021-1: The processing device acquires a sample video resource and divides the sample video resource into at least one sample video resource segment.

[0115] Specifically, after acquiring a sample video resource, the processing device divides the sample video resource into at least one sample video resource segment, which is the same as the total number of parallel input channels of the pre-set initial content recognition model. Each sample video resource segment corresponds to a time range. When dividing the sample video resource segment, the device can selectively divide it into at least one sample video resource segment with the same duration according to the total duration of the sample video resource. Alternatively, it can divide a sample video resource into at least one video resource segment with different durations according to the actual processing needs.

[0116] For example, assuming the processing device obtains a sample video resource segment of 2 minutes and 20 seconds, and the initial content recognition model indicates that the number of input video segments is M, specifically 7, then the duration of the video resource segment can be set to 20 seconds, resulting in 7 sample video resource segments with corresponding time ranges of: 0-20S, 20-30S, 30-40S, 40S-1min, 1min-1min20S, 1min20S-1min40S, 1min40S-2min, and 2min-2min20S.

[0117] Step 2021-2: The processing device extracts a sample keyframe from at least one sample video resource segment through the main task network, and aggregates the context features of each pixel contained in each sample keyframe to obtain the pixel feature matrix of each sample keyframe.

[0118] Specifically, the processing device can use the keyframe extraction network in the main task network to extract a corresponding sample keyframe from at least one sample video resource segment.

[0119] For example, see Figure 2b or Figure 2c As shown in the illustration, the processing device splits a sample video resource into m sample video segments, resulting in m sample video resource segments as shown in V1-Vm. Then, the m sample video resource segments are input into the initial content recognition model. With the help of the keyframe extraction network, a sample keyframe sequence V={V_1, V_2, ..., V_n} is obtained, which consists of sample keyframes.

[0120] Furthermore, by leveraging a pixel feature aggregation network, the contextual features of each pixel contained in each sample keyframe are aggregated to obtain the pixel feature matrix of each sample keyframe.

[0121] Specifically, the pixel feature aggregation network in the main task network is used to perform the following operations for each pixel in each sample keyframe: Determine all other pixels within a specified range centered on a given pixel, and aggregate the context features of these other pixels to obtain the pixel features of the given pixel. Then, based on the processed pixel features of each pixel in each sample keyframe, a pixel feature matrix corresponding to each sample keyframe is generated. The specified range used to define the pixels surrounding a given pixel is set according to actual processing needs; for example, pixels directly adjacent to a given pixel in each direction can be considered as other pixels within a specified range centered on that pixel.

[0122] For example, in specific processing, a CNN network can be used to obtain the high-level feature information sequence VF={VF_1, VF_2, ..., VF_n} for each extracted sample keyframe. The obtained high-level feature information refers to the pixel weighted feature obtained by aggregating other pixels around the pixel in the sample keyframe.

[0123] In this way, by weighted aggregation of the context features of other pixels around a pixel, the mutual influence between pixels in the sample keyframe can be extracted, pixel feature extraction can be performed at the video frame level, and finally the pixel features of each sample keyframe can be presented in the form of a pixel feature matrix.

[0124] Step 2021-3: The processing device aggregates the contextual influence between keyframes of each sample in each pixel feature matrix through the main task network, and obtains the image feature matrix representing the image features of each keyframe of each sample.

[0125] Specifically, the processing device inputs the pixel feature matrix corresponding to each sample keyframe into the image feature aggregation network in the main task network to mine the intrinsic relationship between the sample keyframes with temporal relationship, and through the image feature aggregation network in the main task network, aggregates the contextual influence between the sample keyframes with temporal relationship to obtain the image feature matrix representing the image features of each sample keyframe.

[0126] For example, continue to refer to Figure 2b or Figure 2c As shown, the pixel feature sequence VF = {VF_1, VF_2, ..., VF_m} is fed into the image feature aggregation network in the main task network. The image feature aggregation network aggregates the contextual influence between sample image frames to obtain the vector sequence VF_V = {VF_V_1, VF_V_2, ..., VF_V_m}.

[0127] Step 2021-4: The processing device determines the predicted time range of the target occurrence time based on each image feature matrix.

[0128] In this embodiment of the application, when performing step 2021-4, the processing device first uses the main task network to predict the probability prediction result of the target event occurring within the time range corresponding to each sample key frame based on the image feature matrix of each sample key frame.

[0129] Specifically, the processing device passes the image feature matrix through a fully connected layer in the main task network and then into a softmax layer to predict the probability of the target event occurring in the corresponding sample video resource segment for each sample keyframe.

[0130] See Figure 2g As shown, it is a structural diagram of the fully connected layer and the unified layer in an embodiment of this application. Figure 2b or Figure 2c The technical content described describes how, during the process of passing the vector sequence VF_V, composed of concatenated image feature matrices, through a fully connected layer into a normalized layer, the fully connected layer performs a nonlinear transformation on the input vector. The formula for the linear transformation process is as follows:

[0131] Y = f(Wx + b);

[0132] Where f represents the activation function, W is the weight matrix set for the corresponding input features in the fully connected layer, and b represents the set bias constant. The activation function can be a conventional activation function, such as the sigmoid function, the hyperbolic tangent (Tanh) function, and the rectified linear unit (ReLU) function.

[0133] Furthermore, the output of the fully connected layer is fed into the normalization layer to obtain the probability prediction results of the target event occurring within the time range corresponding to each sample keyframe, transformed by the normalization layer based on the output of the fully connected layer. The calculation formula for the normalization layer is as follows:

[0134]

[0135] Where j takes the value 1-m, and m represents the total number of input paths to the initial content recognition model. The input result of the fully connected layer corresponding to the sample video resource segment of the j-th path is represented. The probability prediction result of the target event occurring in the sample video resource segment of the j-th path is characterized by the prediction result.

[0136] Furthermore, the processing device filters out the target probability prediction results that are higher than the set threshold from among the various probability prediction results, and uses the time range corresponding to the target probability prediction results as the prediction time range.

[0137] Specifically, in this embodiment of the application, the processing device divides the sample video resources into at least one sample video resource segment corresponding to a time range, and after obtaining the probability prediction results output by the main task network of the initial content recognition model for each sample video resource segment, it compares each obtained probability prediction result with a set threshold, filters out the target probability prediction results that are higher than the set threshold, and uses the time range corresponding to the filtered target probability prediction results as the prediction time range.

[0138] For example, assuming a threshold of 0.9 is set, and the three sample video resource segments obtained from splitting a sample video resource are input into the initial content recognition model, the probability prediction results for the corresponding three sample video resource segments are as follows: Sample video resource segment 1: 0.23, Sample video resource segment 2: 0.95, Sample video resource segment 3: 0.92. Then, the time ranges corresponding to Sample video resource segments 2 and 3 can be used as the prediction time range.

[0139] In this way, considering that multiple target events may occur in a video resource, by setting a threshold, video resource segments can be filtered and judged, and video resource segments that may contain target events can be selected, and the corresponding prediction time range can be determined.

[0140] Step 2022: The processing device determines the first predicted video category corresponding to the sample video resource based on the image features of the sample video resource through the first auxiliary task network.

[0141] In the implementation scenario corresponding to Form 1 of step 201 in this application, the processing device obtains the image feature matrix corresponding to at least one sample keyframe for characterizing the image features of the sample video resource through the pixel feature aggregation network and image feature aggregation network shared with the main task network in the first auxiliary task network. After obtaining the weighted sum of the obtained image feature matrices, the processing device passes the result of the weighted sum of the obtained image feature matrices into the normalization layer through the fully connected layer in the first auxiliary task network to obtain the probability prediction result for each video category. Then, the video category with the largest probability prediction result is taken as the first predicted video category for the corresponding video resource prediction.

[0142] Step 2023: The processing device adjusts at least the network parameters in the first sub-network based on the predicted time range and the first predicted video category.

[0143] The processing device obtains the prediction time range for a sample video resource corresponding to the main task network in the initial content recognition model, and the first predicted video category for a sample video resource corresponding to the first auxiliary task network in the initial content recognition model, and then adjusts the parameters in the initial content recognition model.

[0144] Specifically, the processing device uses a preset first loss function to determine the first loss value based on the prediction difference between the prediction time range and the corresponding sample time range, and adjusts the network parameters of the main task network, including the first sub-network, based on the first loss value. The first loss function can be an objective function adapted to multi-label classification (BCEWithLogitsLoss).

[0145] At the same time, the processing device uses a preset second loss function to determine the second loss value based on the prediction difference between the first predicted video category and the corresponding sample video category, and adjusts the network parameters of the first auxiliary task network, including the first sub-network, based on the second loss value. The second loss function used can be the cross-entropy loss function.

[0146] In this way, when the second loss value is determined based on the image features of the inherent relationship between the sample keyframes, and the network parameters in the first auxiliary task network of the initial content recognition model are adjusted, it is equivalent to providing complementary information for the learning process of the main task network in the initial content recognition model. This allows the initial content recognition model to take into account the video category corresponding to the sample video resource during the training process, so that the trained target content recognition model can effectively recognize video resources of different video categories.

[0147] It should be noted that, in the embodiments of this application, when using Figure 2c When processing the model structure of Form 2 as shown, the initial content recognition model also includes a second auxiliary task network that shares a second sub-network with the main task network.

[0148] Therefore, during the iterative training of the initial content recognition model, the following operations are also included: through the main task network, the contextual features of pixels in the sample video resources are aggregated to obtain the pixel features of the sample video resources, wherein the second sub-network is contained in the first sub-network; through the second auxiliary task network, based on the pixel features, the second predicted video category corresponding to the sample video resources is determined; based on the prediction time range and the second predicted video category, at least the network parameters in the second sub-network are adjusted.

[0149] Specifically, in the initial content recognition model, when a second auxiliary task network is further included, it needs to be trained simultaneously. This second auxiliary task network predicts the second predicted video category corresponding to the video resource based on the pixel features of the sample keyframes. Then, a third loss function is used to determine the third loss value based on the second predicted video category and the corresponding sample video category. Based on this third loss value, the second auxiliary task network, including the second sub-network, is adjusted.

[0150] Thus, the initial content recognition model simultaneously contains a second auxiliary task network that predicts video categories based on pixel features of sample video resources, and a first auxiliary task network that predicts video categories based on image features of sample video resources. The multi-granularity features can better assist the initial content recognition model in classifying video categories, and further better assist in the recognition of sample keyframes. After recognizing the sample keyframes, it is equivalent to recognizing the time range to which the sample keyframes belong, and then to recognizing the video resource segment corresponding to the time range. Ultimately, this enables the model to better achieve targeted recognition of different categories of video resources.

[0151] Furthermore, the processing device uses sample video resources obtained from the sample video resource set to repeatedly execute the operation of step 202 until the preset model convergence condition is met.

[0152] Specifically, under the model structure shown in Form 1 of step 201, the preset model convergence condition is: the number of times the sum of the first loss value and the second loss value is lower than the first set value reaches the first threshold value; under the model structure shown in Form 2 of step 201, the preset model convergence condition is: the number of times the sum of the first loss value, the second loss value, and the third loss value is lower than the second set value reaches the second threshold value. The values ​​of the first set value, the second set value, the first threshold value, and the second threshold value are set according to the actual processing needs, and this application does not impose specific limitations.

[0153] Step 203: The processing device establishes a target content recognition model based on the main task network output from the last round of iteration training.

[0154] In this embodiment, the processing device establishes a target content recognition model based on the main task network output from the last round of iteration training.

[0155] Specifically, when using the model structure illustrated in Form 1 of step 201, the initial content recognition model includes a main task network and a first auxiliary task network. When the processing device determines that the preset model convergence condition is met, it can delete the network structure of the first auxiliary task network except for the first sub-network in the initial content recognition model output in the last iteration training, and establish the target content recognition model based on the retained main task network.

[0156] When adopting the model structure shown in Form 2 in step 201, the initial content recognition model includes a main task network, a first auxiliary task network, and a second auxiliary task network. When the processing device determines that the preset model convergence condition is met, it can delete the network structure in the first auxiliary network except for the first sub-network and the network structure in the second auxiliary network except for the second sub-network in the initial content recognition model output in the last iteration training. Based on the retained main task network, a target content recognition model is established.

[0157] Alternatively, in any possible implementation of this application, whether it is the initial content recognition model shown in Form 1 of step 201 or the initial content recognition model shown in Form 2 of step 201, after determining that the initial content recognition model meets the preset model convergence conditions, it is also possible to control the output to only the prediction results for the time range of the target event without reducing the network structure of the model.

[0158] In this way, after determining that the initial content recognition model meets the preset model convergence conditions, the target content recognition model is established by deleting the network structures in the auxiliary task network of the initial content recognition model, except for the main task network. In other words, only the main task network in the converged initial content recognition model is retained. This is equivalent to pruning the initial content recognition model, removing some of the network structures used to train the main task network, which reduces the amount of computation in the content recognition process to a certain extent and helps to improve the processing efficiency of the target content recognition model.

[0159] See Figure 3 As shown, it is a schematic diagram of the content recognition process in an embodiment of this application. The following is in conjunction with the attached diagram. Figure 3 The process of using a trained target content recognition model to achieve content recognition in the embodiments of this application will be described.

[0160] Step 301: The processing device acquires the video resources to be identified.

[0161] In this embodiment of the application, the processing device can obtain various types of video resources to be processed according to actual processing needs. After obtaining the video resources, it can selectively set a processing duration threshold for the duration of the video resources to avoid inputting excessively long video resource segments into the target content recognition model, causing information omissions, making it impossible to accurately identify whether a target event has occurred in the video resources, or to accurately identify the occurrence time of the target event in the video resources.

[0162] Specifically, when the processing device determines that the duration of the video resource to be identified exceeds a preset processing duration threshold, it uses a preset frame extraction algorithm to perform frame extraction processing on the video resource to be identified, so that the duration of the processed video resource does not exceed the processing duration threshold. Frame extraction processing of video resources is a mature technology in this field, and will not be described in detail here.

[0163] It should be noted that when performing frame extraction, in order to avoid affecting the content of the video resource to be identified, a method that can achieve dense frame extraction can be selected, such as extracting a few frames per second, so that the duration of the video resource obtained by frame extraction does not exceed the processing time threshold.

[0164] Optionally, when the processing device determines that the duration of the video resource to be identified exceeds a preset processing duration threshold, it can split the video resource to be identified according to the preset processing duration threshold, and then perform content recognition operation based on the split video resource.

[0165] In this way, by setting a processing time threshold, the video resources to be identified can be processed, avoiding the reduction in processing accuracy caused by the excessive duration of the video resources.

[0166] Step 302: The processing device inputs the video resources into the trained target content recognition model to obtain the time range information of the preset target events occurring in the video resources.

[0167] Specifically, the processing device inputs the video resource to be identified into a trained target content recognition model, and obtains the output of the target content recognition model: the time range information of the preset target events occurring in the video resource to be identified. The target content recognition model employs the aforementioned... Figure 2a The training process shown is used to obtain the results.

[0168] In this way, by leveraging the auxiliary task network during training, video category information of the video resources themselves can be incorporated into the trained target content recognition model. This allows the target content recognition model to specifically focus on the occurrence of target events in video resources of different video categories when performing content recognition. Consequently, the target content recognition model can more accurately identify target events and effectively identify the time periods in which exciting events occur. At the same time, when using the same model to recognize data from different video categories, the data can be processed uniformly according to a unified input information processing method, which improves the processing speed of the video resources to be identified, thereby reducing the difficulty of content recognition of video resources and improving the recognition efficiency of video resources.

[0169] Based on the same inventive concept, see [reference] Figure 4 As shown, this is a schematic diagram of the logical structure of the training device for the content recognition model in this embodiment of the application. The training device 400 for the content recognition model includes an acquisition unit 401, a training unit 402, and a building unit 403.

[0170] The acquisition unit 401 is used to acquire the initial content recognition model to be trained, wherein the initial content recognition model includes a main task network and a first auxiliary task network sharing a first sub-network;

[0171] Training unit 402 is used to perform at least one round of iterative training on the initial content recognition model based on the sample video resource set, wherein, during one round of iteration, the following operations are performed:

[0172] The main task network determines the predicted time range of the target event based on the image features of the sample video resources, and the first auxiliary task network determines the first predicted video category corresponding to the sample video resources based on the image features of the sample video resources, and adjusts the network parameters in the first sub-network based on the predicted time range and the first predicted video category.

[0173] Unit 403 is established to build a target content recognition model based on the main task network output from the last round of iteration training.

[0174] Optionally, if the initial content recognition model also includes a second auxiliary task network that shares a second sub-network with the main task network, then during one round of iterative training, training unit 402 is also used for:

[0175] The main task network aggregates the contextual features of pixels in the sample video resources to obtain the pixel features of the sample video resources. The second sub-network is contained within the first sub-network.

[0176] The second auxiliary task network is used to determine the second predicted video category corresponding to the sample video resource based on pixel features.

[0177] Based on the predicted time range and the second predicted video category, at least the network parameters in the second sub-network are adjusted.

[0178] Optionally, when determining the predicted time range of the target event based on image features of sample video resources through the main task network, training unit 402 is used for:

[0179] Acquire a sample video resource and divide the sample video resource into at least one sample video resource segment, wherein one sample video resource segment corresponds to a time range;

[0180] Through the main task network, a sample keyframe is extracted from at least one sample video resource segment, and the context features of each pixel contained in each sample keyframe are aggregated to obtain the pixel feature matrix of each sample keyframe.

[0181] Through the main task network, the contextual influence between key frames of each sample is aggregated in each pixel feature matrix to obtain the image feature matrix representing the image features of each key frame of each sample.

[0182] Based on the feature matrices of each image, the predicted time range for the occurrence of the target event is determined.

[0183] Optionally, when aggregating the contextual features of each pixel in each keyframe of a sample to obtain the pixel feature matrix of each keyframe, the training unit 402 is used for:

[0184] For each pixel in each sample keyframe, perform the following operations:

[0185] Identify all other pixels within a specified range centered on a single pixel, and aggregate the context features of each other pixel to obtain the pixel feature of the single pixel.

[0186] Based on the pixel features of each pixel in each processed keyframe, a pixel feature matrix corresponding to each keyframe is generated.

[0187] Optionally, when determining the predicted time range for the occurrence of the target event, training unit 402 is used to:

[0188] Using a main task network, based on the image feature matrix of each sample keyframe, the probability prediction result of the target event occurring within the time range corresponding to each sample keyframe is predicted.

[0189] The target probability prediction results that are higher than the set threshold are selected from the various probability prediction results, and the time range corresponding to the target probability prediction results is used as the prediction time range.

[0190] Optionally, when adjusting the network parameters in at least the first sub-network based on the predicted time range and the first predicted video category, training unit 402 is used to:

[0191] A first loss function is used to determine the first loss value based on the prediction difference between the prediction time range and the corresponding sample time range, and the network parameters of the main task network, including the first sub-network, are adjusted based on the first loss value.

[0192] A second loss function is used to determine a second loss value based on the prediction difference between the first predicted video category and the corresponding sample video category. The network parameters of the first auxiliary task network, including the first sub-network, are then adjusted based on the second loss value.

[0193] Optionally, when building the target content recognition model based on the main task network output from the last iteration of training, unit 403 is used for:

[0194] When the preset model convergence condition is met, in the initial content recognition model output by the last iteration training, the network structure except the first sub-network in the first auxiliary task network is deleted, and the target content recognition model is established based on the retained main task network.

[0195] Based on the same inventive concept, see [reference] Figure 5 As shown, this is a schematic diagram of the logical structure of the content recognition device in an embodiment of this application. The content recognition device 500 includes an acquisition unit 501 and an input unit 502, wherein...

[0196] Acquisition unit 501 is used to acquire the video resource to be identified;

[0197] The input unit 502 is used to input video resources into a trained target content recognition model to obtain time range information of a preset target event occurring in the video resources. The target content recognition model is obtained using the training method of the content recognition model described above.

[0198] Optionally, after acquiring the video resource to be identified, and before inputting the video resource into the trained target content recognition model, the acquisition unit 501 is further used for:

[0199] When it is determined that the duration of a video resource exceeds a preset processing duration threshold, a preset frame-sampling algorithm is used to perform frame-sampling processing on the video resource so that the duration of the processed video resource does not exceed the processing duration threshold.

[0200] Having introduced the training of the content recognition model, the content recognition method, and the apparatus according to exemplary embodiments of this application, we will now introduce an electronic device according to another exemplary embodiment of this application.

[0201] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0202] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device, see reference. Figure 6 As shown, this is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. The electronic device 600 may include at least a processor 601 and a memory 602. The memory 602 stores program code, which, when executed by the processor 601, causes the processor 601 to perform the training and content recognition steps of any of the above-mentioned content recognition models.

[0203] In some possible implementations, the computing device according to this application may include at least one processor and at least one memory. The memory stores program code that, when executed by the processor, causes the processor to perform the steps of training and content recognition of the content recognition model according to the various exemplary embodiments of this application described above. For example, the processor may perform actions such as... Figure 2a , Figure 3 The steps are shown in the figure.

[0204] The following reference Figure 7 To describe a computing device 700 according to this embodiment of the present application. For example... Figure 7 As shown, the computing device 700 is presented in the form of a general-purpose computing device. The components of the computing device 700 may include, but are not limited to: at least one processing unit 701, at least one storage unit 702, and a bus 703 connecting different system components (including storage unit 702 and processing unit 701).

[0205] Bus 703 represents one or more of several bus architectures, including a memory bus or memory controller, peripheral bus, processor, or a local bus using any of the various bus architectures.

[0206] Storage unit 702 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 7021 and / or cache storage unit 7022, and may further include read-only memory (ROM) 7023.

[0207] Storage unit 702 may also include a program / utility 7025 having a set (at least one) program module 7024, such program module 7024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0208] The computing device 700 can also communicate with one or more external devices 704 (e.g., keyboard, pointing device, etc.), one or more devices that enable objects to interact with the computing device 700, and / or any device that enables the computing device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 705. Furthermore, the computing device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 706. As shown, network adapter 706 communicates with other modules used in the computing device 700 via bus 703. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the computing device 700, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0209] Based on the same inventive concept as the above-described method embodiments, the training and various aspects of the content recognition model provided in this application can also be implemented as a program product, which includes program code. When the program product runs on an electronic device, the program code causes the electronic device to perform the steps in the business resource sharing method according to the various exemplary embodiments of this application described above. For example, the electronic device can perform actions such as... Figure 2a , Figure 3 The steps are shown in the figure.

[0210] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0211] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, this application is also intended to include the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0212] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations of this application fall within the scope of the equivalent technology of this application, this application also intends to include these modifications and variations.

Claims

1. A method for training a content recognition model, characterized in that, include: An initial content recognition model to be trained is obtained, wherein the initial content recognition model includes a main task network and a first auxiliary task network sharing a first sub-network; the main task network is constructed by a keyframe extraction network, a pixel feature aggregation network, an image feature aggregation network, and a fully connected layer and a normalization layer of the main task; the first auxiliary task network is constructed by the keyframe extraction network, the pixel feature aggregation network, the image feature aggregation network, and a fully connected layer and a normalization layer of the first auxiliary task. The initial content recognition model is trained iteratively for at least one round based on the sample video resource set. During one round of iterative training, the following operations are performed: The main task network determines the predicted time range of the target event based on the image features of the sample video resources, and the first auxiliary task network determines the first predicted video category corresponding to the sample video resources based on the image features of the sample video resources, and adjusts at least the network parameters in the first sub-network based on the predicted time range and the first predicted video category. Based on the main task network output from the last round of iteration training, a target content recognition model is established.

2. The method as described in claim 1, characterized in that, If the initial content recognition model also includes a second auxiliary task network that shares a second sub-network with the main task network, then the process of one round of iterative training also includes: The main task network aggregates the contextual features of pixels in the sample video resources to obtain the pixel features of the sample video resources, wherein the second sub-network is included in the first sub-network; The second auxiliary task network determines the second predicted video category corresponding to the sample video resource based on the pixel features. Based on the predicted time range and the second predicted video category, at least the network parameters in the second sub-network are adjusted.

3. The method as described in claim 1, characterized in that, The step of determining the predicted time range of the target event based on image features of sample video resources through the main task network includes: A sample video resource is acquired and the sample video resource is divided into at least one sample video resource segment, wherein one sample video resource segment corresponds to one time range; Through the main task network, a sample keyframe is extracted from each of the at least one sample video resource segments, and the context features of each pixel contained in each sample keyframe are aggregated to obtain the pixel feature matrix of each sample keyframe. Through the main task network, the contextual influence between the key frames of each sample is aggregated in each pixel feature matrix to obtain the image feature matrix representing the image features of each key frame of each sample. Based on the feature matrices of each image, the predicted time range for the occurrence of the target event is determined.

4. The method as described in claim 3, characterized in that, The step of aggregating the contextual features of each pixel in each sample keyframe to obtain the pixel feature matrix of each sample keyframe includes: For each pixel in each sample keyframe, perform the following operations: Identify all other pixels within a specified range centered on a single pixel, and aggregate the context features of these other pixels to obtain the pixel features of the single pixel. Based on the pixel features of each pixel in each processed keyframe, a pixel feature matrix corresponding to each keyframe is generated.

5. The method as described in claim 3, characterized in that, Determining the predicted time range for the occurrence of the target event includes: Using the main task network, based on the image feature matrix of each sample keyframe, the probability prediction result of the target event occurring within the time range corresponding to each sample keyframe is predicted respectively. The target probability prediction results that are higher than a set threshold are selected from the various probability prediction results, and the time range corresponding to the target probability prediction results is used as the prediction time range.

6. The method according to any one of claims 1-5, characterized in that, The step of adjusting at least the network parameters in the first sub-network based on the predicted time range and the first predicted video category includes: A first loss function is used to determine a first loss value based on the prediction difference between the prediction time range and the corresponding sample time range, and the network parameters of the main task network, including the first sub-network, are adjusted based on the first loss value. A second loss function is used to determine a second loss value based on the prediction difference between the first predicted video category and the corresponding sample video category. The network parameters of the first auxiliary task network, which includes the first sub-network, are then adjusted based on the second loss value.

7. The method as described in claim 1, characterized in that, The target content recognition model is established based on the main task network output from the last round of iteration training, including: When the preset model convergence condition is met, in the initial content recognition model output by the last iteration training, the network structure in the first auxiliary task network except for the first sub-network is deleted, and the target content recognition model is established based on the retained main task network.

8. A content recognition method, characterized in that, include: Obtain the video resource to be identified; The video resource is input into a trained target content recognition model to obtain time range information of a preset target event occurring in the video resource, wherein the target content recognition model is obtained by the training method of the content recognition model according to any one of claims 1-7.

9. The method as described in claim 8, characterized in that, After acquiring the video resource to be identified, and before inputting the video resource into the trained target content recognition model, the method further includes: When it is determined that the duration of the video resource exceeds a preset processing duration threshold, a preset frame-sampling algorithm is used to perform frame-sampling processing on the video resource so that the duration of the processed video resource does not exceed the processing duration threshold.

10. A training device for a content recognition model, characterized in that, include: An acquisition unit is used to acquire an initial content recognition model to be trained, wherein the initial content recognition model includes a main task network and a first auxiliary task network sharing a first sub-network; the main task network is constructed by a keyframe extraction network, a pixel feature aggregation network, an image feature aggregation network, and a fully connected layer and a normalization layer of the main task; the first auxiliary task network is constructed by the keyframe extraction network, the pixel feature aggregation network, the image feature aggregation network, and a fully connected layer and a normalization layer of the first auxiliary task. The training unit is used to perform at least one round of iterative training on the initial content recognition model based on the sample video resource set, wherein, during one round of iterative training, the following operations are performed: The main task network determines the predicted time range of the target event based on the image features of the sample video resources, and the first auxiliary task network determines the first predicted video category corresponding to the sample video resources based on the image features of the sample video resources, and adjusts at least the network parameters in the first sub-network based on the predicted time range and the first predicted video category. A unit is established to build a target content recognition model based on the main task network output from the last round of iteration training.

11. The apparatus as claimed in claim 10, characterized in that, If the initial content recognition model also includes a second auxiliary task network that shares a second sub-network with the main task network, then during one round of iterative training, the training unit is further used for: The main task network aggregates the contextual features of pixels in the sample video resources to obtain the pixel features of the sample video resources, wherein the second sub-network is included in the first sub-network; The second auxiliary task network determines the second predicted video category corresponding to the sample video resource based on the pixel features. Based on the predicted time range and the second predicted video category, at least the network parameters in the second sub-network are adjusted.

12. A content recognition device, characterized in that, include: The acquisition unit is used to acquire the video resources to be identified; An input unit is configured to input the video resource into a trained target content recognition model to obtain time range information of a preset target event occurring in the video resource, wherein the target content recognition model is obtained by the training method of the content recognition model according to any one of claims 1-7.

13. The apparatus as claimed in claim 12, characterized in that, After acquiring the video resource to be identified, and before inputting the video resource into the trained target content recognition model, the method further includes: When it is determined that the duration of the video resource exceeds a preset processing duration threshold, a preset frame-sampling algorithm is used to perform frame-sampling processing on the video resource so that the duration of the processed video resource does not exceed the processing duration threshold.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the training method of the content recognition model as described in any one of claims 1-7 or the content recognition method as described in any one of claims 8-9.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the training method of the content recognition model as described in any one of claims 1-7 or the content recognition method as described in any one of claims 8-9.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of the content recognition model as described in any one of claims 1-7 or the content recognition method as described in any one of claims 8-9.

Images