Content understanding model training, content understanding method, apparatus, medium, and device

Abstract

The present disclosure relates to a content understanding model training, a content understanding method, an apparatus, a medium and an equipment. The method comprises: in response to restarting training after the content understanding model is interrupted, obtaining the model training state information and the reading position of training data saved last time in the model training stage, wherein the content understanding model periodically saves the current model training state information and the current reading position of training data in the model training stage; loading the training data from the reading position saved last time, and continuing to train the content understanding model based on the loaded training data and the model training state information saved last time. The training breakpoint can be restored to continue the training state before the model is interrupted, so that the model can use unstable resources such as tides or preemptible resources for training, and the overall iteration efficiency and output efficiency of the model are accelerated.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a content understanding model training, content understanding method, apparatus, medium, and device. Background Technology

[0002] With the continuous increase in the number of posts by internet users and the ongoing development of content recommendation systems, online content understanding and recognition has always been a crucial task for recommendation systems. However, as business demands increase, content scenarios are becoming increasingly complex. This requires models to cover multiple scenarios, including visual, audio, natural language, and multimodal learning. Simultaneously, user and content features are constantly evolving, leading to an increasingly frequent frequency of model iterations. Therefore, to improve model iteration and output efficiency, and to ensure that model capabilities and data processing results can be horizontally reused across different business scenarios, a model training framework suitable for multi-scenario content understanding is needed.

[0003] Large-scale models are prone to training interruptions due to unstable data reading and resource issues, which may require retraining the model, resulting in low efficiency in model iteration and production. Summary of the Invention

[0004] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] Firstly, this disclosure provides a method for training a content understanding model, the method comprising:

[0006] In response to the restart of training after the content understanding model has been interrupted, the model training state information and the reading position of the training data are obtained from the most recently saved model training phase. The content understanding model periodically saves the current model training state information and the current reading position of the training data during the model training phase.

[0007] The training data is loaded from the most recently saved read position, and the content understanding model is trained again based on the most recently saved model training state information, using the loaded training data as a reference.

[0008] Secondly, this disclosure provides a content understanding method, including:

[0009] Obtain media asset data for content recognition;

[0010] The media asset data is input into the content understanding model, and the content understanding model is used to perform content recognition on the media asset data to obtain the target content understanding result of the media asset data. The content understanding model is trained by the content understanding model training method provided in the first aspect of this disclosure.

[0011] Thirdly, this disclosure provides a content understanding model training apparatus, the apparatus comprising:

[0012] The first acquisition module is used to acquire the model training state information and the reading position of the training data most recently saved during the model training phase in response to the restart of training after the content understanding model has been interrupted. The content understanding model periodically saves the current model training state information and the current reading position of the training data during the model training phase.

[0013] The training module is used to load training data from the most recently saved read position and, based on the loaded training data, continue to train the content understanding model using the most recently saved model training state information as a reference.

[0014] Fourthly, this disclosure provides a content understanding apparatus, comprising:

[0015] The second acquisition module is used to acquire media asset data to be identified.

[0016] The content recognition module is used to input the media asset data into the content understanding model, and to perform content recognition on the media asset data through the content understanding model to obtain the target content understanding result of the media asset data. The content understanding model is trained by the content understanding model training method provided in the first aspect of this disclosure.

[0017] Fifthly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the content understanding model training method provided in the first aspect of this disclosure or the steps of the content understanding method provided in the second aspect of this disclosure.

[0018] Sixthly, this disclosure provides an electronic device, comprising:

[0019] A storage device on which computer programs are stored;

[0020] A processing device is configured to execute the computer program in the storage device to implement the steps of the content understanding model training method provided in the first aspect of this disclosure or the steps of the content understanding method provided in the second aspect of this disclosure.

[0021] In the above technical solution, the content understanding model periodically saves the current model training state information and the current reading position of the training data during the model training phase. In this way, when the content understanding model restarts training after an interruption, it can load the training data from the reading position of the most recently saved training data during the training phase, and continue training the content understanding model based on the most recently saved model training state information, thereby realizing training breakpoint recovery and continuing the training state before the model training was interrupted. This allows the model to utilize unstable resources such as tidal (i.e., resources are scarce in one period and sufficient in other periods) or preemptible resources for training, thus accelerating the overall model iteration efficiency and output efficiency.

[0022] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0023] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0024] Figure 1 This is a flowchart illustrating a content understanding model training method according to an exemplary embodiment.

[0025] Figure 2 This is a schematic diagram illustrating a training breakpoint recovery according to an exemplary embodiment.

[0026] Figure 3 This is a schematic diagram illustrating a model training process according to an exemplary embodiment.

[0027] Figure 4 This is a schematic diagram illustrating a model testing process according to an exemplary embodiment.

[0028] Figure 5 This is a flowchart illustrating a content understanding method according to an exemplary embodiment.

[0029] Figure 6 This is a block diagram illustrating a content understanding model training apparatus according to an exemplary embodiment.

[0030] Figure 7 This is a block diagram illustrating a content understanding device according to an exemplary embodiment.

[0031] Figure 8 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. Detailed Implementation

[0032] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0033] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0034] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0035] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0036] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0037] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0038] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0039] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0040] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0041] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0042] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0043] Figure 1 This is a flowchart illustrating a content understanding model training method according to an exemplary embodiment. For example... Figure 1 As shown, the training method for this content understanding model may include the following S101 and S102.

[0044] In S101, in response to the restart of training after the content understanding model has been interrupted, the model training state information and the location of the training data most recently saved during the model training phase are obtained.

[0045] In this disclosure, due to the scarcity of training resources, the model is trained by using the idle resources of the resource cluster (preemptible training tasks). The resources used for model training are not fixed, so the training tasks that occupy these resources will often be interrupted due to the adjustment of resource usage. Therefore, restarting training requires a mechanism that can continue the previous training state. To this end, this disclosure designs a training breakpoint recovery process.

[0046] In addition, during model training, training data is usually loaded from the Hadoop Distributed File System (HDFS) to the local CPU memory. During the loading of training data, network interruptions may cause unstable reading of training data, thus interrupting model training.

[0047] To enable training breakpoint recovery, the content understanding model periodically saves the current model training state information and the current read position of the training data during the model training phase. The model training state information can include model parameters, optimizer state, current training step number, and current training epoch number. The optimization state includes optimizer weights and gradients. During model training, training data is loaded from the training dataset in batches. The read position of the training data is the position of the data following the last data point in the most recently loaded batch. For example, if the most recently loaded batch of training data contains data points 51 to 100, then the current read position of the training data is the position of the 101st data point.

[0048] For example, during the model training phase, every second preset training step (e.g., 100 training steps) indicates that the model's training progress has reached a preset checkpoint. At this time, the model training state information and the reading position of the training data corresponding to the checkpoint are saved, that is, the current model training state information and the current reading position of the training data are saved.

[0049] For example, during the model training phase, the content understanding model will reach a preset checkpoint every preset time interval (e.g., 3 minutes). At this time, the model training state information and the reading position of the training data corresponding to the checkpoint will be saved, that is, the current model training state information and the current reading position of the training data will be saved.

[0050] In S102, training data is loaded from the most recently saved read position, and the content understanding model is continued to be trained based on the most recently saved model training state information, using the loaded training data as a reference.

[0051] During the model training phase, the content understanding model periodically saves the current model training state information and the current reading position of the training data. In this way, when the content understanding model resumes training after an interruption, it can load the training data from the most recently saved reading position of the training data during the training phase, and continue to train the content understanding model based on the loaded training data and the most recently saved model training state information.

[0052] like Figure 2As shown, the preemptible training task includes two rounds of training, with each round consisting of 300 training steps. During the model training phase, the content understanding model saves the current model training state information and the current reading position of the training data every 100 training steps. In the second round of training, the training resources are preempted. At this time, the model training state information and the reading position of the training data saved most recently during the training phase (i.e., saved between the 301st and 400th training steps) are used as the benchmark for subsequent training.

[0053] In the above technical solution, the content understanding model periodically saves the current model training state information and the current reading position of the training data during the model training phase. In this way, when the content understanding model restarts training after an interruption, it can load the training data from the reading position of the most recently saved training data during the training phase, and continue training the content understanding model based on the most recently saved model training state information, thereby realizing training breakpoint recovery and continuing the training state before the model training was interrupted. This allows the model to utilize unstable resources such as tidal (i.e., resources are scarce in one period and sufficient in other periods) or preemptible resources for training, thus accelerating the overall model iteration efficiency and output efficiency.

[0054] The following is combined Figure 3 Provide a detailed explanation of the model training process. For example... Figure 3 As shown, a data preprocessor (DataPreFetcher) is used to encapsulate the data loader (DataLoader). Without changing the stability of data stream reading, the data loader preloads training data in batches (i.e., batch data) into CPU memory and then loads it from CPU memory into training resource memory (e.g., GPU memory), i.e., asynchronous memory-to-training resource memory transfer. This saves the data reading time from CPU memory to training resource memory during the model training phase, improving model training speed and thus increasing model iteration and output efficiency. The training resource uses the batch data loaded into its memory to train the content understanding model. After the batch data is input into the content understanding model, two types of outputs are obtained: training loss and training performance metrics (e.g., accuracy, recall). To facilitate monitoring of the training process, these two types of outputs can be displayed through a visualization dashboard. The training loss is backpropagated, and the learning rate scheduler and optimizer calculate the magnitude of the model parameters that need to be updated. Finally, the overall model parameters are updated, and the current model training state information and the current reading position of the training data are periodically saved, thus repeating the process.

[0055] Among them, such as Figure 2As shown, when periodically saving the current model training state information and the current read position of the training data, this information is not only stored in CPU memory, but also pushed to HDFS to avoid loss of this information due to local training resource crashes. Thus, when the content understanding model restarts training after an interruption (i.e., continues training), it can retrieve the most recently saved model training state information and the read position of the training data (such as...) from HDFS. Figure 2 and Figure 3 (As shown).

[0056] The following is a detailed explanation of the specific implementation method for continuing training the model based on the most recently saved model training state information according to the loaded training data in S102 above. Specifically, it can be achieved through the following steps (1) and (2):

[0057] Step (1): Divide the loaded training data into N training sub-data that correspond one-to-one with the N first resources currently participating in model training.

[0058] In this disclosure, N≥1, meaning that the first resource currently participating in model training can be one or more. The first resource can be a machine, meaning model training can be performed on one machine or distributed training across multiple machines; the first resource can also be a GPU, meaning model training can be performed on one GPU or distributed training across multiple GPUs on one machine.

[0059] For example, the loaded training data can be divided into N equal parts.

[0060] Step (2): Each first resource continues to train the content understanding model based on its corresponding training sub-data and the most recently saved model training state information.

[0061] Specifically, for each first resource, the most recently saved model training state information can be loaded onto the content understanding model to obtain a baseline model; the first resource can then train the baseline model based on its corresponding training sub-data to obtain new model training state information corresponding to the first resource; and finally, the current model state information of the content understanding model can be determined based on the new model state information corresponding to each first resource.

[0062] When there is only one first resource, the new model state information corresponding to the first resource can be directly used as the current model state information of the content understanding model.

[0063] When there are multiple first resources, each first resource loads its most recently saved model training state information into the content understanding model. This means each first resource copies the baseline model, ensuring that the models on each first resource are completely identical before training begins. Then, each first resource generates a training loss based on its corresponding training sub-data and calculates the optimizer state (including optimizer weights and gradients) based on the training loss. Since the training data for each first resource is different, different first resources correspond to different optimizer states. Next, the optimizer states from all first resources are aggregated, and the average value of the optimizer states for each first resource (including the average optimizer weights and gradients) is calculated. Finally, the model parameters on each first resource are updated based on the average optimizer state. This ensures that the models trained across all first resources are consistent, resulting in new model parameters, and the average optimizer state is used as the new optimizer state. Furthermore, the number of training steps and training epochs is the same for all first resources; therefore, the training steps and training epochs for any given first resource can be used as the new training steps and training epochs, respectively.

[0064] If one or more of the first resources are preempted during model training using multiple first resources, the current parallel training process is terminated, and training can be restarted using the first resources that have not been preempted.

[0065] In the above implementation, the model can be understood by training multiple first resources in parallel to accelerate the model training speed, thereby improving the model iteration and output efficiency.

[0066] In one implementation, the first resource is a GPU. In this case, each of the first resources continues to train the content understanding model based on its corresponding training sub-data and the most recently saved model training state information. The method further includes:

[0067] Each first resource preloads its corresponding training sub-data from CPU memory into its respective memory.

[0068] Each GPU used for training preloads its corresponding training sub-data from CPU memory into its own GPU memory, which saves the data reading time from CPU memory to GPU memory, improves model training speed, and thus improves model iteration and output efficiency.

[0069] To monitor the actual training effect of the model, model validation can be performed periodically. Based on the validation results, it can be determined whether the current model state information of the content understanding model being validated is among the K optimal model training states. If so, the current model training state information of the content understanding model being validated can be added to the K optimal model training states. Specifically, the above method may also include the following three steps:

[0070] Every first preset number of training steps, the content understanding model is validated to obtain the current validation result of the content understanding model;

[0071] Based on the current validation results, determine whether to update the optimal K model training state information of the content understanding model saved during the model validation phase;

[0072] If it is determined that the optimal K model training state information should be updated, then the current model training state information of the currently validated content understanding model will be updated to the optimal K model training state information.

[0073] During the model testing phase, the optimal K training states of the content understanding model can be used for model testing. The current validation results can include validation loss and validation performance metrics.

[0074] In addition, such as Figure 3 As shown, in order to monitor the actual test effect of the model, the content understanding model is validated every first preset training step. The difference between the training process and the regular training process is that the model is only used for forward propagation and the model parameters are not updated. The checkpoint server can determine whether to update the best K model training state information of the content understanding model stored in the model validation stage based on the current validation result. If it is determined to update the best K model training state information, the current model training state information of the content understanding model being validated is updated to the best K model training state information, that is, the best K model training state information of the content understanding model stored in CPU memory and HDFS is updated simultaneously. In the first K rounds of model validation, the number of optimal model training state information of the content understanding model is less than or equal to K. The current model training state information of the content understanding model being validated can be directly used as the optimal K model training state information. In the K+1 rounds of model validation and thereafter, the model training state information corresponding to the worst validation result is replaced with the current model training state information of the content understanding model being validated. The worst validation result is the worst of the historical validation results corresponding to the optimal K model training state information.

[0075] The following provides a detailed description of the specific implementation method for determining whether to update the optimal K model training state information of the content understanding model stored during the model validation phase based on the current validation results. This disclosure can be implemented through various methods. In one implementation, the decision to update the optimal K model training state information of the content understanding model stored during the model validation phase can be determined based on the current validation loss. In another implementation, the decision to update the optimal K model training state information of the content understanding model stored during the model validation phase can be determined based on the current validation loss and the historical validation losses corresponding to the optimal K model training state information. Specifically, if the current validation loss is greater than or equal to the maximum validation loss, then it is determined not to update the optimal K model training state information of the content understanding model saved during the model validation phase. That is, the current model training state information of the content understanding model being validated is not saved. Here, the maximum validation loss is the maximum value among the historical validation losses corresponding to the optimal K model training state information. If the current validation loss is less than the maximum validation loss, then it is determined to update the optimal K model training state information of the content understanding model saved during the model validation phase. In this case, the optimal K model training state information and the model training state information corresponding to the maximum validation loss can be replaced with the current model training state information of the content understanding model being validated.

[0076] In another implementation, the decision to update the optimal K model training states of the content understanding model, stored during the model validation phase, can be based on the current validation metrics. Alternatively, the decision to update the optimal K model training states of the content understanding model, stored during the model validation phase, can be based on the historical validation metrics corresponding to both the current validation metrics and the optimal K model training states.

[0077] For example, if the current validation metric is accuracy or recall, then if the current validation metric is less than or equal to the minimum validation metric, it is determined not to update the best K model training state information of the content understanding model saved during the model validation phase. That is, the current model training state information of the content understanding model being validated is not saved. Here, the minimum validation metric is the minimum value among the historical validation metrics corresponding to each of the best K model training state information. If the current validation metric is greater than the minimum validation metric, it is determined to update the best K model training state information of the content understanding model saved during the model validation phase. In this case, the model training state information corresponding to the minimum validation metric in the best K model training state information can be replaced with the current model training state information of the content understanding model being validated.

[0078] After the model training is completed, the model can be tested. Specifically, the above understanding of the model training method can also include the following two steps:

[0079] The test data is divided into M test sub-data that correspond one-to-one with the M second resources currently participating in the model test, where M≥1;

[0080] Each second resource is tested against the content understanding model based on its corresponding test sub-data.

[0081] In this disclosure, M≥1, meaning that the second resource currently participating in model testing can be one or more. The second resource can be a machine, meaning model testing can be performed on one machine or distributed across multiple machines; the second resource can also be a GPU, meaning model testing can be performed on one GPU or distributed across multiple GPUs on one machine.

[0082] For example, the test data can be divided into M equal parts.

[0083] Specifically, each second resource is tested on the content understanding model based on its corresponding test sub-data, which may include:

[0084] Each second resource generates a test content comprehension result based on its corresponding test sub-data;

[0085] Based on the understanding results of each test content and the test labels in the test data, determine the test performance metrics of the content understanding model.

[0086] The test performance metrics may include at least one of precision, recall, automatic penalty rate, and other custom metrics. Content understanding results refer to the content information obtained through content recognition by the content understanding model. For example, the content understanding result may be a result used to characterize the semantics of the test media assets in the test data.

[0087] For example, such as Figure 4 As shown, there are M second resources participating in the model test, namely second resource A1, second resource A2, ..., second resource A M The test data is divided into M test sub-data, namely test sub-data 1 corresponding to the second resource A1, test sub-data 2 corresponding to the second resource A2, ..., test sub-data 3 corresponding to the second resource A1, ..., test sub-data 4 corresponding to the second resource A1, ..., test sub-data 5 corresponding to the second resource A1, ..., test sub-data 6 corresponding to the second resource A1, ..., test sub-data 7 corresponding to the second resource A1, ..., test sub-data M The corresponding test sub-data M; then, the second resource A1 generates the corresponding test content understanding result based on test sub-data 1, the second resource A2 generates the corresponding test content understanding result based on test sub-data 2, ..., the second resource A M Based on the test sub-data M, generate corresponding test content understanding results; finally, based on the test content understanding results generated by each of the M second resources and the test tags in the test data, determine the test performance indicators of the content understanding model.

[0088] In the above implementation, the model can be understood through multiple parallel test contents of secondary resources, and a distributed backscan link can be realized, thereby quickly verifying the performance of indicators, accelerating the model testing speed, and improving the model iteration and output efficiency.

[0089] When the content understanding model saves the current model training state information and the current reading position of the training data every second preset training step during the model training phase, the above content understanding model training method may also include the following steps:

[0090] After each round of model training is completed, the current model training state information of the content understanding model is saved. The number of training steps in each round of model training is greater than the second preset number of training steps.

[0091] During the model training phase, the model training state information is saved after each round of training. This allows the model to continue training even if training crashes, based on the previous round's training state information. Furthermore, the optimal model training state information saved from each round (e.g., the one with the lowest loss) can be used as the training state information for pre-trained models in other business processes, thereby accelerating the training speed of those models.

[0092] Figure 5 This is a flowchart illustrating a content understanding method according to an exemplary embodiment. For example... Figure 5 As shown, the content understanding method may include the following S201 and S202.

[0093] In S201, the media asset data to be performed for content recognition is obtained.

[0094] In this disclosure, media asset data can be various types of data such as audio data, text data, and video data, or a combination of different types of data. This disclosure does not impose any specific limitations.

[0095] In S202, media asset data is input into the content understanding model, and the content understanding model performs content recognition on the media asset data to obtain the target content understanding result of the media asset data.

[0096] In this disclosure, the content understanding model is trained using the content understanding model training method described above.

[0097] In the above technical solution, the content understanding model periodically saves the current model training state information and the current reading position of the training data during the model training phase. In this way, when the content understanding model restarts training after an interruption, it can load the training data from the reading position of the most recently saved training data during the training phase, and continue training the content understanding model based on the most recently saved model training state information, thereby realizing training breakpoint recovery and continuing the training state before the model training was interrupted. This allows the model to utilize unstable resources such as tidal (i.e., resources are scarce in one period and sufficient in other periods) or preemptible resources for training, thus accelerating the overall model iteration efficiency and output efficiency.

[0098] Figure 6 This is a block diagram illustrating a content understanding model training apparatus according to an exemplary embodiment. Figure 6 As shown, the content understanding model training device 300 includes:

[0099] The first acquisition module 301 is used to acquire the model training state information and the reading position of the training data most recently saved during the model training phase in response to the restart of training after the content understanding model has been interrupted. The content understanding model periodically saves the current model training state information and the current reading position of the training data during the model training phase.

[0100] The training module 302 is used to load training data from the most recently saved read position and, based on the loaded training data, continue to train the content understanding model using the most recently saved model training state information as a reference.

[0101] In the above technical solution, the content understanding model periodically saves the current model training state information and the current reading position of the training data during the model training phase. In this way, when the content understanding model restarts training after an interruption, it can load the training data from the reading position of the most recently saved training data during the training phase, and continue training the content understanding model based on the most recently saved model training state information, thereby realizing training breakpoint recovery and continuing the training state before the model training was interrupted. This allows the model to utilize unstable resources such as tidal (i.e., resources are scarce in one period and sufficient in other periods) or preemptible resources for training, thus accelerating the overall model iteration efficiency and output efficiency.

[0102] Optionally, the training module 302 includes:

[0103] The partitioning submodule is used to divide the loaded training data into N training sub-data that correspond one-to-one with the N first resources currently participating in model training, where N≥1;

[0104] The first training submodule is used to train the content understanding model for each of the first resources based on its corresponding training subdata and the most recently saved model training state information.

[0105] Optionally, the first training submodule includes:

[0106] The first loading submodule is used to load the most recently saved model training state information of each first resource onto the content understanding model to obtain a baseline model; the second training submodule is used to train the baseline model based on the corresponding training sub-data of the first resource to obtain the training loss corresponding to the first resource.

[0107] The first determining submodule is used to determine the current model state information of the content understanding model based on the training loss corresponding to each of the first resources.

[0108] Optionally, the first resource is a GPU;

[0109] The first training submodule also includes:

[0110] The second loading submodule is used for each of the first resources to preload its corresponding training subdata from CPU memory into its respective memory.

[0111] Optionally, the content understanding model training device 300 further includes:

[0112] The partitioning module is used to divide the test data into M test sub-data that correspond one-to-one with the M second resources currently participating in the model test, where M≥1;

[0113] The testing module is used to test the content understanding model for each of the second resources based on its corresponding test sub-data.

[0114] Optionally, the test module includes:

[0115] The generation submodule is used to generate test content understanding results for each of the second resources based on its corresponding test sub-data;

[0116] The second determining submodule is used to determine the test performance index of the content understanding model based on the understanding result of each test content and the test tags in the test data.

[0117] Optionally, the content understanding model training device 300 further includes:

[0118] The verification module is used to perform model verification on the content understanding model every first preset training step to obtain the current verification result of the content understanding model.

[0119] The determination module is used to determine, based on the current verification result, whether to update the optimal K model training state information of the content understanding model stored during the model verification phase;

[0120] An update module is used to update the current model training state information of the currently validated content understanding model to the optimal K model training state information if it is determined that the optimal K model training state information should be updated.

[0121] Optionally, the content understanding model saves the current model training state information and the current reading position of the training data every second preset training step during the model training phase;

[0122] The content understanding model training device 300 further includes:

[0123] The storage module is used to save the current model training state information of the content understanding model after each round of model training is completed, wherein the number of training steps in each round of model training is greater than the second preset number of training steps.

[0124] Figure 7 This is a block diagram illustrating a content understanding device according to an exemplary embodiment. Figure 7 As shown, the content understanding device 400 includes:

[0125] The second acquisition module 401 is used to acquire media asset data to be identified.

[0126] The content recognition module 402 is used to input the media asset data into the content understanding model, and to perform content recognition on the media asset data through the content understanding model to obtain the target content understanding result of the media asset data. The content understanding model is trained by the content understanding model training method provided in this disclosure.

[0127] In the above technical solution, the content understanding model periodically saves the current model training state information and the current reading position of the training data during the model training phase. In this way, when the content understanding model restarts training after an interruption, it can load the training data from the reading position of the most recently saved training data during the training phase, and continue training the content understanding model based on the most recently saved model training state information, thereby realizing training breakpoint recovery and continuing the training state before the model training was interrupted. This allows the model to utilize unstable resources such as tidal (i.e., resources are scarce in one period and sufficient in other periods) or preemptible resources for training, thus accelerating the overall model iteration efficiency and output efficiency.

[0128] This disclosure also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the content understanding model training method or the steps of the content understanding method provided in this disclosure.

[0129] The following is for reference. Figure 8 The diagram illustrates a structural schematic of an electronic device (e.g., a terminal device or a server) 600 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0130] like Figure 8 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0131] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0132] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0133] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0134] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0135] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0136] The aforementioned computer-readable medium carries one or more programs. When the electronic device executes the aforementioned one or more programs, the electronic device causes the following actions: in response to the content understanding model restarting training after a training interruption, the electronic device obtains the most recently saved model training state information and the reading position of the training data during the model training phase, wherein the content understanding model periodically saves the current model training state information and the current reading position of the training data during the model training phase; loads training data from the most recently saved reading position, and continues to train the content understanding model based on the loaded training data and the most recently saved model training state information.

[0137] Alternatively, the aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire media asset data to be content-recognized; input the media asset data into a content understanding model; and perform content recognition on the media asset data through the content understanding model to obtain a target content understanding result for the media asset data, wherein the content understanding model is trained using the content understanding model training method provided in this disclosure.

[0138] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0140] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules are not necessarily limiting in certain circumstances; for example, the second acquisition module can also be described as "a module for acquiring media asset data to be identified for content recognition".

[0141] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0142] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] According to one or more embodiments of this disclosure, Example 1 provides a method for training a content understanding model, the method comprising:

[0144] In response to the restart of training after the content understanding model has been interrupted, the model training state information and the reading position of the training data are obtained from the most recently saved model training phase. The content understanding model periodically saves the current model training state information and the current reading position of the training data during the model training phase.

[0145] The training data is loaded from the most recently saved read position, and the content understanding model is trained again based on the most recently saved model training state information, using the loaded training data as a reference.

[0146] According to one or more embodiments of this disclosure, Example 2 provides the method of Example 1, wherein the step of continuing to train the content understanding model based on the most recently saved model training state information based on the loaded training data includes: dividing the loaded training data into N training sub-data corresponding one-to-one with N first resources currently participating in model training, where N≥1; and each first resource continuing to train the content understanding model based on its corresponding training sub-data and the most recently saved model training state information.

[0147] According to one or more embodiments of this disclosure, Example 3 provides the method of Example 2, wherein each first resource continues to train the content understanding model based on the most recently saved model training state information, according to its corresponding training sub-data, including: for each first resource, the first resource loads the most recently saved model training state information onto the content understanding model to obtain a base model; the first resource trains the base model according to its corresponding training sub-data to obtain the training loss corresponding to the first resource; and determines the current model state information of the content understanding model according to the training loss corresponding to each first resource.

[0148] According to one or more embodiments of this disclosure, Example 4 provides the method of Example 3, wherein the first resource is a GPU; each of the first resources continues to train the content understanding model based on the most recently saved model training state information according to its corresponding training sub-data, and further includes: each of the first resources preloads its corresponding training sub-data from CPU memory into its respective memory.

[0149] According to one or more embodiments of this disclosure, Example 5 provides the method of Example 1, the method further comprising: dividing the test data into M test sub-data that correspond one-to-one with M second resources currently participating in the model test, M≥1; each second resource testing the content understanding model according to its corresponding test sub-data.

[0150] According to one or more embodiments of this disclosure, Example 6 provides the method of Example 5, wherein each of the second resources tests the content understanding model based on its corresponding test sub-data, including: each of the second resources generating a test content understanding result based on its corresponding test sub-data; and determining the test performance index of the content understanding model based on each test content understanding result and the test tags in the test data.

[0151] According to one or more embodiments of this disclosure, Example 7 provides a method as described in any one of Examples 1-6, the method further comprising: performing model validation on the content understanding model every first preset training step to obtain the current validation result of the content understanding model; determining, based on the current validation result, whether to update the optimal K model training state information of the content understanding model stored in the model validation phase; if it is determined to update the optimal K model training state information, then updating the current model training state information of the currently validated content understanding model to the optimal K model training state information.

[0152] According to one or more embodiments of this disclosure, Example 8 provides a method as described in any one of Examples 1-6, wherein the content understanding model saves the current model training state information and the current reading position of the training data every second preset training step during the model training phase; the method further includes: saving the current model training state information of the content understanding model after each round of model training is completed, wherein the number of training steps included in each round of model training is greater than the second preset training step.

[0153] According to one or more embodiments of this disclosure, Example 9 provides a content understanding method, including: acquiring media asset data to be content recognized; inputting the media asset data into a content understanding model, and performing content recognition on the media asset data through the content understanding model to obtain a target content understanding result of the media asset data, wherein the content understanding model is trained by the content understanding model training method of any one of Examples 1-8.

[0154] According to one or more embodiments of this disclosure, Example 10 provides a content understanding model training apparatus, the apparatus comprising: a first acquisition module, configured to acquire, in response to restarting training after a content understanding model has been interrupted, the most recently saved model training state information and the read position of training data during the model training phase, wherein the content understanding model periodically saves the current model training state information and the current read position of training data during the model training phase; and a training module, configured to load training data from the most recently saved read position and continue training the content understanding model based on the loaded training data and the most recently saved model training state information.

[0155] According to one or more embodiments of this disclosure, Example 11 provides a content understanding apparatus, including: a second acquisition module for acquiring media asset data to be content-recognized; and a content recognition module for inputting the media asset data into a content understanding model and performing content recognition on the media asset data through the content understanding model to obtain a target content understanding result of the media asset data, wherein the content understanding model is trained by the content understanding model training method described in any one of Examples 1-8.

[0156] According to one or more embodiments of the present disclosure, Example 12 provides a computer-readable medium having a computer program stored thereon that, when executed by a processing device, implements the steps of the method described in any one of Examples 1-9.

[0157] According to one or more embodiments of this disclosure, Example 13 provides an electronic device including: a storage device having a computer program stored thereon; and a processing device for executing the computer program in the storage device to implement the steps of any one of Examples 1-9.

[0158] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0159] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0160] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A method for training a content understanding model, characterized in that, The method includes: In response to the restart of training after the content understanding model has been interrupted, the model obtains the most recently saved model training state information and the reading position of the training data during the model training phase. The content understanding model periodically saves the current model training state information and the current reading position of the training data during the model training phase. The training data includes at least one data in the training dataset, and the training dataset includes multiple data. The training data is loaded from the most recently saved read position, and the content understanding model is trained again based on the most recently saved model training state information, using the loaded training data as a reference. The content understanding model saves the current model training state information and the current reading position of the training data every second preset training step during the model training phase; the method further includes: after each round of model training, saving the current model training state information of the content understanding model, wherein the number of training steps included in each round of model training is greater than the second preset training step.

2. The method according to claim 1, characterized in that, The step of continuing to train the content understanding model based on the loaded training data and using the most recently saved model training state information as a reference includes: The loaded training data is divided into N training sub-data that correspond one-to-one with the N first resources currently participating in model training, where N≥1; Each of the first resources continues to train the content understanding model based on its corresponding training sub-data and the most recently saved model training state information.

3. The method according to claim 2, characterized in that, Each of the first resources, based on its corresponding training sub-data and using the most recently saved model training state information as a reference, continues to train the content understanding model, including: For each of the first resources, the most recently saved model training state information is loaded into the content understanding model to obtain a baseline model; the first resource trains the baseline model according to its corresponding training sub-data to obtain the training loss corresponding to the first resource. Based on the training loss corresponding to each of the first resources, the current model state information of the content understanding model is determined.

4. The method according to claim 3, characterized in that, The first resource is a GPU; Each of the first resources, based on its corresponding training sub-data and using the most recently saved model training state information as a reference, continues to train the content understanding model, further comprising: Each of the first resources preloads its corresponding training sub-data from CPU memory into its respective memory.

5. The method according to claim 1, characterized in that, The method further includes: The test data is divided into M test sub-data that correspond one-to-one with the M second resources currently participating in the model test, where M≥1; Each of the second resources is tested on the content understanding model based on its corresponding test sub-data.

6. The method according to claim 5, characterized in that, Each of the second resources is tested on the content understanding model based on its corresponding test sub-data, including: Each of the second resources generates a test content understanding result based on its corresponding test sub-data; Based on the understanding results of each test content and the test labels in the test data, the test performance indicators of the content understanding model are determined.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Every first preset number of training steps, the content understanding model is validated to obtain the current validation result of the content understanding model; Based on the current verification results, determine whether to update the optimal K model training state information of the content understanding model stored during the model verification phase; If it is determined that the optimal K model training state information should be updated, then the current model training state information of the currently validated content understanding model will be updated to the optimal K model training state information.

8. A content understanding method, characterized in that, include: Obtain media asset data for content recognition; The media asset data is input into the content understanding model, and the content understanding model is used to perform content recognition on the media asset data to obtain the target content understanding result of the media asset data. The content understanding model is trained by the content understanding model training method of any one of claims 1-7.

9. A content understanding model training device, characterized in that, The device includes: The first acquisition module is used to acquire the model training state information and the reading position of the training data most recently saved during the model training phase in response to the restart of training after the content understanding model has been interrupted. The content understanding model periodically saves the current model training state information and the current reading position of the training data during the model training phase. The training data includes at least one data in the training dataset, and the training dataset includes multiple data. The training module is used to load training data from the most recently saved read position and, based on the loaded training data, continue to train the content understanding model using the most recently saved model training state information as a reference. The storage module is used to save the current model training state information of the content understanding model after each round of model training is completed. The number of training steps in each round of model training is greater than the second preset number of training steps. The content understanding model saves the current model training state information and the current reading position of the training data every second preset number of training steps during the model training phase.

10. A content understanding device, characterized in that, include: The second acquisition module is used to acquire media asset data to be identified. The content recognition module is used to input the media asset data into the content understanding model, and to perform content recognition on the media asset data through the content understanding model to obtain the target content understanding result of the media asset data, wherein the content understanding model is trained by the content understanding model training method of any one of claims 1-7.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processing device, it implements the steps of the method described in any one of claims 1-8.

12. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-8.

Images