A method, apparatus, device, and storage medium for scene label recognition.

Abstract

This application discloses a scene label recognition method, apparatus, device, and storage medium applied in the field of data processing technology. The method acquires scene data to be recognized, extracts a first feature from the scene data, calculates the similarity between the first feature and each matching feature belonging to a first number of scene dimensions in a feature library, determines a second number of second features from the multiple matching features based on the calculated similarity, and determines the preset label of the second feature as the scene label of the scene data to be recognized. In another implementation, the method acquires scene data to be recognized, inputs the scene data into a classification model, obtains the classification result output by the classification model, and determines the scene label of the scene data based on the classification result. Using a feature library or a classification model trained with multi-label training data, more multi-dimensional and richer scene labels can be determined for the scene data to be recognized.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a scene label recognition method, apparatus, device, and storage medium. Background Technology

[0002] Scene recognition involves analyzing data to identify the scene depicted by that data. Based on the identified scene, the data is then processed.

[0003] Currently, scene recognition models can be used to determine the scene of the data to be recognized. However, the content of the scenes that scene recognition models can recognize is relatively limited, making it difficult to meet the needs of scene recognition. Summary of the Invention

[0004] In view of this, this application provides a scene label recognition method, apparatus, device and storage medium, which can obtain multi-dimensional scene recognition results.

[0005] To solve the above problems, the technical solution provided in this application is as follows:

[0006] In a first aspect, embodiments of this application provide a scene label recognition method, the method comprising:

[0007] Acquire scene data to be identified, wherein the scene data to be identified includes one or more of text data, image data, and video data;

[0008] Extract the first feature from the scene data to be identified;

[0009] Calculate the similarity between the first feature and each feature to be matched included in the feature library. The feature library includes multiple features to be matched belonging to a first number of scene dimensions and preset labels for each feature to be matched.

[0010] Based on the similarity, a second number of second features are determined, wherein the second number is less than or equal to the first number, and the second number of second features belong to different scene dimensions.

[0011] The preset label of the second feature is determined as the scene label of the scene data to be identified.

[0012] Secondly, embodiments of this application provide a scene label recognition method, the method comprising:

[0013] Acquire scene data to be identified, wherein the scene data to be identified includes one or more of text data, image data, and video data;

[0014] The scene data to be identified is input into the classification model to obtain the classification result output by the classification model. The classification result includes multiple categories. The classification model is trained using multi-label training data.

[0015] The scene label of the scene data to be identified is determined based on the classification results.

[0016] Thirdly, embodiments of this application provide a scene label recognition device, the device comprising:

[0017] An acquisition unit is used to acquire scene data to be identified, wherein the scene data to be identified includes one or more of text data, image data, and video data;

[0018] The first feature extraction unit is used to extract the first feature of the scene data to be identified;

[0019] The calculation unit is used to calculate the similarity between the first feature and each feature to be matched included in the feature library. The feature library includes multiple features to be matched belonging to a first number of scene dimensions and preset labels for each feature to be matched.

[0020] The first determining unit is configured to determine a second number of second features based on the similarity, wherein the second number is less than or equal to the first number, and the second number of second features belong to different scene dimensions respectively;

[0021] The second determining unit is used to determine the preset label of the second feature as the scene label of the scene data to be identified.

[0022] Fourthly, embodiments of this application provide a scene label recognition device, the device comprising:

[0023] An acquisition unit is used to acquire scene data to be identified, wherein the scene data to be identified includes one or more of text data, image data, and video data;

[0024] The input unit is used to input the scene data to be identified into the classification model to obtain the classification result output by the classification model. The classification result includes multiple categories. The classification model is trained using multi-label training data.

[0025] The determining unit is used to determine the scene label of the scene data to be identified based on the classification result.

[0026] Fifthly, embodiments of this application provide an electronic device, including:

[0027] One or more processors;

[0028] Storage device, on which one or more programs are stored,

[0029] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any of the first aspects above, or implement the method as described in any of the second aspects above.

[0030] Sixthly, embodiments of this application provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any of the first aspects above, or implements the method as described in any of the second aspects above.

[0031] Therefore, this application has the following beneficial effects:

[0032] This application provides a scene label recognition method, apparatus, device, and storage medium. The method acquires scene data to be recognized, extracts a first feature from the scene data, calculates the similarity between the first feature and each matching feature in a feature library, where each matching feature belongs to a first number of scene dimensions, determines a second feature from the multiple matching features based on the calculated similarity, and sets a preset label for the second feature as the scene label for the scene data to be recognized. Matching the first feature with each matching feature in the feature library identifies a second number of second features similar to the first feature. These second features belong to different scene dimensions. Based on the second features, multiple scene labels for the scene data to be recognized are determined, resulting in a multi-dimensional and richer scene recognition result. In another implementation, the method acquires scene data to be recognized, inputs the scene data into a classification model, obtains the classification result output by the model, and determines the scene label based on the classification result. The classification model is trained using multi-label training data. The classification result output by the model includes multiple categories, thus enabling the determination of multi-dimensional and richer scene labels for the scene data to be recognized. Attached Figure Description

[0033] Figure 1 A schematic diagram illustrating an exemplary application scenario provided in the embodiments of this application;

[0034] Figure 2 A schematic diagram illustrating an exemplary application scenario provided in the embodiments of this application;

[0035] Figure 3 A flowchart illustrating a scene label recognition method provided in an embodiment of this application;

[0036] Figure 4 A flowchart illustrating a scene label recognition method provided in an embodiment of this application;

[0037] Figure 5 This is a schematic diagram of the structure of a scene label recognition device provided in an embodiment of this application;

[0038] Figure 6 This is a schematic diagram of another scene label recognition device provided in an embodiment of this application;

[0039] Figure 7 This is a schematic diagram of the basic structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0040] To facilitate understanding and explanation of the technical solutions provided in the embodiments of this application, the background technology of this application will be described first.

[0041] Scene recognition of images or videos transmitted over a network can determine the scene to which they belong. After determining the scene, downstream processing of the images or videos is possible, such as data classification and data analysis within the same scene. Currently, scene recognition models are pre-built according to the needs of the scene recognition task. These models can only recognize input data of specific scene types required by the task, and the scene content they can recognize is relatively limited, failing to meet the full requirements of scene recognition tasks.

[0042] Based on this, embodiments of this application provide a scene label recognition method, apparatus, device, and storage medium. The method acquires scene data to be recognized, extracts a first feature from the scene data, calculates the similarity between the first feature and each matching feature in a feature library, where each matching feature belongs to a first number of scene dimensions, determines a second feature from the matching features in the feature library based on the calculated similarity, and sets a preset label for the second feature as the scene label for the scene data to be recognized. Matching the first feature with each matching feature in the feature library identifies a second number of second features similar to the first feature, which belong to different scene dimensions. Based on the second features, multiple scene labels for the scene data to be recognized are determined, resulting in a multi-dimensional and richer scene recognition result. In another implementation, the method acquires scene data to be recognized, inputs the scene data into a classification model, obtains the classification result output by the model, and determines the scene label for the scene data based on the classification result. The classification model is trained using multi-label training data. The classification model outputs classification results that include multiple categories. Thus, based on the classification results, it is possible to determine multi-dimensional and richer scene labels for the scene data to be identified.

[0043] To facilitate understanding of the scene label recognition method provided in this application embodiment, the following description is provided in conjunction with the scene examples shown in the accompanying drawings.

[0044] In one possible implementation, see Figure 1 As shown, Figure 1 This is a schematic diagram illustrating an exemplary application scenario provided in this application embodiment. A feature library 101 is pre-established. The feature library 101 includes multiple features to be matched and preset labels for each feature. The features to be matched in the feature library 101 belong to three different scene dimensions. The three different scene dimensions include entity dimension, action dimension, and atmosphere dimension. The features to be matched in the feature library 101 can be obtained by pre-extracting features from data with preset labels. Scene data 102 to be identified is obtained, and features of the scene data 102 to be identified are extracted to obtain a first feature 103. The similarity between the first feature 103 and each feature to be matched in the feature library 101 is calculated. The features with higher similarity among the three scene dimensions are respectively used as second features 104, resulting in three second features 104 belonging to the entity dimension, action dimension, and atmosphere dimension, respectively. The preset labels of the second features 104 are used as scene labels 105 for the scene data 102 to be identified.

[0045] In another possible implementation, see Figure 2 As shown, Figure 2 This is a schematic diagram illustrating an exemplary application scenario provided in this application embodiment. A classification model 201 is pre-established. The classification model 201 is trained using multi-label training data. Scene data 202 to be identified is obtained and input into the classification model 201 to obtain the classification result 203 output by the classification model 201. The classification result 203 includes multiple categories, and the scene label 204 of the scene data 201 to be identified can be determined based on the classification result.

[0046] Those skilled in the art will understand that Figure 1 and Figure 2 The schematic diagram shown is merely one example in which embodiments of this application can be implemented. The scope of application of the embodiments of this application is not limited by any aspect of this framework.

[0047] To facilitate understanding of the technical solutions provided in the embodiments of this application, the following description, in conjunction with the accompanying drawings, illustrates a scene label recognition method provided in the embodiments of this application.

[0048] See Figure 3 As shown, this figure is a flowchart of a scene label recognition method provided in an embodiment of this application. Figure 3 As shown, the method may include S301-S305:

[0049] S301: Obtain the scene data to be identified.

[0050] The scene data to be identified is the data for which scene labels need to be determined. The scene data to be identified can be one or more of the following: text data, image data, and video data. This application does not limit the method of acquiring the scene data to be identified. As an example, the scene data to be identified can be data that needs to be processed and transmitted over a network.

[0051] S302: Extract the first feature of the scene data to be identified.

[0052] The feature extraction method for the scene data to be identified depends on the type of scene data. For example, if the scene data is text, the first feature is the text feature. As another example, if the scene data is image data, the first feature is the image feature. And as yet another example, if the scene data is video data, the first feature is the video feature.

[0053] This application does not limit the specific implementation of extracting the first feature from the scene data to be identified. In one possible implementation, the scene data to be identified is image data, and features of the scene data to be identified are extracted using an image feature extraction model to obtain the first feature. In another possible implementation, the scene data to be identified is video data, and features of the scene data to be identified are extracted using a video feature extraction model to obtain the first feature.

[0054] S303: Calculate the similarity between the first feature and each feature to be matched in the feature library.

[0055] The feature library includes multiple features to be matched and preset labels for each feature. The multiple features to be matched in the feature library belong to a first number of scene dimensions. The first number of scene dimensions describes the scene data to be identified from different aspects. This application embodiment does not limit the specific value of the first number; in one possible implementation, the first number is three. The first number of scene dimensions includes, for example, entity dimensions, action dimensions, and atmosphere dimensions. Entity dimensions describe the entities involved in the scene to which the scene data to be identified belongs. Action dimensions describe the actions involved in the scene to which the scene data to be identified belongs. Atmosphere dimensions describe the atmosphere involved in the scene to which the scene data to be identified belongs. The features to be matched in the feature library are obtained by pre-extracting features from sample data. The preset labels of the sample data are the preset labels of the features to be matched. It should be noted that the preset labels are the same as the scene dimensions to which the corresponding features to be matched belong. Taking the above example of a first number of three scene dimensions, such as entity dimensions, action dimensions, and atmosphere dimensions, the preset labels include entity labels belonging to the entity dimension, action labels belonging to the action dimension, and atmosphere labels belonging to the atmosphere dimension. Entity labels refer to nouns related to the scene. For example, a meadow or a lake. Action tags are verbs describing actions performed in the scene, such as playing basketball or swimming. Atmosphere tags are adjectives describing the atmosphere of the scene, such as comfortable or relaxing. This allows for analysis of scene tags across multiple scene dimensions, achieving a more comprehensive analysis of the scene data and improving the accuracy of the obtained scene tags.

[0056] Calculate the similarity between the first feature and each feature to be matched. The similarity is used to measure the degree of similarity between the first feature and the features to be matched. This application does not limit the specific implementation of the similarity calculation method, and can be set according to the calculation needs.

[0057] S304: Determine the second number of second features based on similarity.

[0058] The second feature is a feature to be matched that has a high degree of similarity to the first feature. The number of second features is a second quantity. The second quantity is less than or equal to the first quantity. Furthermore, the second quantity of second features belongs to different scene dimensions. That is, when the second quantity is the same as the first quantity, the second quantity of second features is a feature to be matched belonging to all scene dimensions. When the second quantity is less than the first quantity, the second quantity of second features is a feature to be matched belonging to only some scene dimensions.

[0059] This application does not limit the specific implementation of determining the second number of second features based on similarity. In one possible implementation, after obtaining the similarity between the first feature and each feature to be matched, the features to be matched belonging to each scene dimension with a similarity greater than a threshold are selected as the second features. The threshold is a pre-set fixed value. In another possible implementation, the similarity between the obtained first feature and each feature to be matched is first sorted from largest to smallest. Then, the features to be matched with the highest similarity are selected from the sorted list as the second features. For example, the top 5 features with the highest similarity are selected as the second features.

[0060] S305: Determine the preset label of the second feature as the scene label of the scene data to be identified.

[0061] The second feature is a feature to be matched that is similar to the first feature. The preset labels of the second feature are used as scene labels for the scene data to be identified. In this way, multiple scene labels for the scene data to be identified can be determined based on the features to be matched in the feature library and their preset labels, improving the comprehensive analysis of the scene content and facilitating subsequent processing of the scene data based on scene labels. Furthermore, the pre-established feature library can include features to be matched with multiple preset labels, supporting the recognition of a large number of scene labels. In addition, for specific scene recognition tasks, the features to be matched included in the feature library can be adjusted according to the needs of the task, enabling the rapid establishment of a feature library that matches the requirements of the scene recognition task and providing personalized scene recognition services.

[0062] In some possible implementations, the scene data to be identified includes video data. The video data can be processed into image data. Before extracting the first feature of the scene data to be identified, the method further includes:

[0063] Image frames are extracted from the video data according to the first frame extraction frequency to obtain image data.

[0064] It should be noted that the embodiments of this application do not limit the specific value of the first frame extraction frequency. As an example, the first frame extraction frequency is 1 frame per second. By performing frame extraction processing on the video data, the video data can be converted into image data, and then the first feature can be obtained by extracting features from the image data. Converting video data into image data enables the analysis of multiple scenes included in the video data, and identifies a more comprehensive set of scene labels.

[0065] Figure 3 The corresponding scene label recognition method is based on a pre-established feature library. In another possible implementation, a classification model can be used to identify scene labels for the scene data to be recognized. See [link to relevant documentation]. Figure 4As shown, this figure is a flowchart of a scene label recognition method provided in an embodiment of this application. Figure 4 As shown, the method may include S401-S403.

[0066] S401: Obtain the scene data to be identified.

[0067] The scene data to be identified is the data for which scene labels need to be determined. The scene data to be identified can be one or more of the following: text data, image data, and video data. This application does not limit the method of acquiring the scene data to be identified. As an example, the scene data to be identified can be data that needs to be processed and transmitted over a network.

[0068] S402: Input the scene data to be identified into the classification model and obtain the classification result output by the classification model.

[0069] The classification model is pre-trained using multi-label training data. It is used to classify the input scene data to be identified and output the classification result. It should be noted that different classification models are used for different types of scene data. In some possible implementations, this application provides three training methods for the classification models, which are detailed below.

[0070] The classification model outputs classification results that include multiple categories. These multiple categories can encompass different dimensions of classification. For example, multiple categories can include one or more of the following: entity-related categories, action-related categories, or segmentation-related categories.

[0071] S403: Determine the scene label for the scene data to be identified based on the classification results.

[0072] The classification results include scene labels for the scene data to be identified. Based on the determined scene labels, subsequent analysis of the scene data becomes easier. In this way, multiple scene labels can be determined using the classification model, enabling a more comprehensive analysis of the scene data and obtaining more accurate and complete scene labels.

[0073] Different classification models are used for different types of scene data to be identified. In some possible implementations, embodiments of this application provide three classification models for different scene data to be identified.

[0074] The first type: the classification model is the image classification model.

[0075] Image classification models can classify input image data. As one example, image data refers to the acquired images that need to be processed. As another example, image data is obtained by processing video data.

[0076] The training process of the image classification model provided in this application embodiment will be described below. The training process of the image classification model includes the following three steps.

[0077] A1: Obtain the first training data.

[0078] The first training data includes training image data and multiple first scene labels for the training image data. These first scene labels can be pre-labeled labels or pseudo-labels determined by a pseudo-labeling model. For details on methods for using a pseudo-labeling model to annotate pseudo-labels, please refer to the following text.

[0079] A2: Input the training image data into the image classification model to be trained, and obtain the first predicted classification result output by the image classification model to be trained.

[0080] The image classification model to be trained is used to classify the training image data.

[0081] By inputting training image data into the image classification model to be trained, a first predicted classification result for the training image data can be obtained from the output of the image classification model. This first predicted classification result includes multiple first classification categories.

[0082] A3: Using the first predicted classification result and multiple first scene labels, the image classification model to be trained is adjusted using a loss function to obtain the image classification model.

[0083] Using a loss function, the difference between the first predicted classification result and multiple first scene labels is calculated. Based on the calculated difference, the image classification model to be trained is adjusted. The adjusted image classification model that meets the preset requirements is taken as the image classification model. The preset requirements are the requirements for completing the training. As one example, the preset requirement is that the number of training iterations reaches a preset number. As another example, the preset requirement is that the accuracy of the image classification model to be trained reaches a preset level.

[0084] Among some possible implementations, embodiments of this application provide two specific implementations of using a first predicted classification result and a first scene label to adjust the image classification model to be trained using a loss function.

[0085] In one possible implementation, a first classification category is selected as the target type from a plurality of first classification categories included in the first predicted classification result. The specific implementation method for selecting the target type is not limited in this application embodiment. For example, a random selection method can be used to select the target type from a plurality of first classification categories. Furthermore, the specific type of loss function is not limited in this application embodiment. For example, the loss function is a normalized exponential function.

[0086] In another possible implementation, the variant normalized exponential function is the multi-cross-entropy function. The multi-cross-entropy function can calculate the loss value for multiple labels.

[0087] The following section describes the processing procedures for the two types of scene data to be identified by the image classification model.

[0088] In one possible implementation, the scene data to be identified includes video data. The scene data is input into a classification model, and the classification results output by the model are obtained, including:

[0089] The video data is processed by frame extraction according to the second frame extraction frequency to obtain multiple image data to be identified.

[0090] Multiple image data to be identified are input into an image classification model to obtain the image classification results of each image data to be identified, as output by the image classification model.

[0091] This application does not limit the specific value of the second frame extraction frequency in its embodiments. As an example, the second frame extraction frequency is 1 frame per second. By performing frame extraction processing on video data, the video data can be converted into multiple image data to be identified. The obtained multiple image data to be identified are input into an image classification model to obtain the image classification results of the image classification model data corresponding to each image data to be identified. By performing frame extraction processing on video data, multiple image data to be identified can be obtained. Then, by using an image classification model to classify each image data to be identified, one or more scenes involved in the video can be identified, and the overall scene label of the video data can be obtained.

[0092] It should be noted that some video data includes multiple scenes, each with different labels. Correspondingly, this application provides a specific implementation method for determining scene labels for scene data to be identified based on classification results, including:

[0093] If the image classification result of the image data to be identified includes multiple image classification results, the video data is divided into at least two video sub-data based on the second frame sampling frequency and the image classification results of each image data to be identified, and each video sub-data includes different image classification results of the image data to be identified;

[0094] Based on the image classification results of the image data to be identified included in the video sub-data, the scene label of the video sub-data is determined.

[0095] If the image classification model outputs at least two image classification results for each image data to be identified, it indicates that the video data involves at least two scenes. Based on the second frame sampling frequency, the video data can be divided into multiple video sub-data. Based on the image classification results of the image data to be identified extracted from each video sub-data, video sub-data with the same image classification results are merged to obtain video sub-data with different image classification results. The scene label for each video sub-data is determined based on the image classification results of the image data to be identified included in each video sub-data. The time interval between each video sub-data is the scene transition breakpoint. The scene labels of the video sub-data before and after each scene transition breakpoint are different.

[0096] In another possible implementation, the scene data to be identified includes image data. An image classification model can be used to determine the scene label of the image data.

[0097] The second type: The classification model is the first video classification model.

[0098] The first video classification model can classify and process input video data.

[0099] The training process of the first video classification model provided in this application embodiment will be described below. The training process of the first video classification model includes the following three steps:

[0100] B1: Obtain the second training data.

[0101] The second training data includes the first training video data and multiple second scene labels for the first training video data. These second scene labels can be pre-labeled labels or pseudo-labels determined by a pseudo-labeling model. For details on methods for using a pseudo-labeling model to annotate pseudo-labels, please refer to the following text.

[0102] B2: Input the first training video data into the first video classification model to be trained, and obtain the second predicted classification result output by the first video classification model to be trained.

[0103] The first video classification model to be trained is used to classify the first training video data. The first training video data is input into the first video classification model to be trained, resulting in a second predicted classification result for the first training video data. This second predicted classification result includes multiple second classification categories.

[0104] B3: Using the second predicted classification result and the second scene label, the first video classification model to be trained is adjusted using a loss function to obtain the first video classification model.

[0105] Using a loss function, the difference between the second predicted classification result and multiple second scene labels is calculated. Based on the calculated difference, the first video classification model to be trained is adjusted. The adjusted first video classification model that meets the preset requirements is taken as the first video classification model. The preset requirements are the requirements for completing training. As one example, the preset requirement is that the number of training iterations reaches a preset number. As another example, the preset requirement is that the accuracy of the first video classification model to be trained reaches a preset level.

[0106] In one possible implementation, one second classification category is selected as the target type from among multiple second classification categories included in the second predicted classification result. This application does not limit the specific implementation method for selecting the target type. For example, a random selection method can be used to select the target type from multiple second classification categories. Furthermore, this application does not limit the specific type of loss function. For example, the loss function can be a normalized exponential function.

[0107] In another possible implementation, the variant normalized exponential function is the multi-class cross-entropy function. The multi-class cross-entropy function enables the calculation of loss values ​​for multiple labels.

[0108] The third type: The classification model is the second video classification model.

[0109] The second video classification model can classify input video data and related text data. The text data refers to text data related to the video data. The text data can be the title of the video data or the descriptive text of the video data. This application does not limit the source of the text data; the text data may be obtained by performing OCR (Optical Character Recognition) on the page or text containing the video data, or by performing ASR (Automatic Speech Recognition) on the audio related to the video data.

[0110] The training process of the second video classification model provided in this application embodiment will be described below. The training process of the second video classification model includes the following three steps:

[0111] C1: Obtain the third training data, which includes the second training video data, training text data, and multiple second labels of the second video image data.

[0112] The third training data includes the second training video data, training text data, and multiple second scene labels for the second training video data. The second scene labels can be pre-labeled labels or pseudo-labels determined by a pseudo-labeling model. For details on methods for using a pseudo-labeling model to annotate pseudo-labels, please refer to the following text. The training text data is text data related to the second training video data. The training text data and the second training video data belong to the same scene.

[0113] C2: Input the second training video data and training text data into the second video classification model to be trained, and obtain the third predicted classification result output by the second video classification model to be trained. The third predicted classification result includes multiple third classification categories.

[0114] The second training video classification model is used to classify the second training video data and the training text data. This model extracts features from both the second training video data and the training text data, and generates a third predicted classification result based on these features. The third predicted classification result includes multiple third classification categories.

[0115] C3: Using the third predicted classification result and label, the second video classification model to be trained is adjusted using a loss function to obtain the second video classification model.

[0116] Using a loss function, the difference between the third predicted classification result and multiple third scene labels is calculated. Based on the calculated difference, the second video classification model to be trained is adjusted. The adjusted second video classification model that meets the preset requirements is used as the second video classification model. The preset requirements are the requirements for completing the training. As one example, the preset requirement is that the number of training iterations reaches a preset number. As another example, the preset requirement is that the accuracy of the second video classification model to be trained reaches a preset level.

[0117] In one possible implementation, a third category is selected as the target type from multiple third categories included in the third predicted classification result. This application does not limit the specific implementation method for selecting the target type. For example, a random selection method can be used to select the target type from multiple third categories. The difference between the target model and the third scene label is calculated using a loss function. Furthermore, this application does not limit the specific type of loss function. For example, the loss function can be a normalized exponential function.

[0118] In another possible implementation, the variant normalized exponential function is the multi-class cross-entropy function. The multi-class cross-entropy function enables the calculation of loss values ​​for multiple labels.

[0119] In some possible implementations, the multi-label training data used to train the classification model is obtained by processing single-label training data. This application provides a specific implementation for obtaining multi-label training data, comprising the following five steps:

[0120] D1: Acquire collected data.

[0121] The collected data refers to the raw data used to generate multi-label training data. This application does not limit the source of the collected data. In one possible implementation, the collected data can be retrieved from the network or a database using a pre-defined single label. The collected data can be one or more of text data, image data, and video data.

[0122] D2: Input the collected data into the pseudo-label annotation model to obtain the predicted labels of the collected data output by the pseudo-label annotation model.

[0123] The pseudo-labeling model can output multiple predicted labels for the collected data based on the input data. The pseudo-labeling model can be pre-trained.

[0124] It should be noted that the type of pseudo-labeling model differs depending on the type of collected data. As one example, if the collected data is image data, the pseudo-labeling model is designed for processing image data. As another example, if the collected data is video data, the pseudo-labeling model can be used to process video data. As yet another example, if the collected data includes both video and text data, the pseudo-labeling model can be used to process both video and text data.

[0125] D3: Obtain multiple standard labels for the collected data.

[0126] The standard labels for collected data are labels that are manually annotated to the collected data.

[0127] D4: If the similarity between the predicted label and the standard label is greater than or equal to the threshold, then the standard label and the predicted label will be used as the training labels for the collected data.

[0128] After acquiring multiple standard labels and multiple predicted labels from the collected data, these labels are analyzed. One possible implementation utilizes a confusion matrix for analysis. Based on the confusion matrix analysis results, predictive labels that are identical or similar to the standard labels, as well as those that differ from the standard labels, can be identified. Predictive labels with a similarity greater than or equal to a threshold with the standard label can be used as synonyms of the standard label. Both the standard label and the predictive labels with a similarity greater than or equal to the threshold can be used as training labels for the collected data. This approach expands the range of labels while maintaining accuracy, resulting in more accurate and comprehensive training labels.

[0129] D5: If the similarity between multiple predicted labels and the standard labels is less than the threshold, the collected data will be cleaned.

[0130] If all predicted labels are different from the labeled labels or the similarity is less than the threshold, it indicates that the quality of the collected data is poor, and the collected data needs to be cleaned. One possible implementation is to reacquire data related to the standard labels of the collected data and determine the pseudo-labels for the newly acquired data.

[0131] Based on the scene label recognition method provided in the above-described method embodiments, this application also provides a scene label recognition device, which will be described below with reference to the accompanying drawings.

[0132] See Figure 5 As shown in the figure, this is a structural schematic diagram of a scene label recognition device provided in an embodiment of this application. Figure 5 As shown, the scene label recognition device includes:

[0133] The acquisition unit 501 is used to acquire scene data to be identified, wherein the scene data to be identified includes one or more of text data, image data and video data;

[0134] The first feature extraction unit 502 is used to extract the first feature of the scene data to be identified;

[0135] The calculation unit 503 is used to calculate the similarity between the first feature and each feature to be matched included in the feature library. The feature library includes multiple features to be matched belonging to a first number of scene dimensions and preset labels for each feature to be matched.

[0136] The first determining unit 504 is used to determine a second number of second features based on the similarity, wherein the second number is less than or equal to the first number, and the second number of second features belong to different scene dimensions respectively;

[0137] The second determining unit 505 is used to determine the preset label of the second feature as the scene label of the scene data to be identified.

[0138] In one possible implementation, the first determining unit 504 is specifically used to take the matching features with a similarity greater than a threshold from the matching features belonging to each scene dimension as the second feature.

[0139] In one possible implementation, the scene data to be identified includes video data, and the device further includes:

[0140] The second feature extraction unit is used to extract image frames from the video data according to the first frame extraction frequency to obtain image data;

[0141] The first feature extraction unit is specifically used to extract the first feature of the image data.

[0142] In one possible implementation, the first quantity is three, and the scene dimensions of the first quantity include entity dimension, action dimension, and atmosphere dimension.

[0143] Based on the scene label recognition method provided in the above method embodiments, this application also provides another scene label recognition device, which will be described below with reference to the accompanying drawings.

[0144] See Figure 6 As shown in the figure, this is a schematic diagram of another scene label recognition device provided in an embodiment of this application. Figure 6 As shown, the scene label recognition device includes:

[0145] The acquisition unit 601 is used to acquire scene data to be identified, wherein the scene data to be identified includes one or more of text data, image data and video data;

[0146] The input unit 602 is used to input the scene data to be identified into the classification model to obtain the classification result output by the classification model. The classification result includes multiple categories. The classification model is trained using multi-label training data.

[0147] The determining unit 603 is used to determine the scene label of the scene data to be identified based on the classification result.

[0148] In one possible implementation, the scene data to be identified includes video data. The input unit 602 is specifically used to perform frame extraction processing on the video data according to a second frame extraction frequency to obtain multiple image data to be identified; the multiple image data to be identified are input into an image classification model to obtain the image classification results of each image data to be identified output by the image classification model.

[0149] In one possible implementation, the determining unit 603 is specifically configured to, if the image classification result of the image data to be identified includes multiple image classification results, divide the video data into at least two video sub-data based on the second frame sampling frequency and the image classification results of each of the image data to be identified, wherein the image classification results of the image data to be identified included in each video sub-data are different; and determine the scene label of the video sub-data based on the image classification results of the image data to be identified included in the video sub-data.

[0150] In one possible implementation, the scene data to be identified includes image data, and the classification model is an image classification model.

[0151] In one possible implementation, the classification model is an image classification model, which is trained in the following manner:

[0152] Acquire first training data, the first training data including training image data and multiple first scene labels of the training image data;

[0153] The training image data is input into the image classification model to be trained to obtain the first predicted classification result output by the image classification model to be trained. The first predicted classification result includes multiple first classification categories.

[0154] Using the first predicted classification result and multiple first scene labels, the image classification model to be trained is adjusted using a loss function to obtain the image classification model.

[0155] In one possible implementation, adjusting the image classification model to be trained using a loss function based on the first predicted classification result and the first scene label includes:

[0156] Select the target type from the multiple first classification categories included in the first predicted classification result;

[0157] Using the target type and the first scene label, the image classification model to be trained is adjusted using a loss function.

[0158] In one possible implementation, the loss function is a multi-class cross-entropy function.

[0159] In one possible implementation, the test scenario data includes video data, and the classification model is a first video classification model, which is trained in the following manner:

[0160] Acquire second training data, which includes first training video data and multiple second scene labels of the first training video data;

[0161] The first training video data is input into the first video classification model to be trained, and a second predicted classification result is obtained from the output of the first video classification model to be trained. The second predicted classification result includes multiple second classification categories.

[0162] Using the second predicted classification result and the second scene label, the first video classification model to be trained is adjusted using a loss function to obtain the first video classification model.

[0163] In one possible implementation, the test scenario data includes video data and text data, wherein the text data is related to the video data, and the classification model is a second video classification model, which is trained in the following manner:

[0164] Acquire third training data, which includes second training video data, training text data, and multiple second labels of the second training video data, wherein the training text data is related to the second training video data;

[0165] The second training video data and training text data are input into the second video classification model to be trained to obtain the third predicted classification result output by the second video classification model to be trained. The third predicted classification result includes multiple third classification categories.

[0166] Using the third predicted classification result and the label, the second video classification model to be trained is adjusted using a loss function to obtain the second video classification model.

[0167] In one possible implementation, the multi-label training data is obtained in the following way:

[0168] Acquire collected data;

[0169] The collected data is input into the pseudo-labeling model to obtain the predicted label of the collected data output by the pseudo-labeling model;

[0170] Obtain the standard labels of the collected data;

[0171] If the similarity between the predicted label and the standard label is greater than or equal to a threshold, then the standard label and the predicted label are used as training labels for the collected data.

[0172] If the similarity between the predicted label and the standard label is less than a threshold, the collected data is cleaned.

[0173] Based on the scene label recognition method provided in the above method embodiments, this application also provides an electronic device, including: one or more processors; a storage device storing one or more programs thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the scene label recognition method as described in any of the above embodiments.

[0174] The following is for reference. Figure 7 This document illustrates a structural schematic diagram of an electronic device 700 suitable for implementing embodiments of this application. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Android Devices), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs (televisions), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0175] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0176] Typically, the following devices can be connected to I / O interface 705: input devices 708 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0177] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this application.

[0178] The electronic device provided in this application embodiment and the scene label recognition method provided in the above embodiment belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0179] Based on the scene label recognition method provided in the above-described method embodiments, this application embodiment provides a computer storage medium storing a computer program thereon, wherein the program, when executed by a processor, implements the above-described method. Figure 3 The scene label recognition method described in the corresponding embodiment, or the implementation as described above. Figure 4 The scene label recognition method described in the corresponding embodiment.

[0180] It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0181] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0182] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0183] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the aforementioned scene label recognition method.

[0184] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0185] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0186] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0187] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0188] The units described in the embodiments of this application can be implemented in software or in hardware. The name of the unit / module does not necessarily limit the unit itself; for example, a voice data acquisition module can also be described as a "data acquisition module".

[0189] According to one or more embodiments of this application, [Example 1] provides a scene label recognition method, the method comprising:

[0190] Acquire scene data to be identified, wherein the scene data to be identified includes one or more of text data, image data, and video data;

[0191] Extract the first feature from the scene data to be identified;

[0192] Calculate the similarity between the first feature and each feature to be matched included in the feature library. The feature library includes multiple features to be matched belonging to a first number of scene dimensions and preset labels for each feature to be matched.

[0193] Based on the similarity, a second number of second features are determined, wherein the second number is less than or equal to the first number, and the second number of second features belong to different scene dimensions.

[0194] The preset label of the second feature is determined as the scene label of the scene data to be identified.

[0195] According to one or more embodiments of this application, [Example 2] provides a scene label recognition method, wherein determining a second number of second features based on the similarity includes:

[0196] For each scene dimension, the feature with a similarity greater than the threshold is selected as the second feature.

[0197] According to one or more embodiments of this application, [Example 3] provides a scene label recognition method, wherein the scene data to be recognized includes video data, and before extracting the first feature of the scene data to be recognized, the method further includes:

[0198] Image data is obtained by extracting image frames from the video data according to the first frame extraction frequency.

[0199] The extraction of the first feature of the scene data to be identified includes:

[0200] Extract the first feature from the image data.

[0201] According to one or more embodiments of this application, [Example 4] provides a scene label recognition method, wherein the first quantity is three, and the scene dimensions of the first quantity include entity dimension, action dimension and atmosphere dimension.

[0202] According to one or more embodiments of this application, [Example 5] provides a scene label recognition method, the method comprising:

[0203] Acquire scene data to be identified, wherein the scene data to be identified includes one or more of text data, image data, and video data;

[0204] The scene data to be identified is input into the classification model to obtain the classification result output by the classification model. The classification result includes multiple categories. The classification model is trained using multi-label training data.

[0205] The scene label of the scene data to be identified is determined based on the classification results.

[0206] According to one or more embodiments of this application, [Example Six] provides a scene label recognition method, wherein the scene data to be recognized includes video data, and the step of inputting the scene data to be recognized into a classification model to obtain the classification result output by the classification model includes:

[0207] The video data is processed by frame extraction according to the second frame extraction frequency to obtain multiple image data to be identified.

[0208] Multiple images to be identified are input into an image classification model to obtain the image classification results of each image to be identified, as output by the image classification model.

[0209] According to one or more embodiments of this application, [Example Seven] provides a scene label recognition method, wherein determining the scene label of the scene data to be identified based on the classification result includes:

[0210] If the image classification result of the image data to be identified includes multiple image classification results, the video data is divided into at least two video sub-data based on the second frame sampling frequency and the image classification results of each of the image data to be identified, and the image classification results of the image data to be identified included in each video sub-data are different;

[0211] The scene label of the video sub-data is determined based on the image classification results of the image data to be identified included in the video sub-data.

[0212] According to one or more embodiments of this application, [Example 8] provides a scene label recognition method, wherein the scene data to be recognized includes image data, and the classification model is an image classification model.

[0213] According to one or more embodiments of this application, [Example Nine] provides a scene label recognition method, wherein the classification model is an image classification model, and the image classification model is trained in the following manner:

[0214] Acquire first training data, the first training data including training image data and multiple first scene labels of the training image data;

[0215] The training image data is input into the image classification model to be trained to obtain the first predicted classification result output by the image classification model to be trained. The first predicted classification result includes multiple first classification categories.

[0216] Using the first predicted classification result and multiple first scene labels, the image classification model to be trained is adjusted using a loss function to obtain the image classification model.

[0217] According to one or more embodiments of this application, [Example 10] provides a scene label recognition method, wherein the method utilizes the first predicted classification result and multiple first scene labels, and adjusts the image classification model to be trained using a loss function, including:

[0218] Select the target type from the multiple first classification categories included in the first predicted classification result;

[0219] Using the target type and the first scene label, the image classification model to be trained is adjusted using a loss function.

[0220] According to one or more embodiments of this application, [Example 11] provides a scene label recognition method, wherein the loss function is a multi-class cross-entropy function.

[0221] According to one or more embodiments of this application, [Example Twelve] provides a scene label recognition method, wherein the test scene data includes video data, and the classification model is a first video classification model, which is trained in the following manner:

[0222] Acquire second training data, which includes first training video data and multiple second scene labels of the first training video data;

[0223] The first training video data is input into the first video classification model to be trained, and a second predicted classification result is obtained from the output of the first video classification model to be trained. The second predicted classification result includes multiple second classification categories.

[0224] Using the second predicted classification result and the second scene label, the first video classification model to be trained is adjusted using a loss function to obtain the first video classification model.

[0225] According to one or more embodiments of this application, [Example Thirteen] provides a scene label recognition method, wherein the test scene data includes video data and text data, the text data is related to the video data, and the classification model is a second video classification model, which is trained in the following manner:

[0226] Acquire third training data, which includes second training video data, training text data, and multiple second labels of the second training video data, wherein the training text data is related to the second training video data;

[0227] The second training video data and training text data are input into the second video classification model to be trained to obtain the third predicted classification result output by the second video classification model to be trained. The third predicted classification result includes multiple third classification categories.

[0228] Using the third predicted classification result and the label, the second video classification model to be trained is adjusted using a loss function to obtain the second video classification model.

[0229] According to one or more embodiments of this application, [Example Fourteen] provides a scene label recognition method, wherein the multi-label training data is obtained in the following manner:

[0230] Acquire collected data;

[0231] The collected data is input into the pseudo-labeling model to obtain the predicted label of the collected data output by the pseudo-labeling model;

[0232] Obtain the standard labels of the collected data;

[0233] If the similarity between the predicted label and the standard label is greater than or equal to a threshold, then the standard label and the predicted label are used as training labels for the collected data.

[0234] If the similarity between the predicted label and the standard label is less than a threshold, the collected data is cleaned.

[0235] According to one or more embodiments of this application, [Example Fifteen] provides a scene label recognition device, the device comprising:

[0236] An acquisition unit is used to acquire scene data to be identified, wherein the scene data to be identified includes one or more of text data, image data, and video data;

[0237] The first feature extraction unit is used to extract the first feature of the scene data to be identified;

[0238] The calculation unit is used to calculate the similarity between the first feature and each feature to be matched included in the feature library. The feature library includes multiple features to be matched belonging to a first number of scene dimensions and preset labels for each feature to be matched.

[0239] The first determining unit is configured to determine a second number of second features based on the similarity, wherein the second number is less than or equal to the first number, and the second number of second features belong to different scene dimensions respectively;

[0240] The second determining unit is used to determine the preset label of the second feature as the scene label of the scene data to be identified.

[0241] According to one or more embodiments of this application, [Example Sixteen] provides a scene label recognition device, wherein the first determining unit is specifically used to take the matching features belonging to each scene dimension, the matching features with a similarity greater than a threshold as the second features.

[0242] According to one or more embodiments of this application, [Example Seventeen] provides a scene label recognition device, wherein the scene data to be recognized includes video data, and the device further includes:

[0243] The second feature extraction unit is used to extract image frames from the video data according to the first frame extraction frequency to obtain image data;

[0244] The first feature extraction unit is specifically used to extract the first feature of the image data.

[0245] According to one or more embodiments of this application, [Example 18] provides a scene label recognition device, wherein the first quantity is three, and the scene dimensions of the first quantity include entity dimension, action dimension and atmosphere dimension.

[0246] According to one or more embodiments of this application, [Example 19] provides a scene label recognition device, the device comprising:

[0247] An acquisition unit is used to acquire scene data to be identified, wherein the scene data to be identified includes one or more of text data, image data, and video data;

[0248] The input unit is used to input the scene data to be identified into the classification model to obtain the classification result output by the classification model. The classification result includes multiple categories. The classification model is trained using multi-label training data.

[0249] The determining unit is used to determine the scene label of the scene data to be identified based on the classification result.

[0250] According to one or more embodiments of this application, [Example 20] provides a scene label recognition device, wherein the scene data to be recognized includes video data, and the input unit is specifically used to perform frame extraction processing on the video data according to a second frame extraction frequency to obtain multiple image data to be recognized; input the multiple image data to be recognized into an image classification model to obtain the image classification result of each image data to be recognized output by the image classification model.

[0251] According to one or more embodiments of this application, [Example 21] provides a scene label recognition device. The determining unit is specifically configured to, if the image classification result of the image data to be recognized includes multiple image classification results, divide the video data into at least two video sub-data based on the second frame extraction frequency and the image classification results of each of the image data to be recognized, wherein the image classification results of the image data to be recognized included in each video sub-data are different; and determine the scene label of the video sub-data based on the image classification results of the image data to be recognized included in the video sub-data.

[0252] According to one or more embodiments of this application, [Example 22] provides a scene label recognition device, wherein the scene data to be recognized includes image data, and the classification model is an image classification model.

[0253] According to one or more embodiments of this application, [Example 23] provides a scene label recognition device, wherein the classification model is an image classification model, and the image classification model is trained in the following manner:

[0254] Acquire first training data, the first training data including training image data and multiple first scene labels of the training image data;

[0255] The training image data is input into the image classification model to be trained to obtain the first predicted classification result output by the image classification model to be trained. The first predicted classification result includes multiple first classification categories.

[0256] Using the first predicted classification result and multiple first scene labels, the image classification model to be trained is adjusted using a loss function to obtain the image classification model.

[0257] According to one or more embodiments of this application, [Example 24] provides a scene label recognition device, wherein adjusting the image classification model to be trained using a loss function based on the first predicted classification result and the first scene label includes:

[0258] Select the target type from the multiple first classification categories included in the first predicted classification result;

[0259] Using the target type and the first scene label, the image classification model to be trained is adjusted using a loss function.

[0260] According to one or more embodiments of this application, [Example 25] provides a scene label recognition device, wherein the loss function is a multi-class cross-entropy function.

[0261] According to one or more embodiments of this application, [Example 26] provides a scene label recognition device, wherein the test scene data includes video data, and the classification model is a first video classification model, which is trained in the following manner:

[0262] Acquire second training data, which includes first training video data and multiple second scene labels of the first training video data;

[0263] The first training video data is input into the first video classification model to be trained, and a second predicted classification result is obtained from the output of the first video classification model to be trained. The second predicted classification result includes multiple second classification categories.

[0264] Using the second predicted classification result and the second scene label, the first video classification model to be trained is adjusted using a loss function to obtain the first video classification model.

[0265] According to one or more embodiments of this application, [Example 27] provides a scene label recognition device, wherein the test scene data includes video data and text data, the text data is related to the video data, and the classification model is a second video classification model, which is trained in the following manner:

[0266] Acquire third training data, which includes second training video data, training text data, and multiple second labels of the second training video data, wherein the training text data is related to the second training video data;

[0267] The second training video data and training text data are input into the second video classification model to be trained to obtain the third predicted classification result output by the second video classification model to be trained. The third predicted classification result includes multiple third classification categories.

[0268] Using the third predicted classification result and the label, the second video classification model to be trained is adjusted using a loss function to obtain the second video classification model.

[0269] According to one or more embodiments of this application, [Example 28] provides a scene label recognition device, wherein the multi-label training data is obtained in the following manner:

[0270] Acquire collected data;

[0271] The collected data is input into the pseudo-labeling model to obtain the predicted label of the collected data output by the pseudo-labeling model;

[0272] Obtain the standard labels of the collected data;

[0273] If the similarity between the predicted label and the standard label is greater than or equal to a threshold, then the standard label and the predicted label are used as training labels for the collected data.

[0274] If the similarity between the predicted label and the standard label is less than a threshold, the collected data is cleaned.

[0275] According to one or more embodiments of this application, [Example 29] provides an electronic device, including:

[0276] One or more processors;

[0277] Storage device, on which one or more programs are stored,

[0278] When the one or more programs are executed by the one or more processors, the one or more processors implement the scene label recognition method as described in any of [Example 1]-[Example 4], or implement the scene label recognition method as described in any of [Example 5]-[Example 14].

[0279] According to one or more embodiments of this application, [Example Thirty] provides a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the scene label recognition method as described in any of [Example One]-[Example Four], or implements the scene label recognition method as described in any of [Example Five]-[Example Fourteen].

[0280] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0281] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0282] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0283] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0284] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A scene tag identification method characterized by, The method includes: Acquire scene data to be identified, wherein the scene data to be identified includes one or more of text data, image data, and video data; Extract the first feature from the scene data to be identified; Calculate the similarity between the first feature and each feature to be matched included in the feature library. The feature library includes multiple features to be matched belonging to a first number of scene dimensions and preset labels for each feature to be matched. Based on the similarity, a second number of second features are determined, wherein the second number is less than or equal to the first number, and the second number of second features belong to different scene dimensions. The preset label of the second feature is determined as the scene label of the scene data to be identified; The step of determining a second number of second features based on the similarity includes: In each scene dimension, the feature to be matched with a similarity greater than a threshold is taken as the second feature. When the second quantity is the same as the first quantity, the second feature of the second quantity is the feature to be matched belonging to each scene dimension; When the second quantity is less than the first quantity, the second feature of the second quantity is a feature to be matched belonging to a partial scene dimension.

2. The method of claim 1, wherein, The scene data to be identified includes video data. Before extracting the first feature of the scene data to be identified, the method further includes: Image data is obtained by extracting image frames from the video data according to the first frame extraction frequency. The extraction of the first feature of the scene data to be identified includes: Extract the first feature of the image data; the first feature of the image data is an image feature.

3. The method of claim 1, wherein, The first quantity is three, and the scene dimensions of the first quantity include entity dimension, action dimension and atmosphere dimension; the entity dimension is the dimension describing the entities involved in the scene to which the scene data to be identified belongs; the action dimension is the dimension describing the actions involved in the scene to which the scene data to be identified belongs; and the atmosphere dimension is the dimension describing the atmosphere involved in the scene to which the scene data to be identified belongs.

4. A scene tag identification method characterized by, The method includes: Acquire scene data to be identified, wherein the scene data to be identified includes one or more of text data, image data, and video data; The scene data to be identified is input into the classification model to obtain the classification result output by the classification model. The classification result includes multiple categories. The classification model is trained using multi-label training data. Based on the classification results, determine the scene label for the scene data to be identified; The scene data to be identified includes video data. The step of inputting the scene data to be identified into a classification model to obtain the classification result output by the classification model includes: The video data is processed by frame extraction according to the second frame extraction frequency to obtain multiple image data to be identified. Multiple images to be identified are input into an image classification model to obtain the image classification results of each image to be identified output by the image classification model. Determining the scene label for the scene data to be identified based on the classification result includes: If the image classification result of the image data to be identified includes multiple image classification results, the video data is divided into at least two video sub-data based on the second frame sampling frequency and the image classification results of each of the image data to be identified, and the image classification results of the image data to be identified included in each video sub-data are different; The scene label of the video sub-data is determined based on the image classification results of the image data to be identified included in the video sub-data.

5. The method of claim 4, wherein, The scene data to be identified includes image data, and the classification model is an image classification model.

6. The method according to claim 4 or 5, characterized in that, The classification model is an image classification model, which is trained using the following method: Acquire first training data, which includes training image data and multiple first scene labels of the training image data; The training image data is input into the image classification model to be trained to obtain the first predicted classification result output by the image classification model to be trained. The first predicted classification result includes multiple first classification categories. Using the first predicted classification result and multiple first scene labels, the image classification model to be trained is adjusted using a loss function to obtain the image classification model.

7. The method according to claim 6, characterized in that, The step of adjusting the image classification model to be trained using the first predicted classification result and multiple first scene labels, and employing a loss function, includes: Select the target type from the multiple first classification categories included in the first predicted classification result; Using the target type and the first scene label, the image classification model to be trained is adjusted using a loss function.

8. The method according to claim 7, characterized in that, The loss function is the multi-class cross-entropy function.

9. The method according to claim 4, characterized in that, The scene data to be identified includes video data, and the classification model is a first video classification model, which is trained in the following manner: Acquire second training data, which includes first training video data and multiple second scene labels of the first training video data; The first training video data is input into the first video classification model to be trained, and a second predicted classification result is obtained from the output of the first video classification model to be trained. The second predicted classification result includes multiple second classification categories. Using the second predicted classification result and the second scene label, the first video classification model to be trained is adjusted using a loss function to obtain the first video classification model.

10. The method according to claim 4, characterized in that, The scene data to be identified includes video data and text data, wherein the text data is related to the video data, and the classification model is a second video classification model, which is trained in the following manner: Acquire third training data, which includes second training video data, training text data, and multiple second labels of the second training video data, wherein the training text data is related to the second training video data; The second training video data and training text data are input into the second video classification model to be trained to obtain the third predicted classification result output by the second video classification model to be trained. The third predicted classification result includes multiple third classification categories. Using the third predicted classification result and the label, the second video classification model to be trained is adjusted using a loss function to obtain the second video classification model.

11. The method according to claim 4, characterized in that, The multi-label training data was obtained in the following way: Acquire collected data; The collected data is input into the pseudo-labeling model to obtain the predicted label of the collected data output by the pseudo-labeling model; Obtain the standard labels of the collected data; If the similarity between the predicted label and the standard label is greater than or equal to a threshold, then the standard label and the predicted label are used as training labels for the collected data. If the similarity between the predicted label and the standard label is less than a threshold, the collected data is cleaned.

12. A scene label recognition device, characterized in that, The device includes: An acquisition unit is used to acquire scene data to be identified, wherein the scene data to be identified includes one or more of text data, image data, and video data; The first feature extraction unit is used to extract the first feature of the scene data to be identified; The calculation unit is used to calculate the similarity between the first feature and each feature to be matched included in the feature library. The feature library includes multiple features to be matched belonging to a first number of scene dimensions and preset labels for each feature to be matched. The first determining unit is configured to determine a second number of second features based on the similarity, wherein the second number is less than or equal to the first number, and the second number of second features belong to different scene dimensions respectively; The second determining unit is used to determine the preset label of the second feature as the scene label of the scene data to be identified; The second determining unit is further configured to: In each scene dimension, the feature to be matched with a similarity greater than a threshold is taken as the second feature. When the second quantity is the same as the first quantity, the second feature of the second quantity is the feature to be matched belonging to each scene dimension; When the second quantity is less than the first quantity, the second feature of the second quantity is a feature to be matched belonging to a partial scene dimension.

13. A scene label recognition device, characterized in that, The device includes: An acquisition unit is used to acquire scene data to be identified, wherein the scene data to be identified includes one or more of text data, image data, and video data; The input unit is used to input the scene data to be identified into the classification model to obtain the classification result output by the classification model. The classification result includes multiple categories. The classification model is trained using multi-label training data. A determining unit is configured to determine the scene label of the scene data to be identified based on the classification result; The input unit is also used for: The video data is processed by frame extraction according to the second frame extraction frequency to obtain multiple image data to be identified. Multiple images to be identified are input into an image classification model to obtain the image classification results of each image to be identified output by the image classification model. The determining unit is further configured to: If the image classification result of the image data to be identified includes multiple image classification results, the video data is divided into at least two video sub-data based on the second frame sampling frequency and the image classification results of each of the image data to be identified, and the image classification results of the image data to be identified included in each video sub-data are different; The scene label of the video sub-data is determined based on the image classification results of the image data to be identified included in the video sub-data.

14. An electronic device, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the scene label recognition method as described in any one of claims 1-3, or implement the scene label recognition method as described in any one of claims 4-11.

15. A computer-readable medium, characterized in that, It stores a computer program, wherein when the program is executed by a processor, it implements the scene label recognition method as described in any one of claims 1-3, or implements the scene label recognition method as described in any one of claims 4-11.

Images