Video classification method, device, storage medium and electronic device

Abstract

The present disclosure relates to the field of computer technology, and in particular to a video classification method, device, storage medium and electronic device. The method provided by the embodiment of the present disclosure includes: extracting at least one video frame from a video to be classified, and preprocessing the video frame to obtain a global image; wherein the feature information of the global image includes at least one of environmental features and object features; inputting the global image into a pre-trained environmental feature classification model to obtain an environmental feature classification result based on the environmental features; inputting the global image into a pre-trained object feature classification model to obtain an object feature classification result based on the object features; and obtaining a classification result of the video to be classified based on the fusion of the environmental feature classification result and the object feature classification result. This method significantly improves the accuracy of video classification and can be applied to video classification scenarios involving high-level semantic analysis such as social relationships between characters.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video classification method, a video classification device, a computer-readable storage medium, and an electronic device. Background Art

[0002] Video classification technology generally refers to the use of computer algorithms to categorize a video into predefined categories (such as advertising, sports, news, and film and television) based on its images, audio, and other content. Video classification technology can be used for tasks such as automatic tagging, searching, and content understanding of online videos. Currently, the most accurate classification methods are those based on deep learning models (such as convolutional neural networks and recurrent neural networks). However, existing classification methods based on deep learning models can only learn basic information from videos and are therefore only suitable for general video classification applications. They have difficulty effectively identifying high-level semantic information such as the social relationships between characters in the video, and still have shortcomings in terms of video classification accuracy and effectiveness.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0004] The purpose of the present disclosure is to provide a video classification method, a video classification device, a computer-readable storage medium and an electronic device, thereby overcoming, at least to a certain extent, the technical problem of difficulty in identifying high-level semantic information due to the limitations of related technologies.

[0005] According to one aspect of the present disclosure, a video classification method is provided, the method comprising:

[0006] Extracting at least one video frame from the video to be classified, and preprocessing the video frame to obtain a global image; wherein feature information of the global image includes at least one of an environmental feature and an object feature;

[0007] Inputting the global image into a pre-trained environmental feature classification model to obtain an environmental feature classification result based on the environmental features;

[0008] Inputting the global image into a pre-trained object feature classification model to obtain an object feature classification result based on the object features;

[0009] The classification result of the video to be classified is obtained by fusing the environmental feature classification result and the object feature classification result.

[0010] In some exemplary embodiments of the present disclosure, the object feature classification model includes an object detection model, a feature extraction model, and an object classification model;

[0011] Inputting the global image into a pre-trained object feature classification model to obtain an object feature classification result based on object features includes:

[0012] Inputting the global image into the pre-trained object detection model to obtain a local image corresponding to the physical object;

[0013] Inputting the partial images into the pre-trained feature extraction model to obtain object features of each of the partial images;

[0014] determining object relationship information between the partial images according to the object features and image source information of the partial images;

[0015] The object features and the object relationship information are input into the pre-trained object classification model to obtain an object feature classification result.

[0016] In some exemplary embodiments of the present disclosure, determining the object relationship information between the partial images based on the object features and image source information of the partial images includes:

[0017] Combining the partial images to obtain a relational graph model for representing the object relationship between the partial images;

[0018] An adjacency matrix corresponding to the relationship graph model is determined according to the object features and image source information of each of the local images, and the adjacency matrix is ​​used as the object relationship information.

[0019] In some exemplary embodiments of the present disclosure, determining the adjacency matrix corresponding to the relationship graph model based on the object features and image source information of each of the partial images includes:

[0020] Calculating feature similarity of object features of any two of the partial images;

[0021] Determining the relationship feature values ​​between any two of the partial images according to the feature similarity and the image source information according to a preset rule;

[0022] The relationship eigenvalues ​​are combined to obtain an adjacency matrix corresponding to the relationship graph model.

[0023] In some exemplary embodiments of the present disclosure, the feature similarity is the Euclidean distance or the cosine distance between feature vectors.

[0024] In some exemplary embodiments of the present disclosure, the entity object includes a primary entity object and a secondary entity object, and the partial image includes a primary object partial image corresponding to the primary entity object and a secondary object partial image corresponding to the secondary entity object;

[0025] The determining of the object relationship information between the partial images according to the object features and image source information of the partial images includes:

[0026] determining main object relationship information between the main object partial images according to the object features and image source information of the main object partial images;

[0027] Secondary object relationship information between the primary object partial image and the secondary object partial image is determined according to image source information of the primary object partial image and the secondary object partial image.

[0028] In some exemplary embodiments of the present disclosure, the object feature classification result includes a primary object feature classification result corresponding to the primary entity object and a secondary object feature classification result corresponding to the secondary entity object;

[0029] Inputting the object features and the object relationship information into the pre-trained object classification model to obtain an object feature classification result includes:

[0030] Inputting the object features of the main object partial image and the main object relationship information into the pre-trained object classification model to obtain the main object feature classification result;

[0031] The object features of the primary object partial image, the object features of the secondary object partial image, and the secondary object relationship information are input into the pre-trained object classification model to obtain the secondary object feature classification result.

[0032] In some exemplary embodiments of the present disclosure, the primary object relationship information includes same-body object relationship information for representing object relationships between the same primary entity objects and different-body object relationship information for representing object relationships between different primary entity objects;

[0033] The determining of the main object relationship information between the main object partial images according to the object features and image source information of the main object partial images includes:

[0034] Calculating feature similarity of object features of any two partial images of the main object;

[0035] Determining, according to the feature similarity, same-body relationship feature values ​​of the two partial images of the main object in accordance with a first preset rule, and determining the same-body object relationship information based on the same-body relationship feature values;

[0036] According to the feature similarity and the image source information, the heteromorphic relationship feature value of the two partial images of the main object is determined according to a second preset rule, and the heteromorphic object relationship information is determined based on the heteromorphic relationship feature value.

[0037] In some exemplary embodiments of the present disclosure, determining the same-body relationship feature value of the two partial images of the main object according to the first preset rule includes:

[0038] If the feature similarity is less than a first preset threshold, taking the first feature value as a feature value of the same-body relationship between the two partial images of the main object;

[0039] If the feature similarity is greater than or equal to the first preset threshold, the second feature value is used as the same-body relationship feature value.

[0040] In some exemplary embodiments of the present disclosure, determining the heteromorphic relationship feature value of the two partial images of the main object according to the second preset rule includes:

[0041] If the two partial images of the main object are derived from the same global image, the third eigenvalue is used as a eigenvalue of the heteromorphic relationship between the two partial images of the main object;

[0042] If the two partial images of the main object are derived from different global images and the feature similarity is greater than or equal to a second preset threshold, using the third feature value as the heteromorphic relationship feature value;

[0043] If the two partial images of the main object are derived from different global images and the feature similarity is less than the second preset threshold, the fourth feature value is used as the heteromorphic relationship feature value.

[0044] In some exemplary embodiments of the present disclosure, determining the secondary object relationship information between the primary object partial image and the secondary object partial image based on the image source information of the primary object partial image and the secondary object partial image includes:

[0045] According to the image source information of the main object partial image and the secondary object partial image, a secondary relationship feature value is determined according to a third preset rule, and based on the secondary relationship feature value, the secondary object relationship information between the main object partial image and the secondary object partial image is determined.

[0046] In some exemplary embodiments of the present disclosure, determining the secondary relationship feature value according to the third preset rule includes:

[0047] If the primary object partial image and the secondary object partial image are derived from the same global image, the fifth eigenvalue is used as the secondary relationship eigenvalue;

[0048] If the primary object partial image and the secondary object partial image are derived from different global images, the sixth eigenvalue is used as the secondary relationship eigenvalue.

[0049] In some exemplary embodiments of the present disclosure, inputting the object features and the object relationship information into the pre-trained object classification model to obtain an object feature classification result includes:

[0050] Combining the object features of each of the partial images according to the temporal relationship of the partial images in the video to be classified to obtain an object combination feature;

[0051] The object combination features and the object relationship information are input into the pre-trained object classification model to obtain an object feature classification result.

[0052] In some exemplary embodiments of the present disclosure, after inputting the global image into the pre-trained object detection model to obtain a local image corresponding to the physical object, the method further includes:

[0053] Obtaining a detection confidence of each of the local images obtained by the object detection model;

[0054] The partial images are screened according to the detection confidence to obtain a preset number of the partial images.

[0055] In some exemplary embodiments of the present disclosure, preprocessing the video frame to obtain a global image includes:

[0056] The video frame is scaled using a bilinear interpolation algorithm to obtain a global image with a preset height and a preset width.

[0057] In some exemplary embodiments of the present disclosure, obtaining a classification result of the video to be classified based on the fusion of the environmental feature classification result and the object feature classification result includes:

[0058] The environment feature classification result and the object feature classification result are weighted and summed according to a preset weighting coefficient to obtain a classification result of the video to be classified.

[0059] According to one aspect of the present disclosure, a video classification device is provided, the device comprising:

[0060] an image extraction module configured to extract at least one video frame from the video to be classified and preprocess the video frame to obtain a global image; wherein feature information of the global image includes at least one of environmental features and object features;

[0061] an environment classification module configured to input the global image into a pre-trained environment feature classification model to obtain an environment feature classification result based on the environment feature;

[0062] an object classification module configured to input the global image into a pre-trained object feature classification model to obtain an object feature classification result based on the object features;

[0063] The classification fusion module is configured to obtain a classification result of the video to be classified by fusing the environmental feature classification result and the object feature classification result.

[0064] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. The particularity of the computer program is that when the computer program is executed by a processor, any of the above-mentioned video classification methods is implemented.

[0065] According to one aspect of the present disclosure, an electronic device is provided, which is special in that it includes a processor and a memory; wherein the memory is used to store executable instructions of the processor, and the processor is configured to perform any of the above-mentioned video classification methods by executing the executable instructions.

[0066] In the video classification method, video classification device, computer-readable storage medium, and electronic device provided in this exemplary embodiment, two parallel processing sub-processes are used to extract environmental and object features from the video content of the video to be classified, and the classification results based on these features are then fused. By combining global feature information corresponding to environmental features with detailed feature information corresponding to object features, the system effectively utilizes the rich spatial visual information and dynamic temporal information in the video data, significantly improving the accuracy of video classification. This system is applicable to video classification scenarios involving advanced semantic analysis such as character social relationships.

[0067] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0069] Figure 1 The following schematically illustrates a flowchart of steps of a video classification method in an exemplary embodiment of the present disclosure.

[0070] Figure 2 A flowchart schematically illustrates the steps of obtaining object feature classification results in another exemplary embodiment of the present disclosure.

[0071] Figure 3 A flowchart schematically illustrates the steps of obtaining object feature classification results in another exemplary embodiment of the present disclosure.

[0072] Figure 4 A flowchart schematically illustrates the steps of obtaining a partial image in another exemplary embodiment of the present disclosure.

[0073] Figure 5 The following schematically illustrates a flowchart of steps for obtaining object relationship information in another exemplary embodiment of the present disclosure.

[0074] Figure 6 A flowchart schematically illustrates the steps of obtaining an adjacency matrix in another exemplary embodiment of the present disclosure.

[0075] Figure 7 The following schematically illustrates a flowchart of steps for obtaining object relationship information in another exemplary embodiment of the present disclosure.

[0076] Figure 8 A flowchart schematically illustrates the steps of obtaining object feature classification results in another exemplary embodiment of the present disclosure.

[0077] Figure 9 The following schematically illustrates a flowchart of steps for obtaining object relationship information in another exemplary embodiment of the present disclosure.

[0078] Figure 10 A flowchart schematically illustrates the steps of a video classification method provided by an exemplary embodiment of the present disclosure in an application scenario.

[0079] Figure 11 The following schematically shows a block diagram of a video classification device in an exemplary embodiment of the present disclosure.

[0080] Figure 12 A schematic diagram schematically illustrates a program product in an exemplary embodiment of the present disclosure.

[0081] Figure 13 The module diagram of an electronic device in an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0082] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0083] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0084] In an exemplary embodiment of the present disclosure, a video classification method is first provided. This method primarily utilizes a computer video processing algorithm to extract and learn image features from a video to achieve video classification. This method is particularly suitable for identifying and classifying social relationships (e.g., friends, colleagues, relatives, etc.) or other high-level semantic information involved in a video.

[0085] like Figure 1 As shown, the video classification method provided by this exemplary embodiment may mainly include the following steps:

[0086] Step S110. Extract at least one video frame from the video to be classified, and preprocess the video frame to obtain a global image; wherein the feature information of the global image includes at least one of an environmental feature and an object feature.

[0087] The video to be classified can be a complete video or a video clip with a fixed number of frames or time length. For a video to be classified of a certain length, this step first extracts a certain number of video frames from the video. The method of extracting video frames from the video to be classified can be, for example, uniform extraction according to a preset number of frame intervals, or random extraction from all video frames that make up the video to be classified, or extraction according to specific rules. For example, this step can refer to the video coding standard to extract intra-frame coded frames (Intra-Prediction, I frames) in the video to be classified as key frames for obtaining the global image. In addition, any other key frame extraction method can be used to extract key frames from the video to be classified. By extracting key frames, some interference information in the video to be classified can be removed, while reducing the computational complexity of subsequent video classification processing and retaining the representative key information in the video to be classified to the greatest extent.

[0088] This step preprocesses the extracted video frames to obtain a global image. For example, the preprocessing method may be to scale the video frames using a bilinear interpolation algorithm to obtain a global image with a preset height and width. This preprocessing yields a normalized global image, making the video classification method provided in this exemplary embodiment applicable to videos of various formats and sizes.

[0089] The global image obtained after preprocessing generally includes multiple types of feature information, and each feature information of the global image can include at least one of environmental features and object features. Environmental features primarily include environmental information related to the image content, and can be expressed, for example, as global feature information that can characterize various scenes or environments such as "outdoors," "conference room," and "restaurant." Object features primarily include information about physical objects such as people, animals, and objects involved in the image content, and can be expressed, for example, as detailed feature information that can characterize various object details such as person appearance, object type, interactions between people, and interactions between people and objects.

[0090] Step S120: Input the global image into a pre-trained environmental feature classification model to obtain an environmental feature classification result based on the environmental features.

[0091] The global image obtained by step S110 can be used as an input parameter, and the pre-trained environmental feature classification model is used to analyze and process it to obtain the environmental feature classification result. Among them, the environmental feature classification model mainly focuses on feature extraction of environmental features in the global image, and performs classification processing based on the extracted environmental features according to a plurality of pre-labeled classification labels. The environmental feature classification model used in this step can be, for example, a machine learning model trained based on a temporal segmentation network (TSN). The environmental feature classification result output by the environmental feature classification model can be expressed as a score of various different category entries, which can reflect the probability of the video to be classified being classified into different category entries. Generally speaking, the higher the score, the higher the possibility that the video to be classified is classified into the category entry.

[0092] Step S130: Input the global image into a pre-trained object feature classification model to obtain an object feature classification result based on object features.

[0093] Similar to step S120, this step uses the global image as an input parameter and analyzes and processes it using a pre-trained object feature classification model to obtain object feature classification results. The object feature classification model primarily focuses on extracting object features from the global image and classifying them according to multiple pre-labeled classification labels based on the extracted object features. Because object features have more complex representations and deeper meaning than environmental features, the object feature classification model used in this step can be a machine learning model trained using neural networks such as convolutional neural networks (CNNs) and recursive neural networks (RNNs), or a combined model integrated from multiple sub-models of different types or structures. Furthermore, this step can also classify the global image according to pre-set segmentation criteria, thereby obtaining multiple object feature classification results corresponding to different feature types. Therefore, the object feature classification results obtained in this step can be single-dimensional classification results or a combination of detailed classification results from multiple dimensions. Furthermore, the detailed classification results from different dimensions can be different outputs from the same model or from different sub-models.

[0094] The object feature classification results output by the object feature classification model can also be expressed as scores of various different category entries, and the object feature classification results obtained in this step and the environmental feature classification results obtained in step S130 can use the same or at least partially the same category entries to improve the fusibility of the classification results.

[0095] Step S140: Obtain a classification result of the video to be classified based on the fusion of the environmental feature classification result and the object feature classification result.

[0096] In this exemplary embodiment, the processing of the environmental feature classification model in step S120 and the processing of the object feature classification model in step S130 can be performed in parallel, and the environmental feature classification results and the object feature classification results obtained by processing are fused in this step to obtain the classification result of the video to be classified. Uniform blending, linear blending or any other fusion method can be used in this step, and this exemplary embodiment does not specifically limit this. For example, this step can be to perform a weighted summation of the environmental feature classification results and the object feature classification results according to a preset weighting coefficient to obtain the classification result of the video to be classified. The weighted method can be used to adjust the degree of influence of the environmental features and object features in the final classification result, and the corresponding weighting coefficient can be used as a hyperparameter to adaptively adjust for different videos to be classified to obtain a more accurate classification result.

[0097] The video classification method provided in this exemplary embodiment uses two parallel processing sub-processes to extract environmental and object features from the video content to be classified, and then fuses the classification results based on these features. This method combines global feature information corresponding to environmental characteristics with detailed feature information corresponding to object characteristics, effectively utilizing the rich spatial visual information and dynamic temporal information in video data, significantly improving the accuracy of video classification. It is applicable to video classification scenarios involving advanced semantic analysis such as character social relationships.

[0098] Based on the above exemplary embodiments, the object feature classification model used in step S130 may further include sub-models such as an object detection model, a feature extraction model, and an object classification model. The sub-models are connected to each other in series, and the output of the previous sub-model serves as the input of the next sub-model to finally obtain the object feature classification result.

[0099] like Figure 2 As shown, step S130 in this exemplary embodiment, inputting the global image into a pre-trained object feature classification model to obtain an object feature classification result based on the object feature, may further include the following steps:

[0100] Step S210: Input the global image into a pre-trained object detection model to obtain a local image corresponding to the physical object.

[0101] This step first inputs the global image corresponding to each frame in the video to be classified into a pre-trained object detection model. The object detection model then outputs local images corresponding to the physical objects. A global image typically contains physical objects of varying types and numbers, such as people, animals, and objects. This step utilizes the object detection model to identify and detect the physical objects in the global image and, based on the detection results, obtain local images corresponding to the physical objects. For example, if a global image depicts people A and B conversing at a desk, the object detection model can detect and identify local images a corresponding to person A, b corresponding to person B, and c corresponding to the desk. For another example, if a global image depicts person A using a laptop at the same desk, the object detection model can detect and identify local images d corresponding to person A, e corresponding to the same desk, and f corresponding to the laptop. The object detection model used in this step can be, for example, R-CNN (Region Convolutional Neural Network), Fast R-CNN, Faster R-CNN, Mask R-CNN, or any other machine learning model that can achieve target detection.

[0102] Step S220: Input the local images into a pre-trained feature extraction model to obtain object features of each local image.

[0103] Through step S210, all global images can be subjected to entity object recognition detection to obtain local images corresponding to each entity object. In this step, the local images obtained in step S210 are input into a pre-trained feature extraction model, and the feature extraction model performs feature extraction on each local image to obtain object features of each local image. The object features of each local image can be represented as a feature vector, and the local images obtained from different global images generally have different object features. In addition, even if two local images correspond to the same entity object, such as the local image a and the local image d corresponding to person A, they usually show certain differences in object features. The feature extraction model used in this step can be, for example, ResNet (Residual Neural Network) or any other machine learning model that can realize image feature extraction.

[0104] Step S230: Determine object relationship information between the partial images based on the object features of the partial images and the image source information.

[0105] Based on the object features of each local image and the image source information of each local image obtained in step S220, this step can determine the object relationship information between each local image. The object features of the local image can reflect the spatial visual information of the entity object in the video to be classified to a certain extent, and thus can reflect the object relationship between different local images in a spatial sense. The image source information of the local image mainly refers to the relationship between the local image and the global image. For example, if a global image can obtain three local images after being processed by the object detection model in step S210, then it can be considered that these three local images are derived from the same global image, that is, they have the same image source. Otherwise, they can be considered to have different image sources. The image source information of the local image can reflect the dynamic timing information of the entity object in the video to be classified to a certain extent, and thus can reflect the object relationship between different local images in a temporal sense.

[0106] Step S240: Input the object features and object relationship information into a pre-trained object classification model to obtain an object feature classification result.

[0107] The object features of each partial image obtained in step S220 and the object relationship information between the partial images obtained in step S230 are combined as input parameters and fed into a pre-trained object classification model. The object classification model then performs classification processing to obtain an object feature classification result. Because the object feature and object relationship information are included, the object feature classification result obtained in this step can effectively reflect the inherent relationships between physical objects in the video to be classified, and is particularly suitable for identifying and analyzing the social relationships between people.

[0108] In the video classification method provided in this exemplary embodiment, the complex object feature analysis process can be decomposed into multiple sub-models for implementation by connecting multiple sub-models in series. Each sub-model can also be trained using its own suitable basic model and learning algorithm, which not only improves the training efficiency of the overall model, but also obtains a more optimized object feature classification result based on object features.

[0109] like Figure 3 As shown, in another exemplary embodiment of the present disclosure, step S240: inputting object features and object relationship information into a pre-trained object classification model to obtain an object feature classification result may include the following steps:

[0110] Step S310: Combining the object features of each partial image according to the temporal relationship of the partial images in the video to be classified to obtain an object combination feature.

[0111] Since each local image originates from the same or different global image, and the global image is obtained by processing video frames extracted from the video to be classified, each local image naturally carries the timing information corresponding to each video frame in the video to be classified. In this step, the object features of each local image are combined according to the timing relationship in the video to be classified to obtain the object combination feature. When the object features of the local images are represented as feature vectors, the object combination feature can be correspondingly represented as a feature matrix formed by arranging the feature vectors in chronological order.

[0112] Step S320: Input the object combination features and object relationship information into a pre-trained object classification model to obtain an object feature classification result.

[0113] Based on the object combination features obtained in step S310, this step inputs them and the object relationship information as input parameters into a pre-trained object classification model to obtain an object feature classification result through processing by the object classification model.

[0114] In the video classification method provided in this exemplary embodiment, by combining the object features of each local image according to a temporal relationship to form an object combination feature, time information can be further incorporated into the analysis and processing process of the object classification model in addition to spatial information, thereby improving the accuracy of the classification results of the object classification model.

[0115] like Figure 4 As shown, based on the above exemplary embodiment, after step S210 of inputting the global image into a pre-trained object detection model to obtain a local image corresponding to the physical object, the following steps may also be included:

[0116] Step S410: Obtain the detection confidence of each local image obtained by the object detection model.

[0117] By processing the global image using the object detection model in step S210, local images corresponding to the physical objects are obtained. The detection confidence level for each local image is also obtained. This confidence level can be used to evaluate the accuracy of the recognition and detection of each local image. A higher detection confidence level indicates a higher degree of confidence in the physical objects detected in the local images, i.e., a higher degree of accuracy in the recognition and detection results.

[0118] Step S420: Screen the partial images according to the detection confidence to obtain a preset number of partial images.

[0119] Based on the detection confidence obtained in step S410, this step can filter the local images. The filtering method can be, for example, to sort the local images in descending order of detection confidence, and then retain a preset number of local images with the highest detection confidence, while discarding other local images with lower detection confidence. In addition, the filtering method can also be to retain local images with detection confidence higher than a preset threshold, and discard local images with detection confidence lower than the preset threshold. Regardless of the filtering method used, this step can obtain a certain number of local images with high detection confidence.

[0120] In this exemplary embodiment, by using the detection confidence to filter the partial images, it is possible to balance the accuracy and efficiency of video classification to a certain extent. Of course, in other exemplary embodiments, all partial images may be input into the feature extraction model of step S220 for subsequent analysis and processing without filtering, and this disclosure does not specifically limit this.

[0121] like Figure 5 As shown, in another exemplary embodiment of the present disclosure, step S230 of determining object relationship information between the partial images based on the object features and image source information of the partial images may include the following steps:

[0122] Step S510: Combine the partial images to obtain a relational graph model for representing the object relationship between the partial images.

[0123] With each local image as a node, this step can construct a relationship graph model consisting of each node and the connection relationship between the nodes. The relationship graph model can intuitively and accurately represent the object relationship between each local image.

[0124] Step S520: Determine the adjacency matrix corresponding to the relationship graph model according to the object features of each local image and the image source information, and use the adjacency matrix as the object relationship information.

[0125] This step determines the corresponding adjacency matrix for the relationship graph model constructed in step S510. Each row and column of the adjacency matrix corresponds to a local image. For example, if there are 20 local images, the adjacency matrix can be a 20*20 matrix. The value of each element in the adjacency matrix is ​​determined by the object characteristics of each local image and the image source information.

[0126] The method of constructing a graph model and forming an adjacency matrix can intuitively and accurately represent the object relationship between any two local images. The object relationship information in matrix form is also conducive to subsequent model processing.

[0127] like Figure 6 As shown, based on the above exemplary embodiment, determining the adjacency matrix corresponding to the relationship graph model according to the object features and image source information of each local image in step S520 may include the following steps:

[0128] Step S610: Calculate the feature similarity of the object features of any two partial images.

[0129] This step calculates the object features of each local image to obtain the feature similarity between each object feature. For all local images, a feature similarity can be calculated for the object features of every two local images. When the object features of the local images are represented as feature vectors, the feature similarity can be measured by the vector distance, for example, it can be the Euclidean distance, cosine distance or any other vector similarity representation method between the feature vectors. It should be noted that this step can calculate the feature similarity for all local images. In addition, the local images can be divided into types and the feature similarity can be calculated for some local images in a targeted manner. This exemplary embodiment does not specifically limit this.

[0130] Step S620: Determine the relationship feature value between any two partial images according to the feature similarity and the image source information and the preset rules.

[0131] Based on the feature similarity calculated in step S610 and the image source information of each partial image obtained in advance, this step determines the relationship feature value of the two partial images according to the preset rules. For example, when the feature similarity of the object features of the two partial images exceeds a certain threshold, it can be considered that the entity objects corresponding to the two partial images have a high similarity, and thus the relationship feature value can be determined as a certain set value; and if the feature similarity of the object features of the two partial images does not exceed the threshold, it can be considered that the entity objects corresponding to the two partial images have a relatively low similarity, and thus the relationship feature value between the two can be determined as another set value. For another example, when the two partial images are derived from the same global image, the relationship feature value between the two can be determined as a certain set value; and when the two partial images are derived from different global images, the relationship feature value between the two can be determined as another set value. Of course, depending on the preset rules, there can be multiple ways to determine the relationship feature values ​​in this step. In particular, a combination of multiple determination methods can be used according to the type classification of local images. For example, for a part of the local images, the relationship feature values ​​can be determined solely by using feature similarity, for a part of the local images, the relationship feature values ​​can be determined solely by using image source information, and then for a part of the local images, the relationship feature values ​​can be determined jointly by using feature similarity and image source information. This exemplary embodiment does not make any special limitations on this.

[0132] Step S630: Combine the relationship eigenvalues ​​to obtain an adjacency matrix corresponding to the relationship graph model.

[0133] After the relationship eigenvalues ​​are determined by step S620, this step can combine the various relationship eigenvalues ​​as matrix elements to obtain an adjacency matrix corresponding to the relationship graph model, where each row and column in the adjacency matrix represents the relationship eigenvalues ​​determined between a local image and other local images.

[0134] This exemplary embodiment can form a unified representation form for the object relationship information between each local image by first determining the relationship eigenvalue and then forming an adjacency matrix. In particular, for feature similarities with continuous value characteristics, they can be mapped into relationship eigenvalues ​​with discretization and normalization characteristics. This can not only reduce the processing difficulty and improve the processing efficiency of the subsequent classification process, but also improve the overall stability and universal applicability of the method.

[0135] In another exemplary embodiment of the present disclosure, step S210 utilizes an object detection model to identify and detect multiple different types of entity objects from a global image, such as primary entity objects and secondary entity objects. Accordingly, the local image output by the object detection model may include a primary object local image corresponding to the primary entity object and a secondary object local image corresponding to the secondary entity object. For example, the primary entity object may be a person appearing in the global image, and the corresponding primary object local image is a person image; the secondary entity object may be an object appearing in the global image, and the corresponding secondary object local image is an object image. Furthermore, in other embodiments, entity objects may be divided into more types according to different rules. It should be noted that the terms "primary" and "secondary" in this exemplary embodiment merely represent the classification of different entity objects and local images, and do not define the primary and secondary relationships between various entity objects and various local images.

[0136] like Figure 7 As shown, when the entity object includes a primary entity object and a secondary entity object and the partial image includes a primary object partial image and a secondary object partial image, step S230 of determining object relationship information between the partial images based on the object features and image source information of the partial images may include the following steps:

[0137] Step S710: Determine the main object relationship information between the main object partial images according to the object features and image source information of the main object partial images.

[0138] For the partial images of the main object, this step determines the main object relationship information between any two partial images of the main object based on their object features and image sources. For example, for some partial images of the main object, the main object relationship is determined by calculating feature similarity using object features, while for another partial image of the main object, the main object relationship information is determined based on the feature similarity calculated and the image source information.

[0139] Step S720: Determine secondary object relationship information between the primary object partial image and the secondary object partial image according to the image source information of the primary object partial image and the secondary object partial image.

[0140] For the secondary object partial images, this step determines the secondary object relationship information based on the image source information of the primary object partial images and the secondary object partial images. For example, when a primary object partial image and a secondary object partial image originate from the same global image, a relationship feature value may be determined as the secondary object relationship information between the two; whereas, when the two originate from different global images, another relationship feature value may be determined as the secondary object relationship information between the two.

[0141] Based on this exemplary embodiment, when entity objects are divided into different categories, different object feature classification results will be obtained accordingly. For example, the object feature classification results include primary object feature classification results corresponding to primary entity objects and secondary object feature classification results corresponding to secondary entity objects.

[0142] like Figure 8 As shown, step S240. Inputting the object features and object relationship information into a pre-trained object classification model to obtain an object feature classification result may include the following steps:

[0143] Step S810: Input the object features of the main object partial image and the main object relationship information into a pre-trained object classification model to obtain a main object feature classification result.

[0144] In this step, the object features of the primary object partial image and the primary object relationship information determined in step S710 are input into the pre-trained object classification model to obtain a primary object feature classification result. The primary object feature classification result focuses on the object features of the primary object partial image as the relatively primary classification basis, while excluding the secondary object partial images.

[0145] Step S820: Input the object features of the primary object partial image, the object features of the secondary object partial image, and the secondary object relationship information into a pre-trained object classification model to obtain a secondary object feature classification result.

[0146] For the secondary object partial image, this step inputs the object features of the primary object partial image, the object features of the secondary object partial image, and the secondary object relationship information into the same object classification model to obtain the secondary object feature classification result. The secondary object feature classification result focuses on the object relationship between the primary and secondary object partial images, while combining the object features of each as the primary classification basis.

[0147] This exemplary embodiment classifies entity objects into types, and can selectively and specifically adopt different analysis and processing strategies for different types of entity objects, ultimately obtaining multiple classification results. The multiple classification results can then be fused, which is conducive to mining deep information of entity objects to obtain more accurate classification results.

[0148] Based on the above exemplary embodiments, the main object relationship information may further include same-body object relationship information for representing object relationships between the same main entity objects and different-body object relationship information for representing object relationships between different main entity objects.

[0149] like Figure 9 As shown, step S710. determining the main object relationship information between the main object partial images according to the object features and image source information of the main object partial images may include the following steps:

[0150] Step S910: Calculate the feature similarity of the object features of any two partial images of the main object.

[0151] Before determining the object relationship information, this step first calculates the feature similarity of the object features between the partial images of each main object. The specific calculation method can refer to the method steps in the above exemplary embodiments and will not be repeated here.

[0152] Step S920: Determine the same-body relationship feature values ​​of the two main object partial images according to the feature similarity and the first preset rule, and determine the same-body object relationship information based on the same-body relationship feature values.

[0153] Based on the feature similarity calculated in step S910, this step can determine, according to a first preset rule, a same-body relationship feature value used to characterize the object relationship between the same primary entity objects, and determine corresponding same-body object relationship information based on the same-body relationship feature value. The method for determining the same-body relationship feature value using the first preset rule may, for example, include: if the feature similarity is less than a first preset threshold, using the first feature value as the same-body relationship feature value for the partial images of the two primary objects; if the feature similarity is greater than or equal to the first preset threshold, using the second feature value as the same-body relationship feature value. For example, if the first feature value is 1 and the second feature value is 0, the same-body object relationship information obtained can be represented as a matrix consisting of two elements, 1 and 0.

[0154] Step S930: Determine the heteromorphic relationship feature values ​​of the two main object partial images according to the feature similarity and the image source information and in accordance with a second preset rule, and determine heteromorphic object relationship information based on the heteromorphic relationship feature values.

[0155] Based on the feature similarity calculated in step S910 and the image source information of each primary object partial image, this step can determine, according to a second preset rule, a heteromorphic relationship feature value used to characterize the object relationship between different primary entity objects, and determine heteromorphic object relationship information based on the heteromorphic relationship feature value. The method for determining the heteromorphic relationship feature value using the second preset rule may, for example, include: if the two primary object partial images originate from the same global image, using a third feature value as the heteromorphic relationship feature value for the two primary object partial images; if the two primary object partial images originate from different global images and the feature similarity is greater than or equal to a second preset threshold, using the third feature value as the heteromorphic relationship feature value; and if the two primary object partial images originate from different global images and the feature similarity is less than the second preset threshold, using a fourth feature value as the heteromorphic relationship feature value. For example, if the third feature value is 1 and the fourth feature value is 0, the heteromorphic object relationship information thus obtained can be represented as a matrix consisting of two elements, 1 and 0. In some exemplary embodiments, the heteromorphic object relationship information can also be represented as a matrix consisting of any other feature values ​​as elements, and the corresponding element values ​​can be different from the homomorphic object relationship information, which is not specifically limited in this disclosure.

[0156] Based on the above exemplary embodiments, step S720. determining the secondary object relationship information between the primary object partial image and the secondary object partial image according to the image source information of the primary object partial image and the secondary object partial image may include: according to the image source information of the primary object partial image and the secondary object partial image, determining the secondary relationship feature value according to the third preset rule, and determining the secondary object relationship information between the primary object partial image and the secondary object partial image based on the secondary relationship feature value. Among them, the method for determining the secondary relationship feature value using the third preset rule may include, for example: if the primary object partial image and the secondary object partial image are from the same global image, taking the fifth feature value as the secondary relationship feature value; if the primary object partial image and the secondary object partial image are from different global images, taking the sixth feature value as the secondary relationship feature value. For example, the fifth feature value is 1 and the sixth feature value is 0, and the secondary object relationship information obtained therefrom can be represented as a matrix composed of two elements 1 and 0. In some exemplary embodiments, the secondary object relationship information may also be represented as a matrix composed of other arbitrary feature values as elements, and the corresponding element values may be different from the same-body object relationship information and the different-body object relationship information, and the present disclosure does not make special limitations on this.

[0157] The following uses an application scenario to detail the video classification method provided in the above exemplary embodiments.

[0158] This application scenario involves using computer algorithms to extract and learn image features in a video to achieve the purpose of identifying the social relationships (such as colleagues, friends, relatives, etc.) of people in the video. The video classification method used in this application scenario first performs preprocessing operations such as key frame extraction and normalization on an input video. Then, through two parallel processing sub-processes, the environmental features related to the global and the object features related to the details are respectively extracted. Finally, the environmental features and the object features are fused to classify the social relationships of the people in the video.

[0159] Refer to Figure 10 As shown in the step flow chart, the video classification method used in this application scenario mainly may include the following flow steps:

[0160] 1) Video key frame extraction

[0161] First, for an input video segment V with a length of F frames, K frames (K < F) are evenly extracted as key frames.

[0162] Then, the K key frames are scaled into images with a width of W and a height of H using the bilinear interpolation algorithm for subsequent processing.

[0163] 2) Global feature extraction and classification

[0164] First, we use video data with labeled social relationships to train an environmental feature classification model. We use the Time Series Segmentation Network (TSN) as an example to illustrate this.

[0165] Second, the K key frames in step 1) are input into the trained TSN network for forward propagation to obtain the global classification score s of the input video V. g :

[0166] s g ={a i} C

[0167] Where C is the number of social relationship categories, a i The scores for each social relationship category.

[0168] 3) Object Detection

[0169] Use the pre-trained object detection model to detect physical objects in all video frames. Physical objects here include people and objects. Sort all detected people and objects by detection confidence from large to small, and retain the P person images and O object images with the highest confidence.

[0170] 4) Object feature extraction

[0171] Use the pre-trained feature extraction model to extract feature vectors {f1,…,f P} and O object images to extract feature vectors {g1,…,g O}.

[0172] 5) Graph model construction

[0173] Construct the adjacency matrix A represented as s The intra-person graph (IntraG) is represented by the adjacency matrix A d The inter-person graph (InterG) and the adjacency matrix A o There are three graph models of Person-Object Graph (POG). s and A d A is a P×P matrix, where each row and column corresponds to each character; o It is a (P+O)×(P+O) matrix, where each row and column corresponds to each person and item.

[0174] Calculate the Euclidean distance dist(f between the feature vectors of all characters i ,f j).

[0175] For the fan relationship graph, if dist(f i ,f j ) is less than a threshold τ, then let A s (i,j)=1, otherwise let A s (i,j)=0.

[0176] For the heterosexual relationship graph, if two characters come from the same video frame, let A d (i,j)=1; if the two characters are from different key frames and dist(f i ,f j ) is greater than or equal to the threshold τ, then let A d (i,j)=1; if the two characters are from different key frames and dist(f i ,f j ) is less than the threshold τ, then let A d (i,j)=0.

[0177] For the character relationship diagram, if a character p k and an object o l From the same key frame, let A o (p k ,o l )=1, otherwise let A o (p k ,o l )=0.

[0178] 6) Visual Relationship Reasoning

[0179] All character feature vectors obtained in step 4) are combined into a feature matrix X in the order of time sequence p , the character feature vector and object feature vector obtained in step 4) are combined into a feature matrix X in the order of time sequence o .

[0180] Use pre-trained object classification models to perform visual relationship reasoning, using graph neural networks as an example.

[0181] Assuming that the pre-trained graph neural network contains L layers, the network parameters of each layer are W l , each layer of relational reasoning can be expressed as:

[0182]

[0183] in is the critical matrix The degree matrix, X (l) is the feature matrix, and σ is a nonlinear activation function, such as the ReLU function.

[0184] Take the character feature matrix X p and object feature matrix X o is the initial feature matrix of the 0th layer, and the feature matrix from the lth layer to the l+1th layer is calculated using the above formula.

[0185] After a total of L layers of calculation, the predicted feature matrix is ​​obtained and By averaging the two feature matrices with the sample dimension, we can get the feature vectors and

[0186] 7) Relationship feature classification

[0187] The inferred character traits and object features Input the feature classifier respectively and output the relationship classification score s p and s o :

[0188] s p ={b i} C

[0189] s o ={c i} C

[0190] 8) Classification result fusion

[0191] The three sets of relations s obtained in step 2) and step 7) are g 、s p and s o The final recognition score is obtained by weighted summation according to the category:

[0192] s={α·a i +β·b i +γ·c i} C

[0193] The category with the highest score among the C categories is taken as the output result.

[0194] Based on the classification process in this application scenario, the video classification method provided by the exemplary embodiments of this disclosure can establish visual relationships between people and objects by extracting key frames from the video. Through a series of model analysis and processing, it can quickly and accurately identify the social relationships between people in the video. This method can be used in a variety of application scenarios, including video classification, video understanding, personalized recommendations, and automatic organization and cataloging of videos for mobile users.

[0195] It should be noted that although the above exemplary embodiments describe the various steps of the method of the present disclosure in a specific order, this does not require or imply that the steps must be performed in this specific order, or that all steps must be performed to achieve the desired results. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0196] In an exemplary embodiment of the present disclosure, a video classification device corresponding to the video classification method in the above exemplary embodiment is also provided. Figure 11 As shown, the video classification device 1100 may mainly include: an image extraction module 1110, an environment classification module 1120, an object classification module 1130, and a classification fusion module 1140. The image extraction module 1110 is configured to extract at least one video frame from the video to be classified and pre-process the video frame to obtain a global image; wherein the feature information of the global image includes at least one of environmental features and object features; the environment classification module 1120 is configured to input the global image into a pre-trained environment feature classification model to obtain an environment feature classification result based on the environment features; the object classification module 1130 is configured to input the global image into a pre-trained object feature classification model to obtain an object feature classification result based on the object features; and the classification fusion module 1140 is configured to fuse the environment feature classification result and the object feature classification result to obtain a classification result for the video to be classified.

[0197] The specific details of the above video classification device have been described in detail in the corresponding video classification method, so they will not be repeated here.

[0198] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0199] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it can implement the above-mentioned video classification method of the present disclosure. In some possible embodiments, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code; the program product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or mobile hard drive) or on a network; when the program product is executed on a computing device (such as a personal computer, server, terminal device, or network device), the program code is used to cause the computing device to execute the method steps of the above-mentioned exemplary embodiments of the present disclosure.

[0200] See also Figure 12 As shown, the program product 1200 for implementing the above method according to an embodiment of the present disclosure can be a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on a computing device (such as a personal computer, server, terminal device or network device, etc.). However, the program product of the present disclosure is not limited to this. In this exemplary embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, device or device.

[0201] The program product may adopt any combination of one or more readable media, which may be readable signal media or readable storage media.

[0202] The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0203] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0204] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0205] The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, as a standalone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device, such as via the Internet using an Internet service provider.

[0206] In an exemplary embodiment of the present disclosure, an electronic device is also provided, comprising at least one processor and at least one memory for storing executable instructions of the processor; wherein the processor is configured to execute the method steps in the above-mentioned exemplary embodiments of the present disclosure by executing the executable instructions.

[0207] The following combination Figure 13 The electronic device 1300 in this exemplary embodiment is described. The electronic device 1300 is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0208] See also Figure 13 As shown, electronic device 1300 is implemented as a general-purpose computing device. Components of electronic device 1300 may include, but are not limited to, at least one processing unit 1310, at least one storage unit 1320, a bus 1330 connecting various system components (including processing unit 1310 and storage unit 1320), and a display unit 1340.

[0209] The storage unit 1320 stores program codes, which can be executed by the processing unit 1310, so that the processing unit 1310 performs the method steps in the above-mentioned exemplary embodiments of the present disclosure.

[0210] The storage unit 1320 may include a readable medium in the form of a volatile memory unit, such as a random access memory unit 1321 (RAM) and / or a cache memory unit 1322 , and may further include a read-only memory unit 1323 (ROM).

[0211] The storage unit 1320 may also include a program / utility 1324 having a set (at least one) of program modules 1325, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0212] The bus 1330 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0213] The electronic device 1300 may also communicate with one or more external devices 1400 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and may also communicate with one or more devices that allow a user to interact with the electronic device 1300, and / or any device that enables the electronic device 1300 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 1350. Furthermore, the electronic device 1300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 1360. Figure 13 As shown, the network adapter 1360 can communicate with other modules of the electronic device 1300 via the bus 1330. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 1300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0214] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0215] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

[0216] The features, structures or characteristics described above can be combined in any suitable manner in one or more embodiments, and if possible, the features discussed in each embodiment are interchangeable. In the above description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, materials, etc. can be adopted. In other cases, known structures, materials or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

Claims

1. A video classification method, characterized in that: include: Extracting at least one video frame from a video to be classified, and preprocessing the video frame to obtain a global image; wherein feature information of the global image includes at least one of an environmental feature and an object feature; the environmental feature is environmental information involved in the image content, and the object feature is physical object information involved in the image content; Inputting the global image into a pre-trained environmental feature classification model to obtain an environmental feature classification result based on the environmental features; the environmental feature classification result includes scores representing different environmental category items; Inputting the global image into a pre-trained object feature classification model to obtain an object feature classification result based on the object features; Obtaining a classification result of the video to be classified according to the fusion of the environmental feature classification result and the object feature classification result; The object feature classification model includes an object detection model, a feature extraction model and an object classification model; Inputting the global image into a pre-trained object feature classification model to obtain an object feature classification result based on object features includes: Inputting the global image into the pre-trained object detection model to obtain a local image corresponding to the physical object; Inputting the partial images into the pre-trained feature extraction model to obtain object features of each of the partial images; determining object relationship information between the partial images according to the object features and image source information of the partial images; Inputting the object features and the object relationship information into the pre-trained object classification model to obtain an object feature classification result; The entity objects include a primary entity object and a secondary entity object, and the partial images include a primary object partial image corresponding to the primary entity object and a secondary object partial image corresponding to the secondary entity object; The determining of the object relationship information between the partial images according to the object features and image source information of the partial images includes: determining main object relationship information between the main object partial images according to the object features and image source information of the main object partial images; Secondary object relationship information between the primary object partial image and the secondary object partial image is determined according to image source information of the primary object partial image and the secondary object partial image.

2. The video classification method according to claim 1, characterized in that The determining of the object relationship information between the partial images according to the object features and image source information of the partial images includes: Combining the partial images to obtain a relational graph model for representing the object relationship between the partial images; An adjacency matrix corresponding to the relationship graph model is determined according to the object features and image source information of each of the local images, and the adjacency matrix is ​​used as the object relationship information.

3. The video classification method according to claim 2, characterized in that The determining of the adjacency matrix corresponding to the relationship graph model according to the object features and image source information of each of the partial images includes: Calculating feature similarity of object features of any two of the partial images; Determining the relationship feature values ​​between any two of the partial images according to the feature similarity and the image source information according to a preset rule; The relationship eigenvalues ​​are combined to obtain an adjacency matrix corresponding to the relationship graph model.

4. The video classification method according to claim 3, characterized in that The feature similarity is the Euclidean distance or cosine distance between feature vectors.

5. The video classification method according to claim 1, characterized in that The object feature classification result includes a primary object feature classification result corresponding to the primary entity object and a secondary object feature classification result corresponding to the secondary entity object; Inputting the object features and the object relationship information into the pre-trained object classification model to obtain an object feature classification result includes: Inputting the object features of the main object partial image and the main object relationship information into the pre-trained object classification model to obtain the main object feature classification result; The object features of the primary object partial image, the object features of the secondary object partial image, and the secondary object relationship information are input into the pre-trained object classification model to obtain the secondary object feature classification result.

6. The video classification method according to claim 1, characterized in that The main object relationship information includes same-body object relationship information for representing object relationships between the same main entity objects and different-body object relationship information for representing object relationships between different main entity objects; The determining of the main object relationship information between the main object partial images according to the object features and image source information of the main object partial images includes: Calculating feature similarity of object features of any two partial images of the main object; Determining, according to the feature similarity, same-body relationship feature values ​​of the two partial images of the main object in accordance with a first preset rule, and determining the same-body object relationship information based on the same-body relationship feature values; According to the feature similarity and the image source information, the heteromorphic relationship feature value of the two partial images of the main object is determined according to a second preset rule, and the heteromorphic object relationship information is determined based on the heteromorphic relationship feature value.

7. The video classification method according to claim 6, characterized in that: The determining of the same-body relationship feature value of the two partial images of the main object according to the first preset rule includes: If the feature similarity is less than a first preset threshold, taking the first feature value as a feature value of the same-body relationship between the two partial images of the main object; If the feature similarity is greater than or equal to the first preset threshold, the second feature value is used as the same-body relationship feature value.

8. The video classification method according to claim 6, characterized in that: The determining of the heteromorphic relationship feature values ​​of the two partial images of the main object according to the second preset rule includes: If the two partial images of the main object are derived from the same global image, the third eigenvalue is used as a eigenvalue of the heteromorphic relationship between the two partial images of the main object; If the two partial images of the main object are derived from different global images and the feature similarity is greater than or equal to a second preset threshold, using the third feature value as the heteromorphic relationship feature value; If the two partial images of the main object are derived from different global images and the feature similarity is less than the second preset threshold, the fourth feature value is used as the heteromorphic relationship feature value.

9. The video classification method according to claim 1, characterized in that: The determining of the secondary object relationship information between the primary object partial image and the secondary object partial image according to the image source information of the primary object partial image and the secondary object partial image includes: According to the image source information of the main object partial image and the secondary object partial image, a secondary relationship feature value is determined according to a third preset rule, and based on the secondary relationship feature value, the secondary object relationship information between the main object partial image and the secondary object partial image is determined.

10. The video classification method according to claim 9, characterized in that: The determining of the secondary relationship characteristic value according to the third preset rule includes: If the primary object partial image and the secondary object partial image are derived from the same global image, the fifth eigenvalue is used as the secondary relationship eigenvalue; If the primary object partial image and the secondary object partial image are derived from different global images, the sixth eigenvalue is used as the secondary relationship eigenvalue.

11. The video classification method according to claim 1, wherein: Inputting the object features and the object relationship information into the pre-trained object classification model to obtain an object feature classification result includes: Combining the object features of each of the partial images according to the temporal relationship of the partial images in the video to be classified to obtain an object combination feature; The object combination features and the object relationship information are input into the pre-trained object classification model to obtain an object feature classification result.

12. The video classification method according to claim 1, wherein: After inputting the global image into the pre-trained object detection model to obtain a local image corresponding to the physical object, the method further includes: Obtaining a detection confidence of each of the local images obtained by the object detection model; The partial images are screened according to the detection confidence to obtain a preset number of the partial images.

13. The video classification method according to any one of claims 1 to 12, characterized in that: The preprocessing of the video frame to obtain a global image includes: The video frame is scaled using a bilinear interpolation algorithm to obtain a global image with a preset height and a preset width.

14. The video classification method according to any one of claims 1 to 13, characterized in that: Obtaining a classification result of the video to be classified according to the fusion of the environmental feature classification result and the object feature classification result, including: The environment feature classification result and the object feature classification result are weighted and summed according to a preset weighting coefficient to obtain a classification result of the video to be classified.

15. A video classification device, characterized in that: include: An image extraction module is configured to extract at least one video frame from the video to be classified and preprocess the video frame to obtain a global image; wherein the feature information of the global image includes at least one of an environmental feature and an object feature; the environmental feature is environmental information involved in the image content, and the object feature is physical object information involved in the image content; an environment classification module configured to input the global image into a pre-trained environment feature classification model to obtain an environment feature classification result based on the environment features; the environment feature classification result includes scores representing different environment category items; an object classification module configured to input the global image into a pre-trained object feature classification model to obtain an object feature classification result based on the object features; A classification fusion module is configured to obtain a classification result of the video to be classified based on the fusion of the environmental feature classification result and the object feature classification result; The object feature classification model includes an object detection model, a feature extraction model and an object classification model; The inputting the global image into a pre-trained object feature classification model to obtain an object feature classification result based on object features is configured as follows: Inputting the global image into the pre-trained object detection model to obtain a local image corresponding to the physical object; Inputting the partial images into the pre-trained feature extraction model to obtain object features of each of the partial images; determining object relationship information between the partial images according to the object features and image source information of the partial images; Inputting the object features and the object relationship information into the pre-trained object classification model to obtain an object feature classification result; The entity objects include a primary entity object and a secondary entity object, and the partial images include a primary object partial image corresponding to the primary entity object and a secondary object partial image corresponding to the secondary entity object; The determining of the object relationship information between the partial images according to the object features and image source information of the partial images is configured as follows: determining main object relationship information between the main object partial images according to the object features and image source information of the main object partial images; Secondary object relationship information between the primary object partial image and the secondary object partial image is determined according to image source information of the primary object partial image and the secondary object partial image.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video classification method according to any one of claims 1 to 14 is implemented.

17. An electronic device, characterized in that: include: processor; a memory for storing executable instructions of the processor; The processor is configured to execute the video classification method according to any one of claims 1 to 14 by executing the executable instructions.

Images