Data processing method and related equipment based on artificial intelligence

Abstract

The embodiment of the present application discloses a data processing method based on artificial intelligence and related equipment, wherein the method includes: obtaining multiple videos to be processed; inputting the multiple videos to be processed into a feature vector extraction model to obtain a video feature vector of each video to be processed, wherein the feature vector extraction model is obtained by training the original model using the image information of multiple sample videos and the corresponding feature description information, and the feature description information includes a first classification information for describing the coarse-grained semantic features of the video, and a second classification information for describing the fine-grained semantic features of the video; according to the video feature vector of each video to be processed, the multiple videos to be processed are divided into at least one video cluster. By adopting the present application, the conditional restrictions on extracting video feature vectors can be reduced, and the expression ability and accuracy of the video feature vectors on the video semantics can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an artificial intelligence-based data processing method and related equipment. Background Art

[0002] The rapid development of short videos has accumulated a large amount of video data, which has brought considerable challenges to the understanding and analysis of video content. A common method is to extract the feature vector of the video to analyze the video more accurately and in multiple dimensions. This method is applied to recommendation systems such as user profiling, recall, and sorting, thereby promoting the improvement of product form and function.

[0003] Usually, the feature vector of a video is extracted by using the user behavior vector or understanding the title features of the video. In this case, it is necessary to accumulate user behavior data or the video must have a title. However, for videos, the feature vector extracted in this way is subject to many restrictions and the accuracy is not high. Summary of the invention

[0004] The embodiments of the present application provide an artificial intelligence-based data processing method and related equipment, which can reduce the conditional restrictions on extracting video feature vectors and improve the expressiveness and accuracy of the video feature vectors for video semantics.

[0005] On the one hand, an embodiment of the present application provides a data processing method based on artificial intelligence, including:

[0006] Get multiple videos to be processed;

[0007] Inputting the multiple videos to be processed into a feature vector extraction model to obtain a video feature vector of each video to be processed, wherein the feature vector extraction model is obtained by training an original model using image information of multiple sample videos and corresponding feature description information, the feature description information includes first classification information for describing coarse-grained semantic features of the video, and second classification information for describing fine-grained semantic features of the video, the original model includes a first branch network and a second branch network, the first branch network is used to learn the image information and the first classification information, and the second branch network is used to assist the first branch network in learning the second classification information;

[0008] According to the video feature vector of each video to be processed, the multiple videos to be processed are divided into at least one video cluster.

[0009] On the one hand, an embodiment of the present application provides a data processing device based on artificial intelligence, including:

[0010] An acquisition module, used to acquire multiple videos to be processed;

[0011] An input module, used for inputting the plurality of videos to be processed into a feature vector extraction model to obtain a video feature vector of each video to be processed, wherein the feature vector extraction model is obtained by training an original model using image information of a plurality of sample videos and corresponding feature description information, wherein the feature description information includes first classification information for describing coarse-grained semantic features of the video, and second classification information for describing fine-grained semantic features of the video, and the original model includes a first branch network and a second branch network, wherein the first branch network is used for learning the image information and the first classification information, and the second branch network is used for assisting the first branch network in learning the second classification information;

[0012] The processing module is used to divide the multiple videos to be processed into at least one video cluster according to the video feature vector of each video to be processed.

[0013] An embodiment of the present application provides a computer device, including: a processor and a memory;

[0014] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method in the embodiment of the present application.

[0015] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the method in the embodiment of the present application is executed.

[0016] On one hand, an embodiment of the present application provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in one aspect of the embodiment of the present application.

[0017] In the embodiment of the present application, by using a feature vector extraction model, multiple input videos to be processed are processed to obtain video feature vectors corresponding to the multiple videos to be processed, and then the videos with similar pictures are aggregated according to the multiple video feature vectors, so as to improve the accuracy of the aggregation result. At the same time, the image information of the sample video and the feature description information of the sample video are used to train the model, and the first classification information for describing the coarse-grained semantic features of the video and the second classification information for describing the fine-grained semantic features of the video are introduced, so that the trained model has the ability to understand the high-level semantics of the video, thereby enhancing the expression ability of the feature vector of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a schematic diagram of the architecture of a data processing system provided in an embodiment of the present application;

[0020] Figure 2 It is a flowchart of a data processing method based on artificial intelligence provided in an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a video display interface provided in an embodiment of the present application;

[0022] Figure 4 It is a flowchart of another data processing method based on artificial intelligence provided by an embodiment of the present application;

[0023] Figure 5 This is a flowchart of another data processing method based on artificial intelligence provided by an embodiment of the present application;

[0024] Figure 6 It is a schematic diagram of a label dependency graph structure provided in an embodiment of the present application;

[0025] Figure 7a It is a schematic diagram of a model structure provided in an embodiment of the present application;

[0026] Figure 7b is a schematic diagram of a video feature vector aggregation effect provided in an embodiment of the present application;

[0027] Figure 8 It is a structural schematic diagram of an artificial intelligence-based data processing device provided in an embodiment of the present application;

[0028] Fig. 9 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0031] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0032] The solution provided in the embodiment of the present application belongs to the computer vision technology (Computer Vision, CV) and machine learning (Machine Learning, ML) under the field of artificial intelligence.

[0033] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further processing graphics so that the computer processing becomes an image that is more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0034] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0035] The solution provided in the embodiments of the present application involves artificial intelligence computer vision, machine learning and other technologies, which are specifically described by the following embodiments:

[0036] The feature vector extraction model trained based on the image information of the sample video and the corresponding feature description information is used to extract the feature vector of the input video. Since the feature vector extraction model learns the high-level semantic features of the video, the accuracy of the video feature vector in expressing the video content can be improved. At the same time, visual consistency is taken into account to improve the effect of video aggregation.

[0037] See also Figure 1 , is an architecture diagram of a data processing system provided in an embodiment of the present application. Figure 1 As shown, the architecture of the data processing system may include a server 100 and multiple terminal devices, which may specifically include terminal device 1, terminal device 2, ..., terminal device n. Each terminal device may be connected to the server 100 through a network, and the network connection may include a wired connection or a wireless connection, so that each terminal device may interact with the server 100 through the network connection, and the server 100 may receive business data from each terminal device.

[0038] Each terminal device can be used to display business data to the user, and the business data can specifically include multimedia data such as picture data or video data, and each terminal device can also be used to store the business data. Each terminal device can be integrated with a target application, which can specifically be a video sharing application or a social application. The server 100 can store the business data generated by the target application and interact with each terminal device. For example, the user can upload video data to the server 100 through the target application, and can also download video data from the server 100.

[0039] In the embodiment of the present application, taking the service data as video as an example, the server 100 can filter out similar videos from the videos generated by the target application on each terminal device, and display them to the user through each terminal device. Specifically, it may include: a feature vector extraction model is run on the server 100, and this feature vector extraction model is used to extract video feature vectors. The server 100 processes these videos using the feature vector extraction model to obtain video feature vectors of each video. The videos can be clustered according to the video feature vectors, thereby filtering out similar videos. In addition, the feature vector extraction model is trained based on comprehensive information such as the image content of the sample video (i.e., image information), the corresponding feature description information (i.e., the first classification information and the second classification information), etc. Specifically, the TSM network is trained using the image content of the sample video and the first classification information, and the GCN is trained using the second classification information. The training of the image content of the sample video enables the feature vector extraction model to learn not only frame picture features but also motion information. The first classification information and the second classification information are used to describe the coarse-grained semantic features and fine-grained semantic features of the video, respectively, so that the feature vector extraction model learns high-level semantic information, ensuring that the extracted feature vector can more accurately and completely represent the features of the video, thereby improving the clustering accuracy.

[0040] The terminal devices in the embodiments of the present application may include: smart phones, tablet computers, laptop computers, desktop computers, smart TVs, smart speakers, desktop computers, smart watches and other smart terminal devices, but are not limited to these.

[0041] It is understandable that the method provided in the embodiment of the present application can be executed by a computer device, which includes but is not limited to the above-mentioned server 100. Among them, the server 100 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0042] For further understanding, please see Figure 2 , Figure 2 This embodiment of the application is based on Figure 1 The data processing system shown is a flow chart of a data processing method based on artificial intelligence. The data processing method based on artificial intelligence may at least include the following steps S201-S203:

[0043] S201, obtaining multiple videos to be processed.

[0044] In a possible embodiment, the multiple videos to be processed obtained here are videos in the video library of the target application, wherein the target application may be a video application, a social application, or any application that can process videos, and this embodiment does not limit this. The videos in the video library may be uploaded by multiple users using the target application and stored in the video library corresponding to the target application through the background, or they may be videos pre-stored in the background server, and this embodiment does not limit the specific acquisition method. In addition, for the video to be processed, the embodiment of the present application mainly processes the continuous frame image content contained therein, so the obtained video to be processed may not include other content information, such as the category to which it belongs, the multi-label information it carries, or audio information.

[0045] S202, inputting the multiple videos to be processed into a feature vector extraction model to obtain a video feature vector for each video to be processed, wherein the feature vector extraction model is obtained by training an original model using image information of multiple sample videos and corresponding feature description information, and the feature description information includes first classification information for describing coarse-grained semantic features of the video, and second classification information for describing fine-grained semantic features of the video.

[0046] In a possible embodiment, a plurality of videos to be processed need to be preprocessed before being input into a feature vector extraction model. For example, after frame sampling (i.e., frame extraction), the format of the image obtained by video sampling needs to be processed into the format required by the feature vector extraction model, etc., and then the key frame image obtained after preprocessing is input into the feature vector extraction model for processing. Moreover, since the feature vector extraction model is obtained by training the original model using the image information of a plurality of sample videos and the corresponding feature description information, the feature vector extraction model has the ability to understand the high-level semantic information of the video on the basis of extracting the traditional frame image features. Therefore, the video feature vector of each video to be processed can express the video content more accurately.

[0047] In a possible embodiment, the feature description information corresponding to the sample video includes first classification information for describing the coarse-grained semantic features of the video, and second classification information for describing the fine-grained semantic features of the video. Among them, the coarse-grained semantic features have different emphasis on the description of the video compared to the fine-grained semantic features. More specifically, the video classification will be divided into multiple mutually exclusive categories based on the semantic content of the video, such as human behavior and complex events. These mutually exclusive categories can be regarded as first classification information. The first classification information describes the video in a coarse-grained manner, that is, the first classification information of a sample video includes a single label. Relatively speaking, the second classification information includes multiple labels, which is a more specific and multi-dimensional classification system. The video can be described in a more fine-grained manner through the second classification information. For example, a video of playing the piano is just a Western musical instrument for classification (i.e., the first classification information), but the labels of the second classification information can be piano, indoor, female and other finer-grained features, and they can coexist. For another example, a travel record video can have corresponding labels such as sky, building, person, car, tree and other second classification information. In addition, the feature description information carried may also be different according to different analysis dimensions. For example, from the perspective of human behavior, the second classification information of a travel video may include swimming or running, etc. In general, the first classification information is a coarse-grained semantic feature that is relatively general and mutually exclusive for the video, and the labels included in the second classification information are fine-grained semantic features that specifically represent the video content and have coexistence. Both the first classification information and the second classification information can be represented by labels.

[0048] In a possible embodiment, the original model includes a first branch network and a second branch network, wherein the first branch network is used to learn the image information and the first classification information of the aforementioned sample video, and the second branch network is used to assist the first branch network in learning the second classification information contained in the corresponding feature description information of the sample video. The original model is trained by using an end-to-end training method to obtain a trained original model, which includes a trained first branch network and a trained second branch network. Finally, the trained first branch network can be used as a feature vector extraction model to process the video to be processed to obtain a video feature vector of the video to be processed. The first branch network here includes a temporal shift module (TSM) that can be embedded in a two-dimensional convolutional neural network. The TSM network formed by combining TSM and a two-dimensional convolutional neural network is the first branch network. The first branch network can better integrate the apparent features and temporal features of the video, and can easily achieve video understanding capabilities equivalent to those of a three-dimensional convolutional neural network using a small amount of resources. That is, the video image information is modeled in time and space, and the interaction of the information contained in the video in time is realized, which improves the feature extraction capability of the time series. Optionally, the first branch network may also be other neural networks capable of extracting image information, such as a three-dimensional convolutional neural network.

[0049] S203: Divide the multiple videos to be processed into at least one video cluster according to the video feature vector of each video to be processed.

[0050] In a possible embodiment, when dividing the video into video clusters, the video feature vectors of the video to be processed are mainly used. Since the idea of ​​face recognition is borrowed, that is, each video to be processed is regarded as a different category, just like each person's face is different, so the video feature vector of each video is also unique, but it should be noted that although the video feature vectors have their own characteristics, they also have similarities. Therefore, by measuring the video feature vectors, for example, using distance measurement, videos of similar categories can be aggregated together. Specifically, the distance between the video feature vectors of any two videos to be processed can be calculated. If the distance is less than or equal to a preset distance threshold (for example, 0.1), the two videos can be divided into a video cluster, thereby obtaining at least one video cluster corresponding to the multiple videos to be processed.

[0051] Furthermore, after the video to be processed is divided into at least one video cluster, at least one video cluster can be sent to the terminal device so that the terminal device displays the videos included in each video cluster in the user interface. The videos included in the video cluster have similar images. The specific product presentation form is not limited here, and can be as follows: Figure 3 As shown. Figure 3 The video 1, video 2, ..., video m, etc. shown are all a type of videos with similar pictures and video semantics. The user can select any video among them and browse the specific content. For example, when the user clicks on video 1, the terminal device will display the dynamic picture of video 1 to the user, that is, play video 1; or when the user selects video 2 by sliding, the terminal device will display the dynamic picture of video 2 to the user. The specific selection method is not limited here.

[0052] In a possible embodiment, the method may be executed by a server (such as the server 100 described above), wherein the server processes the video to be processed, and after obtaining the processing result, sends at least one video cluster to the terminal device, so that the terminal device displays the videos included in each video cluster in the user interface.

[0053] In a possible embodiment, the method may also be executed by a terminal device (such as Figure 1 The terminal device 1 shown in the figure). The terminal device 1 processes the video to obtain a video aggregation result (i.e., at least one video cluster), and then displays it on the user interface. Optionally, the terminal device 1 can also send the video aggregation result to other terminal devices so that other terminal devices (such as Figure 1 The terminal device 2) shown displays the videos included in each video cluster in the user interface.

[0054] As an extensible example, by using the feature extraction model in the target application, videos with similar images can be pushed or displayed to users. Since the video feature vector indicates rich information, the feature description information of the video can also be predicted based on the video feature vector, that is, the single-label category it belongs to or the multi-label category it should carry, and other functions can be implemented based on this information, such as automatically classifying the video or correcting the user's classification of the video; analysis of user interests can also be added to the entire model to display video clusters that match the user's interests in a targeted manner.

[0055] In summary, the embodiments of the present application have at least the following advantages:

[0056] The video feature vector of the video to be processed is extracted through the feature vector extraction model. Since the information indicated by the video feature vector can help understand the content of the video, the video to be processed can be aggregated more accurately according to the output video feature vector, and finally at least one video cluster is obtained, that is, a collection of videos with similar pictures, which are then displayed on the user interface to make the push more intelligent. In addition, the feature vector can be obtained according to the visual information of each video without the need for other unnecessary conditions, making the extraction of feature vectors more universal.

[0057] See also Figure 4 , Figure 4 This embodiment of the application is based on Figure 1 The data processing system shown is a flow chart of another data processing method based on artificial intelligence. The data processing method based on artificial intelligence may include at least the following steps:

[0058] S401, obtaining a training sample set, where the training sample set includes a plurality of sample videos.

[0059] In the field of artificial intelligence, such as machine learning tasks, sample sets play a very important role, especially in neural networks. By using sample sets to train models, important parameters can be matched to establish an initial model, and then the parameters are continuously optimized during the training process, so that the model finally trained has good learning ability. In this application, the training sample set also has a similar role. Accordingly, there are multiple sample videos in the training sample set, and a suitable model can be trained to implement some functions through the training sample set. The multiple sample videos can be selected from the video database corresponding to the target application, or from the videos included in some public video resource libraries. There is no restriction here, and the specific acquisition method and the source of the sample video are not limited here. Among them, the target application can be a video application, a social application, or any application that can process videos, and this embodiment does not limit this. It should be noted that the sample videos obtained can include videos of different themes or categories, such as dance, musical instruments, entertainment, scenery, games, etc.

[0060] S402: Training the original model according to the image information and corresponding feature description information of the multiple sample videos.

[0061] In a possible embodiment, the image information of the sample video reflects the key content of the sample video. By training the original model with the image information, the original video can learn the image content features. That is, the video feature vector extracted by the original model contains the image content features, which can represent the content of the sample video. Other related processing can be performed using the video feature vector. In addition, each sample video in the acquired training sample set has corresponding feature description information, such as a single label as the first classification information and multiple labels as the second classification information. The first classification information includes a single label, which is mutually exclusive, and the multiple labels included in the second classification information can coexist. For example, for a piano playing video, the corresponding first classification information is Western musical instruments, which are mutually exclusive labels with national musical instruments. The second classification information can have piano, people, indoor and other labels, and these labels can exist at the same time. For another example, for a travel record video, the corresponding second classification information includes labels such as sky, building, people, car, tree, etc. It should be noted that, as an implementation method, the feature description information corresponding to these sample videos can be machine-labeled. For example, image recognition can be performed on the sample videos, and at least one label corresponding to the sample videos can be determined based on the recognized content, thereby obtaining the feature description information corresponding to the sample videos. As another implementation method, the feature description information corresponding to these sample videos can also be manually labeled, and the labels carried can also be different according to different analysis dimensions. For example, from the perspective of human behavior, the labels can be swimming, running, etc. In general, the single label in the first classification information represents the coarse-grained features of the video, and the multiple labels in the second classification information specifically represent the fine-grained features of the video. Using a fully supervised approach, all sample videos and the second classification information are input into the original model as training data for training, and an optimized model can be obtained, wherein the original model includes a first branch network and a second branch network.

[0062] Specifically, the first branch network and the second branch network included in this original model have different functions, that is, as mentioned above, the first branch network is used to learn the image information and the first classification information, and the second branch network is used to assist the first branch network in learning the second classification information. Among them, the first branch network includes a temporal shift module (TSM). Specifically, TSM and a two-dimensional convolutional neural network (2D Convolutional Neural Network, 2D CNN) are combined into a first branch network, and the first branch network can be called a TSM network. As one of the technologies for video understanding, it can replace the three-dimensional convolutional neural network (3D Convolutional Neural Network, 3DCNN) and easily achieve video understanding capabilities equivalent to those of 3D CNN. Specifically, any video A in the training sample set can be represented as follows:

[0063] (1)

[0064] Wherein, N is the batch size of the sample video, C is the number of channels, T is the time dimension, and H and W are the spatial resolutions. However, the 2D convolutional neural network operates independently in the time dimension and cannot model time. Relatively speaking, the 3D convolutional neural network is more suitable for the learning of spatiotemporal features, and can better integrate the apparent features and temporal features of the video, but it also has the disadvantage of large amount of computation. The TSM network solves this problem by introducing a time displacement module, that is, embedding the time displacement module into the 2D CNN. While simulating the 3D convolutional neural network for modeling, it does not add any additional computation and parameters to achieve the same effect as the 3D convolutional neural network. In an embodiment of the present application, for multiple frame images to be processed, the channels between adjacent frames are alternately spliced ​​so that adjacent frames contain each other's information, thereby obtaining time series features and increasing the capture of motion information on the basis of frame image feature extraction.

[0065] Furthermore, since the TSM network decomposes the convolution operation into two processes: displacement and weight superposition, for example, for a video tensor , TSM moves a portion of the channels forward or backward along the time dimension (i.e., displacement), and the corresponding frame information on the time dimension is not the original, that is, the tensor of each time dimension The channel dimension covers the information of adjacent frames. For example, if the time dimension moves +1, the frame information corresponding to t=2 is the frame information of t=1, and the receptive field of the time dimension reaches 3. Then multiplication is used to accumulate from the time dimension to the channel dimension, that is, for each time dimension tensor The final result (i.e. weight superposition) is obtained after weight sharing.

[0066] Since displacement is a regular address offset pointer operation, it basically does not consume computing resources, which greatly reduces the amount of calculation, making it very easy for the TSM network to achieve performance balance. In addition, since the TSM module introduces contextual interaction in the time dimension, it also improves the modeling ability in the time dimension, allowing the model to learn time series characteristics.

[0067] In addition, the second branch network can be a graph convolutional neural network (GCN), which is mainly responsible for assisting the first branch network (ie, TSM network) in training. Generally speaking, objects that appear simultaneously in a video are correlated, and a suitable model needs to be established for the relationship between various labels corresponding to multiple objects in the video, so that the labels included in the second classification information can be used to improve the accuracy of video clustering. However, if the labels included in the second classification information are regarded as each independent node, then the relationship between the labels appearing in the same video can be represented by a graph structure, which is an irregular data structure that does not belong to the Euclidean space data. It cannot be calculated using a general convolutional neural network, but a graph convolutional neural network can process graph data, so the second branch network uses a graph convolutional neural network to extract features from these graph data, thereby realizing various functions, such as node classification, graph classification, edge prediction, etc.

[0068] S403: Use the trained first branch network as the feature vector extraction model.

[0069] In a possible embodiment, since the second branch network only serves as an auxiliary training structure for the first branch network, the purpose of its training is to enable the first branch network, that is, the TSM network, to have better video semantic understanding capabilities. Therefore, after training, the feature vector can be obtained by simply processing the input video using the TSM network. The trained first branch network can also be called a feature vector extraction model, and its main function is to obtain the feature vector of the input video. This feature vector can not only express the basic image information of the video, but also express more other information, such as coarse-grained video semantic information, fine-grained video semantic information and other advanced semantic information.

[0070] In summary, the embodiments of the present application have at least the following advantages:

[0071] In the method of sampling supervised learning, an optimal original model is trained through known sample videos (including image information and feature description information), so that the original model has predictive ability. The second branch network in the original model assists the first branch network in training and learning the second classification information, and the first branch network adopts the TSM network to increase the capture of motion information (or timing information), that is, by fully modeling the features of the time dimension, these functions are combined to allow the model to learn the high-level semantics of the video, making the information expressed by the feature vector richer and more accurate; in addition, under the same resource conditions, the feature vector extraction model can process more input videos and improve the efficiency of obtaining video feature vectors.

[0072] See also Figure 5 , Figure 5 This embodiment of the application is based on Figure 1 The data processing system shown in FIG. 1 is a flowchart of another data processing method based on artificial intelligence. The process may include:

[0073] S501, obtaining a training sample set, where the training sample set includes a plurality of sample videos.

[0074] The specific implementation method of this step can be found in the above Figure 4 S401 in the corresponding embodiment will not be described in detail here.

[0075] S502: Input image information of each sample video among the multiple sample videos into the first branch network to obtain a video feature vector of each sample video.

[0076] In a possible embodiment, the first branch network may be a TSM network, which includes a TSM module embedded in a two-dimensional convolutional neural network 2D CNN. Such a first branch network has a more efficient semantic understanding capability for sample videos than a 3D CNN. During training, the sample videos are input into the first branch network in units of each sample video. Specifically, the image information contained in each sample is input into the first branch network, and the output is a video feature vector corresponding to each sample video. This video feature vector can represent the image content feature. For example, there are 10 sample videos, {V1, V2...V 10}, first input V1 to obtain the video feature vector D1, then input V2 to obtain the video feature vector D2, and so on, the corresponding 10 feature vectors can be obtained.

[0077] S503: Input the multiple labels included in the second classification information of each sample video into the second branch network to obtain a label feature vector of each sample video.

[0078] In one possible embodiment, the second branch network may be a graph convolutional neural network GCN, and the corresponding label feature vector may be obtained by inputting the multiple labels included in the second classification information of each sample video into the second branch network. Specifically, a mutually dependent target classifier may be learned from the features corresponding to the labels in the second classification information through a GCN-based mapping function, and the target classifier is used to indicate the dependency relationship between the label feature vectors. It should be noted that the multiple labels included in the second classification information input here correspond to the sample video of the first branch network. For example, the sample video currently input to the first branch network is a video of playing the piano, and the labels include piano, female, and indoor. Then these three labels (piano, female, indoor) will be used in the second branch network to obtain the corresponding label feature vector. Since these three labels are correlated, they can be used as training data, and there is no restriction on the specific mathematical expression used to input them into the second branch network.

[0079] In a possible embodiment, the specific processing of the multiple labels included in the second classification information of the training sample set and the process of using the multiple labels included in the second classification information to input the second branch network may include: obtaining all labels included in the second classification information of multiple sample videos; generating a label dependency matrix based on all labels and the multiple labels included in the second classification information of each sample video, wherein the label dependency matrix includes the correlation coefficient between any two labels; and inputting the label feature matrix and the label dependency matrix corresponding to the multiple labels included in the second classification information of each sample video into the second branch network to obtain a label feature vector for each sample video.

[0080] Specifically, before inputting the second branch network GCN, it is necessary to construct the dependency relationship between the labels included in the second classification information of multiple sample videos, which can also be called the correlation between labels. Therefore, it is necessary to first obtain all the labels included in the second classification information of the training sample set. These labels are non-repetitive and can coexist, representing different contents. Then, the dependency graph structure between these labels is constructed based on the multiple labels contained in each video. For example, if there are 3 sample videos, namely {V1, V2, V3}, and all the labels included in the second classification information include people, ties, skiing, badminton, and badminton rackets, and the labels in the sample video V1 include people and ties; the labels in V2 include people and skiing; the labels in V3 include people, badminton, and badminton rackets. In other words, the multiple labels included in the second classification information correspond to each sample video, so the specific dependency graph structure can be as follows Figure 6As shown in the figure, the dependency relationship can be represented by the arrows in the figure. For example, "badminton" points to "person", which means that when the label "badminton" appears, the label "person" must appear, but if the label "person" appears, the label "badminton" may not appear. Optionally, the mathematical expression of the graph structure can be implemented using a label dependency matrix. Specifically, first, the adjacency matrix is ​​constructed by counting the number of times any two labels appear at the same time. Therefore, this adjacency matrix is dimensional matrix, where N represents all the labels of the sample videos, and then count the total number of occurrences of each label (that is, the number of sample videos corresponding to the label), and then combine the values ​​and total number in this adjacency matrix to calculate the conditional probability, that is, under the label L i Under the conditions of L j The probability of occurrence P(L j |L i ), that is, the association coefficient, is used to update the matrix information, that is, to replace the original number of times with the calculated conditional probability value, and set the diagonal to 1, representing that the probability of each occurrence is 1. The updated adjacency matrix can be called the label dependency matrix.

[0081] Optionally, considering that the input multi-layer GCN may have the problem of overfitting, the label dependency matrix can be improved to be better applied to the graph convolutional neural network, for example, by introducing parameter calibration to assign weights to the node itself and other related nodes.

[0082] In addition, a label feature matrix can be obtained based on the multiple labels included in the second classification information of each sample video. This label feature matrix can be regarded as the concatenation of feature vectors extracted from the multiple labels included in the second classification information. Specifically, the sentence2vector technology can be used to extract the natural language processing (NLP) features of the label, which can be called sentence vectors, and based on the sentence vectors, the co-occurrence vectors of the relevant labels, that is, the label feature matrix, are constructed. Among them, the granularity of the sentence vector is a natural sentence. The sentence vector model is used after being trained with a large amount of text. The feature extraction of the label, especially the feature extraction of more complex labels, will be more accurate. Of course, as a non-limiting example, the word embedding vector of the label can also be extracted. The word embedding vectors extracted from each label are concatenated to form a label feature matrix. This label feature matrix can be input into the second branch network as prior knowledge of the label. For example, if a sample video has 2 labels, the corresponding label feature matrix should be dimensional, where d is the dimension size of the vectors corresponding to the multiple labels (such as word embedding vectors) included in the second classification information. The feature vector of each node can be updated through each GCN layer. For example, a sample video has three labels: piano, female, and indoor. The sentence vector r1, r2, and r3 of each label can be obtained by using sentence2vector technology. The label feature matrix {r1, r2, r3} can be formed by splicing based on the sentence vector.

[0083] By inputting the label dependency matrix and label feature matrix mentioned above into the second branch network GCN, we can get the label feature vector of each sample video. The dependency between these label feature vectors is indicated by the target classifier, where "classification" refers to all labels included in the second classification information of the sample video. The graph convolutional neural network maps the prior features of the label into a set of interdependent classifiers, which can be regarded as a matrix with a size of , where the dimension represented by D is consistent with the dimension of the video feature vector extracted by the first branch network TSM network, and can be 512, 2048, etc., which is determined after processing according to the size of the image extracted from the video. C is the number of all labels in the sample set, and the vector of the i-th label is mapped to On, among which, C i Represents the i-th column in the matrix. GCN is introduced to establish a directed graph between labels (label dependency graph) in a data-driven way and GCN maps the labels to the corresponding target classifiers to model the relationship between labels, so that the first branch network learns the high-level semantic information of the video, the label, and improves the representation learning ability of the model. This target classifier is applied to the image features generated by the TSM network to achieve end-to-end training.

[0084] S504: Use the video feature vector and the label feature vector to adjust the network parameters of the original model to train the original model.

[0085] Specifically, by continuously inputting sample videos, corresponding video feature vectors and label feature vectors are obtained to adjust the network parameters of the original model to achieve training of the original model.

[0086] In a possible embodiment, sample videos of a batch size (batch_size) may be input in sequence, and the network parameters of the original model may be adjusted after the corresponding video feature vectors and label feature vectors are obtained. For example, batch_size is 10, 10 sample videos are input into the original model in sequence, 10 video feature vectors and 10 label feature vectors are output, and then the network parameters are adjusted again. This is more convenient and effective from the perspective of processing efficiency and resource usage, and the number of repeated scheduling of related resources is greatly reduced.

[0087] Optionally, the image information of each sample video can be input into the TSM network to obtain the corresponding video feature vector, and the second classification information of each sample video can be input into the GCN network to obtain the label feature vector. In this way, the network parameters of the original model are adjusted once. That is to say, if there are N sample videos, the network parameters are adjusted N times. However, there is still room for improvement in performance of this method.

[0088] The specific model structure can be as follows Figure 7a As shown in the figure, it mainly includes two branches, TSM network and GCN network. After the video is frame sampled, a frame image is obtained, which is input into the TSM network. The video feature vector extracted by the TSM network and the label feature vector are vector dot product to obtain the fusion feature vector C, which is used as the predicted feature vector. The total loss value is calculated using the loss function, and the network parameters of the model are continuously adjusted according to the total loss value to achieve the optimal value. There are three main loss functions, two of which are for the classification of the video (these two loss functions can be recorded as the first category loss function and the second category loss function respectively), and the third loss function is for the label carried by the video (the third loss function can be recorded as the label loss function).

[0089] In one possible embodiment, the first classification information of each sample video can be obtained, and then the total loss value can be determined based on the video feature vector, the label feature vector and the first classification information. The total loss value includes the loss value of the first category loss function and the loss value of the second category loss function. The total loss value is then used to adjust the network parameters of the first branch network to train the original model. Furthermore, the total loss value can also include the loss value of the label loss function. Specifically, the network parameters of the first branch network of the original model can be adjusted according to the total loss value, and the network parameters of the second branch network can be adjusted using the loss value of the label loss function to train the second branch network of the original model, thereby achieving training of the original model.

[0090] Specifically, the first classification information and the second classification information of the sample video are different. The second classification information is a more fine-grained description of the video, and the first classification information can only be classified into one category for each sample video, that is, the first classification information is mutually exclusive. For example, if a piano-playing video is classified as a Western instrument, then the video cannot be classified as a national instrument. The first classification information of each sample video is obtained in order to obtain the feature vector corresponding to the first classification information.

[0091] In a possible embodiment, the loss value of the first category loss function can be determined based on the video feature vector of each sample video and the feature vector corresponding to the first classification information. Since the embodiment of the present application also involves the problem of the classification to which the video belongs, the first category loss function can use the loss function corresponding to the normalized exponential function Softmax. Specifically, the cross entropy loss function for processing multiple classifications in the loss function corresponding to Softmax is more appropriate in the embodiment of the present application. For a sample video, the specific definition of the function is as follows:

[0092] (2)

[0093] Among them, L1 is the loss value of a sample video; T represents the number of all categories (or categories, that is, the first classification information) in the training sample set; is the feature vector of the jth category, which is a The vector has T values, and only one value is 1. The position corresponding to this value represents the true label, and the remaining T-1 values ​​are all 0; is the probability of predicting that the sample video belongs to category j, and its value range is .

[0094] Among them, for Softmax loss Specifically, the video feature vector needs to be normalized. Specifically, all the values ​​in the vector are converted into positive numbers through exponential operation, and then the sum of these positive numbers is used as the denominator, and the positive numbers corresponding to the vector are used as the numerator to calculate the corresponding probability, and finally a The value in the range is the probability of predicting that the sample video belongs to category j. The specific expression is as follows:

[0095] (3)

[0096] Among them, S j is the output of softmax, that is, the probability that each sample video belongs to the jth category, and a represents the original value of the category to which the video feature vector may belong.

[0097] The specific calculation of the above loss function is to determine the loss value L1 of the first category loss function based on the video feature vector obtained after each sample video is input into the feature extraction model and the feature vector corresponding to the first classification information.

[0098] In a possible embodiment, multiple sample videos of a batch_size can be input into the TSM network to obtain multiple video feature vectors, which can be spliced ​​into a matrix and then calculated to determine the loss value of the first category loss function. The loss function expression at this time is:

[0099] (4)

[0100] Among them, n represents the size of batch_size, that is, a batch of sample videos; represents the probability that sample i belongs to category j, It is the probability that the model predicts sample i as category j, thereby obtaining the loss value loss1 of the first category loss function.

[0101] As a non-limiting example, the first category loss function may also adopt other loss functions, such as the loss function of a support vector machine (SVM), etc., which is not limited here.

[0102] In a possible embodiment, the loss value of the second category loss function can be determined according to the video feature vectors of multiple sample videos and the first classification information. The second category loss function can use an additive margin softmax (AM-Softmax) to calculate the loss value, and the specific expression is:

[0103] (5)

[0104] Where loss2 represents the loss value, s is the hyperparameter of the cosine distance scaling factor, m is the interval coefficient, n represents the batch_size of the training sample set, and C is the total number of all categories of sample videos (i.e., the first classification information); y i Represents the predicted category of the i-th sample video; It means that the video feature vector of the i-th sample video is predicted to be y i When the class is , the angle between its corresponding eigenvector and the eigenvector of the i-th class; It represents the angle between the corresponding feature vector and the feature vector of the j-th category when the video feature vector of the i-th sample video is predicted to be the j-th category.

[0105] Compared with Softmax loss, AM-Softmax adds a boundary margin to the angle discrimination boundary, making the classification boundary stricter. The loss is calculated according to the normalized exponential function that increases the boundary, and the parameters are updated until the training is completed. The distance between the obtained video feature vectors of similar categories is reduced, while the distance between the video feature vectors of different categories is enlarged, which has a better improvement on the final clustering effect based on the video feature vector. In the embodiment of the present application, the idea of ​​face recognition is borrowed, that is, each video is regarded as a different classification, just like each face is different. By training different frames of the sampled video, the fingerprint granularity characteristics of the video can be learned, that is, each video feature vector is different, so that similar feature vectors are more closely grouped.

[0106] In a possible embodiment, a total loss value can be determined based on the loss value of the first category loss function and the loss value of the second category loss function as a reference for adjusting the network parameters of the original model. Specifically, when determining the total loss value, it can be calculated based on the weight coefficients and loss values ​​of the above two loss functions, and then the network parameters of the first branch network, that is, the TSM network, are adjusted based on the total loss value to train the first branch network of the original model, so that the feature vector extracted by the TSM network has better expression ability.

[0107] In one possible embodiment, the loss value of the label loss function can be determined based on the fused feature vector and the feature vector of each label included in the second classification information of each sample video. The fused feature vector is obtained by performing a vector dot product on the video feature vector and the label feature vector, that is, applying the label feature vector to the video feature vector output by the TSM network, which can also be understood as matrix multiplication to obtain a fused feature vector. The fused feature vector is used as a vector representing the predicted label, and the predicted score is obtained. This score can be a positive or negative value, and of course it can also be in other forms, such as the probability of the predicted label.

[0108] Optionally, the label loss function here uses the binary cross entropy (BCE) loss function, which can be used for multi-label classification problems. Specifically, the activation function sigmoid function is used in the output layer to convert the values ​​in the fused feature vector into a number between 0 and 1, but these values ​​are not additive, that is, the sum is not necessarily 1. In a more specific implementation process, the sigmoid function and the BCE process can be put together, and the built-in BCEWith Logits Loss function can be used to calculate the loss value. The expression of this loss function is:

[0109] (6)

[0110] in, Represents the fusion feature vector of the i-th sample video after the sigmoid function activation processing, the value ; The feature vector of each label included in the second classification information of the i-th sample video is represented, where the value is either 0 or 1, and there can be multiple 1s and multiple 0s, indicating whether there is a corresponding label; n represents the batch_size of the training sample set.

[0111] As a non-limiting example, the label loss function can also be calculated using a multi-label loss function (which is also a form of BCE loss), and the specific expression is:

[0112] (7)

[0113] Wherein, y represents the feature vector of each label included in the second classification information of each sample video, and , that is, the value is either 0 or 1, and there can be multiple 1s and multiple 0s, representing whether label i appears in the sample video; C represents the number of all labels; is the fused feature vector; This is the Sigmod function.

[0114] The specific expression of Sigmod function is:

[0115] (8)

[0116] In one possible embodiment, after obtaining the loss value of the first category loss function, the loss value of the label loss function, and the loss value of the second category loss function, the total loss value can be determined based on the loss value of the first category loss function, the loss value of the label loss function, and the loss value of the second category loss function. When determining the total loss value, it can be calculated based on the weight coefficients and loss values ​​of the above three loss functions, and then the network parameters of the first branch network, i.e., the TSM network, are adjusted according to the total loss value to train the first branch network of the original model, and the network parameters of the second branch network, i.e., the GCN network, are adjusted according to the loss value of the label loss function to train the second branch network of the original model, thereby enabling the model to have the ability to understand the label features in the high-level semantic features of the video.

[0117] It should be noted that the first and second classification information of the video are excellent supervisory information, and the two complement each other and are indispensable. If there is only single-label first classification information, the feature granularity learned by the model is too coarse, but if there is only multi-label second classification information, the video feature vector generated by the trained model will not have strong performance.

[0118] Optionally, the first branch network and the second branch network can be adjusted once according to the loss value of each loss function. For example, if the loss value of the first category loss function is determined, the network parameters of the first branch network are adjusted once using the loss value of the first category loss function; if the loss value of the second category loss function is determined, the network parameters of the first branch network are adjusted once using the loss value of the second category loss function; if the loss value of the label loss function is determined, the network parameters of the first branch network and the second branch network are adjusted once using the loss value of the label loss function.

[0119] S505: Use the trained first branch network as the feature vector extraction model.

[0120] The present application embodiments and the aforementioned Figure 4 Similar to the corresponding S403, after calculating multiple loss functions and adjusting or optimizing the network parameters of each branch network, the obtained first branch network has the ability to understand the high-level semantics of the video, that is, the multi-label and single-label classification information of the video can be extracted through the first branch network, and the picture consistency of each video (i.e., fingerprint information) can be learned. Therefore, the first branch network is used as a feature vector extraction model. The feature vector obtained by the feature vector extraction model can represent relatively rich information, such as single-label classification, multi-label classification, and the characteristics of the image itself.

[0121] Optionally, since the supervisory information used in the training video feature vector in the embodiment of the present application is the strong supervisory information of the video label and classification, the accuracy of the feature description information is used as the criterion for judging the effect of the video feature vector. Taking the animal category as an example, after experiments, the initial accuracy was 75% without adding GCN in the experiment. After adding GCN, the feature description information reached 80.3% accuracy on the same test set. Figure 7b As shown, it contains two sub-graphs, namely (A) and (B), where sub-graph (A) is the aggregation effect of the video feature vector generated without GCN after PCA dimensionality reduction, and sub-graph (B) is the aggregation effect of the video feature vector generated after adding GCN and feature description information after principal component analysis PCA dimensionality reduction. It can be seen that the aggregation effect of the video feature vector extracted after adding the GCN training model is better.

[0122] In addition, the above embodiment only generates a video feature vector from the visual perspective, which is a single-mode information. As an expandable example, the title information of the video (non-picture information of the video) can also be added, and the audio information in the video is obtained through the automatic speech recognition technology ASR. The text information in the video, such as the subtitles, the barrage that appears, etc., is obtained through optical character recognition. Multimodal fusion is performed in multiple aspects such as vision and hearing, and the original model is trained using the information after multimodal fusion to obtain a feature extraction model with stronger learning ability, thereby obtaining a video feature vector with richer information. Similarly, an end-to-end training method can also be used, and a model is trained separately for each branch (that is, each modality), and finally an optimized distributed gradient enhancement library XGBOOST and other classifier fusion models are used, so that it is lighter during training and reduces the interference of mutual coupling between branches.

[0123] S506, obtaining multiple videos to be processed.

[0124] S507, input the multiple videos to be processed into a feature vector extraction model to obtain a video feature vector for each video to be processed, wherein the feature vector extraction model is obtained by training an original model using image information of multiple sample videos and corresponding feature description information, and the feature description information includes first classification information for describing coarse-grained semantic features of the video, and second classification information for describing fine-grained semantic features of the video.

[0125] S508: Divide the multiple videos to be processed into at least one video cluster according to the video feature vector of each video to be processed.

[0126] The specific implementation of the above steps S506-S508 can be found in the above Figure 2 The description of steps S201-S203 in the corresponding embodiment will not be repeated here. In addition, it should be noted that step S506 can be executed after S505 or before S505, and the execution order of S505 and S506 is not limited here.

[0127] In summary, the embodiments of the present application have at least the following advantages:

[0128] An end-to-end training method is adopted to directly learn the mapping from sample videos to the desired output video feature vector. At the same time, the prior knowledge of labels in the feature description information is used to introduce a second branch network (GCN network) to assist model training. The label feature vector is obtained by mapping the dependencies between the labels contained in the second classification information, so that the final label feature vector maintains a meaningful semantic topological structure. The label feature vector and the video feature vector are used to obtain a fusion vector. Finally, the parameters of the original model are adjusted based on the loss values ​​of the three loss functions to obtain the optimal feature vector extraction model, so that the feature vector extraction of the video is more accurate. A normalized exponential function with increased boundaries is used, and visual similarity is taken into account, so that the videos clustered according to the video feature vectors have screen similarities.

[0129] See also Figure 8 , is a structural diagram of an artificial intelligence-based data processing device 80 provided in an embodiment of the present application. The artificial intelligence-based data processing device may be a computer program (including program code) running in a computer device. For example, the artificial intelligence-based data processing device is an application software; the device may be used to execute the corresponding steps in the method provided in an embodiment of the present application. The device 80 may include: an acquisition module 801, an input module 802, and a processing module 803, wherein:

[0130] The acquisition module 801 is used to acquire multiple videos to be processed.

[0131] An input module 802 is used to input the multiple videos to be processed into a feature vector extraction model to obtain a video feature vector for each video to be processed, wherein the feature vector extraction model is obtained by training an original model using image information of multiple sample videos and corresponding feature description information, the feature description information includes first classification information for describing coarse-grained semantic features of the video, and second classification information for describing fine-grained semantic features of the video, the original model includes a first branch network and a second branch network, the first branch network is used to learn the image information and the first classification information, and the second branch network is used to assist the first branch network in learning the second classification information.

[0132] The processing module 803 is configured to divide the multiple videos to be processed into at least one video cluster according to the video feature vector of each video to be processed.

[0133] Optionally, the apparatus 80 may further include: a sending module 804, wherein:

[0134] The sending module 804 is configured to send the at least one video cluster to a terminal device, so that the terminal device displays the videos included in each video cluster in a user interface.

[0135] Optionally, the apparatus 80 may further include: a training module 805, wherein:

[0136] The training module 805 is used to obtain a training sample set, where the training sample set includes multiple sample videos.

[0137] The training module 805 is further used to train the original model according to the image information of the multiple sample videos and the corresponding feature description information.

[0138] The processing module 803 is used to use the trained first branch network as the feature vector extraction model.

[0139] Optionally, the training module 805 is also used to: input the image information of each sample video in the multiple sample videos into the first branch network to obtain the video feature vector of each sample video; input the multiple labels included in the second classification information of each sample video into the second branch network to obtain the label feature vector of each sample video; and use the video feature vector and the label feature vector to adjust the network parameters of the original model to train the original model.

[0140] Optionally, the training module 805 is also used to: divide each sample video of the multiple sample videos into K video segments, where K is a positive integer, and randomly sample a picture from each of the K video segments to obtain K pictures of each sample video, and then input the K pictures of each sample video as image information into the first branch network to obtain a video feature vector of each sample video.

[0141] Optionally, the training module 805 is also used to: obtain all labels included in the second classification information of the multiple sample videos; generate a label dependency matrix based on all the labels and the multiple labels included in the second classification information of each sample video, wherein the label dependency matrix includes the correlation coefficient between any two labels; and input the label feature matrix corresponding to the multiple labels included in the second classification information of each sample video and the label dependency matrix into the second branch network to obtain the label feature vector of each sample video.

[0142] Optionally, the training module 805 is also used to: obtain the first classification information of each sample video; determine a total loss value based on the video feature vector, the label feature vector and the first classification information, the total loss value including the loss value of the first category loss function and the loss value of the second category loss function; and use the total loss value to adjust the network parameters of the first branch network to train the original model.

[0143] Optionally, the training module 805 is also used to: determine the loss value of the first category loss function based on the video feature vector of each sample video and the feature vector corresponding to the first classification information; determine the loss value of the second category loss function based on the video feature vectors of the multiple sample videos and the first classification information, the second category loss function including a normalized exponential function with an increased boundary; determine the total loss value based on the loss value of the first category loss function and the loss value of the second category loss function.

[0144] As for the device embodiment, since it is basically similar to the method embodiment, the relevant parts may refer to the partial description of the method embodiment.

[0145] See also Fig. 9 , is a schematic diagram of the structure of a computer device provided in an embodiment of the present application, the computer device 90 may include a processor 901, a memory 902, a network interface 903 and at least one communication bus 904. Among them, the processor 901 is used to schedule computer programs, and may include a central processing unit, a controller, and a microprocessor; the memory 902 is used to store computer programs, and may include a high-speed random access memory, a non-volatile memory, such as a disk storage device, a flash memory device; the network interface 903 provides a data communication function, and the communication bus 904 is responsible for connecting various communication elements.

[0146] The processor 901 may be used to call a computer program in the memory to perform the following operations:

[0147] Acquire multiple videos to be processed; input the multiple videos to be processed into a feature vector extraction model to obtain a video feature vector of each video to be processed, wherein the feature vector extraction model is obtained by training an original model using image information of multiple sample videos and corresponding feature description information, the feature description information includes first classification information for describing coarse-grained semantic features of the video, and second classification information for describing fine-grained semantic features of the video, the original model includes a first branch network and a second branch network, the first branch network is used to learn the image information and the first classification information, and the second branch network is used to assist the first branch network in learning the second classification information; according to the video feature vector of each video to be processed, divide the multiple videos to be processed into at least one video cluster.

[0148] In a possible embodiment, the processor 901 is further configured to: send the at least one video cluster to a terminal device, so that the terminal device displays the videos included in each video cluster in a user interface.

[0149] In a possible embodiment, the processor 901 is also used to: obtain a training sample set, the training sample set including multiple sample videos; train the original model according to the image information of the multiple sample videos and the corresponding feature description information; and use the trained first branch network as the feature vector extraction model.

[0150] In a possible embodiment, the processor 901 is specifically used to: input the image information of each sample video among the multiple sample videos into the first branch network to obtain the video feature vector of each sample video; input the multiple labels included in the second classification information of each sample video into the second branch network to obtain the label feature vector of each sample video; and use the video feature vector and the label feature vector to adjust the network parameters of the original model to train the original model.

[0151] In a possible embodiment, the processor 901 is specifically used to: divide each sample video of the multiple sample videos into K video segments, where K is a positive integer; randomly sample a picture from each of the K video segments to obtain K pictures of each sample video; and input the K pictures of each sample video as image information into the first branch network to obtain a video feature vector of each sample video.

[0152] In a possible embodiment, the processor 901 is specifically used to: obtain all labels included in the second classification information of the multiple sample videos; generate a label dependency matrix based on all the labels and the multiple labels included in the second classification information of each sample video, and the label dependency matrix includes the correlation coefficient between any two labels; input the label feature matrix corresponding to the multiple labels included in the second classification information of each sample video and the label dependency matrix into the second branch network to obtain the label feature vector of each sample video.

[0153] In a possible embodiment, the processor 901 is specifically used to: obtain the first classification information of each sample video; determine a total loss value based on the video feature vector, the label feature vector and the first classification information, the total loss value including the loss value of the first category loss function and the loss value of the second category loss function; use the total loss value to adjust the network parameters of the first branch network to train the original model.

[0154] In a possible embodiment, the processor 901 is specifically used to: determine the loss value of the first category loss function based on the video feature vector of each sample video and the feature vector corresponding to the first classification information; determine the loss value of the second category loss function based on the video feature vectors of the multiple sample videos and the first classification information, the second category loss function including a normalized exponential function with an increased boundary; determine the total loss value based on the loss value of the first category loss function and the loss value of the second category loss function.

[0155] It should be understood that the computer device described in the embodiments of the present application can implement the description of the data processing method in the embodiments, and can also execute the description of the data processing device in the corresponding embodiments, which will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here.

[0156] In addition, it should be noted that the embodiment of the present application also provides a storage medium, in which a computer program of the aforementioned data processing method is stored, and the computer program includes program instructions. When one or more processors load and execute the program instructions, the description of the data processing method in the embodiment can be implemented, which will not be repeated here, and the description of the beneficial effects of using the same method will not be repeated here. It is understandable that the program instructions can be deployed and executed on one or more computer devices that can communicate with each other.

[0157] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the steps performed in the embodiments of the above-mentioned methods.

[0158] Finally, it should be noted that the terms in the specification and claims of the present application and the above-mentioned drawings, such as first and second, etc., are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0159] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A data processing method based on artificial intelligence, characterized in that: include: Get multiple videos to be processed; Inputting the plurality of videos to be processed into a feature vector extraction model to obtain a video feature vector of each video to be processed; According to the video feature vector of each video to be processed, the multiple videos to be processed are divided into at least one video cluster; The training process of the feature vector extraction model includes: Acquire image information and corresponding feature description information of a plurality of sample videos, wherein the feature description information includes first classification information for describing coarse-grained semantic features of the videos and second classification information for describing fine-grained semantic features of the videos; Inputting image information of each sample video of the plurality of sample videos into a first branch network included in the original model to obtain a video feature vector of each sample video; Inputting a plurality of labels included in the second classification information of each sample video into the second branch network included in the original model for dependency mapping processing to obtain a label feature vector of each sample video; Training the first branch network according to the video feature vector, the label feature vector and the first classification information; The trained first branch network is used as the feature vector extraction model.

2. The method according to claim 1, characterized in that The first classification information includes a single label, and the second classification information includes a plurality of labels.

3. The method according to claim 1, characterized in that After dividing the plurality of videos to be processed into at least one video cluster according to the video feature vector of each video to be processed, the method further comprises: The at least one video cluster is sent to a terminal device, so that the terminal device displays the videos included in each video cluster in a user interface.

4. The method according to claim 1, characterized in that The step of obtaining image information of a plurality of sample videos includes: Divide each sample video among the multiple sample videos into K video segments, where K is a positive integer; Randomly sampling a picture from each of the K video clips to obtain K pictures of each sample video; The K pictures of each sample video are used as image information.

5. The method according to claim 1, characterized in that The step of inputting the multiple labels included in the second classification information of each sample video into the second branch network included in the original model for dependency mapping processing to obtain the label feature vector of each sample video includes: Generate a label dependency matrix according to all labels included in the second classification information of the plurality of sample videos and the plurality of labels included in the second classification information of each sample video, wherein the label dependency matrix includes a correlation coefficient between any two labels; The label feature matrix corresponding to the multiple labels included in the second classification information of each sample video and the label dependency matrix are input into the second branch network included in the original model for dependency mapping processing to obtain the label feature vector of each sample video.

6. The method according to any one of claims 1 to 5, characterized in that: The training of the first branch network according to the video feature vector, the label feature vector and the first classification information includes: Determine a total loss value according to the video feature vector, the label feature vector and the first classification information; The network parameters of the first branch network are adjusted using the total loss value to train the original model.

7. The method according to claim 6, characterized in that The total loss value includes a loss value of a first category loss function and a loss value of a second category loss function, and determining the total loss value according to the video feature vector, the label feature vector and the first classification information includes: Determining a loss value of the first category loss function according to the video feature vector of each sample video and the feature vector corresponding to the first classification information; Determining a loss value of the second category loss function according to the video feature vectors of the plurality of sample videos and the first classification information, wherein the second category loss function includes a normalized exponential function that increases a boundary; A total loss value is determined according to the loss value of the first category loss function and the loss value of the second category loss function.

8. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a network communication function, the memory is used to store program codes, and the processor is used to call the program codes to execute the method described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 7 is executed.

10. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method according to any one of claims 1 to 7.

Images