Video classification method and device, storage medium and electronic device

Abstract

The present invention discloses a video classification method and device, storage medium, and electronic device related to artificial intelligence and cloud servers. The method comprises: upon obtaining a target video to be classified, obtaining image features of each frame in the target video; clustering all video frames in the target video based on the feature distances between the image features of each video frame to obtain multiple video segments; obtaining the corresponding segment temporal fusion features of each of the multiple video segments; determining the video features of the target video based on the corresponding segment temporal fusion features of each video segment; and identifying the video features to obtain the target type of the target video. The present invention solves the technical problem of inaccurate video classification.

Description

Technical Field

[0001] The present invention relates to the field of computer computing, and in particular to a video classification method and device, a storage medium, and an electronic device. Background Art

[0002] In the prior art, when classifying a video, if the video is a long video, the classification result of the video may be inaccurate because the content of the long video includes different scenes.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] Embodiments of the present invention provide a video classification method and apparatus, a storage medium, and an electronic device to at least solve the technical problem of inaccurate video classification.

[0005] According to one aspect of an embodiment of the present invention, a video classification method is provided, comprising: upon obtaining a target video to be classified, obtaining image features of each video frame in the target video; clustering all video frames in the target video according to feature distances between the image features of the respective video frames to obtain a plurality of video segments, wherein each of the plurality of video segments includes a plurality of ordered video frames; obtaining a segment timing fusion feature corresponding to each of the plurality of video segments, wherein the segment timing fusion feature is a feature determined according to a target video frame in the video segment, and the target video frame is a portion of the video frames in the video segment; determining the video features of the target video according to the segment timing fusion features corresponding to each of the video segments; and identifying the video features to obtain a target type of the target video.

[0006] According to another aspect of an embodiment of the present invention, a video classification device is also provided, including: a first acquisition unit, for acquiring image features of each video frame in the target video when a target video to be classified is acquired; a clustering unit, for clustering all video frames in the target video according to feature distances between the image features of each video frame, so as to obtain multiple video segments, wherein each of the multiple video segments includes multiple ordered video frames; a second acquisition unit, for acquiring segment timing fusion features corresponding to each of the multiple video segments, wherein the segment timing fusion features are features determined based on the target video frames in the video segments, and the target video frames are part of the video frames in the video segments; a determination unit, for determining video features of the target video according to the segment timing fusion features corresponding to each of the video segments; an identification unit, for identifying the video features to obtain the target type of the target video.

[0007] As an optional example, the clustering unit includes: a first determination module for determining multiple target features; a second processing module for taking each of the above-mentioned image features of the above-mentioned target video as a current feature, performing the following operations on the above-mentioned current feature to obtain the above-mentioned multiple video clips: determining the feature distance between the above-mentioned current feature and each of the above-mentioned target features in the above-mentioned multiple target features, and determining the first target feature corresponding to the minimum value of the above-mentioned feature distance and the above-mentioned current feature as two features in a group of features; the second determination module is used to determine the video frame corresponding to the above-mentioned image feature in the same group of features as a video frame in the above-mentioned video clip after traversing each of the above-mentioned image features of the above-mentioned target video.

[0008] As an optional example, the above-mentioned first determination module includes: a first determination submodule, used to determine multiple initial features; a first processing submodule, used to perform the following operations on the above-mentioned current feature to obtain multiple first distances: determine the first distance between each of the above-mentioned current features and the first initial feature among the above-mentioned multiple initial features, wherein the above-mentioned first initial feature is the feature with the smallest distance from the above-mentioned current feature among the above-mentioned multiple initial features, and each of the above-mentioned current features corresponds to one of the above-mentioned first distances; a second determination submodule, used to determine the sum of the multiple first distances; an adjustment module, used to adjust the above-mentioned multiple initial features to obtain the re-determined above-mentioned sum; and a third determination submodule, used to determine the above-mentioned multiple initial features corresponding to the smallest above-mentioned sum as the above-mentioned multiple target features.

[0009] As an optional example, the above-mentioned determination unit includes: a first acquisition module, used to obtain the above-mentioned fragment timing fusion features of each of the above-mentioned video fragments; a third processing module, used to perform an averaging operation on multiple above-mentioned fragment timing fusion features to obtain an average feature; and a third determination module, used to use the above-mentioned average feature as the above-mentioned video feature.

[0010] As an optional example, the above-mentioned first acquisition module includes: a second processing sub-module, which is used to take each of the above-mentioned video segments as the current segment, and perform the following operations on the above-mentioned current segment: determine the target frame features of each of the above-mentioned target video frames in the above-mentioned current segment; determine each of the above-mentioned target frame features as the current frame features, and perform the following operations on the above-mentioned current frame features: use the features of the first part of the first frame features to replace the features of the above-mentioned first part of the current frame features and use the features of the second part of the second frame features to replace the features of the above-mentioned second part of the current frame features, wherein the above-mentioned first frame features are the frame features of the previous frame of the video frame where the above-mentioned current frame features are located in the above-mentioned target video frame, and the above-mentioned second frame features are the frame features of the next frame of the video frame where the above-mentioned current frame features are located in the above-mentioned target video frame; each of the above-mentioned target frame features after processing are fused in chronological order into the above-mentioned segment temporal fusion features of the above-mentioned current segment.

[0011] As an optional example, the above-mentioned third processing module includes: a first pooling sub-module, used to perform a pooling operation on each of the multiple fragment timing fusion features in the time dimension to obtain the pooled features after pooling; a fourth determination sub-module, used to average the multiple pooling features to determine the features as the above-mentioned average features.

[0012] As an optional example, the above-mentioned second acquisition unit includes: a second acquisition module, used to use the feature extraction layer of the target neural network model to extract the above-mentioned fragment timing fusion features; the above-mentioned recognition unit includes: a first recognition module, used to use the fully connected layer of the above-mentioned target neural network model to recognize the above-mentioned video features to obtain the above-mentioned target type.

[0013] As an optional example, the second acquisition unit further includes: a third acquisition module, used to acquire a sample video before using the feature extraction layer of the target neural network model to extract the fragment temporal fusion feature, wherein the sample video is marked with a sample type; an input module, used to input the sample video into the target neural network model to obtain the estimated type of the sample video output by the target neural network model; a first adjustment module, used to adjust the first parameter of the feature extraction layer and the second parameter of the fully connected layer of the target neural network model when the estimated type is different from the sample type.

[0014] As an optional example, the sample video is also marked with the first position of the key frame, and the second acquisition unit also includes: a fourth acquisition module, which is used to obtain the sample segment temporal fusion features of each sample video segment of the sample video when the sample video is input into the target neural network model to obtain the video type of the sample video output by the target neural network model; a pooling module, which is used to perform a maximum pooling operation on each of the sample segment temporal fusion features to obtain the pooled sample features after pooling; a splicing module, which is used to splice multiple pooled sample features into two-dimensional features; a second recognition module, which is used to identify the two-dimensional features by the fully connected layer of the target neural network model to obtain the estimated position of the key frame of the sample video; a second adjustment module, which is used to adjust the first parameter and the second parameter when the estimated position is different from the first position.

[0015] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the above-mentioned video classification method when running.

[0016] According to another aspect of an embodiment of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the video classification method through the computer program.

[0017] In an embodiment of the present invention, when a target video to be classified is obtained, image features of each video frame in the target video are obtained; all video frames in the target video are clustered according to feature distances between the image features of each video frame to obtain a plurality of video segments, wherein each of the plurality of video segments includes a plurality of ordered video frames; and segment temporal fusion features corresponding to each of the plurality of video segments are obtained, wherein the segment temporal fusion features are features determined according to the target video frame in the video segment, and the target video frame is a portion of the video frame in the video segment. frequency frames; determining the video features of the target video according to the temporal fusion features corresponding to each of the above video clips; identifying the video features to obtain the target type of the target video. Because in the above method, when the target video to be classified is obtained, it is first clustered into multiple video clips according to the feature distance of the image features of the video frames in the target video, and then the video features of the target video are determined based on the temporal fusion features of the video clips. The target type of the target video is identified according to the video features, thereby taking into account the contents of different scenes in the long video, improving the accuracy of video classification, and then solving the technical problem of inaccurate video classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1 is a schematic diagram of an application environment of an optional video classification method according to an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of an application environment of another optional video classification method according to an embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of a process of an optional video classification method according to an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of obtaining a target video frame according to an optional video classification method according to an embodiment of the present invention;

[0023] Figure 5 is a schematic diagram of replacement features of an optional video classification method according to an embodiment of the present invention;

[0024] Figure 6 is a schematic diagram of replacement features of an optional video classification method according to an embodiment of the present invention;

[0025] Figure 7 is a system diagram of an optional video classification method according to an embodiment of the present invention;

[0026] Figure 8 is a schematic structural diagram of an optional video classification device according to an embodiment of the present invention;

[0027] Figure 9 FIG. 4 is a schematic structural diagram of an optional electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] According to one aspect of an embodiment of the present invention, a video classification method is provided. Optionally, as an optional implementation, the video classification method can be applied to, but is not limited to, Figure 1 in the environment shown.

[0031] like Figure 1As shown, the terminal device 102 includes a memory 104 for storing various data generated during the operation of the terminal device 102, a processor 106 for processing and calculating the above-mentioned data, and a display 108 for displaying the classification results of the target video. The terminal device 102 can exchange data with a server 112 via a network 110. The server 112 includes a database 114 for storing various data and a processing engine 116 for processing the above-mentioned data. Through steps S102 to S106, the terminal device 102 can send the target video to the server 112, and the server 112 identifies the target type of the target video and sends the target type.

[0032] As an optional implementation, the above video classification method can be applied to, but is not limited to, Figure 2 in the environment shown.

[0033] like Figure 2 As shown, terminal device 202 includes a memory 204 for storing various data generated during the operation of terminal device 202, a processor 206 for processing and calculating the aforementioned data, and a display 208 for displaying the classification results of the target video. Terminal device 202 can perform steps S202 to S210 to identify the target video and obtain the target type.

[0034] Optionally, in this embodiment, the terminal device 102 or the terminal device 202 may be a terminal device configured with a target client, which may include but is not limited to at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop computer, a tablet computer, a PDA, an MID (Mobile Internet Devices), a PAD, a desktop computer, a smart TV, etc. The target client may be a video client, an instant messaging client, a browser client, an education client, etc. The network 110 may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network, and a wide area network, and the wireless network includes: Bluetooth, WIFI, and other networks that implement wireless communication. The server 112 may be a single server, or a server cluster consisting of multiple servers, or a cloud server. The above is only an example, and this embodiment does not impose any limitation on this.

[0035] The server 112 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to these. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0036] The video classification method described in this embodiment involves artificial intelligence. Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI is the study of the design principles and implementation methods of various intelligent machines, enabling them to have the capabilities of perception, reasoning, and decision-making.

[0037] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0038] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, where cameras and computers replace the human eye in identifying, tracking, and measuring objects. This involves further processing the images, transforming them into images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0039] Optionally, the above-mentioned video classification method can be applied to blockchain nodes. The blockchain in this embodiment is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a string of data blocks generated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of the information (anti-counterfeiting) and generate the next block. Blockchain can include the blockchain underlying platform, the platform product service layer, and the application service layer.

[0040] The underlying blockchain platform can include processing modules such as user management, basic services, smart contracts, and operation monitoring. Among them, the user management module is responsible for the identity information management of all blockchain participants, including maintaining public and private key generation (account management), key management, and maintaining the corresponding relationship between the user's real identity and the blockchain address (authority management), etc., and under authorization, it supervises and audits the transactions of certain real identities and provides risk control rule configuration (risk control audit); the basic service module is deployed on all blockchain node devices to verify the validity of business requests, and records the valid requests to the storage after consensus is reached. For a new business request, the basic service first adapts the interface to parse and authenticate the request (interface adaptation), and then encrypts the business information through the consensus algorithm (consensus management). The smart contract module is responsible for the registration, issuance, triggering and execution of contracts. Developers can define the contract logic in a programming language and publish it to the blockchain (contract registration). According to the logic of the contract terms, the contract logic is triggered by calling keys or other events to trigger execution. The contract logic is completed, and the contract upgrade and cancellation functions are also provided. The operation monitoring module is mainly responsible for the deployment, configuration modification, contract setting, cloud adaptation and real-time status visualization output of the product during the product release process, such as alarms, network status monitoring, and node device health monitoring.

[0041] The platform's product service layer provides the basic capabilities and implementation framework for typical applications. Developers can build on these basic capabilities, overlay business features, and complete the blockchain implementation of business logic. The application service layer provides application services based on blockchain solutions for business participants to use.

[0042] Alternatively, as an optional implementation, Figure 3 As shown, the above video classification method includes:

[0043] S302, when a target video to be classified is obtained, obtaining image features of each video frame in the target video;

[0044] S304, performing clustering processing on all video frames in the target video based on feature distances between image features of each video frame to obtain a plurality of video segments, wherein each of the plurality of video segments includes a plurality of ordered video frames;

[0045] S306, obtaining a segment temporal fusion feature corresponding to each video segment in the plurality of video segments, wherein the segment temporal fusion feature is a feature determined based on a target video frame in the video segment, and the target video frame is a portion of the video frames in the video segment;

[0046] S308, determining the video features of the target video based on the segment temporal fusion features corresponding to each video segment;

[0047] S310: Identify video features to obtain a target type of the target video.

[0048] Optionally, this embodiment can be applied, but is not limited to, to video classification. For example, it can be applied to network security monitoring to classify videos into normal and illegal videos. For example, it can be applied to video players to identify the target video type and assign different types of labels to the target video. For example, it can be applied to learning and education to identify the target video and determine the subject type of the target video, thereby classifying the target video.

[0049] Taking the application of the above-mentioned video classification method to a video player as an example, after the video player obtains the video uploaded by the personal media account, it can identify the video. First, the target video is clustered into multiple video segments through the image features of each video frame, and a segment timing fusion feature is determined for each video segment. The segment timing fusion feature is used to determine the video feature of the video, and finally the video feature is identified to determine the type of the video. The video is labeled with a corresponding type, and the video can be searched when searching for resources with the label on the user side. Since the content of long videos in different scenarios is taken into account in the above-mentioned video classification process, the above-mentioned method improves the accuracy of video classification.

[0050] Taking the application of the above-mentioned video classification method to network security monitoring as an example, after obtaining videos uploaded by personal media accounts, the videos can be identified. First, the target video is clustered into multiple video segments based on the image features of each video frame. For each video segment, a segment temporal fusion feature is determined. The segment temporal fusion feature is used to determine the video features of the video. Finally, the video features are identified to determine the type of video. The video is labeled with a corresponding type, which can be a label indicating that the video is legal or illegal. Illegal videos can be blocked, while legal videos can be allowed to be viewed.

[0051] Optionally, the target video may be of a plurality of pre-set types. For example, the target video may be a normal video or an illegal video, a video containing language content or a video containing mathematics content, or a suspense type, a thriller type, a comedy type, etc. The plurality of types may be classified according to different classification methods, which is not limited in this embodiment.

[0052] Optionally, the target video frame is a portion of the video frame in the video clip. One frame can be intercepted from the video clip every other frame, and the intercepted video frame can be used as the target video frame. Alternatively, multiple frames can be randomly extracted from the target video in order as the target video frame. For example, Figure 4 As shown, Figure 4 For a video of 30 frames, for the 30 frames of the video, one frame of picture is intercepted every 10 frames as the target video frame, and three frames of target video frames 404 are extracted. The above-mentioned intercepting one frame of picture every 10 frames as the target video frame is an example, and one frame of picture can also be intercepted every 5 frames. Optionally, in this embodiment, a continuous section of video frames can also be intercepted in the video clip as the target video frame. For example, the 10th to 15th frames of the 30-frame video clip are intercepted as the target video frame. Optionally, in this embodiment, when intercepting the target video frame, the target video frame can also be intercepted according to the scene of the video clip. For example, the scene in the video clip can be identified, multiple scenes can be identified, and one or more frames of video frames can be intercepted for each scene.

[0053] Optionally, the recognition of the above-mentioned scenes can be achieved by identifying the colors of video frames in a video clip. The color differences of pixels at the same position in two adjacent video frames in the video clip are counted. If the weighted sum of the color differences is greater than a predetermined threshold, the two adjacent video frames are considered to belong to video frames in different scenes. Multiple video frames in which the weighted sum results of the two adjacent video frames are less than or equal to the predetermined threshold can be determined as video frames of one scene. Through this embodiment, through the above-mentioned method, when identifying the video type, the content of different scenes in the target video can be taken into consideration, thereby improving the accuracy of identifying the video type.

[0054] Optionally, in this embodiment, after acquiring the target video, when acquiring the image features of each video frame of the target video, the histogram features of each video frame can be acquired. The method can be: determining each video frame of the target video as the current video frame, performing the following operations on the current video frame: acquiring multiple target grayscale values, determining each target grayscale value as the current grayscale value, counting the number of pixels in the current video frame whose grayscale value is the same as the current grayscale value to obtain a first number, wherein in the current video frame, each target grayscale value corresponds to a first number; and determining a sequence consisting of multiple first numbers as the image feature of the current video frame.

[0055] Optionally, the target grayscale value can be a manually set value or a value obtained by calculation during the operation of the system. There can be multiple target grayscale values, and the number is not limited. For example, there can be 16 target grayscale values, and the 16 target grayscale values ​​are different. Then, for each frame of video, the grayscale value of each pixel in the frame of video can be obtained. Count the number of grayscale values ​​of each pixel in the frame of video that are the same as any of the above 16 target grayscale values. For example, the 16 target grayscale values ​​include grayscale value 30, grayscale value 55, and grayscale value 126... a total of 16 values ​​(for example only). Then, for a frame of the target video, count the number of pixels in the frame of video that have grayscale values ​​of the above 16 values. For example, there are 3 pixels with a grayscale value of 30, 20 pixels with a grayscale value of 55, 80 pixels with a grayscale value of 126, and so on. 16 numbers are obtained. The 16 grayscale values ​​are pulled into a vector to obtain a 16-dimensional histogram vector feature. This histogram vector feature is the image feature of the video frame of the target video.

[0056] Optionally, in this embodiment, when acquiring the image features of each frame of image, color features of the image can also be extracted. The color features can be color moments of the color. Color moments are divided into first-order moments, second-order moments, and third-order moments. The first-order moment is the mean, which in this embodiment can be the mean of the pixel positions of the pixels corresponding to the color in the image. The second-order moment is the variance, which in this embodiment can be the variance of the pixel positions of the pixels corresponding to the color in the image. The third-order moment is the slope, which in this embodiment can be the slope of the pixel positions of the pixels corresponding to the color in the image. When acquiring the image features of a frame of image, the colors included in the image can be determined. For each color, the first-order moment, second-order moment, and third-order moment of that color can be determined to obtain the features of that color. The features of each color can be combined to obtain the image features of the image. In other words, for a frame of image, a first-order moment, second-order moment, and third-order moment can be obtained for each color. For example, if a frame of image has five colors, there are five first-order moments, five second-order moments, and five third-order moments in total. The combination of the five first-order moments, five second-order moments, and five third-order moments constitutes the image features of the frame of image.

[0057] Optionally, in this embodiment, image features are clustered based on feature distances between the image features to obtain multiple video segments of the target video. This may include: determining multiple target features; using each image feature of the target video as a current feature, and performing the following operations on the current feature to obtain multiple video segments: determining a feature distance between the current feature and each of the multiple target features, and determining the first target feature corresponding to the minimum value of the feature distance and the current feature as two features in a set of features; after traversing each image feature of the target video, determining the video frames corresponding to the image features in the same set of features as video frames in a video segment.

[0058] That is to say, in this embodiment, when clustering is performed, multiple target features can be determined first. For all video frames of the target video, each video frame is used as the current video frame, and then the feature distance between the video feature of the current video frame and each target feature in the target feature is determined. As a result, the feature distance between the video feature of the current video frame and the first target feature in the target feature is the smallest. The current video frame is then determined to be a group with the first target feature. Finally, the video frames of the target video can be divided into multiple groups. For a video frame in a group, the feature distance between the image feature of the video frame and the first target feature of the group is smaller than the feature distance between the image feature of the video frame and the first target feature of any other group. The video frames of each group are regarded as video frames in a video clip.

[0059] Optionally, the videos in each group may be sorted in chronological order to obtain a video clip.

[0060] Optionally, in this embodiment, the aforementioned multiple target features may be pre-calculated. For example, multiple initial features may be determined; the following operations may be performed on the current feature to obtain multiple first distances: determining a first distance between each current feature and a first initial feature among the multiple initial features, where the first initial feature is the feature with the smallest distance from the current feature among the multiple initial features, and each current feature corresponds to a first distance; determining the sum of the multiple first distances; adjusting the multiple initial features to obtain a re-determined sum; and determining the multiple initial features corresponding to the minimum sum as the multiple target features.

[0061] In this embodiment, after determining multiple initial features and determining the sum, the initial features can be adjusted according to a strategy, and then the sum can be determined again. If the determined sum is larger than the previous sum, the adjustment strategy is switched. If the determined sum is smaller than the previous sum, the multiple initial features are continued to be adjusted according to the strategy. For example, a portion of the multiple initial features can be adjusted first, such as adjusting one first, and then the sum is re-determined. If the determined sum is larger than the sum before adjustment, another feature of the initial features is adjusted. If the determined sum is smaller than the sum before adjustment, the adjustment of a portion of the initial features is continued.

[0062] In other words, multiple initial features can be randomly determined, and then the sum of the first distances of these initial features can be calculated. After adjustment, multiple sums are obtained. The initial features corresponding to the minimum sum are determined as the multiple target features. This method can effectively ensure the degree of correlation between the video frames in the clustered video clips, thereby improving clustering accuracy.

[0063] Optionally, in this embodiment, when clustering the video frames extracted from the target video, the image features can be clustered, and each image feature can be regarded as a class. Then, the distance between each two classes of image features is determined, and the two classes of image features with the smallest distance are merged into one class of image features. The process of determining the distance between each two classes of image features and merging the two classes of image features with the smallest distance into one class of image features is repeated until the proportion of the number of remaining classes to the initial number of classes is less than a predetermined value, for example, less than 10%.

[0064] For example, if there are 10 images, each image corresponds to an image feature, for a total of 10 image features. Each image feature is divided into a category, and the distance between each category of image features is calculated. If the distance between the third and sixth categories of image features is the smallest, the two categories of image features are merged into one category, resulting in 9 new categories of image features. The distance calculation and merging steps are repeated. If the predetermined value is 30%, then when the 10 categories of image features of the 10 image features are merged into 3 categories of image features, the distance calculation and merging steps are stopped. At this time, the 10 image features are clustered into 3 categories of image features. The 10 images corresponding to the 10 image features are then clustered into 3 video clips.

[0065] Optionally, in this embodiment, after acquiring multiple video clips, a temporal fusion feature of each video clip can be acquired, and then the video features of the target video can be obtained based on the temporal fusion feature. For example, a temporal fusion feature of each video clip can be acquired; the multiple temporal fusion features can be averaged to obtain an average feature; and the average feature can be used as the video feature.

[0066] For example, for the temporal fusion features H1*W1*C1*t1 and H2*W2*C2*t2, averaging them yields the following average feature: (H1+H2) / 2*(W1+W2) / 2*(C1+C2) / 2*(t1+t2) / 2. To average the temporal fusion features of multiple clips, the above method can also be used to average the individual features.

[0067] The above averaging operation can be used to calculate the average of the features to obtain the average feature.

[0068] Optionally, in this embodiment, obtaining the segment timing fusion features of each video segment includes: taking each video segment as the current segment, and performing the following operations on the current segment: determining the target frame features of each target video frame in the current segment; determining each target frame feature as the current frame feature, and performing the following operations on the current frame feature: using the features of the first part of the first frame feature to replace the features of the first part of the current frame feature and using the features of the second part of the second frame feature to replace the features of the second part of the current frame feature, wherein the first frame feature is the frame feature of the previous frame of the video frame where the current frame feature is located in the target video frame, and the second frame feature is the frame feature of the next frame of the video frame where the current frame feature is located in the target video frame; fusing each processed target frame feature into the segment timing fusion feature of the current segment in chronological order.

[0069] That is, for each video clip, multiple target video frames can be first determined from the video clip. The determination method is mentioned above and will not be repeated here. After determining multiple target video frames, for each target video frame, the target frame features are partially replaced with the target frame features of the previous target video frame, and partially replaced with the target frame features of the next target video frame, thereby obtaining the replaced target frame features. The replaced target frame features are used as the segment temporal fusion features.

[0070] For example, for a video clip containing 60 frames, one frame is extracted every 20 frames, and 3 target video frames are extracted. For the 3 target video frames, each target video frame corresponds to a target frame feature. For the second target frame feature, the first part of the feature is replaced by the first part of the feature of the first target frame feature, and the second part of the feature is replaced by the second part of the feature of the third target frame feature. For example, Figure 5 As shown, Figure 5The target frame features of three target video frames are included, namely target frame feature 502, target frame feature 504, and target frame feature 506. The target video frame where target frame feature 502 is located is located before the target video frame where target frame feature 504 is located, and the target video frame where target frame feature 506 is located is located after the target video frame where target frame feature 504 is located. The features of the first part 502-1 of the target frame feature 502 are replaced with the features of the first part 504-1 of the target frame feature 504, and the features of the second part 506-2 of the target frame feature 506 are replaced with the features of the second part 504-2 of the target frame feature 504. After the replacement, for target frame feature 502, if it is the target frame feature of the first frame in the target video frame, its first part 502-1 is padded with zeros. For target frame feature 506, if it is the target frame feature of the last frame in the target video frame, its second part 506-2 is padded with zeros. If the target frame feature 502 and the target frame feature 506 are not in the first frame and the last frame of the target video frame, the target frame feature 504 is processed using the same processing method.

[0071] Optionally, in this embodiment, after determining the target video frame and replacing the frame features of the target video frame to obtain a segment temporal fusion feature, an averaging operation can be performed on multiple segment temporal fusion features to obtain an average feature. For example, each of the multiple segment temporal fusion features can be pooled in the time dimension to obtain a pooled feature; and the feature determined by averaging the multiple pooled features is used as the average feature.

[0072] For example, the temporal fusion features of each clip are pooled over time to obtain pooled features. The pooled features are then averaged to obtain average features. The average features are used as the video features of the target video, and the video features are identified to determine the target video's object type.

[0073] Optionally, in this embodiment, a target neural network model can be used to extract segment timing fusion features from the clustered video segments, and to determine the video features of the target video based on the segment timing fusion features corresponding to each video segment and to identify the video features of the target video to obtain a target type. The target neural network model in this embodiment can be a pre-trained model. During training, a sample video can be obtained, wherein the sample video is labeled with a sample type; the sample video is input into the target neural network model to obtain an estimated type of the sample video output by the target neural network model; when the estimated type and the sample type are different, the first parameter of the feature extraction layer and the second parameter of the fully connected layer of the target neural network model are adjusted.

[0074] For example, the sample video may be a video labeled with a sample type. For example, a sample video is labeled with a comedy type. The sample video first undergoes the above-mentioned steps of obtaining image features and clustering to obtain multiple video segments, and then uses the target neural network model to obtain the segment timing fusion features of each video segment of the sample video, and determines the video features of the sample video based on the segment timing fusion features, and finally identifies the estimated type of the sample video. If the estimated type and the sample type are different, it means that the result of the target neural network model recognition is inaccurate. The first parameter and the second parameter of the target neural network model can be adjusted. In this embodiment, the feature extraction layer can be a convolutional layer, which is responsible for extracting features and calculations, and the fully connected layer is responsible for identifying features.

[0075] As another optional method, it is also possible to determine whether to adjust the first parameter and the second parameter based on the recognition accuracy of the target neural network model. For example, for a batch of sample videos, such as 100 sample videos, each sample video is labeled with a type. Each of the 100 sample videos is subjected to the above-mentioned steps of obtaining image features and clustering to obtain multiple video segments. Then, the target neural network model is used to obtain the segment timing fusion features of each video segment of the sample video, and the video features of the sample video are determined based on the segment timing fusion features. Finally, the estimated type of the sample video is identified, and the estimated type is compared with the label of the sample video. It can be checked how many of the 100 sample videos have accurate recognition results and how many have inaccurate recognition results. The recognition accuracy is determined. For example, if 90 sample videos are accurately recognized, the recognition accuracy is 90%.

[0076] In this embodiment, the first and second parameters do not need to be adjusted after each sample video is recognized. Instead, a batch of sample videos can be recognized, and the determination of whether to adjust the first and second parameters is based on the recognition results of the batch of sample videos. During this process, if the recognition accuracy is less than a predetermined value, such as 90% as described above, the recognition is considered inaccurate and the first and second parameters need to be adjusted. Of course, the recognition accuracy of each batch of sample videos can also be compared. If the recognition accuracy is increasing, even if it has not reached 90%, the first and second parameters can be temporarily adjusted. Adjustment can be made only when the recognition accuracy of the sample videos no longer increases. For example, if the recognition accuracy of several batches of sample videos is 50%, 60%, and 70%, respectively, the recognition accuracy is increasing, so the first and second parameters can be adjusted. If the recognition accuracy no longer increases, such as 50%, 60%, and 59%, the first and second parameters can be adjusted. Or if the increase in recognition accuracy is small, such as a preset increase of 5% and the recognition accuracy is 50%, 60%, and 61%, the first and second parameters can be adjusted.

[0077] In this embodiment, after the target neural network model inputs multiple sample videos, the first parameter and the second parameter can be adjusted when the recognition accuracy is low, thereby improving the recognition accuracy of the target neural network model.

[0078] Optionally, this embodiment also provides a method for training a target neural network model. If the sample video is also marked with the first position of the key frame, this embodiment can obtain the sample segment temporal fusion features of each sample video segment of the sample video; perform a maximum pooling operation on each sample segment temporal fusion feature to obtain a pooled sample feature after pooling; splice multiple pooled sample features into a two-dimensional feature; use the fully connected layer of the target neural network model to identify the two-dimensional feature to obtain the estimated position of the key frame of the sample video; and adjust the first parameter and the second parameter if the estimated position is different from the first position.

[0079] That is, in this embodiment, not only are the first and second parameters adjusted based on a comparison between the estimated type and the sample type output by the target neural network model, but the first position of the key frame of the sample video is also marked. After the sample video is input into the target neural network model, the target neural network model also identifies the estimated position of the key frame of the sample video. If the first position and the estimated position are inconsistent, or the similarity is less than a first threshold, the recognition result of the target neural network model is considered inaccurate. In this case, the first and second parameters can be adjusted until the recognition accuracy of the target neural network model exceeds the required recognition accuracy.

[0080] In this embodiment, when marking the first position of the key frame, the accuracy of marking the first position may also be determined, and the above method may be used to determine whether to adjust the first parameter and the second parameter.

[0081] The following is an explanation with a specific example.

[0082] This embodiment involves video semantic classification and feature maps.

[0083] Video semantic classification: refers to a type of algorithm that uses deep learning methods to learn image information and temporal information in videos to classify videos / recognize actions.

[0084] Feature map: The feature map obtained by convolving an image with a filter. A feature map can be convolved with a filter to generate a new feature map.

[0085] For a target video of a target type to be identified. After acquiring the target video, first parse out each frame of the target video, and then use the histogram algorithm to extract image features from each frame of the target video. Specifically, the target video can be split into a series of video frames first, such as a one-minute video is split into 60 frames per second. For each frame of the video, first calculate the histogram of the video frame. For example, if the parameter bin of the histogram is set to 16, then a video frame can obtain 16 grayscale pixels. The 16 values ​​are pulled into a vector, and then a video frame corresponds to a 16-dimensional histogram vector feature. The histogram vector feature is the image feature of the video frame. 60 histogram features can be obtained in a target video, and the 60 histogram features are directly divided into k parts using the kmeans algorithm. Frames of the same video clip category may be scattered in different time periods, and the smaller clips in the same category are directly classified as a class of video frames in the nearest time. Finally, a target video can obtain k video clips. As Figure 6 The video frames in different video clips are arranged in chronological order. Each video clip uses stride frame to sample the input video clip and convert the clip into a fixed number of frames. For example, in 60 frames, every 20 frames are sampled to obtain 3 target video frames. The target video frame of each video clip is obtained by TSM algorithm to obtain the temporal fusion feature of the clip. Figure 6 The temporal fusion features of all video clips will be averaged and pooled in the time dimension. For example, the feature map size of each clip is H*W*C*t (since the features in the H and W directions do not participate in the feature transfer of TSM, for the convenience of representation, Figure 6 (Only the features of the channel C and time series t dimensions are shown in the figure.) After pooling each segment in the t dimension, a feature map of size H*W*C is obtained. Then, k feature maps of size H*W*C are averaged again to obtain a feature map of size H*W*C. This feature map is then classified through a fully connected layer to determine the target type of the target video. After identifying the target type, the correct identification can be determined based on the feedback, thereby determining whether the first and second parameters need to be adjusted. Furthermore, key segment training can be used to further improve the recognition accuracy of the target recognition model.

[0086] The purpose of key segment training is to improve the recognition accuracy of the model. For a sample video, the start and end time periods of the key frames in the sample video, that is, the first position of the key frames in the sample video, are marked (the sample type of the sample video is also marked). After the sample video is input into the target neural network model and clustered into multiple sample video segments, for each sample video segment, a feature map (temporal fusion feature of the sample segment) with a size of H*W*C*t can be obtained through TSM. Figure 7 As shown. For K features, the maximum value pooling is performed on each H*W*C size Feature map in the t dimension, and finally the Feature map of each sample video segment is converted into a 1*1*1*t feature. The 1*1*1*t feature of each segment can be considered as the information proportion of the segment in the classification process. Each value in 1*1*1*t will be returned and multiplied with the H*W Feature map of the corresponding channel (the C dimension) to achieve the effect of improving the attention of the segment information of the sample video segment. The 1*1*1*t features of the k segments are spliced ​​into K*t two-dimensional features in time sequence, and the final decision feature vector, that is, the K*1 feature, is obtained through the fully connected layer to obtain the estimated position of the key frame. Specifically, the key frame label can be calculated based on the key frame annotation. During training, the sample video is pre-divided into k sample video segments by clustering. In each segment, the number of video frames belonging to the key frame is divided by the total number of video frames to obtain the key frame label of the current sample video segment; (2) Figure 7 The k*1 features obtained in are normalized using sigmoid, and then the loss is calculated using the following mean-squared loss function, and gradient backpropagation is performed to adjust the first and second parameters.

[0087]

[0088] where Y i , They represent the label and the predicted result respectively, and n represents the number of label categories.

[0089] During the entire training process, keyframe segment training is performed only on data with keyframe annotations. To ensure training effectiveness, keyframe-annotated videos must be collected in each training batch. This training approach influences the model's semantic classification predictions with keyframe training, improving the accuracy of semantic classification. Furthermore, testing eliminates the need for keyframe information and produces classification results that include keyframe segment information.

[0090] The above video classification method can sample information according to the length of the video, reduce information loss in long videos, and can simultaneously collect scene information and timing information, thereby strengthening the control of video information. In addition, only a small amount of key frame label data is required for the model to have perception of key frames while learning semantic classification, thereby improving the semantic classification effect.

[0091] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0092] According to another aspect of the embodiment of the present invention, a video classification device for implementing the above-mentioned video classification method is also provided. Figure 8 As shown, the device includes:

[0093] The first acquisition unit 802 is configured to acquire image features of each video frame in the target video when the target video to be classified is acquired;

[0094] a clustering unit 804 configured to perform clustering processing on all video frames in the target video based on feature distances between image features of the respective video frames to obtain a plurality of video segments, wherein each of the plurality of video segments includes a plurality of ordered video frames;

[0095] A second acquiring unit 806 is configured to acquire a segment temporal fusion feature corresponding to each of the plurality of video segments, wherein the segment temporal fusion feature is a feature determined based on a target video frame in the video segment, and the target video frame is a portion of the video frames in the video segment;

[0096] A determining unit 808 is configured to determine the video features of the target video based on the temporal fusion features of the respective video segments;

[0097] The recognition unit 810 is used to recognize video features and obtain a target type of the target video.

[0098] Optionally, this embodiment can be applied, but is not limited to, to video classification. For example, it can be applied to network security monitoring to classify videos into normal and illegal videos. For example, it can be applied to video players to identify the target video type and assign different types of labels to the target video. For example, it can be applied to learning and education to identify the target video and determine the subject type of the target video, thereby classifying the target video.

[0099] Taking the application of the above-mentioned video classification device to a terminal and a video player as an example, the video player can identify the video after obtaining the video uploaded by the personal media account. First, the target video is clustered into multiple video segments through the image features of each video frame, and a segment timing fusion feature is determined for each video segment. The segment timing fusion feature is used to determine the video feature of the video, and finally the video feature is identified to determine the type of the video. The video is labeled with a corresponding type, and the video can be searched when searching for resources with the label on the user side. Since the content of long videos in different scenarios is taken into account in the above-mentioned video classification process, the above-mentioned method improves the accuracy of video classification.

[0100] Optionally, the target video may be of a plurality of pre-set types. For example, the target video may be a normal video or an illegal video, a video containing language content or a video containing mathematics content, or a suspense type, a thriller type, a comedy type, etc. The plurality of types may be classified according to different classification methods, which is not limited in this embodiment.

[0101] Optionally, the target video frame is a portion of a video clip. A frame may be captured every other frame in the video clip, and the captured frame may be used as the target video frame. Alternatively, multiple frames may be randomly selected from the target video in a sequential order as the target video frames. This embodiment, through the above method, allows the content of different scenes in the target video to be considered when identifying the video type, thereby improving the accuracy of video type identification.

[0102] According to another aspect of the embodiment of the present invention, an electronic device for implementing the above-mentioned video classification method is also provided. The electronic device may be Figure 9 The terminal device or server shown in FIG. This embodiment is described by taking the electronic device as a terminal as an example. Figure 9 As shown, the electronic device includes a memory 902 and a processor 904. The memory 902 stores a computer program, and the processor 904 is configured to execute the steps in any of the above method embodiments through the computer program.

[0103] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0104] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0105] When a target video to be classified is obtained, image features of each frame of the target video are obtained;

[0106] performing clustering processing on all video frames in the target video according to feature distances between image features of each video frame to obtain a plurality of video segments, wherein each of the plurality of video segments includes a plurality of ordered video frames;

[0107] Obtaining a segment temporal fusion feature corresponding to each video segment in the plurality of video segments, wherein the segment temporal fusion feature is a feature determined according to a target video frame in the video segment, and the target video frame is a portion of the video frames in the video segment;

[0108] Determine the video features of the target video based on the temporal fusion features of the respective video clips;

[0109] Identify video features and obtain the target type of the target video.

[0110] Alternatively, those skilled in the art will appreciate that Figure 9 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 9 It does not limit the structure of the electronic device. For example, the electronic device may also include Figure 9 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 9 Different configurations shown.

[0111] Among them, the memory 902 can be used to store software programs and modules, such as the program instructions / modules corresponding to the video classification method and device in the embodiment of the present invention. The processor 904 executes various functional applications and data processing by running the software programs and modules stored in the memory 902, that is, realizing the above-mentioned video classification method. The memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 902 may further include a memory remotely located relative to the processor 904, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 902 can be used specifically but not limited to store target video and other information. As an example, if Figure 9As shown, the memory 902 may include, but is not limited to, the first acquisition unit 802, clustering unit 804, second acquisition unit 806, determination unit 808, and identification unit 810 in the video classification device. In addition, it may also include, but is not limited to, other module units in the video classification device, which will not be repeated in this example.

[0112] Optionally, the transmission device 906 is used to receive or send data via a network. Specific examples of the network may include wired networks and wireless networks. In one embodiment, the transmission device 906 includes a network interface controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In one embodiment, the transmission device 906 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0113] In addition, the electronic device further includes: a display 908 for displaying the target type of the target video; and a connection bus 910 for connecting various module components in the electronic device.

[0114] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting multiple nodes through network communication. The nodes may form a peer-to-peer (P2P) network, and any computing device, such as a server, terminal, or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0115] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the steps of any of the above method embodiments when run.

[0116] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0117] When a target video to be classified is obtained, image features of each frame of the target video are obtained;

[0118] performing clustering processing on all video frames in the target video according to feature distances between image features of each video frame to obtain a plurality of video segments, wherein each of the plurality of video segments includes a plurality of ordered video frames;

[0119] Obtaining a segment temporal fusion feature corresponding to each video segment in the plurality of video segments, wherein the segment temporal fusion feature is a feature determined according to a target video frame in the video segment, and the target video frame is a portion of the video frames in the video segment;

[0120] Determine the video features of the target video based on the temporal fusion features of the respective video clips;

[0121] Identify video features and obtain the target type of the target video.

[0122] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0123] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0124] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above-mentioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing one or more computer devices (such as personal computers, servers, or network devices) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0125] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, and can be electrical or other forms.

[0127] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0128] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0129] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A video classification method, characterized in that: include: When a target video to be classified is obtained, image features of each video frame in the target video are obtained; performing clustering processing on all video frames in the target video according to feature distances between the image features of the respective video frames to obtain a plurality of video segments, wherein each of the plurality of video segments includes a plurality of ordered video frames; Each of the multiple video segments is taken as a current segment, and the following operations are performed on the current segment: a target video frame is determined from the current segment, and frame features of each target video frame are determined, wherein the target video frame is a portion of the video frames in the current segment; each of the target video frames in the current segment is taken as a current frame, and the following operations are performed on the current frame: features of the first portion of the frame features of the current frame are replaced with features of the first portion of the first frame features, and features of the second portion of the frame features of the current frame are replaced with features of the second portion of the second frame features, so as to obtain processed frame features of the current frame, wherein the first frame features are frame features of a video frame preceding the current frame, and the second frame features are frame features of a video frame following the current frame; based on the processed frame features of each of the target video frames, segment temporal fusion features of the current segment are determined; Determining the video features of the target video according to the segment temporal fusion features corresponding to each of the video segments; The video features are identified to obtain the target type of the target video.

2. The method according to claim 1, characterized in that When a target video to be classified is obtained, obtaining image features of each frame of the target video includes: Determine each video frame of the target video as a current video frame, and perform the following operations on the current video frame: Acquire multiple target grayscale values, determine each of the target grayscale values ​​as a current grayscale value, and count the number of pixels in the current video frame whose grayscale value is the same as the current grayscale value to obtain a first number, wherein each target grayscale value in the current video frame corresponds to one first number; A sequence consisting of a plurality of the first numbers is determined as the image feature of the current video frame.

3. The method according to claim 1, characterized in that The clustering process is performed on all video frames in the target video according to the feature distances between the image features of each video frame to obtain a plurality of video segments, including: Identify multiple target features; Taking each of the image features of the target video as a current feature, performing the following operations on the current feature to obtain the multiple video clips: determining a feature distance between the current feature and each of the multiple target features, and determining a first target feature corresponding to a minimum value of the feature distances and the current feature as two features in a set of features; After traversing each of the image features of the target video, the video frames corresponding to the image features in the same set of features are determined as video frames in one of the video segments.

4. The method according to claim 3, characterized in that Determining a plurality of target features comprises: determining a plurality of initial features; Performing the following operations on the current feature to obtain multiple first distances: determining a first distance between each of the current features and a first initial feature among the multiple initial features, wherein the first initial feature is a feature among the multiple initial features that has the smallest distance from the current feature, and each current feature corresponds to one first distance; determining a sum of a plurality of said first distances; adjusting the plurality of initial features to obtain the re-determined sum; The plurality of initial features corresponding to the minimum sum are determined as the plurality of target features.

5. The method according to claim 1, characterized in that The determining of the video features of the target video according to the segment temporal fusion features corresponding to each of the video segments comprises: Obtaining the segment temporal fusion feature of each of the video segments; Performing an averaging operation on the temporal fusion features of the plurality of segments to obtain an average feature; The average feature is used as the video feature.

6. The method according to claim 5, characterized in that Determining the segment temporal fusion feature of the current segment according to the processed frame feature of each target video frame includes: The processed frame features of each target video frame are fused in time sequence to form the segment temporal fusion features of the current segment.

7. The method according to claim 5, characterized in that The performing an averaging operation on the plurality of temporal fusion features of the fragments to obtain an average feature comprises: Performing a pooling operation on each of the plurality of fragment temporal fusion features in the time dimension to obtain a pooled feature; A feature determined by averaging the multiple pooled features is used as the average feature.

8. The method according to claim 1, characterized in that The method further comprises: extracting the segment temporal fusion feature using a feature extraction layer of a target neural network model; The identifying the video features to obtain the target type of the target video includes: using the fully connected layer of the target neural network model to identify the video features to obtain the target type.

9. The method according to claim 8, characterized in that Before extracting the segment temporal fusion features using the feature extraction layer of the target neural network model, the method further includes: Acquire a sample video, wherein the sample video is labeled with a sample type; Inputting the sample video into the target neural network model to obtain an estimated type of the sample video output by the target neural network model; When the estimation type and the sample type are different, adjust the first parameter of the feature extraction layer and the second parameter of the fully connected layer of the target neural network model.

10. The method according to claim 9, characterized in that The sample video is also marked with the first position of a key frame. When the sample video is input into the target neural network model to obtain the video type of the sample video output by the target neural network model, the method further includes: Obtaining a sample segment temporal fusion feature of each sample video segment of the sample video; Performing a maximum pooling operation on the temporal fusion features of each sample segment to obtain a pooled sample feature; Splicing a plurality of the pooled sample features into a two-dimensional feature; Recognizing the two-dimensional features by the fully connected layer of the target neural network model to obtain an estimated position of the key frame of the sample video; When the estimated position is different from the first position, the first parameter and the second parameter are adjusted.

11. A video classification device, characterized in that: include: A first acquisition unit is configured to acquire image features of each video frame in the target video when a target video to be classified is acquired; a clustering unit, configured to perform clustering processing on all video frames in the target video according to feature distances between the image features of the respective video frames, so as to obtain a plurality of video segments, wherein each of the plurality of video segments includes a plurality of ordered video frames; A second acquisition unit is configured to take each of the multiple video segments as a current segment, and perform the following operations on the current segment: determine a target video frame from the current segment, and determine the frame features of each target video frame, wherein the target video frame is a portion of the video frames in the current segment; take each of the target video frames in the current segment as a current frame, and perform the following operations on the current frame: replace the features of the first portion of the frame features of the current frame with the features of the first portion of the first frame features, and replace the features of the second portion of the frame features of the current frame with the features of the second portion of the second frame features, so as to obtain the processed frame features of the current frame, wherein the first frame features are the frame features of a video frame preceding the current frame, and the second frame features are the frame features of a video frame following the current frame; determine the segment temporal fusion features of the current segment based on the processed frame features of each of the target video frames; A determining unit, configured to determine the video features of the target video according to the segment temporal fusion features corresponding to each of the video segments; The recognition unit is used to recognize the video features and obtain the target type of the target video.

12. The device according to claim 11, characterized in that The clustering unit further includes: A first determination module is used to determine multiple target features; a second processing module, configured to take each of the image features of the target video as a current feature, and perform the following operations on the current feature to obtain the multiple video clips: determining a feature distance between the current feature and each of the multiple target features, and determining a first target feature corresponding to a minimum value of the feature distances and the current feature as two features in a set of features; The second determining module is configured to, after traversing each of the image features of the target video, determine the video frames corresponding to the image features in the same set of features as video frames in one of the video segments.

13. The device according to claim 12, characterized in that The first determining module further includes: A first determination submodule is used to determine a plurality of initial features; a first processing submodule, configured to perform the following operations on the current feature to obtain a plurality of first distances: determining a first distance between each of the current feature and a first initial feature among the plurality of initial features, wherein the first initial feature is a feature among the plurality of initial features having the smallest distance from the current feature, and each current feature corresponds to one first distance; a second determining submodule, configured to determine a sum of a plurality of said first distances; an adjustment module, configured to adjust the plurality of initial features to obtain the re-determined sum; The third determining submodule is configured to determine the multiple initial features corresponding to the minimum sum as the multiple target features.

14. The device according to claim 11, characterized in that The determining unit further includes: A first acquisition module is used to obtain the segment temporal fusion feature of each of the video segments; A third processing module is configured to perform an averaging operation on the plurality of temporal fusion features of the fragments to obtain an average feature; The third determining module is configured to use the average feature as the video feature.

15. The device according to claim 11, characterized in that The device further comprises: The second processing submodule is configured to fuse the processed frame features of each target video frame into the segment temporal fusion features of the current segment in chronological order.

16. The device according to claim 14, characterized in that The third processing module includes: A first pooling submodule is configured to perform a pooling operation on each of the plurality of fragment temporal fusion features in a time dimension to obtain a pooled feature; The fourth determining submodule is configured to determine a feature by averaging the plurality of pooled features as the average feature.

17. The device according to claim 11, characterized in that The apparatus further includes: a second acquisition module, configured to extract the segment temporal fusion features using a feature extraction layer of a target neural network model; The recognition unit includes: a first recognition module, which is used to use the fully connected layer of the target neural network model to identify the video features and obtain the target type.

18. The device according to claim 17, characterized in that The second acquiring unit further includes: A third acquisition module is configured to acquire a sample video before extracting the segment temporal fusion feature using the feature extraction layer of the target neural network model, wherein the sample video is labeled with a sample type; An input module, configured to input the sample video into the target neural network model to obtain an estimated type of the sample video output by the target neural network model; The first adjustment module is used to adjust the first parameter of the feature extraction layer and the second parameter of the fully connected layer of the target neural network model when the estimation type and the sample type are different.

19. The device according to claim 18, characterized in that The second acquiring unit further includes: a fourth acquisition module, configured to, when inputting the sample video into the target neural network model and obtaining the video type of the sample video output by the target neural network model, obtain a sample segment temporal fusion feature of each sample video segment of the sample video, wherein the sample video is also marked with a first position of a key frame; A pooling module is used to perform a maximum pooling operation on the temporal fusion features of each sample segment to obtain a pooled sample feature; a splicing module for splicing the plurality of pooled sample features into a two-dimensional feature; a second recognition module for recognizing the two-dimensional feature by the fully connected layer of the target neural network model to obtain an estimated position of the key frame of the sample video; The second adjustment module is configured to adjust the first parameter and the second parameter when the estimated position is different from the first position.

20. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed, the method according to any one of claims 1 to 10 is executed.

21. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 10 through the computer program.

Images