Video data identification method and device, electronic equipment and storage medium

Abstract

This application provides a method, apparatus, electronic device, and storage medium for identifying video data. The method includes: acquiring a target video segment to be processed; inputting the target video segment into a target model to obtain video features, wherein the target model is used to derive feature information characterizing the video contained within the target video segment, and the target model is obtained by adjusting the model parameters of an initial model; and determining the identification result of the video data in the target video segment based on the video features. This application solves the problems of low automation and inaccurate identification of video content features in related technologies.

Description

Technical Field

[0001] This application relates to the field of video image processing, and more particularly to a method and apparatus for identifying video data, an electronic device, and a storage medium. Background Technology

[0002] Currently, when identifying the mediums (such as people, backgrounds, actions, etc.) that can characterize video features in a video clip, it is necessary to compare the images appearing in the video clip with some existing images. In this case, the existing image comparison usually involves comparing the similarity of at least two images, such as using hash algorithms, histogram algorithms, structural similarity algorithms, etc., to measure the similarity between the images and thus determine the content features in the video clip images.

[0003] However, the aforementioned technologies require comparative learning of images, have a low degree of automation, and the comparison results have a large error with the actual images.

[0004] Therefore, the relevant technologies suffer from low automation in video frame data recognition and processing, and cannot accurately identify the video content features of video segments. Summary of the Invention

[0005] This application provides a method and apparatus for identifying video data, an electronic device, and a storage medium to at least solve the problem in related technologies that have a low degree of automation and cannot accurately identify the video content features of video segments.

[0006] According to one aspect of the embodiments of this application, a method for recognizing video data is provided, the method comprising:

[0007] Obtain the target video segment to be processed;

[0008] The target video segment is input into the target model to obtain video features. The target model is used to derive feature information that can characterize the video contained in the target video segment. The target model is obtained by adjusting the model parameters of the initial model.

[0009] Based on the video features, the recognition result of the video data in the target video segment is determined.

[0010] According to another aspect of the embodiments of this application, a video data recognition device is also provided, the device comprising:

[0011] The first acquisition unit is used to acquire the target video segment to be processed;

[0012] The first input unit is used to input the target video segment into the target model to obtain video features. The target model is used to derive feature information that can characterize the video contained in the target video segment. The target model is obtained by adjusting the model parameters of the initial model.

[0013] The determining unit is used to determine the recognition result of the video data in the target video segment based on the video features.

[0014] Optionally, the device further includes:

[0015] The second acquisition unit is used to acquire an initial video segment before inputting the target video segment into the target model to obtain video features, wherein the initial video segment is a sample set for training the initial model;

[0016] A segmentation unit is used to divide the initial video segment into multiple video sub-segments with equal frequency frame counts. Each video sub-segment contains multiple video frames, and the number of video frames in each video sub-segment is the same.

[0017] The masking unit is used to mask the video sub-segment using a masking scheme to obtain a masked first video frame and an unmasked second video frame, wherein the number of the first video frame and the number of the second video frame are both at least one.

[0018] An adjustment unit is used to adjust the model parameters of the initial model based on the first video frame and the second video frame to obtain the target model.

[0019] Optionally, the masking unit includes:

[0020] The masking module is used to mask the video sub-segment according to the masking ratio in the masking scheme to obtain the first video frame and the second video frame.

[0021] Optionally, the adjustment unit includes:

[0022] The first determining module is used to determine the target position of the first video frame based on the first video frame.

[0023] The second determining module is used to determine a first sequence composed of the second video frames based on the second video frames;

[0024] An adjustment module is used to adjust the model parameters according to the target position and the first sequence to obtain the target model.

[0025] Optionally, the adjustment module includes:

[0026] The first input subunit is used to input the first sequence into the encoder to obtain the encoded second sequence;

[0027] A subunit is added to add multiple masking identifiers at the target position in the second sequence to obtain a third sequence, wherein the masking identifiers are used to increase the integrity of the second sequence;

[0028] The second input subunit is used to input the third sequence into the decoder to obtain the decoded feature vector;

[0029] The adjustment subunit is used to adjust the model parameters using the feature vector and loss function to obtain the target model.

[0030] Optionally, adjusting the subunit includes:

[0031] The calculation result of the feature vector is obtained using the loss function;

[0032] The model parameters are updated by backpropagation based on the calculation results until the calculation results are greater than or equal to a preset threshold. At this point, the adjustment of the model parameters is stopped, and the target model is obtained. The preset threshold is the lowest value at which the adjustment of the model parameters is stopped.

[0033] Optionally, the device further includes:

[0034] The third acquisition unit is used to acquire downstream task information;

[0035] The second input unit is used to input the original video frame sequence corresponding to the target video segment into the encoder to obtain the encoded fourth sequence.

[0036] The combining unit is used to combine the fourth sequence with downstream task information to obtain the combined application result.

[0037] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein the memory is used to store a computer program; and the processor is used to execute the method steps of any of the above embodiments by running the computer program stored in the memory.

[0038] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to execute the method steps of any of the above embodiments when running.

[0039] This application embodiment can be applied to computer vision in the field of deep learning technology. In this application embodiment, a target video segment to be processed is acquired; the target video segment is input into a target model to obtain video features, wherein the target model is used to derive feature information that can characterize the video contained in the target video segment, and the target model is obtained by adjusting the model parameters of the initial model; based on the video features, the recognition result of the video data in the target video segment is determined. Since this application embodiment uses a trained target model to process the acquired target video segment to obtain corresponding video features, the content features of the video data in the target video segment can be identified through these video features, such as background, people, or certain actions, thereby achieving the effect of automatic recognition. At the same time, a video self-supervised encoder pre-training is also implemented before the entire video data recognition. Thus, the recognition result of the video data obtained by the target model after the initial model pre-training is more accurate and convenient, thereby solving the problem of low automation and inaccurate identification of video content features of video segments in related technologies. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the hardware environment for an optional video data recognition method according to an embodiment of the present invention;

[0043] Figure 2 This is a flowchart illustrating an optional video data recognition method according to an embodiment of this application;

[0044] Figure 3 This is a schematic diagram of an optional video data recognition model according to an embodiment of this application;

[0045] Figure 4 This is a structural block diagram of an optional video data recognition device according to an embodiment of this application;

[0046] Figure 5 This is a structural block diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0047] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0048] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0049] According to one aspect of the embodiments of this application, a method for recognizing video data is provided. Optionally, in this embodiment, the above-described video data recognition method can be applied to, for example... Figure 1 In the hardware environment shown. For example... Figure 1 As shown, terminal 102 may include memory 104, processor 106, and display 108 (optional component). Terminal 102 can communicate with server 112 via network 110. Server 112 can provide services (such as application services) to the terminal or clients installed on the terminal. Database 114 can be set up on or independently of server 112 to provide data storage services to server 112. In addition, server 112 may run a processing engine 116, which can be used to execute the steps performed by server 112.

[0050] Optionally, terminal 102 may be, but is not limited to, a terminal capable of computing data, such as a mobile terminal (e.g., mobile phone, tablet computer), laptop computer, PC (Personal Computer), etc. The aforementioned network may include, but is not limited to, a wireless network or a wired network. The wireless network includes Bluetooth, Wi-Fi (Wireless Fidelity), and other networks that enable wireless communication. The aforementioned wired network may include, but is not limited to, a wide area network (WAN), a metropolitan area network (MAN), and a local area network (LAN). The aforementioned server 112 may include, but is not limited to, any hardware device capable of computing.

[0051] Furthermore, in this embodiment, the video data recognition method described above can also be applied to, but is not limited to, a powerful independent processing device without requiring data interaction. For example, the processing device can be, but is not limited to, a powerful terminal device; that is, the various operations in the video data recognition method described above can be integrated into a single independent processing device. The above is merely an example, and no limitation is made in this embodiment.

[0052] Optionally, in this embodiment, the video data recognition method described above can be executed by server 112, by terminal 102, or jointly by server 112 and terminal 102. The video data recognition method of this embodiment can also be executed by a client installed on terminal 102.

[0053] Taking running on a server as an example, Figure 2 This is a flowchart illustrating an optional video data recognition method according to an embodiment of this application, such as... Figure 2 As shown, the process of this method may include the following steps:

[0054] Step S201: Obtain the target video segment to be processed;

[0055] Step S202: Input the target video segment into the target model to obtain video features. The target model is used to obtain feature information that can characterize the video contained in the target video segment. The target model is obtained by adjusting the model parameters of the initial model.

[0056] Step S203: Based on the video features, determine the recognition result of the video data in the target video segment.

[0057] Optionally, in the embodiments of this application, a trained target model is first obtained. This target model is a model pre-trained by a self-supervised encoder on a video segment sample set, that is, the final model obtained after adjusting the model parameters of the initial model. The target model can be used to determine the recognition result of video data in the video segment.

[0058] More specifically, the server acquires the target video segment to be processed, which can be any one or more segments requiring video data recognition. The target video segment is then input into a trained target model, which outputs video features. These video features are typically vector features, which allow for the identification of video data within the target video segment, such as identifying whether the current target video segment contains background, people, or performing actions.

[0059] In this embodiment, a target video segment to be processed is acquired; the target video segment is input into a target model to obtain video features. The target model is used to derive feature information that characterizes the video contained within the target video segment. The target model is obtained by adjusting the model parameters of an initial model. Based on the video features, the recognition result of the video data in the target video segment is determined. Since this embodiment uses a trained target model to process the acquired target video segment to obtain corresponding video features, these features can identify the content features of the video data within the target video segment, such as background, people, or certain actions, thereby achieving automatic recognition. Furthermore, before the entire video data recognition process, a video self-supervised encoder is pre-trained. This makes the recognition result of the video data obtained by the target model after pre-training the initial model more accurate and convenient, thus solving the problem of low automation and inaccurate identification of video content features in related technologies.

[0060] As an optional embodiment, before inputting the target video segment into the target model to obtain video features, the method further includes:

[0061] Obtain the initial video segments, which serve as the sample set for training the initial model;

[0062] The initial video segment is divided into multiple video sub-segments by equal frequency frame counts. Each video sub-segment contains multiple video frames, and the number of video frames in each video sub-segment is the same.

[0063] The video sub-segment is masked using a masking scheme to obtain a masked first video frame and an unmasked second video frame, wherein there is at least one first video frame and at least one second video frame.

[0064] Based on the first and second video frames, the model parameters of the initial model are adjusted to obtain the target model.

[0065] Optionally, in the embodiments of this application, the process of training the initial model of the video encoder is described. Specifically, a training sample set is first selected, such as obtaining the initial video segments (i.e., Figure 3 The initial video segments are divided into multiple video sub-segments based on equal-frequency frame counts. It should be noted that since this embodiment uses equal-frequency frame count division, each video sub-segment contains the same number of video frames. That is, all video sub-segments contain N frames. The frame data N*C*H*W of each video sub-segment is obtained, where N is the frame count (a hyperparameter that can be adjusted), C is the number of channels in the input image, and W and H are the width and height of the input image.

[0066] Then, the video sub-segments are masked using the masking ratio set in the masking scheme, such as... Figure 3 Masking K video frames yields the occluded and unoccluded portions. The occluded portion is called the first video frame, represented by N * the masking ratio, and the unoccluded portion is called the second video frame, represented by N * (1 - masking ratio). There must be at least one first and one second video frame. The masking ratio is determined based on historical experience and multiple experiments, and can be 80%, 70%, 60%, etc. It should be noted that to better train the initial model, a larger masking ratio can be set to increase the training difficulty and improve the accuracy of the target model.

[0067] Then, the model parameters of the initial model are adjusted based on the first and second video frames obtained, thereby determining the final trained target model.

[0068] In this embodiment, by setting the masking ratio of video frames, the initial model can be trained better, the training difficulty can be increased, and the accuracy of the target model can be improved.

[0069] As an optional embodiment, the target model is obtained by adjusting the model parameters of the initial model based on the first video frame and the second video frame, including:

[0070] Based on the first video frame, determine the current target location of the first video frame;

[0071] Based on the second video frame, determine the first sequence composed of the second video frame;

[0072] Based on the target location and the first sequence, the model parameters are adjusted to obtain the target model.

[0073] Optionally, the location of the currently occluded video frame can be determined from the first video frame, referred to as the target location. The second video frame can be used to construct a first sequence, which is then input into the encoder to obtain the encoded second sequence. Multiple masking markers (such as...) are then added to the corresponding occluded target location in the second sequence. Figure 3 The encoder adds mask tokens (such as Mask Token 1) to obtain a third sequence, which is then input into the decoder to obtain the decoded feature vector. For example, if the original video frame sequence is 1, 2, 3, 4, 5, and even-numbered frames are masked, then 1, 3, 5 are input into the encoder to obtain video features 11, 33, 55. Mask tokens are added to these video features to obtain 11, MaskToken2, 33, Mask Token4, 55. This sequence is then input into the decoder to obtain the decoded feature vector.

[0074] It is understood that the aforementioned target position corresponds to a masking identifier, meaning that the masked video frames are replaced with masking identifiers so that the resulting third sequence can completely represent the corresponding video sub-segment. The representation of the masking identifier in this application embodiment can be arbitrarily set, and this application does not impose specific limitations on it.

[0075] Finally, the decoded feature vectors are combined with the loss function, and the network is updated through backpropagation. The initial video segments are continuously reconstructed, the model parameters of the initial model are adjusted, and the target model is finally determined.

[0076] In this embodiment of the application, setting a mask identifier to completely correspond to the video sub-segment is beneficial for the training of the initial model and the generation of the target model.

[0077] As an optional implementation, the target model is obtained by adjusting the model parameters using feature vectors and a loss function, including:

[0078] The loss function is used to obtain the calculation results of the feature vector;

[0079] The model parameters are updated by backpropagation based on the calculation results until the calculation results are greater than or equal to a preset threshold. At this point, the adjustment of the model parameters is stopped, and the target model is obtained. The preset threshold is the lowest value at which the adjustment of the model parameters is stopped.

[0080] Optionally, in this embodiment, a preset threshold is set, which serves to stop the infinite loop of adjusting model parameters; that is, it represents the minimum value at which adjusting model parameters stops. Specifically, a loss function is used to calculate the loss, and the calculated result is backpropagated to update the model parameters. The model parameters are continuously adjusted until the calculated result is greater than or equal to the preset threshold, at which point the adjustment of the model parameters stops, and the currently trained initial model can be used as the target model.

[0081] As an optional embodiment, the method further includes:

[0082] Obtain downstream task information;

[0083] The original video frame sequence corresponding to the target video segment is input into the encoder to obtain the encoded fourth sequence.

[0084] The fourth sequence is combined with downstream task information to obtain the combined application result.

[0085] Optionally, in this embodiment, the encoded video features need to be applied to downstream tasks. Therefore, the server first obtains the downstream tasks, which are typically those that can be combined with the identified video features. Here, the downstream task discards the decoder and retains only the encoder, inputting the original video frame sequence corresponding to the target video segment into the encoder to obtain the encoded fourth sequence. The fourth sequence is then combined with the downstream task information to obtain the combined application result.

[0086] In the embodiments of this application, the encoded video features (i.e., the fourth sequence) are applied to downstream tasks, which can improve the processing speed of downstream tasks.

[0087] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0088] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0089] According to another aspect of the embodiments of this application, a video data recognition device for implementing the above-described video data recognition method is also provided. Figure 4 This is a structural block diagram of an optional video data recognition device according to an embodiment of this application, such as... Figure 4 As shown, the device may include:

[0090] The first acquisition unit 401 is used to acquire the target video segment to be processed;

[0091] The first input unit 402 is connected to the first acquisition unit 401 and is used to input the target video segment into the target model to obtain video features. The target model is used to obtain feature information that can characterize the video contained in the target video segment. The target model is obtained by adjusting the model parameters of the initial model.

[0092] The determining unit 403, connected to the first input unit 402, is used to determine the recognition result of video data in the target video segment based on video features.

[0093] It should be noted that the first acquisition unit 401 in this embodiment can be used to execute the above step S201, the first input unit 402 in this embodiment can be used to execute the above step S202, and the determination unit 403 in this embodiment can be used to execute the above step S203.

[0094] Through the modules described above, the acquired target video segments are processed using a trained target model to obtain corresponding video features. These features can be used to identify the content features of the video data within the target video segment, such as background, people, or certain actions, thereby achieving automatic recognition. Furthermore, before the entire video data recognition process, a video self-supervised encoder is pre-trained. This results in more accurate and convenient video data recognition from the target model after pre-training the initial model, thus solving the problems of low automation and inaccurate identification of video content features in related technologies.

[0095] As an optional embodiment, the device further includes:

[0096] The second acquisition unit is used to acquire an initial video segment before inputting the target video segment into the target model to obtain video features, wherein the initial video segment is a sample set for training the initial model;

[0097] The segmentation unit is used to divide the initial video segment into equal-frequency frame numbers to obtain multiple video sub-segments. Each video sub-segment contains multiple video frames, and the number of video frames in each video sub-segment is the same.

[0098] The masking unit is used to mask video segments using a masking scheme to obtain a masked first video frame and an unmasked second video frame, wherein the number of the first video frame and the second video frame is at least one.

[0099] The adjustment unit is used to adjust the model parameters of the initial model based on the first video frame and the second video frame to obtain the target model.

[0100] As an optional embodiment, the masking unit includes:

[0101] The masking module is used to mask video segments according to the masking ratio in the masking scheme to obtain the first video frame and the second video frame.

[0102] As an optional embodiment, the adjustment unit includes:

[0103] The first determining module is used to determine the target position of the first video frame based on the first video frame.

[0104] The second determining module is used to determine a first sequence composed of the second video frames based on the second video frames;

[0105] The adjustment module is used to adjust the model parameters based on the target location and the first sequence to obtain the target model.

[0106] As an optional embodiment, the adjustment module includes:

[0107] The first input subunit is used to input the first sequence into the encoder to obtain the encoded second sequence;

[0108] Add sub-units to add multiple masking identifiers at the target positions in the second sequence to obtain a third sequence, wherein the masking identifiers are used to increase the completeness of the second sequence;

[0109] The second input subunit is used to input the third sequence into the decoder to obtain the decoded feature vector;

[0110] The adjustment sub-unit is used to adjust the model parameters using feature vectors and loss functions to obtain the target model.

[0111] As an optional embodiment, the adjustment subunit includes:

[0112] The loss function is used to obtain the calculation results of the feature vector;

[0113] The model parameters are updated by backpropagation based on the calculation results until the calculation results are greater than or equal to a preset threshold. At this point, the adjustment of the model parameters is stopped, and the target model is obtained. The preset threshold is the lowest value at which the adjustment of the model parameters is stopped.

[0114] As an optional embodiment, the device further includes:

[0115] The third acquisition unit is used to acquire downstream task information;

[0116] The second input unit is used to input the original video frame sequence corresponding to the target video segment into the encoder to obtain the encoded fourth sequence.

[0117] The combination unit is used to combine the fourth sequence with downstream task information to obtain the combined application result.

[0118] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of a device, can operate in environments such as... Figure 1 The hardware environment shown can be implemented through software or hardware, and the hardware environment includes the network environment.

[0119] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described video data recognition method is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0120] Figure 5 This is a structural block diagram of an optional electronic device according to an embodiment of this application, such as... Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504. The processor 501, communication interface 502, and memory 503 communicate with each other via the communication bus 504.

[0121] Memory 503 is used to store computer programs;

[0122] When processor 501 executes a computer program stored in memory 503, it performs the following steps:

[0123] Obtain the target video segment to be processed;

[0124] The target video segment is input into the target model to obtain video features. The target model is used to derive the feature information that can characterize the video contained in the target video segment. The target model is obtained by adjusting the model parameters of the initial model.

[0125] Based on video features, determine the recognition results of video data in the target video segment.

[0126] Optionally, in this embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0127] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0128] The memory may include RAM, or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0129] As an example, such as Figure 5 As shown, the memory 503 may include, but is not limited to, the first acquisition unit 401, the first input unit 402, and the determination unit 403 of the video data recognition device. Furthermore, it may include, but is not limited to, other module units of the video data recognition device, which will not be elaborated upon in this example.

[0130] The processors mentioned above can be general-purpose processors, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; they can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0131] In addition, the aforementioned electronic device also includes a display for displaying the recognition results of video data.

[0132] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0133] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only. The device that implements the above video data recognition method can be a terminal device, such as a smartphone (e.g., Android phone, iOS phone), tablet computer, PDA, mobile Internet Device (MID), PAD, etc. Figure 5 This does not limit the structure of the aforementioned electronic devices. For example, the terminal device may also include components that are more advanced than those described above. Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0134] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0135] According to another aspect of the embodiments of this application, a storage medium is also provided. Optionally, in this embodiment, the storage medium can be used to execute program code for a video data recognition method.

[0136] Optionally, in this embodiment, the storage medium may be located on at least one of the network devices in the network shown in the above embodiment.

[0137] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:

[0138] Obtain the target video segment to be processed;

[0139] The target video segment is input into the target model to obtain video features. The target model is used to derive the feature information that can characterize the video contained in the target video segment. The target model is obtained by adjusting the model parameters of the initial model.

[0140] Based on video features, determine the recognition results of video data in the target video segment.

[0141] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated in this embodiment.

[0142] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0143] According to another aspect of the embodiments of this application, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the video data recognition method steps in any of the above embodiments.

[0144] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0145] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the video data recognition method of the various embodiments of this application.

[0146] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0148] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the solution provided in this embodiment, depending on actual needs.

[0149] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0150] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method of identifying video data, characterized by, The method comprises: acquiring a target video segment to be processed; inputting the target video segment into a target model to obtain video features, wherein the target model is used to derive feature information capable of representing a video contained in the target video segment, and the target model is obtained by adjusting model parameters of an initial model; the target model is a video encoder model obtained through self-supervised pre-training, and a pre-training process of the target model comprises: acquiring an initial video segment, wherein the initial video segment is a sample set for training the initial model; dividing the initial video segment into a plurality of video sub-segments with an equal number of frames, wherein the video sub-segments contain a plurality of video frames, and the number of video frames in each video sub-segment is the same; performing mask processing on the video sub-segments using a preset mask ratio in a mask scheme to obtain occluded first video frames and unoccluded second video frames, wherein the number of the first video frames and the second video frames is at least one; adjusting model parameters of the initial model according to the first video frames and the second video frames to obtain the target model, comprising: determining a target position currently occupied by the first video frames according to the first video frames; determining a first sequence composed of the second video frames according to the second video frames; inputting the first sequence into an encoder to obtain an encoded second sequence; adding a plurality of mask identifiers to the target position in the second sequence to obtain a third sequence, wherein the mask identifiers are used to increase the integrity of the second sequence; inputting the third sequence into a decoder to obtain a decoded feature vector; adjusting the model parameters using the feature vector and a loss function to obtain the target model; determining a recognition result of video data in the target video segment according to the video features.

2. The method of claim 1, wherein, The adjusting the model parameters using the feature vector and the loss function to obtain the target model comprises: obtaining a calculation result of the feature vector using the loss function; updating the model parameters through back propagation according to the calculation result until the calculation result is greater than or equal to a preset threshold value, stopping adjusting the model parameters to obtain the target model, wherein the preset threshold value is the lowest value for stopping adjusting the model parameters.

3. The method of claim 1, wherein, The method further comprises: acquiring downstream task information; inputting an original video frame sequence corresponding to the target video segment into an encoder to obtain an encoded fourth sequence; combining the fourth sequence with the downstream task information to obtain a combined application result.

4. An apparatus for identifying video data, characterized by comprising: The device comprises: a first acquisition unit configured to acquire a target video segment to be processed; The first input unit is configured to input the target video segment into a target model to obtain video features, wherein the target model is configured to derive feature information capable of representing a video contained in the target video segment, and the target model is obtained by adjusting model parameters of an initial model; the target model is a video encoder model obtained by self-supervised pre-training, and a pre-training process of the target model includes: obtaining an initial video segment, wherein the initial video segment is a sample set for training the initial model; dividing the initial video segment into a plurality of video sub-segments with an equal number of frames, wherein the video sub-segments contain a plurality of video frames, and the number of the video frames in each of the video sub-segments is the same; performing mask processing on the video sub-segments by using a preset mask ratio in a mask scheme to obtain occluded first video frames and unoccluded second video frames, wherein the number of the first video frames and the number of the second video frames are both at least one; adjusting the model parameters of the initial model according to the first video frames and the second video frames to obtain the target model, including: determining a target position at which the first video frames are currently located according to the first video frames; determining a first sequence composed of the second video frames according to the second video frames; inputting the first sequence into an encoder to obtain an encoded second sequence; adding a plurality of mask identifiers to the target position in the second sequence to obtain a third sequence, wherein the mask identifiers are configured to increase the integrity of the second sequence; inputting the third sequence into a decoder to obtain a decoded feature vector; adjusting the model parameters by using the feature vector and a loss function to obtain the target model; A determination unit is configured to determine an identification result of video data in the target video segment according to the video features.

5. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein, The processor, the communication interface, and the memory complete mutual communication through the communication bus, and the method comprises the following steps: The memory is configured to store a computer program. The processor is configured to execute the method steps of any one of claims 1 to 3 by running the computer program stored in the memory.

6. A computer readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is executed by the processor to implement the method steps in any one of claims 1 to 3.

Images