Video characterization method and device

Abstract

The present application discloses a video characterization method and device, which belongs to the field of computer technology. The method comprises: obtaining a video segment to be processed; inputting the video segment to be processed into a target video characterization model, and obtaining first video characterization data corresponding to the video segment to be processed. In the technical solution provided by the embodiment of the present application, the similarity information between the video segments can be determined by the video characterization data corresponding to the video segments in the training samples, and the temporal distance information between the video segments is introduced on the basis of the similarity information to determine the distance attenuation loss information, so as to adjust the negative correlation between the temporal distance information and the similarity information during the model training process, so that the video characterization model learns the correlation between the distance of different video segments in the time dimension and their similarity, and outputs more accurate video characterization data to perform feature representation on the video to be processed, thereby effectively improving the accuracy of video characterization.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video characterization method and device. Background Art

[0002] With the development of digital technology and Internet technology, video content services have become popular among the public, and the amount of videos on the Internet has increased dramatically. Therefore, video representation learning has become more important.

[0003] In the related art, a video representation model is trained with a large amount of labeled category data to learn video representation, so that the feature distances between the corresponding video representation data of videos with the same or similar labeled category data are also similar. After the above video representation model is trained, it can output video representation data to represent the features of the input video and apply it to downstream video classification tasks.

[0004] However, video representation learning in related technologies relies on labeled category data, and video representation is only related to the labeled category data, so the accuracy of video representation is low. Summary of the invention

[0005] The embodiments of the present application provide a video characterization method and device, which can improve the accuracy of video characterization.

[0006] According to one aspect of an embodiment of the present application, a video characterization method is provided, the method comprising:

[0007] Get the video clip to be processed;

[0008] Inputting the to-be-processed video segment into a target video representation model to obtain first video representation data corresponding to the to-be-processed video segment;

[0009] Among them, the target video representation model is a machine learning model obtained by training based on training samples and distance attenuation loss information, the distance attenuation loss information is determined based on the corresponding temporal distance information and similarity information between the video clips in the training samples, the similarity information is determined based on the video representation data corresponding to the video clips, and the distance attenuation loss information is used to adjust the degree of negative correlation between the temporal distance information and the similarity information.

[0010] According to one aspect of an embodiment of the present application, a video representation device is provided, the device comprising:

[0011] A video acquisition module is used to acquire the video clips to be processed;

[0012] A video representation module, configured to input the video segment to be processed into a target video representation model to obtain first video representation data corresponding to the video segment to be processed;

[0013] Among them, the target video representation model is a machine learning model obtained by training based on training samples and distance attenuation loss information, the distance attenuation loss information is determined based on the corresponding temporal distance information and similarity information between the video clips in the training samples, the similarity information is determined based on the video representation data corresponding to the video clips, and the distance attenuation loss information is used to adjust the degree of negative correlation between the temporal distance information and the similarity information.

[0014] According to one aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the above-mentioned video characterization method.

[0015] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the above-mentioned video characterization method.

[0016] According to one aspect of the embodiments of the present application, a computer program product is provided, the computer program product comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes to implement the above-mentioned video characterization method.

[0017] The technical solution provided in the embodiments of the present application can bring the following beneficial effects:

[0018] Through the video representation data corresponding to the video clips in the training samples, the similarity information between the video clips can be determined, and the temporal distance information between the video clips can be introduced on the basis of the similarity information to determine the distance attenuation loss information. Therefore, during the model training process, the negative correlation between the temporal distance information and the similarity information can be adjusted according to the distance attenuation loss information, so that the video representation model can learn the correlation between the distance of different video clips in the time dimension and their similarity, so that more accurate video representation data can be output for feature representation of the processed video, which effectively improves the accuracy of video representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a schematic diagram of an application program operating environment provided by an embodiment of the present application;

[0021] Figure 2 is a flow chart of a video characterization method provided by an embodiment of the present application;

[0022] Figure 3 is a flowchart of a similar video determination method provided by an embodiment of the present application;

[0023] Figure 4 is a flow chart of a video clip positioning method provided by an embodiment of the present application;

[0024] Figure 5 This is a process for training a video representation model provided by an embodiment of the present application. Figure 1 ;

[0025] Figure 6 This is a process for training a video representation model provided by an embodiment of the present application. Figure 2 ;

[0026] Figure 7 This is a process for training a video representation model provided by an embodiment of the present application. Figure 3 ;

[0027] Figure 8 This is a process for training a video representation model provided by an embodiment of the present application. Figure 4 ;

[0028] Fig. 9 An exemplary diagram of the training process of a video representation model is shown Figure 1 ;

[0029] Fig.10 An exemplary diagram of the training process of a video representation model is shown Figure 2 ;

[0030] Fig.11 is a block diagram of a video representation device provided by an embodiment of the present application;

[0031] Fig.12 It is a structural block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0032] The video characterization method provided in the embodiment of the present application involves artificial intelligence technology, which is briefly described below to facilitate understanding by those skilled in the art.

[0033] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0034] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0035] Computer Vision (CV): Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further process images so that the computer processes them into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D (3-Dimension) technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0036] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0037] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0038] Please refer to Figure 1 , which shows a schematic diagram of an application program running environment provided by an embodiment of the present application. The application program running environment may include: a terminal 10 and a server 20.

[0039] The terminal 10 includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, game consoles, e-book readers, multimedia playback devices, wearable devices, aircraft and other electronic devices. The client of the application program can be installed in the terminal 10. The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0040] In an embodiment of the present application, the above-mentioned application can be any application that can provide video content services. Typically, the application is a video application. Of course, in addition to video applications, other types of applications can also provide video content services. For example, news applications, social applications, interactive entertainment applications, browser applications, shopping applications, content sharing applications, virtual reality (VirtualReality, VR) applications, augmented reality (Augmented Reality, AR) applications, etc., are not limited to this in the embodiment of the present application. In addition, for different applications, the videos pushed will also be different, and the corresponding functions will also be different, which can be pre-configured according to actual needs, and the embodiment of the present application is not limited to this. Optionally, a client of the above-mentioned application is running in the terminal 10.

[0041] The server 20 is used to provide background services for the client of the application in the terminal 10. For example, the server 20 can be the background server of the above application. The server 20 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the server 20 provides background services for applications in multiple terminals 10 at the same time.

[0042] Optionally, the terminal 10 and the server 20 may communicate with each other via a network 30. The terminal 10 and the server 20 may be directly or indirectly connected via wired or wireless communication, which is not limited in the present application.

[0043] Please refer to Figure 2 , which shows a flow chart of a video characterization method provided by an embodiment of the present application. The method can be applied to a computer device, which refers to an electronic device with data calculation and processing capabilities. For example, the execution subject of each step can be Figure 1 The terminal 10 or server 20 in the application running environment shown. The method may include the following steps (210-220).

[0044] Step 210: Obtain the video segment to be processed.

[0045] Optionally, the video segment to be processed includes a video clip (Clip). In some possible application scenarios, the video segment to be processed is a video segment (Query) used for query in a video retrieval task.

[0046] Step 220: input the to-be-processed video segment into the target video representation model to obtain first video representation data corresponding to the to-be-processed video segment.

[0047] Among them, the target video representation model is a machine learning model obtained by training based on training samples and distance attenuation loss information. The distance attenuation loss information is determined based on the corresponding temporal distance information and similarity information between the video clips in the training samples. The similarity information is determined based on the video representation data corresponding to the video clips. The distance attenuation loss information is used to adjust the negative correlation between the temporal distance information and the similarity information.

[0048] In a possible implementation, the target video representation model may be a video feature extraction network model, including but not limited to a convolutional neural network model (CNN), a three-dimensional convolutional network model (3-DimensionCNN, C3D), a three-dimensional deep residual network model (3-DimensionDeep residualnetwork, R3D), a convolutional network model of a composite of two-dimensional convolution and one-dimensional convolution (R(2+1)D), and a separable three-dimensional convolutional neural network model (Separable 3D CNN, S3D).

[0049] Optionally, the target video representation model is used to determine the video representation data corresponding to the video to perform feature representation on the video. The video representation data includes but is not limited to the video representation vector and probability distribution data corresponding to the video, which are not limited in the present embodiment. The first video representation data refers to the video representation data corresponding to the video clip to be processed.

[0050] For a detailed description of the above video representation model, please refer to the introduction in the embodiment of the video representation model training method below.

[0051] In an exemplary embodiment, the training samples include a target video segment and a sample video segment associated with the target video segment. The training process of the target video representation model includes the content corresponding to the video representation model training method embodiment described below.

[0052] In an exemplary embodiment, if Figure 3 As shown, the above method may further include the following steps (230-260): Figure 3 A flowchart of a similar video determination method provided by an embodiment of the present application is shown.

[0053] Step 230: Obtain at least two video clips.

[0054] Optionally, the at least two video clips are video clips in existing videos. The existing videos include existing videos in a server video library and videos on the Internet, which is not limited in the embodiment of the present application.

[0055] Step 240: input at least two video clips into a target video representation model to obtain second video representation data corresponding to the at least two video clips.

[0056] Optionally, the at least two video clips are respectively input into the target video representation model, and the target video representation model performs video feature extraction processing on the at least two video clips to obtain video representation data corresponding to each of the at least two video clips, such as video representation vectors, probability distribution data, etc.

[0057] Optionally, the second video representation data includes video representation data corresponding to at least two video clips.

[0058] Step 250 : determining the video similarity between the to-be-processed video segment and at least two video segments based on the first video representation data and the second video representation data.

[0059] In a possible implementation, the first video representation data includes a video representation vector corresponding to the video to be processed, and the first video representation data includes video representation vectors corresponding to each of the at least two video segments. The vector similarities between the video representation vector corresponding to the video to be processed and the video representation vectors corresponding to each of the at least two video segments are calculated respectively, and the video similarity can be determined based on the vector similarity. The video similarity is used to characterize the similarity between different videos. Optionally, the video similarity is positively correlated with the vector similarity.

[0060] In one example, the video representation vector corresponding to video clip 1 is recorded as u, and the video representation vector corresponding to video clip 2 is recorded as v. The above vector similarity can be recorded as sim(u,v). T v / (||u||2||v||2). The above vector similarity can be used as the above video similarity.

[0061] It should be noted that vector similarity can also be calculated using other calculation methods, such as determining the cosine distance, Euclidean distance, etc. between two video representation vectors, and according to a certain mapping relationship, the video similarity can be determined based on the above cosine distance or Euclidean distance.

[0062] Step 260: Determine the video segments whose video similarity is greater than or equal to the similarity threshold as similar video segments corresponding to the video segment to be processed.

[0063] In the above steps, the video similarity between the to-be-processed video and the at least two video segments is determined, so the video segment similar to the to-be-processed video in the at least two video segments can be determined according to the similarity threshold.

[0064] Optionally, it is determined whether the similarity of each video is greater than or equal to a similarity threshold. If so, it is determined that the video segment corresponding to the video similarity is a similar video segment corresponding to the video segment to be processed.

[0065] Optionally, the similarity threshold may be a preset threshold. The similarity threshold may be set according to the actual scene, and the embodiment of the present application does not limit this.

[0066] In an exemplary embodiment, if Figure 4 As shown, the above method may further include the following step 270: Figure 4 A flow chart of a video segment positioning method provided by an embodiment of the present application is shown.

[0067] Step 270: Determine the segment location information corresponding to the video segment to be processed based on the similar video segments.

[0068] Optionally, the meta information of the similar video clips is obtained. The meta information includes attribute information such as the source video identifier corresponding to the similar video clips, the position information of the similar video clips in the source video, etc. Optionally, the position information includes the start frame number and the end frame number corresponding to the similar video clips in the source video, or the start time or the end time.

[0069] Optionally, based on the meta information, the segment location information of the video segment to be processed may be determined. The segment location information includes source video information and position information corresponding to the video segment to be processed.

[0070] In a possible implementation, the source video identifier and the position information in the meta information corresponding to the similar video segment are determined as the segment positioning information.

[0071] In some application scenarios, the similar video segment and the to-be-processed video segment are the same video segment, or may be different but similar video segments.

[0072] In summary, the technical solution provided by the embodiment of the present application can determine the similarity information between video clips through the video representation data corresponding to the video clips in the training samples, and introduce the temporal distance information between the video clips on the basis of the similarity information to determine the distance attenuation loss information, so that during the model training process, the negative correlation between the temporal distance information and the similarity information can be adjusted according to the distance attenuation loss information, so that the video representation model learns the correlation between the distance of different video clips in the time dimension and their similarity, so that more accurate video representation data can be output for feature representation of the processed video, which effectively improves the accuracy of video representation.

[0073] In addition, since the above-mentioned target video representation model can output more accurate video representation data, similar videos can be judged based on the above-mentioned target video representation, and video clips with higher precision requirements can be located, thereby improving the accuracy of similar video judgment and the positioning accuracy of video clips.

[0074] Please refer to Figure 5 , which shows the process of the video representation model training method provided by an embodiment of the present application Figure 1The method can be applied to a computer device, which refers to an electronic device with data calculation and processing capabilities. For example, the execution subject of each step can be Figure 1 The terminal 10 or server 20 in the application running environment shown. The method may include the following steps (501-550).

[0075] Step 510: Determine temporal distance information based on the target video segment and the sample video segment.

[0076] Optionally, the temporal distance information is used to characterize the position distance relationship between the target video segment and the sample video segment.

[0077] In an exemplary embodiment, the training sample includes a target video segment and a sample video segment associated with the target video segment. Optionally, the target video segment (Anchor) can be any video segment in the training sample.

[0078] Optionally, the sample video clips include a first video clip, a second video clip and a third video clip. The first video clip refers to a video clip in the target video that is different from the target video clip; the second video clip refers to a video clip in the first sample video that is different from the target video; and the third video clip refers to a video clip in the second sample video that is enhanced in a different manner than the target video.

[0079] The target video refers to the video clip where the target video clip is located, the first sample video is a sample video different from the target video, and the second sample video is a video that is the same as the source video corresponding to the target video but has a different enhancement method. The enhancement methods correspond to different data enhancement processing processes.

[0080] In an exemplary embodiment, if Figure 6 As shown, the process of obtaining the training samples may include the following steps (560-580): Figure 6 The process of the video representation model training method provided by one embodiment of the present application is shown Figure 2 .

[0081] Step 560: Obtain at least two sample videos.

[0082] Optionally, the at least two sample videos are obtained from a sample video dataset, which includes but is not limited to UCF101 and HMDB51.

[0083] UCF101 is an action recognition dataset of real action videos, collected from YouTube, providing 13,320 videos from 101 action categories. Among them, the sample videos in UCF101 include five categories of actions: human-object interaction, simple body movements, human-human interaction, playing musical instruments, and sports.

[0084] HMDB51 is an action recognition dataset. The 51 action categories in HMDB51 can be divided into the following five categories: common facial actions, complex facial actions, common body movements, complex body movements, and multi-person interactive body movements.

[0085] Step 570, determine the random sampling interval.

[0086] This embodiment performs application data expansion from the spatial and temporal dimensions. In the temporal dimension, considering the influence of the random stride representing the number of frames between two segments on the similarity between the segments, this embodiment determines the random sampling interval while ensuring that all video segments (clips) have enough frames, so that the starting position corresponding to each sampled video segment is random.

[0087] The random sampling interval can also be called a random stride, which is a data enhancement method for video clips. It means that in the sampling process, in order to maintain the richness of the samples, there is a random interval between clips. Optionally, the random stride [0, 5] means that the interval is [0, 5].

[0088] Step 580: Perform video segment sampling processing on at least two sample videos according to a random sampling interval to obtain training samples.

[0089] In a possible implementation, four 16-frame clips (ie, video segments) are extracted from each sample video at a random step size of [0, 5] to obtain training samples.

[0090] Optionally, before the video clips in the training samples are input into the video representation model (encoder), data enhancement processing is performed on them to obtain enhanced video clips.

[0091] Optionally, the data enhancement processing includes but is not limited to: random cropping, resizing, horizontal flipping, Gaussian blurring, color jittering and other data enhancement processing methods corresponding to the spatial dimension. Through different data enhancement processing methods, the video representation model can focus on learning the time evolution of each pixel position in the video, rather than information such as color changes.

[0092] Optionally, this embodiment retains the same hyperparameter group for a video. In other words, clips from the same video share a set of enhancement hyperparameters. The enhancement hyperparameters in the hyperparameter group are used to determine the enhancement method corresponding to the target video. According to the enhancement hyperparameters, the target data enhancement method corresponding to the enhancement hyperparameters can be determined, so that the data enhancement processing corresponding to the target data enhancement method can be performed to obtain the enhanced video clips.

[0093] By performing sampling at random intervals as described above, the richness of sample video clips can be improved, as well as the diversity of temporal distances between sample video clips, thereby helping to improve the accuracy of the model in representing videos.

[0094] In an exemplary embodiment, the sample video segment includes a positive sample corresponding to the target video segment. Figure 6 As shown, the implementation process of the above step 510 may include the following steps (511-513).

[0095] Step 511: determine the segment distance between the target video segment and the positive sample.

[0096] Optionally, a first segment distance between the target video segment and the first positive sample is determined. For example, a segment distance between the target video segment and the first video segment, that is, the first segment distance is determined.

[0097] Optionally, a second segment distance between the target video segment and the second positive sample is determined. For example, a segment distance between the target video segment and the third video segment, ie, the second segment distance, is determined.

[0098] The above-mentioned segment distance includes but is not limited to the interval duration and the number of interval frames between video segments. The embodiment of the present application does not limit this and can be selected according to the specific application scenario.

[0099] Step 512: determine the overall video length corresponding to the target video segment.

[0100] Optionally, according to the target video segment, the target video in which the target video segment is located is determined; the overall video length of the target video is obtained, and the overall video length of the target video can be used as the overall video length corresponding to the target video segment. The overall video length includes the total duration and total number of frames of the target video.

[0101] Step 513, determining temperature index data based on the segment distance and the overall video length.

[0102] The temporal distance information includes temperature index data, and the temperature index data is negatively correlated with the fragment distance.

[0103] In a possible implementation (implementation 1), the positive sample includes a first positive sample, the first positive sample includes a first video segment in the target video that is different from the target video segment, and the segment distance includes a first segment distance between the target video segment and the first video segment.

[0104] Accordingly, in the above-mentioned embodiment 1, if Figure 7 As shown, the implementation process of the above step 513 may include the following step 513a: Figure 7 The process of the video representation model training method provided by one embodiment of the present application is shown Figure 3 .

[0105] Step 513a, determining first temperature indicator data based on the first segment distance and the overall video length.

[0106] The temperature index data includes first temperature index data, and the first temperature index data is used to adjust the similarity between the target video segment and the first video segment.

[0107] Optionally, the first temperature indicator data may be a first attenuation temperature coefficient. Optionally, the first attenuation temperature coefficient may be determined according to the following formula (1).

[0108]

[0109] Among them, l represents the segment distance between the target video segment and the positive sample, L represents the overall video length of the target video where the target video segment is located, τ represents the temperature coefficient, and τ i,j represents the first attenuation temperature coefficient. Optionally, the temperature coefficient τ is a constant, which can be selected and determined according to a specific application scenario.

[0110] Optionally, the smaller the timing distance (i.e., segment distance), the larger the first attenuation temperature coefficient, and the larger the timing distance (i.e., segment distance), the smaller the first attenuation temperature coefficient. The first attenuation temperature coefficient decays with the increase of the segment distance, that is, the first attenuation temperature coefficient is negatively correlated with the segment distance.

[0111] In another possible implementation (implementation 2), based on the above implementation 1, the positive sample also includes a second positive sample, the second positive sample includes a third video segment in a second sample video that adopts a different enhancement method from the target video, and the segment distance includes a second segment distance between the target video segment and the third video segment.

[0112] Accordingly, in the above-mentioned embodiment 2, if Figure 8 As shown, the implementation process of the above step 513 may also include the following steps (513b-513c): Figure 8The process of the video representation model training method provided by one embodiment of the present application is shown Figure 4 .

[0113] Step 513b, obtaining constraint indicator data.

[0114] Optionally, the constraint index data is a constraint coefficient, and the constraint index data can be determined according to an actual application scenario. Optionally, the constraint coefficient can be a constant less than 1. Optionally, the preset index data is used to constrain the attenuation amount and attenuation rate of the second temperature index data described below.

[0115] Step 513c, determining the second temperature indicator data based on the constraint indicator data, the second segment distance and the overall video length.

[0116] The temperature index data includes second temperature index data, and the second temperature index data is used to constrain the similarity between the target video segment and the first video segment.

[0117] Optionally, the second temperature index data may be a second attenuation temperature coefficient. Optionally, the second attenuation temperature coefficient may be determined according to the following formula (2).

[0118]

[0119] Among them, l represents the segment distance between the target video segment and the positive sample, L represents the overall video length of the target video where the target video segment is located, ρ represents the constraint coefficient, τ represents the temperature coefficient, and τ′ i,j Optionally, the temperature coefficient τ and the constraint coefficient ρ are constants, which can be selected and determined according to specific application scenarios.

[0120] Step 520: Input the target video segment and the sample video segment into the video representation model to be trained to obtain third video representation data corresponding to the target video segment and fourth video representation data corresponding to the sample video segment.

[0121] The third video representation data refers to the video representation data corresponding to the target video segment, including but not limited to the video representation vector and probability distribution data corresponding to the target video segment. Optionally, the target video segment is input into the video representation model to be trained for video feature extraction processing to obtain the video representation vector corresponding to the target video segment (probability distribution data may also be output).

[0122] The fourth video representation data refers to the video representation data corresponding to the sample video clip, including but not limited to the video representation vector and probability distribution data corresponding to the sample video clip. Optionally, the sample video clip is input into the video representation model to be trained for video feature extraction processing to obtain the video representation vector corresponding to the sample video clip (probability distribution data may also be output).

[0123] Optionally, the sample video clips include but are not limited to a first video clip, a second video clip, and a third video clip. Accordingly, the first video clip, the second video clip, and the third video clip are input into the video representation model for video feature extraction to obtain a video representation vector corresponding to the first video clip, a video representation vector corresponding to the second video clip, and a video representation vector corresponding to the third video clip.

[0124] Step 530: Determine similarity information based on the third video representation data and the fourth video representation data.

[0125] Optionally, the similarity information includes a first similarity between the target video segment and the first video segment, a second similarity between the target video segment and the second video segment, and a third similarity between the target video segment and the third video segment.

[0126] Optionally, the first similarity is determined based on the video representation vectors corresponding to the target video segment and the first video segment respectively; the second similarity is determined based on the video representation vectors corresponding to the target video segment and the second video segment respectively; and the third similarity is determined based on the video representation vectors corresponding to the target video segment and the third video segment respectively.

[0127] Step 540: Determine distance attenuation loss information based on the temporal distance information and the similarity information.

[0128] In the above-mentioned embodiment 1, the sample video segment also includes a first negative sample corresponding to the first positive sample, the first negative sample includes a second video segment in the first sample video different from the target video, and the similarity information includes a first similarity between the target video segment and the first video segment, and a second similarity between the target video segment and the second video segment.

[0129] Accordingly, in the above-mentioned embodiment 1, if Figure 7 As shown, the implementation process of the above step 540 may include the following step 541.

[0130] Step 541 : determining external attenuation contrast loss data based on the first temperature index data, the first similarity, and the second similarity.

[0131] The distance attenuation loss information includes external attenuation contrast loss data, and the external attenuation contrast loss data is used to adjust the negative correlation between the first segment distance and the first similarity.

[0132] The mapping relationship between the external decaying contrast loss data and the first temperature index data (first decaying temperature coefficient), the first similarity and the second similarity can be determined by an external decaying contrast loss (ExdNCE) function.

[0133] Optionally, the external attenuation contrast loss function is shown in the following formula (3).

[0134]

[0135] Among them, Z i Represents the video representation data corresponding to the target video segment, Z j represents the video representation data corresponding to the first video clip (first positive sample), Z n represents the video representation data corresponding to the second video clip (first negative sample), sim(Z i ,Z j ) represents the first similarity, sim(Z i ,Z n ) represents the second similarity, τ i,j represents the first attenuation temperature coefficient, τ represents the temperature coefficient, N represents the batch size, k represents the sequence number of the video clip, Represents the above external attenuation versus loss data.

[0136] Optionally, the similarity between any two video clips u and v is sim(u,v).

[0137] Optionally, sim(u,v)=u T v / (||u||2||v||2).

[0138] Since the first attenuation temperature coefficient is negatively correlated with the segment distance, the longer the temporal distance is, the smaller the first attenuation temperature coefficient is, and the external attenuation contrast loss is relatively reduced, which means that the degree of attention is smaller, thereby reducing the adjustment degree of the negative correlation between the first segment distance and the second similarity, that is, the smaller the change in similarity between the target video segment and the first negative sample, the model will not close the feature representation distance between the target video segment and the first negative sample. On the other hand, the smaller the temporal distance is, the larger the first attenuation temperature coefficient is, and the external attenuation contrast loss is relatively increased, which means that the degree of attention is greater, thereby causing the model to deepen the positive adjustment degree of the negative correlation between the first segment distance and the first similarity, that is, the change in similarity between the target video segment and the first negative sample increases, and the model will close the feature representation distance between the target video segment and the first negative sample. In this way, the negative correlation between the first segment distance and the first similarity will be deepened, ensuring that the similarity between the first positive sample and the target video segment has a temporal relationship, and improving the accuracy of video representation.

[0139] In some possible implementations, based on the similarity between the target video segment and the positive sample and the similarity between the target video segment and the negative sample, the contrast loss data is determined to train the video representation model. However, this does not take into account the differences between different segments in the same video, which will result in the feature representation of the target video segment and the positive sample being indistinguishably close. The video representation data output by the video representation model trained in this way is suitable for video classification tasks, but is difficult to be applied to fine-grained downstream tasks such as video segment positioning.

[0140] In the above-mentioned implementation mode 1, the first temperature index data associated with the segment distance between the video segments is introduced into the loss function, and then the external attenuation contrast loss data is determined in combination with the above-mentioned first similarity and second similarity. The video representation model is trained by the external attenuation contrast loss data determined in this way, and the negative correlation between the first segment distance and the first similarity can be adjusted, so that the similarity between the target video segment in the target video and other video segments in the target video is associated with the temporal distance between them, showing a change in which the closer the temporal distance, the higher the similarity. By introducing the attenuation temperature coefficient that decays with the temporal distance between segments into the loss function, the video representation model can learn the differences between different segments in the same video during the model training process, improve the accuracy of the video representation, and improve the accuracy of the video representation data output by the video representation model, which is more suitable for fine-grained downstream tasks such as video segment positioning.

[0141] In one example, if Fig. 9 As shown, it exemplarily shows a training process diagram of a video representation model Figure 1 . Fig. 9The two clip sequences (sequence 91 and sequence 92) shown in are examples of input video representation model 93. Sequence 91 includes segments A1, A2, A3 and A4; sequence 92 includes segments B1, B2, B3 and B4. A and B indicate that they are from different videos. Segments A1, A2, A3 and A4 are from video A; segments B1, B2, B3 and B4 are from video B. The clip sequence number (1 / 2 / 3 / 4) after the letter indicates the relative position of the video segment on the timeline.

[0142] Exemplarily, segment A1 can be used as the target video segment (Anchor). According to the external attenuation contrast loss function, different video segments in the same video as the target video segment are used as positive samples, and video segments in different videos from the target video segment are used as negative samples. Fig. 9 In the example, segments A2, A3, and A4 are positive samples corresponding to segment A1, and segments B1, B2, B3, and B4 are negative samples corresponding to segment A1.

[0143] The lines between segment A1 and other video segments represent the correlation adjustment relationship between them. The above lines are of two types, one is a solid line representing an attraction relationship, and the other is a dotted line representing a repulsion relationship. The thickness of the lines indicates different attraction strengths or repulsion degrees. The thicker the line, the greater the attraction strength or repulsion degree.

[0144] Above Fig. 9 The connection relationship in is determined based on the external attenuation contrast loss, the attraction strength is determined based on the first attenuation temperature coefficient, and the dynamic first attenuation temperature coefficient acts on the first positive sample. Fig. 9 As can be seen from the example of , the external attenuated contrast loss aims to increase the relative similarity between video clips from the same video and distinguish video clips from different videos.

[0145] Optionally, in Fig. 9 In the figure, as the distances between fragment A1 and fragment A2, fragment A3 and fragment A4 increase, the attraction strengths between fragment A1 and fragment A2, fragment A3 and fragment A4 decrease successively, that is, the attraction strength between fragment A1 and fragment A2 is greater than the attraction strength between fragment A1 and fragment A3, and the attraction strength between fragment A1 and fragment A3 is greater than the attraction strength between fragment A1 and fragment A4.

[0146] In another possible implementation 2, based on the above implementation 1, the sample video segment also includes a second negative sample corresponding to the second positive sample, the second negative sample includes the first video segment, and the similarity information also includes a third similarity between the target video segment and the third video segment.

[0147] Accordingly, in the above-mentioned embodiment 2, if Figure 8 As shown, the implementation process of the above step 540 may also include the following steps (542-543).

[0148] Step 542, determining internal attenuation contrast loss data based on the second temperature index data, the first similarity, and the third similarity.

[0149] The internal attenuated contrastive loss data is used to regularize the external attenuated contrastive loss data.

[0150] The mapping relationship between the internal decaying contrast loss data and the second temperature index data, the first similarity and the third similarity may be determined according to an internal decaying contrastive loss (IndNCE) function.

[0151] Optionally, the above internal attenuation contrast loss function is shown in the following formula (4).

[0152]

[0153] Among them, Z i Represents the video representation data corresponding to the target video segment, Z p represents the video representation data corresponding to the third video clip (the second positive sample), Z j represents the video representation data corresponding to the first video clip (the second negative sample), sim(Z i ,Z p ) represents the third similarity, sim(Z i ,Z j ) represents the first similarity, τ′ i,j represents the second attenuation temperature coefficient, τ represents the temperature coefficient, N represents the batch size, k represents the sequence number of the video clip, Represents the above internal attenuation contrast loss data.

[0154] In one example, if Fig.10 As shown, it exemplarily shows a training process diagram of a video representation model Figure 2 . Fig.10The two clip sequences (sequence 91 and sequence 94) shown in are examples of input video representation model 93. Sequence 91 includes fragment A1, fragment A2, fragment A3 and fragment A4; sequence 94 includes fragment A1', fragment A2', fragment A3' and fragment A4'. A and B indicate that they come from different videos. Fragments A1, A2, A3 and A4 are from video A; fragments A1', A2', A3' and A4' are from video B. Video A and video B have the same original video but different enhancement methods, that is, video A and video B are videos with different enhancement methods obtained after the same original video has been processed with different data enhancements. Optionally, fragments A1 and A1' only have different enhancement methods, the video content is the same, and the other fragments are similar. The clip sequence number (1 / 2 / 3 / 4) after the letter indicates the relative position of the video fragment on the timeline.

[0155] For example, segment A1 can be used as the target video segment (Anchor). According to the internal attenuation contrast loss function, different video segments in the same target video as the target video segment are used as negative samples, and video segments in videos that use different enhancement methods than the target video are used as negative samples. Fig.10 In , segments A2, A3 and A4 are negative samples corresponding to segment A1, and segments A1', A2', A3' and A4' are positive samples corresponding to segment A1.

[0156] The lines between segment A1 and other video segments represent the correlation adjustment relationship between them. The lines are of two types, one is a solid line representing an attraction relationship, and the other is a dotted line representing a repulsion relationship. The thickness of the lines indicates different attraction strengths or repulsion degrees. The thicker the line, the greater the attraction strength or repulsion degree. Fig.10 The connection relationship in is determined based on the external attenuation contrast loss, the above-mentioned repulsion strength is determined based on the above-mentioned second attenuation temperature coefficient, and the above-mentioned dynamic second attenuation temperature coefficient acts on the second negative sample. The internal attenuation contrast loss is used as a regularization of the above-mentioned external attenuation contrast loss. It aims to constrain the upper bound of the relative similarity between different video clips from the same video.

[0157] Optionally, in Fig.10In the figure, for the negative sample side, as the segment distances between segment A1 and segment A2, segment A3 and segment A4 increase, the repulsion degree between segment A1 and segment A2, segment A3 and segment A4 also increases accordingly, that is, the repulsion degree between segment A1 and segment A4 is greater than the repulsion degree between segment A1 and segment A3, and the repulsion degree between segment A1 and segment A3 is greater than the repulsion degree between segment A1 and segment A2. On the other hand, for the positive sample side, the segment distance between segment A1 and segment A1' is smaller than the segment distance between segment A1 and segment A2', so the attraction strength between segment A1 and segment A1' is higher than the attraction strength between segment A1 and segment A2'; as the segment distances between segment A1 and segment A3' and segment A4' increase, segment A1 and segment A3' and segment A4' become repulsive relationships, and the repulsion degree between segment A1 and segment A3' and segment A4' also increases accordingly, that is, the repulsion degree between segment A1 and segment A3' is less than the repulsion degree between segment A1 and segment A4'.

[0158] Step 543: fuse the external attenuation contrast loss data and the internal attenuation contrast loss data to obtain distance attenuation contrast loss data.

[0159] The distance attenuation contrast loss data is used to characterize the distance attenuation loss information.

[0160] Optionally, a Decaying Contrastive Learning of Visual Representation (DCLR) function is as shown in the following formula (5).

[0161]

[0162] Among them, L DCLR Represents the distance attenuation contrast loss data, represents the above external attenuation contrast loss data, represents the above internal attenuation contrast loss data, and E represents expectation.

[0163] Step 550 , performing model training on the video representation model to be trained according to the model loss information to obtain a target video representation model.

[0164] In one possible implementation, such as the above-mentioned implementation 1, based on the above-mentioned external attenuation contrast loss data, the above-mentioned video representation model to be trained is subjected to parameter adjustment processing to obtain an adjusted video representation model; then the above-mentioned target video segment and the sample video segment are input into the adjusted video representation model, and the external attenuation contrast loss data is re-determined; when the external attenuation contrast loss data is in the first preset interval, the current model parameters are saved to obtain the above-mentioned target video representation model.

[0165] In another possible implementation, such as the above-mentioned implementation 2, based on the above-mentioned distance attenuation contrast loss data, the parameters of the above-mentioned video representation model to be trained are adjusted to obtain the adjusted video representation model; then the above-mentioned target video clip and the sample video clip are input into the adjusted video representation model, and the distance attenuation contrast loss data is re-determined; when the distance attenuation contrast loss data is in the second preset interval, the current model parameters are saved to obtain the above-mentioned target video representation model.

[0166] Optionally, the first preset interval and the second preset interval may be determined according to actual scenarios, which is not limited in the embodiment of the present application.

[0167] In a possible implementation, after obtaining the target video representation model, the target video representation model is tested to obtain a test result. Optionally, the test result includes video retrieval accuracy. Optionally, the query video segment and the video segments in the preset video library are respectively input into the target video representation model, and the target video representation model outputs the video representation data of each video segment. Based on the video representation data of each video segment, a video search can be performed to obtain a search result corresponding to the query video segment, and the search result includes the retrieved video segment. The index information corresponding to the query video segment and the index information corresponding to the retrieved video segment are respectively obtained, and the video retrieval accuracy (which can also be understood as the positioning accuracy of the video segment) can be determined based on the above two types of index information. Optionally, the index information includes but is not limited to the source video identifier corresponding to the video segment and the segment position information of the video segment in the source video. Optionally, the segment position information includes a segment order identifier. The index information is used to represent the timing information of the video segment in the source video. For example, index information 11 can indicate that video segment 11 is the first video segment in video 1.

[0168] Optionally, the video retrieval accuracy can be recorded as La. Optionally, La can be determined by the following formula (6).

[0169]

[0170] Wherein, i represents the index of the query video segment q, j represents the index of the retrieved video segment r, lq represents the overall video length of the source video where the query video segment q is located, and lr represents the overall video length of the source video where the retrieved video segment r is located. It can be seen that the distance between the query video segment q and the retrieved video segment r can be judged by the above video retrieval accuracy, and the distance between the query video segment q and the retrieved video segment r is negatively correlated with the video retrieval accuracy La. The higher the above video retrieval accuracy, the more accurate the video representation data output by the target video representation model. In summary, the technical solution provided by the embodiment of the present application can determine the similarity information between the video segments through the video representation data corresponding to the video segments in the training samples, and introduce the temporal distance information between the video segments on the basis of the similarity information to determine the distance attenuation loss information, so that in the model training process, the negative correlation between the temporal distance information and the similarity information can be adjusted according to the distance attenuation loss information, so that the video representation model learns the correlation between the distance of different video segments in the time dimension and their similarity, so that more accurate video representation data can be output to represent the features of the processed video, which effectively improves the accuracy of the video representation.

[0171] In addition, the embodiment of the present application designs an attenuation strategy based on the soft probability distribution generated by the temperature parameter in the knowledge distillation, introduces a variable temperature coefficient that changes dynamically with the time sequence distance in the contrast loss, thereby determining two attenuation contrast losses, and formulates different positive and negative sample rules to train the video representation model, so that the video representation model learns the gradual evolution of the video representation in the time dimension. One of the attenuation contrast losses is the external attenuation loss, which is used to deepen the negative correlation between the temporal distance and the similarity between the target video segment and the positive sample, and reduce the similarity between the target video segment and the negative sample; the other attenuation contrast loss is the internal attenuation loss. The positive sample corresponding to the external attenuation loss is adjusted to a negative sample in the internal attenuation loss, thereby regularizing the external attenuation loss, constraining the upper bound of the similarity between the target video segment and the positive sample, and further improving the accuracy of the video representation. Through the contrast learning strategy of attenuation along the temporal distance, not only can different segments in the same video have good similarity, but also the video segment with a closer temporal distance to the video segment to be retrieved can have a higher similarity, which can better meet the downstream tasks of video retrieval and positioning.

[0172] The application effect of the video characterization method provided by the embodiment of the present application is described below in combination with some specific experimental data. Please refer to the following Table 1, which exemplarily shows the result data obtained by conducting experiments on two mainstream data sets, UCF101 and HMDB51, using three common video feature extraction network models, R3D, C3D, and R(2+1)D, in the embodiment of the present application, as well as the corresponding experimental result data of related technologies.

[0173] The brackets in the dataset contain the total video duration (d means days); R@k means that for a certain goal, we find the first k most similar samples in all test sets. If one of the k samples is hit, it is a success (R@1, R@5, R@10, and R@20 all have similar explanations).

[0174] Table 1

[0175]

[0176]

[0177] It can be seen from the data in the above table that compared with some related technologies, such as STS, CSJ, SpeedNet (video object speed prediction network), MemDPC (Memory-augmented Dense Predictive Coding), Pace Pred (speed prediction), Var.PSP (Pyramid Scene Parsing Network), TempTrans, RSPNet (Relative Speed ​​Perception), TCLR (Temporal Contrastive Learning for Video Representation), the gain effect obtained by applying the video representation model based on distance attenuation contrast loss training in the embodiment of the present application is as follows:

[0178] Based on the premise that the R(2+1)D network structure is pre-trained on UCF101, the embodiment of the present application has achieved the current best results in the nearest neighbor video retrieval R@1, R@5, and R@10 on the test set of UCF101, which are 1.4%, 2.1%, and 0.4% higher than the TCLR training method under the same experimental conditions and network structure; Based on the premise that the R3D network structure is pre-trained on UCF101, the embodiment of the present application has achieved the current best result in the nearest neighbor video retrieval R@5 of HMDB51, which is 1.3% higher than TCLR under the same experimental conditions and network structure; In the network structure of C3D and under the same experimental conditions, the results of the embodiment of the present application in the nearest neighbor video retrieval R@1, R@5, R@10 and R@20 all exceed those of other methods (the comparison methods under the same conditions include: MemDPC, Pace Pred, RSPNet).

[0179] The last set of data in Table 1 above lists the results of the positioning accuracy of R@k in TCLR and DCLR (this application). It can be seen that compared with TCLR, the positioning accuracy of the video clip positioning using the video representation model based on distance attenuation contrast loss training in the embodiment of this application has been greatly improved.

[0180] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0181] Please refer to Fig.11 , which shows a block diagram of a video characterization device provided by an embodiment of the present application. The device has the function of implementing the above-mentioned video characterization method, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be a computer device, or can be set in a computer device. The device 1100 can include: a video acquisition module 1110, a video characterization module 1120.

[0182] The video acquisition module 1110 is used to acquire the video segment to be processed.

[0183] The video representation module 1120 is configured to input the to-be-processed video segment into a target video representation model to obtain first video representation data corresponding to the to-be-processed video segment.

[0184] Among them, the target video representation model is a machine learning model obtained by training based on training samples and distance attenuation loss information, the distance attenuation loss information is determined based on the corresponding temporal distance information and similarity information between the video clips in the training samples, the similarity information is determined based on the video representation data corresponding to the video clips, and the distance attenuation loss information is used to adjust the degree of negative correlation between the temporal distance information and the similarity information.

[0185] In an exemplary embodiment, the apparatus 1100 further includes: a similarity determination module and a similar video determination module.

[0186] The video acquisition module 1110 is further configured to acquire at least two video clips.

[0187] The video representation module 1120 is further configured to input the at least two video segments into the target video representation model to obtain second video representation data corresponding to the at least two video segments.

[0188] The similarity determination module is used to determine the video similarity between the to-be-processed video segment and the at least two video segments based on the first video representation data and the second video representation data.

[0189] The similar video determination module is used to determine the video segments whose video similarity is greater than or equal to a similarity threshold as similar video segments corresponding to the video segment to be processed.

[0190] In an exemplary embodiment, the apparatus 1100 further includes: a video segment positioning module.

[0191] The video segment positioning module is used to determine the segment positioning information corresponding to the video segment to be processed based on the similar video segment.

[0192] In an exemplary embodiment, the training samples include a target video segment and a sample video segment associated with the target video segment. The device 1100 also includes a model training module, which includes: a temporal distance determination unit, a video segment representation unit, a similarity determination unit, a loss information determination unit, and a model training unit.

[0193] The temporal distance determining unit is used to determine the temporal distance information based on the target video segment and the sample video segment.

[0194] The video segment representation unit is used to input the target video segment and the sample video segment into the video representation model to be trained to obtain third video representation data corresponding to the target video segment and fourth video representation data corresponding to the sample video segment.

[0195] The similarity determination unit is configured to determine the similarity information based on the third video representation data and the fourth video representation data.

[0196] A loss information determining unit is used to determine the distance attenuation loss information based on the time series distance information and the similarity information.

[0197] A model training unit is used to perform model training on the video representation model to be trained according to the model loss information to obtain the target video representation model.

[0198] In an exemplary embodiment, the sample video segment includes a positive sample corresponding to the target video segment, and the temporal distance determination unit includes: a segment distance determination subunit, a video length determination subunit, and a temperature index determination subunit.

[0199] The segment distance determination subunit is used to determine the segment distance between the target video segment and the positive sample.

[0200] The video length determination subunit is used to determine the overall video length corresponding to the target video segment.

[0201] The temperature index determination subunit is used to determine temperature index data based on the segment distance and the overall video length, the temporal distance information includes the temperature index data, and the temperature index data is negatively correlated with the segment distance.

[0202] In an exemplary embodiment, the positive sample includes a first positive sample, the first positive sample includes a first video segment in the target video that is different from the target video segment, and the segment distance includes a first segment distance between the target video segment and the first video segment.

[0203] The temperature index determination subunit is specifically used to determine first temperature index data based on the first segment distance and the overall video length.

[0204] The temperature index data includes the first temperature index data, and the first temperature index data is used to adjust the similarity between the target video segment and the first video segment.

[0205] In an exemplary embodiment, the sample video segment also includes a first negative sample corresponding to the first positive sample, the first negative sample includes a second video segment in a first sample video different from the target video, and the similarity information includes a first similarity between the target video segment and the first video segment, and a second similarity between the target video segment and the second video segment.

[0206] The loss information determining unit includes: an external attenuation loss determining subunit.

[0207] The external attenuation loss determination subunit is used to determine external attenuation contrast loss data based on the first temperature index data, the first similarity and the second similarity.

[0208] The distance attenuation loss information includes the external attenuation contrast loss data, and the external attenuation contrast loss data is used to adjust the negative correlation between the first segment distance and the first similarity.

[0209] In an exemplary embodiment, the positive sample also includes a second positive sample, the second positive sample includes a third video segment in a second sample video that adopts a different enhancement method from the target video, and the segment distance includes a second segment distance between the target video segment and the third video segment.

[0210] The temperature index determination subunit is further specifically used for:

[0211] Obtain constraint index data; determine second temperature index data based on the constraint index data, the second segment distance and the overall video length.

[0212] The temperature index data includes the second temperature index data, and the second temperature index data is used to constrain the similarity between the target video segment and the first video segment.

[0213] In an exemplary embodiment, the sample video clip also includes a second negative sample corresponding to the second positive sample, the second negative sample includes the first video clip, and the similarity information also includes a third similarity between the target video clip and the third video clip; the loss information determination unit also includes: an internal attenuation loss determination subunit and a contrast loss data fusion subunit.

[0214] The internal attenuation loss determination subunit is used to determine internal attenuation contrast loss data based on the second temperature index data, the first similarity and the third similarity, wherein the internal attenuation contrast loss data is used to regularize the external attenuation contrast loss data.

[0215] The contrast loss data fusion subunit is used to fuse the external attenuation contrast loss data and the internal attenuation contrast loss data to obtain distance attenuation contrast loss data, and the distance attenuation contrast loss data is used to characterize the distance attenuation loss information.

[0216] In an exemplary embodiment, the model training module further includes: a sample video acquisition unit, a sampling interval determination unit, and a video segment sampling unit.

[0217] The sample video acquisition unit is used to acquire at least two sample videos.

[0218] The sampling interval determination unit is used to determine the random sampling interval.

[0219] The video segment sampling unit is used to perform video segment sampling processing on the at least two sample videos according to the random sampling interval to obtain the training samples.

[0220] In summary, the technical solution provided by the embodiment of the present application can determine the similarity information between video clips through the video representation data corresponding to the video clips in the training samples, and introduce the temporal distance information between the video clips on the basis of the similarity information to determine the distance attenuation loss information, so that during the model training process, the negative correlation between the temporal distance information and the similarity information can be adjusted according to the distance attenuation loss information, so that the video representation model learns the correlation between the distance of different video clips in the time dimension and their similarity, so that more accurate video representation data can be output for feature representation of the processed video, which effectively improves the accuracy of video representation.

[0221] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0222] Please refer to Fig.12 , which shows a structural block diagram of a computer device provided by an embodiment of the present application. The computer device may be a server for executing the above-mentioned video characterization method. Specifically:

[0223] The computer device 1200 includes a central processing unit (CPU) 1201, a system memory 1204 including a random access memory (RAM) 1202 and a read only memory (ROM) 1203, and a system bus 1205 connecting the system memory 1204 and the central processing unit 1201. The computer device 1200 also includes a basic input / output system (I / O (Input / Output) system) 1206 that helps to transfer information between various components in the computer, and a large-capacity storage device 1207 for storing an operating system 1213, application programs 1214, and other program modules 1215.

[0224] The basic input / output system 1206 includes a display 1208 for displaying information and an input device 1209 such as a mouse and a keyboard for user inputting information. The display 1208 and the input device 1209 are connected to the central processing unit 1201 through an input / output controller 1210 connected to the system bus 1205. The basic input / output system 1206 may also include an input / output controller 1210 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1210 also provides output to a display screen, a printer, or other types of output devices.

[0225] The mass storage device 1207 is connected to the central processing unit 1201 through a mass storage controller (not shown) connected to the system bus 1205. The mass storage device 1207 and its associated computer readable media provide non-volatile storage for the computer device 1200. That is, the mass storage device 1207 may include a computer readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0226] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 1204 and mass storage device 1207 can be collectively referred to as memory.

[0227] According to various embodiments of the present application, the computer device 1200 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1200 can be connected to the network 1212 through the network interface unit 1211 connected to the system bus 1205, or the network interface unit 1211 can be used to connect to other types of networks or remote computer systems (not shown).

[0228] The memory further includes a computer program, which is stored in the memory and configured to be executed by one or more processors to implement the above-mentioned video characterization method.

[0229] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and when the at least one instruction, the at least one program, the code set or the instruction set is executed by a processor, the above-mentioned video characterization method is implemented.

[0230] Optionally, the computer readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives) or optical disks, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0231] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned video characterization method.

[0232] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0233] In addition, in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0234] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A video characterization method, characterized in that: The method comprises: Get the video clip to be processed; Inputting the to-be-processed video segment into a target video representation model to obtain first video representation data corresponding to the to-be-processed video segment; The training samples of the target video representation model include a target video segment and a sample video segment associated with the target video segment, the sample video segment includes a positive sample corresponding to the target video segment, the positive sample includes a first positive sample and a second positive sample, the first positive sample includes a first video segment in the target video that is different from the target video segment, and the second positive sample includes a third video segment in a second sample video that uses a different enhancement method from the target video; the sample video segment includes a first negative sample corresponding to the first positive sample and a second negative sample corresponding to the second positive sample, the first negative sample includes a second video segment in the first sample video that is different from the target video; the second negative sample includes the first video segment; the training process of the target video representation model includes: Determine the segment distance between the target video segment and the positive sample; determine the overall video length corresponding to the target video segment; Based on the segment distance and the overall video length, temperature index data is determined to obtain temporal distance information including the temperature index data, wherein the temperature index data is negatively correlated with the segment distance, and the temperature index data includes first temperature index data and second temperature index data, wherein the first temperature index data is used to adjust the similarity between the target video segment and the first video segment, and the second temperature index data is used to constrain the similarity between the target video segment and the first video segment; Inputting the target video segment and the sample video segment into a video representation model to be trained, obtaining third video representation data corresponding to the target video segment and fourth video representation data corresponding to the sample video segment; Determine similarity information based on the third video representation data and the fourth video representation data, the similarity information including a first similarity between the target video segment and the first video segment, a second similarity between the target video segment and the second video segment, and a third similarity between the target video segment and the third video segment; Determining external attenuation contrast loss data based on the first temperature index data, the first similarity, and the second similarity; Based on the second temperature index data, the first similarity and the third similarity, determine internal attenuation contrast loss data, wherein the internal attenuation contrast loss data is used to regularize the external attenuation contrast loss data; fusing the external attenuation contrast loss data and the internal attenuation contrast loss data to obtain distance attenuation loss information; Model training is performed on the video representation model to be trained according to the distance attenuation loss information to obtain the target video representation model.

2. The method according to claim 1, characterized in that The method further comprises: Obtain at least two video clips; Inputting the at least two video clips into the target video representation model to obtain second video representation data corresponding to the at least two video clips; Determining a video similarity between the to-be-processed video segment and the at least two video segments based on the first video representation data and the second video representation data; The video segment whose video similarity is greater than or equal to the similarity threshold is determined as a similar video segment corresponding to the video segment to be processed.

3. The method according to claim 2, characterized in that The method further comprises: Based on the similar video segments, segment positioning information corresponding to the video segment to be processed is determined.

4. The method according to any one of claims 1 to 3, characterized in that: The segment distance includes a first segment distance between the target video segment and the first video segment; and determining the temperature indicator data based on the segment distance and the overall video length includes: The first temperature indicator data is determined based on the first segment distance and the overall video length.

5. The method according to claim 4, characterized in that The external attenuation contrast loss data is used to adjust the negative correlation between the first segment distance and the first similarity.

6. The method according to claim 5, characterized in that The segment distance includes a second segment distance between the target video segment and the third video segment; and determining the temperature indicator data based on the segment distance and the overall video length further includes: Obtain constraint indicator data; The second temperature indicator data is determined based on the constraint indicator data, the second segment distance and the overall video length.

7. The method according to claim 1, characterized in that The method further comprises: Obtain at least two sample videos; Determine the random sampling interval; The at least two sample videos are subjected to video segment sampling processing according to the random sampling interval to obtain the training samples.

8. A video representation device, characterized in that: The device comprises: A video acquisition module is used to acquire the video clips to be processed; A video representation module, configured to input the video segment to be processed into a target video representation model to obtain first video representation data corresponding to the video segment to be processed; The training samples of the target video representation model include a target video segment and a sample video segment associated with the target video segment, the sample video segment includes a positive sample corresponding to the target video segment, the positive sample includes a first positive sample and a second positive sample, the first positive sample includes a first video segment in the target video that is different from the target video segment, and the second positive sample includes a third video segment in a second sample video that uses a different enhancement method from the target video; the sample video segment includes a first negative sample corresponding to the first positive sample and a second negative sample corresponding to the second positive sample, the first negative sample includes a second video segment in the first sample video that is different from the target video; the second negative sample includes the first video segment; the training process of the target video representation model includes: Determine the segment distance between the target video segment and the positive sample; determine the overall video length corresponding to the target video segment; Based on the segment distance and the overall video length, temperature index data is determined to obtain temporal distance information including the temperature index data, wherein the temperature index data is negatively correlated with the segment distance, and the temperature index data includes first temperature index data and second temperature index data, wherein the first temperature index data is used to adjust the similarity between the target video segment and the first video segment, and the second temperature index data is used to constrain the similarity between the target video segment and the first video segment; Inputting the target video segment and the sample video segment into a video representation model to be trained, obtaining third video representation data corresponding to the target video segment and fourth video representation data corresponding to the sample video segment; Determine similarity information based on the third video representation data and the fourth video representation data, the similarity information including a first similarity between the target video segment and the first video segment, a second similarity between the target video segment and the second video segment, and a third similarity between the target video segment and the third video segment; Determining external attenuation contrast loss data based on the first temperature index data, the first similarity, and the second similarity; Based on the second temperature index data, the first similarity and the third similarity, determine internal attenuation contrast loss data, wherein the internal attenuation contrast loss data is used to regularize the external attenuation contrast loss data; fusing the external attenuation contrast loss data and the internal attenuation contrast loss data to obtain distance attenuation loss information; Model training is performed on the video representation model to be trained according to the distance attenuation loss information to obtain the target video representation model.

Images