Video retrieval method, device, equipment and storage medium

Abstract

The present application relates to the field of computers, and in particular to the field of artificial intelligence, and provides a method, apparatus, device and storage medium for video retrieval to solve the problems of low quantization efficiency and low accuracy. The method includes: extracting features of the image features of the video to be retrieved to obtain a first quantized feature, and then based on the first quantized feature, obtaining a second candidate video with a high category similarity to the video to be retrieved, and finally using the second candidate video with a high content similarity to the video to be retrieved as the target video. The quantization control parameters are adjusted according to the texture feature loss value corresponding to each training sample, so that the target quantization processing sub-model learns the sorting ability of the target texture feature sub-model, ensuring that the sorting effects of the two sub-models tend to be consistent, and the end-to-end model architecture enables the target quantization processing sub-model to obtain the corresponding quantized features based on the image features, thereby improving the generation accuracy of the quantized features and the recall performance of the video retrieval.

Description

Background Art

[0002] In related technologies, quantitative features are usually used as index tags of videos to retrieve corresponding videos. Generally, any of the following methods is used to obtain the corresponding quantitative features:

[0003] Method 1: Obtain corresponding quantitative features based on the k-means clustering algorithm. However, when clustering large-scale sample data, in order to ensure the accuracy of index retrieval, a large amount of resources are required to obtain enough quantitative features;

[0004] Method 2: Obtain the corresponding quantitative features based on product quantization (PQ). However, the quantitative features obtained by this method will reduce the accuracy of the quantitative features due to the loss in the generation process, thus affecting the recall performance of video retrieval.

[0005] Method three obtains the corresponding quantitative features based on a deep learning neural network. However, this neural network first extracts the embedding features of the video image, and then performs feature extraction processing on the embedding features to obtain the corresponding quantitative features. This will reduce the accuracy of the generation of quantitative features due to losses in the generation process, thereby affecting the recall performance of video retrieval. Summary of the invention

[0006] The embodiments of the present application provide a method, apparatus, device and storage medium for video retrieval to solve the problems of low quantization efficiency and low accuracy.

[0007] In a first aspect, an embodiment of the present application provides a method for video retrieval, comprising:

[0008] The target image processing sub-model of the trained target video retrieval model is used to extract features of the video to be retrieved and obtain corresponding image features;

[0009] The target quantization processing submodel of the target video retrieval model is used to extract features of the image features to obtain corresponding first quantization features, and based on the first quantization features, at least one second candidate video that has a category similarity with the video to be retrieved and meets the set category similarity requirements is screened out from each first candidate video; wherein the quantization control parameter of the target quantization processing submodel is adjusted based on the texture feature loss value corresponding to each training sample during the training process, and the texture feature loss value is determined based on the texture control parameter preset for the texture processing submodel to be trained during the parameter adjustment process of the texture processing submodel to be trained;

[0010] Based on the content similarity between the video to be retrieved and the at least one second candidate video, the second candidate video whose content similarity meets the set content similarity requirement is output as the corresponding target video.

[0011] In a second aspect, an embodiment of the present application further provides a video retrieval device, including:

[0012] An image processing unit, used for extracting features of the video to be retrieved by using a target image processing sub-model of a trained target video retrieval model to obtain corresponding image features;

[0013] A quantization processing unit, used to use the target quantization processing sub-model of the target video retrieval model to extract features of the image features to obtain corresponding first quantization features, and based on the first quantization features, screen out at least one second candidate video that has a category similarity with the video to be retrieved and meets the set category similarity requirements from each first candidate video; wherein the quantization control parameter of the target quantization processing sub-model is adjusted based on the texture feature loss value corresponding to each training sample during the training process, and the texture feature loss value is determined based on the texture control parameter preset for the texture processing sub-model to be trained during the parameter adjustment process of the texture processing sub-model to be trained;

[0014] The retrieval unit is used to output the second candidate video whose content similarity meets the set content similarity requirement as the corresponding target video based on the content similarity between the video to be retrieved and the at least one second candidate video.

[0015] Optionally, the device further comprises a model training unit, wherein the model training unit obtains the trained target video retrieval model by executing the following method:

[0016] Obtaining a plurality of sample triplets, each sample triplet comprising a sample video, and a positive label and a negative label associated with the sample video;

[0017] Each sample triplet is used as training data and sequentially input into the image processing sub-model to be trained and the texture processing sub-model to be processed to obtain a corresponding first texture set; wherein, each time a first texture set is obtained, a corresponding image processing texture feature loss value is generated based on a plurality of first sample texture features contained in the first texture set, and based on the image processing texture feature loss value, parameters of the image processing sub-model to be trained and the texture processing sub-model to be trained are adjusted until the image processing texture feature loss value is not higher than a preset image processing texture feature loss threshold value, thereby obtaining a candidate image processing sub-model and a candidate texture processing sub-model;

[0018] Each sample triplet is used as training data and is input into the candidate image processing sub-model, the candidate texture processing sub-model and the quantization processing sub-model to be trained in turn to obtain a corresponding second texture set and a quantization feature group; wherein, each time a second texture set is obtained, a corresponding texture feature loss value is generated based on a plurality of second sample texture features contained in the second texture set, and based on the texture feature loss value, the parameter of the candidate texture processing sub-model is adjusted; and each time a quantization feature group is obtained, a corresponding quantization feature loss value is generated based on a plurality of sample quantization features contained in the quantization feature group and the texture feature loss value, and based on the quantization feature loss value, the parameter of the candidate image processing sub-model and the quantization processing sub-model to be trained is adjusted, until the texture feature loss value and the quantization feature loss value are both no higher than a preset feature loss threshold value, thereby obtaining the target image processing sub-model, the target texture processing sub-model, and the target quantization processing sub-model.

[0019] Optionally, the model training unit is used to:

[0020] Based on the texture feature loss value, adjusting the quantization control parameters of the quantization processing sub-model to be trained;

[0021] Based on the multiple sample quantization features included in the one quantization feature group and the quantization control parameter, respectively determine the training sample loss value and the symbol quantization loss value of the quantization processing sub-model to be trained;

[0022] Based on the training sample loss value and the symbol quantization loss value of the quantization processing sub-model to be trained, a corresponding quantization feature loss value is generated.

[0023] Optionally, the model training unit is used to:

[0024] Based on the multiple sample quantization features included in the one quantization feature group, respectively determine the training sample loss value and the symbol quantization loss value of the quantization processing sub-model to be trained;

[0025] Based on the training sample loss value, the symbol quantization loss value and the texture feature loss value of the quantization processing sub-model to be trained, a corresponding quantization feature loss value is generated.

[0026] In a third aspect, an embodiment of the present application further provides a computer device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of any one of the above-mentioned video retrieval methods.

[0027] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of any one of the above-mentioned video retrieval methods.

[0028] The beneficial effects of this application are as follows:

[0029] The embodiment of the present application provides a method, device, equipment and storage medium for video retrieval, the method comprising: extracting features of the image features of the video to be retrieved, obtaining a first quantitative feature, and then obtaining a second candidate video with a high similarity to the category of the video to be retrieved based on the first quantitative feature, and finally outputting the second candidate video with a high content similarity to the video to be retrieved as the target video. Since the quantization control parameters of the target quantization processing submodel are adjusted according to the texture feature loss value corresponding to each training sample, the target quantization processing submodel learns the sorting ability of the target texture feature submodel, ensuring that the sorting effects of the two submodels tend to be consistent, and avoiding the situation where the target quantization processing submodel has random sorting due to fixed quantization control parameters. Due to the end-to-end model architecture, the target quantization processing submodel trained in the above manner can obtain corresponding quantitative features based on image features, reduce the loss in the process of generating quantitative features, and improve the accuracy of generating quantitative features. In addition, the embodiment of the present application also optimizes the sorting ability of the target quantization processing submodel, which further improves the recall performance of video retrieval.

[0030] Other features and advantages of the present application will be described in the subsequent description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0032] Figure 1a This is an optional schematic diagram of an application scenario in an embodiment of the present application;

[0033] Figure 1b A schematic diagram of the first display interface provided in the embodiment of the present application;

[0034] Figure 1c A schematic diagram of a second display interface provided in an embodiment of the present application;

[0035] Figure 1dA schematic diagram of the architecture of a target video retrieval model provided in an embodiment of the present application;

[0036] Figure 1e A schematic diagram of the architecture of a quantization processing model used in related technologies;

[0037] Figure 2a A schematic diagram of a process for training a target video retrieval model provided in an embodiment of the present application;

[0038] Figure 2b A schematic diagram of a process for mining multiple sample triples provided in an embodiment of the present application;

[0039] Figure 2c A schematic diagram of a first process for generating a quantized feature loss value provided in an embodiment of the present application;

[0040] Figure 2d A schematic diagram of a second process for generating a quantized feature loss value provided in an embodiment of the present application;

[0041] Figure 3a A schematic diagram of the process of establishing an index table and a mapping table provided in an embodiment of the present application;

[0042] Figure 3b A logical schematic diagram of establishing an index table and a mapping table provided in an embodiment of the present application;

[0043] Figure 4a A schematic diagram of a flow chart of a video retrieval method provided in an embodiment of the present application;

[0044] Figure 4b A logical schematic diagram of a specific embodiment of the application video retrieval method provided in the embodiment of the present application;

[0045] Figure 5 A schematic diagram of the structure of a video retrieval device provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application;

[0047] Figure 7 Schematic diagram of the structure of a computing device in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0049] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0050] The embodiments of the present application relate to the field of artificial intelligence (AI) and are designed based on machine learning (ML) and computer vision (CV) technologies.

[0051] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It studies the design principles and implementation methods of various machines, attempts to understand the essence of intelligence, and produces a new intelligent machine that can respond in a similar way to human intelligence, so that the machine has the functions of perception, reasoning and decision-making.

[0052] Artificial intelligence is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation interaction systems, mechatronics and other technologies. The software technology of artificial intelligence mainly includes computer vision technology, speech processing technology, natural language processing technology, machine learning / deep learning, automatic driving, smart transportation and other major directions. With the development and progress of artificial intelligence, artificial intelligence has been able to carry out research and application in many fields, such as common smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, smart wearable devices, unmanned driving, automatic driving, drones, robots, smart medical care, Internet of Vehicles, automatic driving, smart transportation and other fields. It is believed that with the further development of future technology, artificial intelligence will be applied in more fields and play an increasingly important role. The solution provided in the embodiment of this application involves technologies such as deep learning and augmented reality of artificial intelligence, which are further explained by the following embodiments.

[0053] Machine learning is an interdisciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers can acquire new knowledge or skills and reorganize existing knowledge structures by simulating human learning behaviors, so that computers can continuously improve their performance.

[0054] Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. The core of machine learning is deep learning, a technology that implements machine learning. Machine learning usually includes deep learning, reinforcement learning, transfer learning, inductive learning, artificial neural networks, and self-learning. Deep learning includes convolutional neural networks (CNN), deep belief networks, recursive neural networks, autoencoders, generative adversarial networks, and other technologies.

[0055] Computer vision is a comprehensive subject that integrates computer science, signal processing, physics, applied mathematics, statistics, neurophysiology and other disciplines. It is also an important and challenging research direction in the field of science. Computer vision is a subject that studies how to make machines "see". To put it more specifically, this subject refers to the use of various imaging systems such as cameras and computers to replace human visual organs, and to perform machine vision processing such as target recognition, tracking and measurement, and through further graphics processing, the collected images are processed into images that are more suitable for human observation or transmission to instrument detection.

[0056] As a scientific discipline, computer vision attempts to enable computers to observe and understand the world through visual organs like humans by studying related theories and technologies, and to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, autonomous driving, smart transportation and other technologies. In addition, computer vision technology also includes common biometric recognition technologies such as face recognition and fingerprint recognition.

[0057] The following is a brief introduction to the design concept of the embodiment of the present application:

[0058] In related technologies, quantitative features are usually used as index tags of videos to retrieve corresponding videos. Generally, any of the following methods is used to obtain the corresponding quantitative features:

[0059] Method 1: Obtain corresponding quantitative features based on the k-means clustering algorithm. However, when clustering large-scale sample data, in order to ensure the accuracy of index retrieval, a large amount of resources are required to obtain enough quantitative features;

[0060] Method 2: Obtain the corresponding quantitative features based on product quantization (PQ). However, the quantitative features obtained by this method will reduce the accuracy of the quantitative features due to the loss in the generation process, thus affecting the recall performance of video retrieval.

[0061] Method three obtains the corresponding quantitative features based on a deep learning neural network. However, this neural network first extracts the embedding features of the video image, and then performs feature extraction processing on the embedding features to obtain the corresponding quantitative features. This will reduce the accuracy of the generation of quantitative features due to losses in the generation process, thereby affecting the recall performance of video retrieval.

[0062] In view of this, the embodiments of the present application provide a method, apparatus, device and storage medium for video retrieval to solve the problems of low quantization efficiency and low accuracy. The embodiments of the present application can be applied to various video retrieval scenarios. For example, in a video infringement scenario, the video retrieval method provided by the embodiments of the present application is used to recall a batch of videos with high content similarity to the video to be retrieved, and the recalled videos are determined to be infringing videos.

[0063] The method includes: extracting features from the image features of the video to be retrieved to obtain a first quantitative feature, and then based on the first quantitative feature, obtaining a second candidate video with a high similarity to the category of the video to be retrieved, and finally outputting the second candidate video with a high content similarity to the video to be retrieved as the target video. Since the quantization control parameters of the target quantization processing submodel are adjusted according to the texture feature loss value corresponding to each training sample, the target quantization processing submodel learns the sorting ability of the target texture feature submodel, ensuring that the sorting effects of the two submodels tend to be consistent, and avoiding the situation where the target quantization processing submodel has random sorting due to fixed quantization control parameters. Due to the end-to-end model architecture, the target quantization processing submodel trained in the above manner can obtain corresponding quantitative features based on image features, reducing the loss in the process of generating quantitative features and improving the accuracy of generating quantitative features. In addition, the embodiment of the present application also optimizes the sorting ability of the target quantization processing submodel, further improving the recall performance of video retrieval.

[0064] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0065] See also Figure 1a and Figure 1b The schematic diagram shown includes two physical terminal devices 110 and a server 130 in the application scenario of the embodiment of the present application.

[0066] The target object can log in to the video retrieval client through the physical terminal device 110, and present the retrieval interface on the display screen 120 of the physical terminal device 110; thereafter, the target object inputs the image of the video to be retrieved in the retrieval interface, so that the target video retrieval model running on the target server 130 obtains the target video with a high content similarity with the video to be retrieved from the huge video library connected to the backend port based on the image of the video to be retrieved; after receiving all the target videos returned by the target server 130, the physical terminal device 110 presents each target video on the display interface of the display screen 120, and the user can also view the video details of the selected target video through gesture operations such as clicking on the page, and the progress bar will also mark the similar or repeated segments of the target video and the video to be retrieved.

[0067] like Figure 1b The display interface shown presents excerpts from a TV series. For the played clips, the corresponding progress bar color is white; for the unplayed clips, the corresponding progress bar color is black; and for similar clips or repeated clips, the corresponding progress bar color is gray. In this way, users can roughly estimate the similarity between the target video and the video to be retrieved by the color of the progress bar, which is convenient for users to make infringement judgments on video creation.

[0068] like Figure 1c The display interface shown presents excerpts from a TV series. For the played segments, the corresponding progress bar is white; for the unplayed segments, the corresponding progress bar is black; and for similar or repeated segments, triangular markers or other shaped markers are used on the progress bar to mark the starting and ending points of these segments. In this way, the user can jump directly to the corresponding plot by clicking the marker. Similarly, the user can also roughly estimate the similarity between the target video and the video to be retrieved by the number of markers on the progress bar.

[0069] In the embodiment of the present application, the physical terminal device 110 is an electronic device used by a user, and the electronic device may be a personal computer, a mobile phone, a tablet computer, a laptop computer, an e-book reader, a smart home device, or other computer device.

[0070] Each physical terminal device 110 communicates with the target server 130 through a communication network. In an optional implementation, the communication network is a wired network or a wireless network, so each physical terminal device 110 can directly or indirectly establish a communication connection with the target server 130 through a wired network or a wireless network, and this application does not limit this.

[0071] The target server 130 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), big data and artificial intelligence platforms, etc. This application does not impose any restrictions on this.

[0072] The target video retrieval model is deployed on the target server 130, such as Figure 1d As shown, the target video retrieval model includes a target image processing sub-model, a target texture processing sub-model and a target quantization processing sub-model.

[0073] The target image processing sub-model and the target texture processing sub-model are both deep learning network models built using the ResNet_101 network architecture, and are pre-trained based on imageNet. ImageNet is a large-scale open source dataset for general object recognition. ImageNet contains a large amount of pre-annotated image data, and imageNet contains about 1,000 types of image data. Therefore, the deep learning network model obtained based on imageNet pre-training has better model parameter stability and overall model versatility.

[0074] In addition, you can also use network architectures other than ResNet_101 to build deep learning network models, and pre-train deep learning network models based on other large-scale data sets, such as deep learning network models obtained based on openimage pre-training.

[0075] The target quantization processing sub-model is used to perform binary quantization processing on complex high-dimensional image features, compressing high-dimensional image features into binary codes of specified bits (i.e., quantized features). When performing video retrieval, the corresponding target video is recalled using the quantized features as indexes, which greatly reduces the calculation time and complexity, is more conducive to calculation, and is very beneficial for the retrieval of massive data.

[0076] In addition, for binary coding, the value of each bit is 0 or 1, such as compressing the 128-dimensional image feature into a 4-bit binary code 0100.

[0077] and Figure 1e Different from the traditional quantization processing model shown in the figure, the target texture processing sub-model and the target quantization processing sub-model in the embodiment of the present application are two sub-models placed in parallel. The advantage of such deployment is that, in the training stage, the quantization control parameters of the target quantization processing sub-model will be adjusted according to the texture feature loss value corresponding to each training sample, so that the target quantization processing sub-model learns the sorting ability of the target texture feature sub-model, ensuring that the sorting effects of the two sub-models tend to be consistent, avoiding the situation where the target quantization processing sub-model has random sorting due to fixed quantization control parameters;

[0078] In the application stage, compared with Figure 1e Compared with the non-end-to-end model architecture shown, the embodiment of the present application adopts an end-to-end model architecture, which can obtain corresponding quantitative features based on image features, reduce the loss in the process of generating quantitative features, and improve the accuracy of generating quantitative features. In addition, the embodiment of the present application also optimizes the sorting capability of the target quantitative processing sub-model, which further improves the recall performance of video retrieval.

[0079] Moreover, the target video retrieval model built based on artificial intelligence technology in the embodiment of the present application has better processing speed and recall performance and consumes less resources when processing large-scale retrieval videos compared to the traditional k-means clustering algorithm.

[0080] In the training process of the target video retrieval model, it is specifically divided into a pre-training stage and a fine-tuning joint learning stage. However, the training data used in the two training stages are the same. The difference is that the network parameters that need to be learned and the loss values ​​generated in the two training stages are different.

[0081] Among them, in the pre-training stage, the image processing texture feature loss value is used to adjust the parameters of the image processing sub-model to be trained and the texture processing sub-model to be trained, so as to obtain a candidate image processing sub-model and a candidate texture processing sub-model; in the fine-tuning joint learning stage, the texture feature loss value is used to adjust the parameters of the candidate texture processing sub-model, and the quantization feature loss value is used to adjust the parameters of the candidate image processing sub-model and the quantization processing sub-model to be trained, so as to obtain a target image processing sub-model, a target texture processing sub-model, and a target quantization processing sub-model.

[0082] For easier understanding, see Figure 2a The flowchart shown introduces the training process of the target video retrieval model.

[0083] S201: Obtain multiple sample triplets, each sample triplet comprising a sample video, and a positive label and a negative label associated with the sample video.

[0084] Compared with the traditional quantization processing method, the embodiment of the present application uses labeled training data, so that the quantization processing sub-model to be trained can learn positive labels and negative labels at the same time, thereby improving the recall effect of the target quantization processing sub-model.

[0085] Among them, the positive label refers to the sample video with a high content similarity with the sample video, while the negative label refers to the sample video with only a small amount of content identical or similar to the sample video.

[0086] See also Figure 2b As shown in the flowchart, the embodiment of the present application can obtain multiple sample triples by performing the following operations:

[0087] S2011: Obtain a similar sample set including a plurality of similar sample pairs, each similar sample pair including a sample video and a positive label associated with the sample video.

[0088] S2012: Input each similar sample pair in the similar sample set into the image processing sub-model to be trained and the texture processing sub-model to be trained in sequence to obtain a corresponding texture feature group.

[0089] The image processing sub-model to be trained and the texture processing sub-model to be trained are both deep learning network models obtained by pre-training on the large-scale general object recognition open source dataset ImageNet. Each similar sample pair is input into the above two sub-models in turn to obtain the texture feature group corresponding to each sample similar pair. Each texture feature group includes the texture features of the sample video and the texture features of the positive label.

[0090] S2013: Read a sample video c of a similar sample pair;

[0091] S2014: For each other similar sample pair in the similar sample set, perform the following operations respectively: based on the texture feature of the sample video c and the texture feature of any other sample video in another similar sample pair, obtain the corresponding texture feature distance.

[0092] In an optional implementation, the Euclidean distance between two texture features is used as the corresponding texture feature distance. The smaller the value of the Euclidean distance, the higher the content similarity of the two sample videos; conversely, the larger the value of the Euclidean distance, the lower the content similarity of the two sample videos.

[0093] S2015: Arrange other sample videos in order of distance of texture features.

[0094] S2016: After eliminating the top k% of other sample videos, determine the top m other sample videos as negative labels of the sample videos.

[0095] Arrange the other sample videos in order of texture feature distance from near to far. According to the previous introduction to texture feature distance, the first k% of other sample videos have a very high content similarity with the sample video. In the embodiment of the present application, the negative label to be mined is the sample video that has only a small amount of content that is the same or similar to the sample video. Obviously, the first k% of other sample videos do not meet the definition of negative labels and are removed as interference noise. Among them, k is a controllable value. The larger the interference noise, the larger the corresponding k value.

[0096] The other sample videos that are ranked extremely low do not meet the definition of negative labels because they have almost no identical or similar content to the sample video. Therefore, the embodiment of the present application determines the other sample videos ranked in the top m as negative labels for the sample video.

[0097] For example, assuming that the similar sample pair is (sample video 1, sample video 2), Table 1 shows the texture feature distances between other sample videos and sample video 1, and finally the following multiple sample triplets are screened: (sample video 1, sample video 2, other sample video 7), (sample video 1, sample video 2, other sample video 4), (sample video 1, sample video 2, other sample video 5)

[0098] Table 1

[0099] Other sample video names Texture feature distance Other sample videos 3 0.5 Other sample videos 6 0.55 Other sample videos 7 0.8 Other sample videos 4 0.83 Other sample videos 5 0.9 Other sample videos 8 1.2 Other sample videos 9 1.5

[0100] S2017: Determine whether all similar sample pairs in the similar sample set have been read. If so, execute step 2018; otherwise, return to step 2013.

[0101] S2018: Determine whether all similar sample sets have been read. If so, output all sample triplets; otherwise, return to step 2011.

[0102] S202: Read a sample triplet d, and use the sample triplet d as training data, and input it into the image processing sub-model to be trained and the texture processing sub-model to be processed in sequence to obtain a corresponding first texture set.

[0103] S203: Generate corresponding image processing texture feature loss values ​​based on the multiple first sample texture features included in the first texture set, and adjust parameters of the image processing sub-model to be trained and the texture processing sub-model to be trained based on the image processing texture feature loss values.

[0104] Formula 1 is used to generate the corresponding image processing texture feature loss value; then based on the image processing texture feature loss value, the stochastic gradient descent (SGD) method is used to adjust the parameters of the image processing sub-model to be trained and the texture processing sub-model to be trained.

[0105] Among them, L in formula 1 em is the texture feature loss value of image processing, x a is the first sample texture feature of the sample video, x p is the first sample texture feature of the positive label, x n Then it is the texture feature of the first sample of the negative label, ‖x a -x p ‖ represents the texture feature distance between positive sample pairs, ‖x a -x n ‖ represents the texture feature distance between negative sample pairs, and margin_em represents the texture control parameter.

[0106] L em =max(‖x a -x p ‖-‖x a -x n ‖+margin_em) Formula 1;

[0107] S204: Determine whether the image processing texture feature loss value is higher than a preset image processing texture feature loss threshold value. If so, return to step 202; otherwise, execute step 205.

[0108] S205: Stop iterating the training sub-model, and output the candidate image processing sub-model and the candidate texture processing sub-model obtained in the last round of iteration.

[0109] S206: Read a sample triplet e, and use the sample triplet e as training data, and input it into the candidate image processing sub-model, the candidate texture processing sub-model and the quantization processing sub-model to be trained in sequence to obtain the corresponding second texture set and quantization feature group.

[0110] S207: Generate corresponding texture feature loss values ​​based on the multiple second sample texture features included in the second texture set, and adjust parameters of the candidate texture processing sub-model based on the texture feature loss values.

[0111] Formula 2 is used to generate the corresponding texture feature loss value; then based on the texture feature loss value, the SGD method is used to adjust the parameters of the candidate texture processing sub-model.

[0112] Among them, L in Formula 2em ′ is the texture feature loss value of image processing, x a ′ is the second sample texture feature of the sample video, x p ′ is the second sample texture feature of the positive label, x n ′ is the texture feature of the first sample of the negative label, ‖x a ′-x p ′‖ represents the texture feature distance between the positive sample pairs, ‖x a ′-x n ′‖ represents the texture feature distance between negative sample pairs, and margin_em represents the texture control parameter.

[0113] L em ′=max(‖x a ′-x p ′‖-‖x a ′-x n ′‖+margin_em) Formula 2;

[0114] S208: Based on the multiple sample quantization features and texture feature loss values ​​included in the quantization feature group, a corresponding quantization feature loss value is generated, and based on the quantization feature loss value, parameters of the candidate image processing sub-model and the quantization processing sub-model to be trained are adjusted.

[0115] Optional, see Figure 2c The flowchart shown introduces the first method of generating quantized feature loss values.

[0116] S2081: Based on the texture feature loss value, adjust the quantization control parameters of the quantization processing sub-model to be trained.

[0117] Formula 3 is used to calculate the quantization control parameter of the i-th sample triplet. Specifically, margin_i represents the quantization control parameter of the i-th sample triplet, margin0 is the preset Hamming distance, Mem is the ratio between the texture feature distance and the Hamming distance, and L em _i is the texture feature loss value of the i-th sample triplet.

[0118] margin_i=margin0*L em _i / Mem Formula 3;

[0119] S2082: Based on multiple sample quantization features and quantization control parameters included in a quantization feature group, respectively determine the training sample loss value and the symbol quantization loss value of the quantization processing sub-model to be trained.

[0120] Among them, Formula 4 is the calculation formula for the loss value of the training sample. triplet is the loss value of the training sample, is the sample quantization feature of the sample video, is the sample quantization feature of the positive label, is the sample quantization feature of the negative label, Characterizes the quantitative feature distance between positive sample pairs, It represents the quantitative feature distance between negative sample pairs, and margin_i represents the quantitative control parameter of the i-th sample triplet.

[0121]

[0122] First, use Formula 5 to perform symbol quantization on each bit in the sample quantization feature to obtain the symbol quantization feature; then use Formula 6 to generate the corresponding symbol quantization loss value based on the sample quantization feature and the symbol quantization feature.

[0123] Among them, L coding is the sign quantization loss value, ui represents the i-th bit in the sample quantization feature, bi represents the i-th bit of the sign quantization feature, if ui is a negative number, then the value of bi is -1, otherwise the value of bi is 1.

[0124]

[0125]

[0126] S2083: Generate corresponding quantization feature loss values ​​based on the training sample loss values ​​and symbol quantization loss values ​​of the quantization processing sub-model to be trained.

[0127] Formula 7 is used to generate the corresponding quantization feature loss value. Then, based on the quantization feature loss value, the SGD method is used to adjust the parameters of the candidate image processing sub-model and the quantization processing sub-model to be trained.

[0128] Where, L in Formula 7 q To quantify the feature loss value, L triplet is the training sample loss value of the quantization processing sub-model to be trained, w 21 is the weight assigned to the loss value of the training sample, L coding is the symbolic quantization loss value of the quantization processing sub-model to be trained, w 22 is the weight assigned to the symbolic quantization loss value.

[0129] L q =w 21 L triplet +w 22 L coding Formula 7;

[0130] Optional, see Figure 2dAs shown in the flow chart, the embodiment of the present application also provides a second method for generating a quantized feature loss value.

[0131] S2081′: Based on multiple sample quantization features included in a quantization feature group, respectively determine the training sample loss value and the symbol quantization loss value of the quantization processing sub-model to be trained.

[0132] Substitute multiple sample quantization features into Formula 4 to generate the corresponding training sample loss value, but at this time the value of margin_i in Formula 4 is the same as the value of margin_em.

[0133] Substitute multiple sample quantization features into formulas 5 to 6 in sequence to obtain the corresponding symbol quantization loss values. The relevant formulas have been introduced in the previous article and will not be repeated here.

[0134] S2082′: Generate a corresponding quantization feature loss value based on the training sample loss value, symbol quantization loss value and texture feature loss value of the quantization processing sub-model to be trained.

[0135] Formula 8 is used to generate the corresponding quantization feature loss value. Where L q To quantify the feature loss value, L triplet is the training sample loss value of the quantization processing sub-model to be trained, w 21 , L em ′ are the weights of the loss value of the training samples, L coding is the symbolic quantization loss value of the quantization processing sub-model to be trained, w 22 is the weight assigned to the symbolic quantization loss value.

[0136] L q =w 21 (L em ′*L triplet )+w 22 L coding Formula 8;

[0137] S209: Determine whether the texture feature loss value and the quantization feature loss value are both higher than a preset image processing texture feature loss threshold value. If so, return to step 206; otherwise, execute step 210.

[0138] S210: Stop iterative training of the sub-models, and output the target image processing sub-model, target texture processing sub-model, and target quantization processing sub-model obtained in the last round of iterations.

[0139] Next, see Figure 3a The flowchart shown is Figure 3b The logical schematic diagram shown uses the trained target video retrieval model to establish the index table and mapping table of the video database.

[0140] S301: Read a first candidate video s;

[0141] S302: Input the first candidate video s into the target video retrieval model to obtain the corresponding initial quantitative features and second texture features;

[0142] S303: adding the second texture feature to the mapping table, and determining the quantized feature distances between the initial quantized feature and each second quantized feature recorded in the index table;

[0143] S304: adding the first candidate video s to the second quantitative feature corresponding to the minimum quantitative feature distance;

[0144] S305: Determine whether all first candidate videos in the video database have been read, if so, execute step 306; otherwise, return to step 301;

[0145] S306: Output the mapping table and index table obtained in the last round of iteration.

[0146] Wherein, when executing step 303, if the index table is empty, the initial quantization feature of the first candidate video s is added to the index table as the second quantization feature. The index table is shown as Lindex:[q1:[img1,img2,img6],q2:[img3],q3:[img4]], and the table includes multiple second quantization features, each quantization feature corresponds to at least one first candidate video, so each second quantization feature represents the video category to which the corresponding at least one first candidate video belongs; the mapping table is shown as T:[[img1,embedding1],[img2,embedding2],……,[img6,embedding6]], and the table includes multiple first candidate videos and corresponding second texture features.

[0147] In addition, for the first candidate video newly added to the video database, the following operations can also be performed: Figure 3a According to the process shown, corresponding index relationships and mapping relationships are established.

[0148] Next, see Figure 4a The flow chart shown applies the video retrieval method provided in the embodiment of the present application to the trained target video retrieval model.

[0149] S401: Using the trained target image processing sub-model of the target video retrieval model, extract features from the video to be retrieved to obtain corresponding image features.

[0150] When executing step 401, the complete video of the video to be retrieved can be input into the target image processing sub-model to obtain a corresponding image feature; or key frames can be first extracted from the video to be retrieved, and then the obtained multiple key frames are input into the target image processing sub-model to obtain corresponding multiple image features.

[0151] S402: Using the target quantization processing sub-model of the target video retrieval model, perform feature extraction on the image features to obtain the corresponding first quantization features, and based on the first quantization features, screen out from each first candidate video at least one second candidate video that has a category similarity with the video to be retrieved and meets the set category similarity requirements; wherein the quantization control parameters of the target quantization processing sub-model are adjusted during the training process based on the texture feature loss value corresponding to each training sample, and the texture feature loss value is determined based on the texture control parameters preset for the texture processing sub-model to be trained during the parameter adjustment process of the texture processing sub-model to be trained.

[0152] The quantization control parameters of the target quantization processing submodel are adjusted according to the texture feature loss value corresponding to each training sample, so that the target quantization processing submodel learns the sorting ability of the target texture feature submodel, ensuring that the sorting effects of the two submodels tend to be consistent, and avoiding the situation where the target quantization processing submodel has random sorting due to fixed quantization control parameters. The end-to-end model architecture enables the target quantization processing submodel to obtain corresponding quantization features based on image features, reducing the loss in the process of generating quantization features and improving the accuracy of generating quantization features. In addition, the embodiment of the present application also optimizes the sorting ability of the target quantization processing submodel, further improving the recall performance of video retrieval.

[0153] According to the foregoing introduction, the index table contains multiple second quantitative features, each quantitative feature corresponds to at least one first candidate video. Therefore, when executing step 402, the quantitative feature distance between the first quantitative feature and the second quantitative feature of each first candidate video is determined respectively, and then the first candidate video whose quantitative feature distance is lower than the preset quantitative feature distance threshold value is determined as the second candidate video.

[0154] S403: Based on the content similarity between the video to be retrieved and at least one second candidate video, the second candidate video whose content similarity meets the set content similarity requirement is output as the corresponding target video.

[0155] In an embodiment of the present application, either the complete video of the video to be retrieved can be used as the model input, or the obtained multiple key frames can be used as the model input. Therefore, for different model inputs, the following methods of obtaining the target video are provided.

[0156] Method 1: Applicable to both of the above two model inputs, it is to filter and obtain the target video according to the texture feature distance.

[0157] Optionally, a target texture processing sub-model is used to extract features from image features to obtain corresponding first texture features; then, for at least one second candidate video, the following operations are performed: determine the texture feature distance between the first texture feature and the second texture feature of a second candidate video; if the texture feature distance is lower than a preset texture feature distance threshold, determine that the content similarity between the video to be retrieved and the second candidate video meets the set content similarity requirement, and determine the second candidate video as the target video output; wherein the second texture feature represents the texture information of the second candidate video. In the embodiment of the present application, a variety of distance calculation methods such as Euclidean distance and Hamming distance can be used to calculate the quantitative feature distance and texture feature distance. Regardless of the distance calculation method used, if the value of the distance is small, it indicates that the content similarity of the two videos is high; conversely, if the value of the distance is small, it indicates that the content similarity of the two videos is low, which will not be described in detail later.

[0158] Method 2: Take the complete video as the model input and filter the target video according to the content repetition.

[0159] Optionally, for at least one second candidate video, the following operations are performed respectively:

[0160] The ratio between the total matching duration and the comparison duration is determined as the content repetition between the video to be retrieved and a second candidate video; wherein the total matching duration is obtained based on the matching duration between at least one second candidate video and the video to be retrieved, and the comparison duration is the duration value of the shorter video duration between the video to be retrieved and the second candidate video;

[0161] If the content repetition exceeds the set content repetition threshold, the content similarity between the video to be retrieved and the second candidate video is determined to meet the set content similarity requirement, and the second candidate video is determined to be the target video output.

[0162] For example, assuming that the video length of the video to be retrieved is 30 seconds, and the matching durations between each second candidate video and the video to be retrieved are as shown in Table 2, then method 2 is adopted to obtain the content repetition between the video to be retrieved and each second candidate video, and finally the second candidate videos 1 to 3 are returned to the user as target videos.

[0163] Table 2

[0164] The name of the second candidate video Video Length Match duration Content duplication Second candidate video 1 15s 5s 6 Second candidate video 2 20s 10s 4.5 Second candidate video 3 25s 20s 3.6 Second candidate video 4 60s 35s 3 Second candidate video 5 120s 20s 3

[0165] Method 3: Use multiple key frames as model input and filter the target video according to the content repetition.

[0166] Each key frame corresponds to a first quantitative feature, and each first quantitative feature can recall a second candidate video with the same feature. Therefore, the ratio between the number of identical quantitative features and the comparison duration can be determined as the content repetition between the video to be retrieved and the second candidate video.

[0167] Optionally, for at least one second candidate video, the following operations are performed respectively:

[0168] Determining the number of identical quantitative features between the video to be retrieved and a second candidate video;

[0169] The ratio between the number of identical quantitative features and the comparison duration is determined as the content repetition between the video to be retrieved and the second candidate video; wherein the comparison duration is the duration value of the shorter video duration between the video to be retrieved and the second candidate video;

[0170] If the content repetition exceeds the set content repetition threshold, the content similarity between the video to be retrieved and the second candidate video is determined to meet the set content similarity requirement, and the second candidate video is determined to be the target video output.

[0171] For example, assuming that the video length of the video to be retrieved is 30 seconds, and a total of 10 key frames are extracted. The number of identical quantitative features between each second candidate video and the video to be retrieved is shown in Table 3. Then, method 2 is adopted to obtain the content repetition between the video to be retrieved and each second candidate video, and finally the second candidate videos 1 to 2 are returned to the user as the target video.

[0172] Table 3

[0173] The name of the second candidate video Video Length The number of identical quantitative features Content duplication Second candidate video 1 15s 5 0.33 Second candidate video 2 20s 8 0.4 Second candidate video 3 25s 2 0.08 Second candidate video 4 60s 3 0.1 Second candidate video 5 120s 1 0.03

[0174] For easier understanding, see Figure 4b The logical schematic diagram shown introduces the process of applying the video retrieval method in a specific embodiment.

[0175] The complete video of the video to be retrieved is input into the trained target video retrieval model to obtain the corresponding first texture feature and first quantitative feature; based on the quantitative feature distance between the first quantitative feature and each second quantitative feature in the index table, a plurality of candidate videos with high category similarity to the video to be retrieved are obtained; and based on the texture feature distance between the first texture feature and the second texture feature of each candidate video recalled in the previous round, the candidate videos ranked in the top N are returned to the user as target videos with high content similarity to the video to be retrieved.

[0176] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides a video retrieval device, such as Figure 5 As shown, the apparatus 500 may include:

[0177] An image processing unit 501 is used to extract features of the video to be retrieved by using a target image processing sub-model of a trained target video retrieval model to obtain corresponding image features;

[0178] The quantization processing unit 502 is used to use the target quantization processing sub-model of the target video retrieval model to extract features of the image features to obtain corresponding first quantization features, and based on the first quantization features, screen out at least one second candidate video that has a category similarity with the video to be retrieved and meets the set category similarity requirements from each first candidate video; wherein the quantization control parameter of the target quantization processing sub-model is adjusted based on the texture feature loss value corresponding to each training sample during the training process, and the texture feature loss value is determined based on the texture control parameter preset for the texture processing sub-model to be trained during the parameter adjustment process of the texture processing sub-model to be trained;

[0179] The retrieval unit 503 is used to output the second candidate video whose content similarity meets the set content similarity requirement as the corresponding target video based on the content similarity between the video to be retrieved and at least one second candidate video.

[0180] Optionally, the target video retrieval model further includes a target texture processing sub-model, and the retrieval unit 503 is used to:

[0181] Using the target texture processing sub-model, extracting features of the image to obtain corresponding first texture features;

[0182] For at least one second candidate video, the following operations are performed respectively: determining a texture feature distance between a first texture feature and a second texture feature of a second candidate video; if the texture feature distance is lower than a preset texture feature distance threshold, determining whether the content similarity between the video to be retrieved and the second candidate video meets the set content similarity requirement, and determining a second candidate video as a target video output; wherein the second texture feature represents texture information of a corresponding second candidate video.

[0183] Optionally, the retrieval unit 503 is used to:

[0184] For at least one second candidate video, perform the following operations respectively:

[0185] The ratio between the total matching duration and the comparison duration is determined as the content repetition between the video to be retrieved and a second candidate video; wherein the total matching duration is obtained based on the matching duration between at least one second candidate video and the video to be retrieved, and the comparison duration is the duration value of the shorter video duration between the video to be retrieved and the second candidate video;

[0186] If the content repetition exceeds the set content repetition threshold, the content similarity between the video to be retrieved and a second candidate video is determined to meet the set content similarity requirement, and the second candidate video is determined to be output as the target video.

[0187] Optionally, the retrieval unit 503 is used to:

[0188] For at least one second candidate video, perform the following operations respectively:

[0189] Determining the number of identical quantitative features between the video to be retrieved and a second candidate video;

[0190] The ratio between the number of identical quantitative features and the comparison duration is determined as the content repetition between the video to be retrieved and a second candidate video; wherein the comparison duration is the duration value of the shorter video duration between the video to be retrieved and the second candidate video;

[0191] If the content repetition exceeds the set content repetition threshold, the content similarity between the video to be retrieved and a second candidate video is determined to meet the set content similarity requirement, and the second candidate video is determined to be output as the target video.

[0192] Optionally, the quantization processing unit 502 is used for:

[0193] Determine respectively the quantitative feature distance between the first quantitative feature and the second quantitative feature of each first candidate video;

[0194] The first candidate video whose quantitative feature distance is lower than a preset quantitative feature distance threshold is determined as the second candidate video; wherein each second quantitative feature represents the video category to which the corresponding at least one first candidate video belongs.

[0195] Optionally, the device 500 further includes a model training unit 504, and the model training unit 504 obtains a trained target video retrieval model by executing the following method:

[0196] Obtain multiple sample triplets, each sample triplet comprising a sample video, and a positive label and a negative label associated with the sample video;

[0197] Each sample triplet is used as training data and sequentially input into the image processing sub-model to be trained and the texture processing sub-model to be processed to obtain a corresponding first texture set; wherein, each time a first texture set is obtained, a corresponding image processing texture feature loss value is generated based on a plurality of first sample texture features contained in the first texture set, and based on the image processing texture feature loss value, parameters of the image processing sub-model to be trained and the texture processing sub-model to be trained are adjusted until the image processing texture feature loss value is not higher than a preset image processing texture feature loss threshold value, thereby obtaining a candidate image processing sub-model and a candidate texture processing sub-model;

[0198] Each sample triplet is used as training data and input into the candidate image processing sub-model, the candidate texture processing sub-model and the quantization processing sub-model to be trained in turn to obtain the corresponding second texture set and quantization feature group; wherein, each time a second texture set is obtained, a corresponding texture feature loss value is generated based on a plurality of second sample texture features contained in a second texture set, and based on the texture feature loss value, the parameter of the candidate texture processing sub-model is adjusted; and each time a quantization feature group is obtained, a corresponding quantization feature loss value is generated based on a plurality of sample quantization features and texture feature loss value contained in a quantization feature group, and based on the quantization feature loss value, the parameter of the candidate image processing sub-model and the quantization processing sub-model to be trained is adjusted, until the texture feature loss value and the quantization feature loss value are both no higher than the preset feature loss threshold value, thereby obtaining the target image processing sub-model, the target texture processing sub-model and the target quantization processing sub-model.

[0199] Optionally, the model training unit 504 is used to:

[0200] Based on the texture feature loss value, adjusting the quantization control parameters of the quantization processing sub-model to be trained;

[0201] Based on a plurality of sample quantization features and quantization control parameters included in a quantization feature group, respectively determining a training sample loss value and a symbol quantization loss value of a quantization processing sub-model to be trained;

[0202] Based on the training sample loss value and the symbol quantization loss value of the quantization processing sub-model to be trained, the corresponding quantization feature loss value is generated.

[0203] Optionally, the model training unit 504 is used to:

[0204] Based on a plurality of sample quantization features included in a quantization feature group, respectively determining a training sample loss value and a symbol quantization loss value of a quantization processing sub-model to be trained;

[0205] Based on the training sample loss value, symbol quantization loss value and texture feature loss value of the quantization processing sub-model to be trained, a corresponding quantization feature loss value is generated.

[0206] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0207] After introducing the method and apparatus for accessing the service platform according to the exemplary embodiment of the present application, next, a computer device according to another exemplary embodiment of the present application is introduced.

[0208] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0209] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides a computer device, see Figure 6 As shown, the computer device 600 may include at least a processor 601 and a memory 602. The memory 602 stores program codes, and when the program codes are executed by the processor 601, the processor 601 executes the steps of any one of the above-mentioned video retrieval methods.

[0210] In some possible implementations, the computing device according to the present application may include at least one processor and at least one memory. The memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the video retrieval method according to various exemplary implementations of the present application described above in this specification. For example, the processor may execute the steps shown in FIG4.

[0211] Refer to the following Figure 7 hereinafter, a computing device 700 according to this embodiment of the present application is described. Figure 7 The computing device 700 is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0212] like Figure 7 As shown, the computing device 700 is in the form of a general computing device. The components of the computing device 700 may include but are not limited to: at least one processing unit 701, at least one storage unit 702, and a bus 703 connecting different system components (including the storage unit 702 and the processing unit 701).

[0213] Bus 703 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0214] The storage unit 702 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 7021 and / or a cache memory unit 7022 , and may further include a read-only memory (ROM) 7023 .

[0215] The storage unit 702 may also include a program / utility 7025 having a set (at least one) of program modules 7024, such program modules 7024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0216] The computing device 700 may also communicate with one or more external devices 704 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the computing device 700, and / or communicate with any device that enables the computing device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 705. In addition, the computing device 700 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 706. As shown, the network adapter 706 communicates with other modules for the computing device 700 via a bus 703. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computing device 700, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0217] Based on the same inventive concept as the above-mentioned method embodiments, various aspects of the video retrieval method provided in the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the video retrieval method according to various exemplary embodiments of the present application described above in this specification. For example, the electronic device can execute the steps shown in Figure 4.

[0218] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0219] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0220] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A video retrieval method, characterized in that: include: The target image processing sub-model of the trained target video retrieval model is used to extract features of the video to be retrieved and obtain corresponding image features; The target quantization processing submodel of the target video retrieval model is used to extract features of the image features to obtain corresponding first quantization features, and based on the first quantization features, at least one second candidate video that has a category similarity with the video to be retrieved and meets the set category similarity requirements is screened out from each first candidate video; wherein the quantization control parameter of the target quantization processing submodel is adjusted based on the texture feature loss value corresponding to each training sample during the training process, and the texture feature loss value is determined based on the texture control parameter preset for the texture processing submodel to be trained during the parameter adjustment process of the texture processing submodel to be trained; Based on the content similarity between the video to be retrieved and the at least one second candidate video, outputting the second candidate video whose content similarity meets the set content similarity requirement as the corresponding target video; Wherein, the trained target video retrieval model is obtained by executing the following method: performing multiple rounds of iterative parameter adjustment based on the first texture sets of each of the multiple sample triplets output by the image processing submodel to be trained and the texture processing submodel to be processed, so as to obtain a candidate image processing submodel and a candidate texture processing submodel; performing multiple rounds of iterative parameter adjustment based on the second texture sets and quantization feature groups of each of the multiple sample triplets output by the candidate image processing submodel, the candidate texture processing submodel and the quantization processing submodel to be trained, so as to obtain the target image processing submodel, the target texture processing submodel, and the target quantization processing submodel.

2. The method according to claim 1, characterized in that The target video retrieval model also includes a target texture processing sub-model; The step of outputting, based on the content similarity between the video to be retrieved and the at least one second candidate video, a second candidate video whose content similarity meets a set content similarity requirement as a corresponding target video comprises: Using the target texture processing sub-model, extracting features from the image features to obtain corresponding first texture features; For the at least one second candidate video, the following operations are performed respectively: determining the texture feature distance between the first texture feature and the second texture feature of a second candidate video; if the texture feature distance is lower than a preset texture feature distance threshold, determining that the content similarity between the video to be retrieved and the second candidate video meets the set content similarity requirement, and determining the second candidate video as the target video output; wherein the second texture feature represents the texture information of the corresponding second candidate video.

3. The method according to claim 1, characterized in that The step of outputting, based on the content similarity between the video to be retrieved and the at least one second candidate video, a second candidate video whose content similarity meets a set content similarity requirement as a corresponding target video comprises: For the at least one second candidate video, the following operations are performed respectively: The ratio between the total matching duration and the comparison duration is determined as the content repetition between the video to be retrieved and a second candidate video; wherein the total matching duration is obtained based on the matching duration between each of the at least one second candidate video and the video to be retrieved, and the comparison duration is the duration value of the shorter video duration between the video to be retrieved and the second candidate video; If the content repetition exceeds the set content repetition threshold, the content similarity between the video to be retrieved and the second candidate video is determined to meet the set content similarity requirement, and the second candidate video is determined to be the target video output.

4. The method according to claim 1, characterized in that The step of outputting, based on the content similarity between the video to be retrieved and the at least one second candidate video, a second candidate video whose content similarity meets a set content similarity requirement as a corresponding target video comprises: For the at least one second candidate video, the following operations are performed respectively: Determining the number of identical quantitative features between the video to be retrieved and a second candidate video; The ratio between the number of the same quantitative features and the comparison duration is determined as the content repetition between the video to be retrieved and a second candidate video; wherein the comparison duration is the duration value of the shorter video duration between the video to be retrieved and the second candidate video; If the content repetition exceeds the set content repetition threshold, the content similarity between the video to be retrieved and the second candidate video is determined to meet the set content similarity requirement, and the second candidate video is determined to be the target video output.

5. The method according to any one of claims 1 to 4, characterized in that: The step of selecting, based on the first quantitative feature, from each of the first candidate videos at least one second candidate video that has a category similarity with the video to be retrieved and meets a set category similarity requirement comprises: respectively determining the quantitative feature distances between the first quantitative feature and the second quantitative features of each of the first candidate videos; The first candidate video whose quantitative feature distance is lower than a preset quantitative feature distance threshold is determined as the second candidate video; wherein each second quantitative feature represents the video category to which the corresponding at least one first candidate video belongs.

6. The method according to any one of claims 1 to 4, characterized in that: The trained target video retrieval model is obtained by executing the following method: Obtaining a plurality of sample triplets, each sample triplet comprising a sample video, and a positive label and a negative label associated with the sample video; Each sample triplet is used as training data and sequentially input into the image processing sub-model to be trained and the texture processing sub-model to be processed to obtain a corresponding first texture set; wherein, each time a first texture set is obtained, a corresponding image processing texture feature loss value is generated based on a plurality of first sample texture features contained in the first texture set, and based on the image processing texture feature loss value, parameters of the image processing sub-model to be trained and the texture processing sub-model to be trained are adjusted until the image processing texture feature loss value is not higher than a preset image processing texture feature loss threshold value, thereby obtaining a candidate image processing sub-model and a candidate texture processing sub-model; Each sample triplet is used as training data and is input into the candidate image processing sub-model, the candidate texture processing sub-model and the quantization processing sub-model to be trained in turn to obtain a corresponding second texture set and a quantization feature group; wherein, each time a second texture set is obtained, a corresponding texture feature loss value is generated based on a plurality of second sample texture features contained in the second texture set, and based on the texture feature loss value, the parameter of the candidate texture processing sub-model is adjusted; and each time a quantization feature group is obtained, a corresponding quantization feature loss value is generated based on a plurality of sample quantization features contained in the quantization feature group and the texture feature loss value, and based on the quantization feature loss value, the parameter of the candidate image processing sub-model and the quantization processing sub-model to be trained is adjusted, until the texture feature loss value and the quantization feature loss value are both no higher than a preset feature loss threshold value, thereby obtaining the target image processing sub-model, the target texture processing sub-model, and the target quantization processing sub-model.

7. The method according to claim 6, characterized in that Based on the plurality of sample quantization features included in the one quantization feature group and the texture feature loss value, generating a corresponding quantization feature loss value, including: Based on the texture feature loss value, adjusting the quantization control parameters of the quantization processing sub-model to be trained; Based on the multiple sample quantization features included in the one quantization feature group and the quantization control parameter, respectively determining a training sample loss value and a symbol quantization loss value of the quantization processing sub-model to be trained; Based on the training sample loss value and the symbol quantization loss value of the quantization processing sub-model to be trained, a corresponding quantization feature loss value is generated.

8. The method according to claim 6, characterized in that Based on the plurality of sample quantization features included in the one quantization feature group and the texture feature loss value, generating a corresponding quantization feature loss value, including: Based on the multiple sample quantization features included in the one quantization feature group, respectively determine the training sample loss value and the symbol quantization loss value of the quantization processing sub-model to be trained; Based on the training sample loss value, the symbol quantization loss value and the texture feature loss value of the quantization processing sub-model to be trained, a corresponding quantization feature loss value is generated.

9. A video retrieval device, characterized in that: include: An image processing unit is used to extract features of the video to be retrieved by using a target image processing sub-model of the trained target video retrieval model to obtain corresponding image features; A quantization processing unit, used to use the target quantization processing sub-model of the target video retrieval model to extract features of the image features to obtain corresponding first quantization features, and based on the first quantization features, screen out at least one second candidate video that has a category similarity with the video to be retrieved and meets the set category similarity requirements from each first candidate video; wherein the quantization control parameter of the target quantization processing sub-model is adjusted based on the texture feature loss value corresponding to each training sample during the training process, and the texture feature loss value is determined based on the texture control parameter preset for the texture processing sub-model to be trained during the parameter adjustment process of the texture processing sub-model to be trained; A retrieval unit, configured to output, as a corresponding target video, a second candidate video whose content similarity meets a set content similarity requirement based on the content similarity between the video to be retrieved and the at least one second candidate video; The model training unit is used to obtain the trained target video retrieval model by executing the following methods: performing multiple rounds of iterative parameter adjustment based on the first texture sets of each of the multiple sample triplets output by the image processing submodel to be trained and the texture processing submodel to be processed, so as to obtain a candidate image processing submodel and a candidate texture processing submodel; performing multiple rounds of iterative parameter adjustment based on the second texture sets and quantization feature groups of each of the multiple sample triplets output by the candidate image processing submodel, the candidate texture processing submodel and the quantization processing submodel to be trained, so as to obtain the target image processing submodel, the target texture processing submodel, and the target quantization processing submodel.

10. The device according to claim 9, characterized in that The target video retrieval model also includes a target texture processing sub-model, and the retrieval unit is used to: Using the target texture processing sub-model, extracting features from the image features to obtain corresponding first texture features; For the at least one second candidate video, the following operations are performed respectively: determining the texture feature distance between the first texture feature and the second texture feature of a second candidate video; if the texture feature distance is lower than a preset texture feature distance threshold, determining that the content similarity between the video to be retrieved and the second candidate video meets the set content similarity requirement, and determining the second candidate video as the target video output; wherein the second texture feature represents the texture information of the corresponding second candidate video.

11. The device according to claim 9, characterized in that The retrieval unit is used for: For the at least one second candidate video, the following operations are performed respectively: The ratio between the total matching duration and the comparison duration is determined as the content repetition between the video to be retrieved and a second candidate video; wherein the total matching duration is obtained based on the matching duration between each of the at least one second candidate video and the video to be retrieved, and the comparison duration is the duration value of the shorter video duration between the video to be retrieved and the second candidate video; If the content repetition exceeds the set content repetition threshold, the content similarity between the video to be retrieved and the second candidate video is determined to meet the set content similarity requirement, and the second candidate video is determined to be the target video output.

12. The device according to claim 9, characterized in that The retrieval unit is used for: For the at least one second candidate video, the following operations are performed respectively: Determining the number of identical quantitative features between the video to be retrieved and a second candidate video; The ratio between the number of the same quantitative features and the comparison duration is determined as the content repetition between the video to be retrieved and a second candidate video; wherein the comparison duration is the duration value of the shorter video duration between the video to be retrieved and the second candidate video; If the content repetition exceeds the set content repetition threshold, the content similarity between the video to be retrieved and the second candidate video is determined to meet the set content similarity requirement, and the second candidate video is determined to be the target video output.

13. The device according to any one of claims 9 to 12, characterized in that: The quantization processing unit is used for: respectively determining the quantitative feature distances between the first quantitative feature and the second quantitative features of each of the first candidate videos; The first candidate video whose quantitative feature distance is lower than a preset quantitative feature distance threshold is determined as the second candidate video; wherein each second quantitative feature represents the video category to which the corresponding at least one first candidate video belongs.

14. A computer device, characterized in that: It comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 8.

15. A computer-readable storage medium, characterized in that: It includes program codes. When the program product is run on a computer device, the program codes are used to enable the computer device to execute the steps of the method according to any one of claims 1 to 8.

Images