Video recommendation method, device, electronic device and storage medium

Abstract

The present application relates to the field of computers, and in particular to the field of artificial intelligence technology, and discloses a video recommendation method, device, electronic device, and storage medium for improving the accuracy of video sorting. The method includes: obtaining input query information and determining at least two videos that match the query information; obtaining the similarity between the at least two videos and the query information based on the text features of the query information and the visual features corresponding to the at least two videos; and sorting the at least two videos based on the similarity between the at least two videos and the query information to obtain the video sorting result. In this way, based on the dynamic features extracted from the video and the text features of the query information, the similarity between the video and the query information is recalculated, and the behavior contained in the video is also taken into consideration, effectively improving the accuracy of the sorting system.

Description

Background Art

[0002] In the era of big data, recommendation systems generally recall videos that match the query information entered by the user from a large number of videos, and sort the recalled videos to obtain a corresponding video list. However, when sorting, the recommendation system calculates the relevance between the video and the query information based on the image features of a single video frame and the text features of the query information, without considering the dynamic characteristics of the video, which is prone to inaccurate sorting. Summary of the invention

[0003] The embodiments of the present application provide a video recommendation method, device, electronic device and storage medium to improve the accuracy of video sorting.

[0004] The present application provides a video recommendation method, which includes:

[0005] Obtaining input query information and determining at least two videos matching the query information;

[0006] Based on the text features of the query information and the visual features corresponding to the at least two videos, respectively obtaining the similarity between the at least two videos and the query information, wherein the visual features of each video are at least obtained based on the dynamic features of the corresponding video, and the dynamic features are obtained based on the motion state of at least one object in continuous video frames;

[0007] Based on the similarity between the at least two videos and the query information, the at least two videos are sorted to obtain a video sorting result.

[0008] The present invention provides a video recommendation device, the method comprising:

[0009] A query module, used to obtain input query information and determine at least two videos matching the query information;

[0010] A processing module, configured to obtain, based on the text features of the query information and the visual features corresponding to the at least two videos, the similarities between the at least two videos and the query information, respectively, wherein the visual features of each video are obtained based on at least a dynamic feature of the corresponding video, and the dynamic feature is obtained based on a motion state of at least one object in consecutive video frames;

[0011] The sorting module is used to sort the at least two videos based on the similarity between each of the at least two videos and the query information to obtain a video sorting result.

[0012] Optionally, when determining the similarity of each video, the processing module performs the following operations:

[0013] Performing feature extraction on the query information to obtain corresponding text features, and performing feature extraction on a video to obtain corresponding visual features;

[0014] Determine a first context-related feature of the text feature according to the first attention weight of the text feature, and determine a second context-related feature of the visual feature according to the second attention weight of the visual feature; wherein the first attention weight of the text feature represents the degree of association between the text feature and each feature in a first splicing matrix, and the second attention weight of the visual feature represents the degree of association between the visual feature and each feature in the first splicing matrix, and the first splicing matrix is ​​obtained by splicing the text feature and the visual feature;

[0015] The similarity is determined based on the first context-related feature and the second context-related feature.

[0016] Optionally, the processing module is used to:

[0017] Segmenting the query information to obtain an initial word sequence;

[0018] Adding a classification word at the head of the word sequence and adding a segmentation word at the end of the word sequence to form a target word sequence;

[0019] Performing vectorization processing on the target word sequence to obtain a word vector for each word in the target word sequence;

[0020] According to the third attention weights of the respective character vectors, the text sub-features corresponding to the respective character vectors are respectively determined, and the respective text sub-features are concatenated to obtain the text feature; wherein the third attention weight of each character vector represents the degree of association between a corresponding character vector and each of the other character vectors.

[0021] Optionally, the processing module is used to:

[0022] Performing feature extraction on the video to obtain image features and the dynamic features of each video frame in the video;

[0023] According to the fourth attention weight of the dynamic feature and the fifth attention weight of each image feature, the visual sub-features corresponding to the dynamic feature and each image feature are determined respectively, and the visual sub-features are spliced ​​to obtain the visual feature; wherein the fourth attention weight represents the degree of association of the dynamic feature with each feature in the second splicing matrix, and the fifth attention weight of each image feature represents the degree of association of a corresponding image feature with each feature in the second splicing matrix, and the second splicing matrix is ​​obtained by splicing the dynamic feature and each image feature.

[0024] Optionally, the processing module is used to:

[0025] Input each video frame of the video into the image feature extraction sub-model, perform convolution processing on each video frame respectively, and obtain image features corresponding to each video frame; and

[0026] The video is input into the feature extraction layer of the dynamic feature extraction sub-model for convolution processing, the motion state of at least one object in continuous video frames in the video is determined, and the dynamic features of the video are obtained based on the motion state of the at least one object.

[0027] Optionally, the processing module is used to:

[0028] According to the first splicing matrix and a plurality of preset weight matrices, respectively obtaining a query vector of the first splicing matrix, a key vector of the first splicing matrix, and a value vector of the first splicing matrix, wherein the query vector of the first splicing matrix includes query vectors of each text sub-feature in the text feature;

[0029] According to the obtained query vectors of the respective text sub-features and the key vectors of the first concatenation matrix, the sixth attention weights of the respective text sub-features are obtained respectively, and the sixth attention weights of the respective text sub-features are concatenated to obtain the first attention weight of the text feature;

[0030] A first context-related feature of the text feature is obtained according to the obtained first attention weight of the text feature and the value vector of the first splicing matrix.

[0031] Optionally, the processing module is used to:

[0032] According to the first splicing matrix and a plurality of preset weight matrices, respectively obtaining a query vector of the first splicing matrix, a key vector of the first splicing matrix, and a value vector of the first splicing matrix, wherein the query vector of the first splicing matrix includes query vectors of each visual sub-feature in the visual feature;

[0033] Obtaining a seventh attention weight of each visual sub-feature according to the obtained query vector of each visual sub-feature and a key vector of the first splicing matrix, and splicing the seventh attention weights of each visual sub-feature to obtain a second attention weight of the visual feature;

[0034] A second context-related feature of the visual feature is obtained according to the obtained second attention weight of the visual feature and the value vector of the first splicing matrix.

[0035] Optionally, the processing module is used to:

[0036] Performing subtitle recognition and speech recognition on the video to obtain corresponding subtitle information and speech-to-text information;

[0037] Feature extraction is performed on the obtained subtitle information and speech-to-text information respectively to obtain video description sub-features, and each video description sub-feature is concatenated to obtain video description features;

[0038] According to the eighth attention weight of the text feature, the third context-related feature of the text feature is determined, according to the ninth attention weight of the visual feature, the fourth context-related feature of the visual feature is determined, and according to the tenth attention weight of the video description feature, the fifth context-related feature of the video description feature is determined; wherein the eighth attention weight of the text feature represents the degree of association of the text feature with each feature in the third splicing matrix, the ninth attention weight of the visual feature represents the degree of association of the visual feature with each feature in the third splicing matrix, and the tenth attention weight of the video description feature represents the degree of association of the visual description feature with each feature in the third splicing matrix, and the third splicing matrix is ​​obtained by splicing the text feature, the visual feature and the video description feature;

[0039] The similarity is determined based on the third context-related feature, the fourth context-related feature, and the fifth context-related feature.

[0040] An electronic device provided by an embodiment of the present application includes a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of any one of the above-mentioned video recommendation methods.

[0041] An embodiment of the present application provides a computer-readable storage medium, which includes a program code. When the program code is executed on an electronic device, the program code is used to enable the electronic device to execute the steps of any one of the above-mentioned video recommendation methods.

[0042] The beneficial effects of this application are as follows:

[0043] The embodiments of the present application provide a video recommendation method, device, electronic device and storage medium. In the embodiments of the present application, input query information is obtained, and at least two videos matching the query information are determined; based on the text features of the query information and the visual features corresponding to the at least two videos, the similarity between the at least two videos and the query information is obtained, respectively, wherein the visual features of each video are at least based on the dynamic features of the corresponding video, and the dynamic features are obtained based on the motion state of at least one object in the continuous video frames; then based on the similarity between the at least two videos and the query information, the at least two videos are sorted to obtain the video sorting result. In this way, based on the dynamic features extracted from the video and the text features of the query information, the similarity between the video and the query information is recalculated, and the behavior contained in the video is also taken into consideration, effectively improving the accuracy of the sorting system.

[0044] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0046] Figure 1 An optional schematic diagram of an application scenario in an embodiment of the present application;

[0047] Figure 2a This is a schematic diagram of the first structure of the recommendation system;

[0048] Figure 2b Schematic diagram of the process of the video recommendation method in the embodiment of the present application;

[0049] Figure 2c This is a page diagram of the task interface on the terminal;

[0050] Figure 3a A schematic diagram of a process for determining the similarity of a video;

[0051] Figure 3b This is a structural diagram of the text Transformer network;

[0052] Figure 3c Schematic diagram of query vector, key vector and value vector for generating word vector matrix;

[0053] Figure 4This is a schematic diagram of the second structure of the recommendation system;

[0054] Figure 5 This is a schematic diagram of video sorting results;

[0055] Figure 6 This is a schematic diagram of the structure of a video recommendation device in an embodiment of the present application;

[0056] Figure 7 This is another structural schematic diagram of the video recommendation device in the embodiment of the present application;

[0057] Figure 8 A schematic diagram of the hardware structure of an electronic device to which an embodiment of the present application is applied. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0059] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein.

[0060] The embodiments of the present application relate to the field of artificial intelligence (AI) and are designed based on natural language processing (NLP) technology, computer vision technology, and cross-modal fusion technology in machine learning (ML).

[0061] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence.

[0062] Artificial intelligence is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation interaction systems, mechatronics and other technologies; the software technologies of artificial intelligence generally include computer vision technology, natural language processing technology, and machine learning / deep learning and other major directions. With the development and progress of artificial intelligence, artificial intelligence is being studied and applied in many fields, such as common smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, robots, smart medical care, etc. It is believed that with the further development of future technology, artificial intelligence will be applied in more fields and play an increasingly important role.

[0063] Machine learning is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.

[0064] Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Deep learning is the core of machine learning and a technology to achieve machine learning. Machine learning usually includes deep learning, reinforcement learning, transfer learning, inductive learning and other technologies. Deep learning includes convolutional neural networks (CNN), deep belief networks, recursive neural networks, autoencoders, generative adversarial networks and other technologies.

[0065] Natural language processing is a discipline that integrates linguistics, computer science, and mathematics. It is also an important research direction in the field of artificial intelligence. It is used to study various theoretical and practical methods for people to use natural language to communicate effectively with computers. Natural language processing usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graphs and other technologies.

[0066] Computer vision is a comprehensive subject that integrates computer science, signal processing, physics, applied mathematics, statistics, neurophysiology and other disciplines. It is also an important and challenging research direction in the field of science. This subject uses various imaging systems instead of visual organs as input means, and computers replace the brain to complete processing and interpretation, so that computers can have the ability to observe and understand the world through vision like humans.

[0067] In an embodiment of the present application, a recurrent neural network (RNN) can be used to perform natural language processing and computer vision processing, or a Transformer network based on an attention mechanism can be used to perform natural language processing and computer vision processing.

[0068] RNN is widely used in computer vision, natural language processing and other fields. The essence of this network is that there are both internal feedback connections and feedforward connections between processing units. Therefore, compared with feedforward neural networks, RNN has stronger dynamic behavior and computing power in the calculation process. However, it is precisely because of this characteristic of RNN that RNN is prone to long-term dependency problems-that is, the current system state is not only affected by the system state at the previous moment, but may also be affected by the system state a long time ago. In theory, RNN can learn information from a long time ago, but it has been proven in practice that RNN will lose information from a long time ago, resulting in long-term memory failure. In order to avoid the long-term dependency problem of RNN, the embodiment of the present application can also use an improved RNN-that is, a long short-term memory network (Long Short-Term Memory, LSTM). LSTM is a special RNN, which is a neural network designed mainly to solve the gradient disappearance and gradient explosion generated during long sequence training.

[0069] The Transformer network is a sequence processing network based on the attention mechanism. It abandons the recursive structure and relies entirely on the attention mechanism to explore the relationship between input and output, transforming from "global attention" to "local attention", so that the feedforward neural network can pay more attention to features with high attention weights during decoding. This can not only flexibly capture the relationship between the global and local, but also achieve fast parallel computing, reduce network training time, and improve the shortcomings of slow RNN training.

[0070] When performing computer vision processing, in addition to using any one of the RNN, Transformer network and LSTM, it is also necessary to use the inception v4 neural network and the three-dimensional convolution expansion (i3d) neural network to extract the image features and dynamic features of each video frame from a video. The dynamic features are obtained based on the motion state of at least one object in a continuous video frame. A residual learning layer is added to the inception v4 neural network to solve the degradation problem of deep neural networks and improve the performance of neural networks; in order to extract spatiotemporal features from the video and allow the neural network to better learn the behavior contained in the video, the convolution kernel and pooling kernel in the neural network are expanded from 2D to 3D based on the two-dimensional convolution expansion neural network.

[0071] In an embodiment of the present application, first, based on the query information input by the target account, at least two videos matching the query information are recalled from the network, and then the text features of the query information and the video features corresponding to the at least two videos are extracted respectively. Then, a cross-modal interaction network is used for multi-task learning to obtain the similarity between the at least two videos and the query information respectively, and then the at least two recalled videos are reordered according to the similarity.

[0072] Modality refers to the form in which data exists, such as text, audio, image, and video file formats. Although some data exist in different forms, they are all used to describe the same thing or event. In information retrieval, it is not only for the retrieval of data of a single modality in the same thing or event, but also may require data-assisted retrieval of other modalities in the same thing or event, so that the retrieval network can enrich the cognition of the thing or event. The process of using data-assisted retrieval of other modalities in the same thing or event can be called cross-modality. Multi-task learning is a type of transfer learning algorithm. By sharing knowledge between multiple similar tasks, it accelerates the learning efficiency of the target neural network and improves the generalization of the target neural network.

[0073] The following is a brief introduction to the design concept of the embodiment of the present application:

[0074] In the era of big data, recommendation systems are generally divided into two steps: "recall" and "sort". First, videos that match the query information input by the user are recalled from a large amount of videos. Then, based on the image features of a single video frame and the text features of the query information, the relevance between the video and the query information is calculated, and the recalled videos are re-sorted according to the relevance of each video. However, in the sorting process, the related technology only uses a single video frame in the video, without considering the dynamic characteristics of the video, which is prone to inaccurate sorting. In view of this, the embodiments of the present application propose a new video recommendation method, device, electronic device and storage medium.

[0075] The method includes: obtaining input query information and determining at least two videos that match the query information; obtaining the similarity between the at least two videos and the query information based on the text features of the query information and the visual features corresponding to the at least two videos, wherein the visual features of each video are at least obtained based on the dynamic features of the corresponding video, and the dynamic features are obtained based on the motion state of at least one object in the continuous video frames; and then sorting the at least two videos based on the similarity between the at least two videos and the query information to obtain the video sorting result. In this way, based on the dynamic features extracted from the video and the text features of the query information, the similarity between the video and the query information is recalculated, and the behavior contained in the video is also taken into consideration, effectively improving the accuracy of the sorting system.

[0076] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0077] See also Figure 1 The application scenario diagram shown includes two terminals 110 and a server 130.

[0078] The user can log in to the application operation interface 120 through the terminal 110, and the application operation interface 120 presents the query entry and the ranking results obtained based on the query information to the user. In the embodiment of the present application, the terminal 110 is an electronic device used by the user, and the electronic device can be a personal computer, a mobile phone, a tablet computer, a notebook, an e-book reader, a smart home, and other computer devices.

[0079] Each terminal 110 communicates with the server 130 through a communication network. In an optional embodiment, the communication network is a wired network or a wireless network, so the terminal 110 can directly or indirectly establish a communication connection with the server 130 through a wired network or a wireless network, and this application does not limit this. The server 130 can be an independent physical server 130, or a server 130 cluster or distributed system composed of multiple physical servers 130, or a cloud server 130 that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc., and this application does not limit this.

[0080] Among them, the recommendation system in the embodiment of the present application is deployed on the server 130, the server 130 receives the query information sent by the terminal 110, recalls at least two videos matching the query information, and then sorts the recalled videos according to the similarity between the at least two videos and the query information, and returns the sorting result to the terminal 110, so that the terminal 110 presents the sorting result to the user. The query information can be part of the text or the complete title of the video, or a video tag (such as video type, video content keywords), or the name of the account that uploaded the video, etc., and the present application does not limit this.

[0081] See also Figure 2a The structure diagram of the recommendation system shown in the figure is composed of three parts: query information processing model, video processing model and cross-modal interaction model. The query information processing model is used to extract text features from the read query information, the video processing model is used to extract visual features from the read video, and the cross-modal interaction model is used to generate the similarity of the video based on the text features and visual features.

[0082] The video recommendation method proposed in the embodiment of the present application can be divided into two parts, namely the training stage and the application stage. Among them, in the training stage, a large number of training samples are prepared in advance. A training sample includes initial sample query information and a corresponding video set. In order to improve the performance of the query information processing model, it is necessary to block some characters in the initial sample query information to obtain the target sample query information. For example, the initial sample query information is "Today is sunny", and the target sample query information after the occlusion processing is "Today is x day" (x means that the character at this position is blocked); the recommendation system continuously reads the target sample query information and the corresponding video set, obtains the similarity of the blocked character recognition result and each video in the video set, and continuously adjusts the model parameters through the optimization algorithm until the iteration stop condition is met, and obtains the trained recommendation system. In the application stage, the trained recommendation system is used to recall at least two videos that match the query information, and then the sorting results of at least two videos are obtained according to the text features of the query information and the visual features corresponding to the at least two videos.

[0083] See also Figure 2b The flowchart shown introduces the process of using the recommendation system to recommend videos.

[0084] S201: Obtain input query information, and determine at least two videos matching the query information.

[0085] See also Figure 2c In the page schematic diagram shown, the terminal displays a task interface for executing a video search task in response to a trigger operation, and then the terminal starts the operation in response to the task triggered by the task interface to execute the video search task. Specifically, the target account can enter query information through the search bar on the task interface. In addition to manually entering query information, the target account can also query by clicking on the title of a popular video on the search page, and the target account can also limit the query scope, which can be a public account, a mini program, and a content sharing space.

[0086] S202: Based on the text features of the query information and the visual features corresponding to the at least two videos, respectively obtain the similarity between the at least two videos and the query information, wherein the visual features of each video are at least obtained based on the dynamic features of the corresponding video, and the dynamic features are obtained based on the motion state of at least one object in continuous video frames.

[0087] For ease of description, see Figure 3a The flowchart shown takes a video as an example to describe the operations performed when determining the similarity of the video:

[0088] S2021: Perform feature extraction on the query information to obtain corresponding text features, and perform feature extraction on the video to obtain corresponding visual features.

[0089] When executing step 221, the query information is first segmented to obtain an initial character sequence. For example, if the query information is "last year's Spring Festival Gala", after the segmentation process, an initial character sequence of the form (last, year, of, spring, festival, evening, meeting) is obtained;

[0090] Then, a classification word is added to the head of the word sequence, and a segmentation word is added to the tail of the word sequence to form a target word sequence. Continuing with the above example, the target word sequence after processing is ([CLS], past, year, of, spring, festival, evening, meeting, [SEP]), where [CLS] represents the classification word (CLS is the abbreviation of classification), and [SEP] represents the segmentation word (SEP is the abbreviation of segmentation);

[0091] Then, the target word sequence is vectorized to obtain the word vector of each word in the target word sequence;

[0092] Finally, according to the third attention weight of each character vector, the text sub-features corresponding to each character vector are determined respectively, and the text sub-features are concatenated to obtain the text features; wherein the third attention weight of each character vector represents the degree of association between a corresponding character vector and other character vectors.

[0093] See also Figure 2a As can be seen from the structural diagram, the query information processing model includes a preprocessing module and a text Transformer network. Figure 3b The schematic diagram of the structure of the text Transformer network shown in FIG. 1 mainly includes an attention module and a feedforward neural network connected to the attention module. The attention module includes three preset weight matrices, which are used to generate the query vector, key vector, and value vector of each word vector. In order to capture more relationships between input and output, multiple attention modules can be added to the text Transformer network.

[0094] In cognitive neuroscience, attention is an indispensable and complex cognitive function of humans. It refers to the ability of people to pay attention to some information while ignoring other information. In daily life, people can receive a large amount of sensory input through vision, hearing, touch, etc., but the brain can still work in an orderly manner under the bombardment of a large amount of external information. This is because the brain can intentionally or unintentionally select a small part of useful information from a large amount of input information for key processing and ignore other information. For example, when people read, usually only a small number of important words are paid attention to and processed. Similarly, the attention mechanism can also enable neural networks to have the ability to focus on their input features-that is, to select specific input features. In this way, under limited computing power, the attention mechanism is used to allocate computing resources to more important tasks, effectively solving the problem of information overload.

[0095] In order to speed up the processing, each word vector is concatenated into a word vector matrix. If there is only one attention module in the text Transformer network, the word vector matrix is ​​input into the attention module to obtain the third attention weight of each word vector, and then the text features corresponding to each word vector are determined according to the third attention weight of each word vector; if there are multiple attention modules in the text Transformer network, the same word vector matrix is ​​input into each attention module respectively, and after obtaining the text features corresponding to each word vector in each attention module, the multiple text features of the same word vector are added together, and the text features obtained after the addition are output as the target text features of the same word vector.

[0096] For the sake of ease of description, this article takes an attention module as an example to introduce the process of generating text features.

[0097] First, generate the query vector, key vector, and value vector of the word vector matrix.

[0098] See also Figure 3c The vector generation diagram shown in the figure takes the word vector matrix as the input matrix and uses the linear transformation matrix W Q , W K , W V , obtain the query vector, key vector and value vector of the word vector matrix, where a row of query vectors represents the query vector of a word vector in the word vector matrix, a row of key vectors represents the key vector of a word vector in the word vector matrix, and a row of value vectors represents the value vector of a word vector in the word vector matrix.

[0099] Next, read the query vector of a word vector.

[0100] Again, the query vector of the word vector is multiplied by the key vector of the word vector matrix to obtain the third attention weight of the word vector, and the third attention weight of the word vector is multiplied by the value vector of the word vector weight to obtain the text sub-feature of the word vector.

[0101] Finally, the above steps are repeated until all word vectors in the word vector matrix are read, and each text sub-feature is concatenated to obtain the text feature.

[0102] For example, if the query information is "last year's Spring Festival Gala", the target word sequence corresponding to the query information is ([CLS], last, year, Spring, Festival, Gala, [SEP]), and the word vector matrix corresponding to the target word sequence is a matrix of the form 9-row n-dimensional matrix, W Q , W K and W V are the same weight matrices, both of which are n-row 9-dimensional matrices. Then the generated query vector Q = key vector K = value vector Read the query vector [1 2 2 20] of the first word vector and multiply it with the key vector K to get the third attention weight as Then multiply the third attention weight by the value vector to get the text sub-feature: The numbers in the matrix are for illustration only and are not accurate data.

[0103] After introducing the generation process of text features, the generation process of visual features will be introduced next.

[0104] First, feature extraction is performed on a video to obtain image features and dynamic features of each video frame in the video.

[0105] See also Figure 2aAs can be seen from the structural diagram, the video processing model includes an image feature extraction submodel, a dynamic feature extraction submodel and a visual Transformer network. In an embodiment of the present application, the image feature extraction submodel is an inception v4 neural network, the video is input into the inception v4 neural network, and convolution processing is performed on each video frame in the video to obtain the image features of each video frame in the video. In an embodiment of the present application, the dynamic feature extraction submodel is an i3d neural network. While obtaining the image features of each video frame, the video is input into the i3d neural network, and after convolution processing of continuous video frames in the video, the motion state of at least one object in the continuous video frames in the video is determined, and according to the motion state of at least one object, the dynamic features of the video are obtained. Objects include but are not limited to people, animals, and inanimate objects such as vehicles; motion states include but are not limited to running, driving, playing musical instruments, etc. Assuming that the query information is the keyword "weight loss", a video is input into the i3d neural network for video behavior recognition, and the recognition result is "yoga", which can better associate the video containing the "yoga" behavior with the keyword "weight loss", which is conducive to improving the accuracy of the similarity calculation between the video and the query information, and then better re-ranking the recalled videos.

[0106] Secondly, according to the fourth attention weight of the dynamic feature and the fifth attention weight of each image feature, the visual sub-features corresponding to the dynamic feature and each image feature are determined, and the visual sub-features are spliced ​​to obtain the visual feature; wherein the fourth attention weight represents the degree of association of the dynamic feature to each feature in the second splicing matrix, and the fifth attention weight of each image feature represents the degree of association of a corresponding image to each feature in the second splicing matrix, and the second splicing matrix is ​​obtained by splicing the dynamic feature and each image feature.

[0107] The specific steps performed by the visual Transformer network are similar to those of the text Transformer network. The specific expression has been introduced in the process of generating text sub-features of word vectors, so it will not be repeated here.

[0108] S2022: Determine a first context-associated feature of the text feature based on the first attention weight of the text feature, and determine a second context-associated feature of the visual feature based on the second attention weight of the visual feature; wherein the first attention weight of the text feature represents the degree of association between the text feature and each feature in a first splicing matrix, and the second attention weight of the visual feature represents the degree of association between the visual feature and each feature in the first splicing matrix, and the first splicing matrix is ​​obtained by splicing the text feature and the visual feature.

[0109] When executing step 2022, in order to speed up the processing, the text features and the visual features are first concatenated to obtain a first concatenated matrix, and the first concatenated matrix is ​​input into the cross-modal interaction model for subsequent processing. Figure 2a As can be seen from the structural diagram, the cross-modal interaction model mainly includes a fused Transformer network, which can contain one or more attention modules.

[0110] Next, the process of determining the first context-related feature by fusing the Transformer network is introduced.

[0111] First, according to the first splicing matrix and a plurality of preset weight matrices, a query vector of the first splicing matrix, a key vector of the first splicing matrix, and a value vector of the first splicing matrix are obtained respectively, wherein the query vector of the first splicing matrix includes query vectors of each text self-feature in the text feature;

[0112] Secondly, according to the query vector of each text sub-feature obtained and the key vector of the first concatenation matrix, the sixth attention weight of each text sub-feature is obtained respectively, and the sixth attention weight of each text sub-feature is concatenated to obtain the first attention weight of the text feature;

[0113] Finally, the first context-related features of the text features are obtained according to the first attention weights of the obtained text features and the value vectors of the first concatenation matrix.

[0114] The method for generating the first context-related feature is similar to the method for generating the text sub-feature of the word vector. The specific expression method has been introduced in the process of generating the text sub-feature of the word vector and will not be repeated here.

[0115] Next, the process of determining the second contextual features by fusing the Transformer network is introduced.

[0116] First, according to the first splicing matrix and a plurality of preset weight matrices, a query vector of the first splicing matrix, a key vector of the first splicing matrix, and a value vector of the first splicing matrix are respectively obtained, wherein the query vector of the first splicing matrix includes query vectors of each visual sub-feature in the visual feature;

[0117] Secondly, according to the query vector of each visual sub-feature and the key vector of the first concatenation matrix, the seventh attention weight of each visual sub-feature is obtained, and the seventh attention weights of each visual sub-feature are concatenated to obtain the second attention weight of the visual feature;

[0118] Finally, a second context-related feature of the visual feature is obtained according to the obtained second attention weight of the visual feature and the value vector of the first concatenation matrix.

[0119] The method for generating the second context-related feature is similar to the method for generating the text sub-feature of the word vector. The specific expression method has been introduced in the process of generating the text sub-feature of the word vector and will not be repeated here.

[0120] S2023: Determine similarity based on the first context-related feature and the second context-related feature.

[0121] See also Figure 2a As can be seen from the structural diagram of , the cross-modal interaction model includes not only a fused Transformer network, but also a fully connected layer and a Softmax layer. The first context-related features and the second context-related features output by step 2022 pass through the fully connected layer and the Softmax layer in sequence to obtain the similarity between the video and the query information.

[0122] In addition to the recommendation system shown in Figure 2, the subtitle information and speech-to-text information of the video can also be used as supplementary features of the video to form a Figure 4 The recommendation system shown in the figure includes a query information processing model, a video processing model, a video auxiliary processing model and a cross-modal interaction model, wherein the video auxiliary processing model is used to perform subtitle recognition and speech recognition on the video to obtain the corresponding subtitle information and speech-to-text information, and perform feature extraction on the subtitle information and speech-to-text information to obtain video description features.

[0123] For ease of description, take a video as an example. Figure 4 The process of generating the similarity of the video by the recommendation system shown is introduced.

[0124] First, perform subtitle recognition and speech recognition on the video to obtain corresponding subtitle information and speech-to-text information.

[0125] See also Figure 4 As shown in the structural diagram, the video-assisted processing model includes an optical character recognition (OCR) module and an automatic speech recognition (ASR) module. The OCR module is used to perform subtitle recognition operations to obtain subtitle information of the video, and the ASR module is used to perform speech recognition to obtain speech-to-text information of the video. Assuming that the query information is an artificial intelligence technology seminar, if only the video screen is captured, the recommendation system can only recognize the behavior of people speaking in the video. In the absence of subtitle information and speech-to-text information, it is difficult to associate the video with the query information.

[0126] Secondly, feature extraction is performed on the obtained subtitle information and speech-to-text information to obtain video description sub-features, and the various video description sub-features are concatenated to obtain video description features.

[0127] Again, according to the eighth attention weight of the text feature, the third context-related feature of the text feature is determined, according to the ninth attention weight of the visual feature, the fourth context-related feature of the visual feature is determined, and according to the tenth attention weight of the video description feature, the fifth context-related feature of the video description feature is determined; wherein, the eighth attention weight of the text feature represents the degree of association of the text feature with each feature in the third splicing matrix, the ninth attention weight of the visual feature represents the degree of association of the visual feature with each feature in the third splicing matrix, and the tenth attention weight of the video description feature represents the degree of association of the visual description feature with each feature in the third splicing matrix, and the third splicing matrix is ​​obtained by splicing the text features, visual features and video description features.

[0128] See also Figure 4 As shown in the structural diagram, the video-assisted processing model also includes a text Transformer network, which is used to calculate the attention weights of each feature to obtain the third context-related feature, the fourth context-related feature, and the fifth context-related feature. The generation method of the third context-related feature, the fourth context-related feature, and the fifth context-related feature is similar to the generation method of the text sub-feature of the word vector. The specific expression method has been introduced in the process of generating the text sub-feature of the word vector, and will not be repeated here.

[0129] Finally, based on the third context-related feature, the fourth context-related feature, and the fifth context-related feature, a similarity is determined.

[0130] See also Figure 4 From the structural diagram, we can see that the cross-modal interaction model not only includes the fused Transformer network, but also includes a fully connected layer and a Softmax layer. The third context-related feature, the fourth context-related feature and the fifth context-related feature pass through the fully connected layer and the Softmax layer in sequence to obtain the similarity between the video and the query information.

[0131] S203: Sort the at least two videos based on the similarities corresponding to the at least two videos to obtain a video sorting result.

[0132] According to the similarity corresponding to each video, at least two recalled videos are sorted to obtain the video sorting result, and the video sorting result is returned to the terminal so that the terminal displays the corresponding video sorting result to the target account in the task interface. The video sorting result is as follows: Figure 5 shown.

[0133] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides a structural schematic diagram of a video recommendation device. Figure 6 As shown, the video recommendation device 600 may include:

[0134] A query module 601 is used to obtain input query information and determine at least two videos matching the query information;

[0135] The processing module 602 is used to obtain the similarity between each of the at least two videos and the query information based on the text features of the query information and the visual features corresponding to each of the at least two videos, wherein the visual features of each video are obtained based on at least a dynamic feature of the corresponding video, and the dynamic feature is obtained based on the motion state of at least one object in continuous video frames;

[0136] The sorting module 603 is used to sort the at least two videos based on the similarity between each of the at least two videos and the query information to obtain a video sorting result.

[0137] Optionally, when determining the similarity of each video, the processing module 602 performs the following operations:

[0138] Performing feature extraction on the query information to obtain corresponding text features, and performing feature extraction on a video to obtain corresponding visual features;

[0139] Determine a first context-related feature of the text feature according to the first attention weight of the text feature, and determine a second context-related feature of the visual feature according to the second attention weight of the visual feature; wherein the first attention weight of the text feature represents the degree of association between the text feature and each feature in a first splicing matrix, and the second attention weight of the visual feature represents the degree of association between the visual feature and each feature in the first splicing matrix, and the first splicing matrix is ​​obtained by splicing the text feature and the visual feature;

[0140] The similarity is determined based on the first context-related feature and the second context-related feature.

[0141] Optionally, the processing module 602 is used to:

[0142] Segmenting the query information to obtain an initial word sequence;

[0143] Adding a classification word at the head of the word sequence and adding a segmentation word at the end of the word sequence to form a target word sequence;

[0144] Performing vectorization processing on the target word sequence to obtain a word vector for each word in the target word sequence;

[0145] According to the third attention weights of the respective character vectors, the text sub-features corresponding to the respective character vectors are respectively determined, and the respective text sub-features are concatenated to obtain the text feature; wherein the third attention weight of each character vector represents the degree of association between a corresponding character vector and each of the other character vectors.

[0146] Optionally, the processing module 602 is used to:

[0147] Performing feature extraction on the video to obtain image features of each video frame in the video and the dynamic features;

[0148] According to the fourth attention weight of the dynamic feature and the fifth attention weight of each image feature, the visual sub-features corresponding to the dynamic feature and each image feature are determined respectively, and the visual sub-features are spliced ​​to obtain the visual feature; wherein the fourth attention weight represents the degree of association of the dynamic feature with each feature in the second splicing matrix, and the fifth attention weight of each image feature represents the degree of association of a corresponding image feature with each feature in the second splicing matrix, and the second splicing matrix is ​​obtained by splicing the dynamic feature and each image feature.

[0149] Optionally, the processing module 602 is used to:

[0150] Input each video frame of the video into the image feature extraction sub-model, perform convolution processing on each video frame respectively, and obtain image features corresponding to each video frame; and

[0151] The video is input into the feature extraction layer of the dynamic feature extraction sub-model for convolution processing, the motion state of at least one object in continuous video frames in the video is determined, and the dynamic features of the video are obtained based on the motion state of the at least one object.

[0152] Optionally, the processing module 602 is used to:

[0153] According to the first splicing matrix and a plurality of preset weight matrices, respectively obtaining a query vector of the first splicing matrix, a key vector of the first splicing matrix, and a value vector of the first splicing matrix, wherein the query vector of the first splicing matrix includes query vectors of each text sub-feature in the text feature;

[0154] According to the obtained query vectors of the respective text sub-features and the key vectors of the first concatenation matrix, the sixth attention weights of the respective text sub-features are obtained respectively, and the sixth attention weights of the respective text sub-features are concatenated to obtain the first attention weight of the text feature;

[0155] A first context-related feature of the text feature is obtained according to the obtained first attention weight of the text feature and the value vector of the first splicing matrix.

[0156] Optionally, the processing module 602 is used to:

[0157] According to the first splicing matrix and a plurality of preset weight matrices, respectively obtaining a query vector of the first splicing matrix, a key vector of the first splicing matrix, and a value vector of the first splicing matrix, wherein the query vector of the first splicing matrix includes query vectors of each visual sub-feature in the visual feature;

[0158] Obtaining a seventh attention weight of each visual sub-feature according to the obtained query vector of each visual sub-feature and a key vector of the first splicing matrix, and splicing the seventh attention weights of each visual sub-feature to obtain a second attention weight of the visual feature;

[0159] A second context-related feature of the visual feature is obtained according to the obtained second attention weight of the visual feature and the value vector of the first splicing matrix.

[0160] Optionally, the processing module 602 is used to:

[0161] Performing subtitle recognition and speech recognition on the video to obtain corresponding subtitle information and speech-to-text information;

[0162] Feature extraction is performed on the obtained subtitle information and speech-to-text information respectively to obtain video description sub-features, and each video description sub-feature is concatenated to obtain video description features;

[0163] According to the eighth attention weight of the text feature, the third context-related feature of the text feature is determined, according to the ninth attention weight of the visual feature, the fourth context-related feature of the visual feature is determined, and according to the tenth attention weight of the video description feature, the fifth context-related feature of the video description feature is determined; wherein the eighth attention weight of the text feature represents the degree of association of the text feature with each feature in the third splicing matrix, the ninth attention weight of the visual feature represents the degree of association of the visual feature with each feature in the third splicing matrix, and the tenth attention weight of the video description feature represents the degree of association of the visual description feature with each feature in the third splicing matrix, and the third splicing matrix is ​​obtained by splicing the text feature, the visual feature and the video description feature;

[0164] The similarity is determined based on the third context-related feature, the fourth context-related feature, and the fifth context-related feature.

[0165] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides a structural schematic diagram of a video recommendation device. Figure 7 As shown, the video recommendation device 700 may include:

[0166] Search module 701, for displaying a task interface for executing a video search task in response to a trigger operation;

[0167] The display module 702 is used to respond to the task start operation triggered by the task interface, execute the video search task, obtain the sorting result, and display the sorting result in the task interface.

[0168] After introducing the video recommendation method and device according to an exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0169] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0170] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. The electronic device can be used to sort the recalled multiple videos. In one embodiment, the electronic device can be a server, such as Figure 1 The server 130 shown may also be other electronic devices, such as Figure 1In this embodiment, the structure of the electronic device can be as follows: Figure 8 As shown, it includes a memory 801 , a communication module 803 and one or more processors 802 .

[0171] The memory 801 is used to store computer programs executed by the processor 802. The memory 801 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0172] The memory 801 may be a volatile memory, such as a random-access memory (RAM); the memory 801 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 801 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 801 may be a combination of the above memories.

[0173] The processor 802 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 802 is configured to implement the above-mentioned video recommendation method when calling the computer program stored in the memory 801 .

[0174] The communication module 803 is used to communicate with terminal devices and other servers.

[0175] The specific connection medium between the memory 801, the communication module 803 and the processor 802 is not limited in the embodiment of the present application. Figure 8 In the embodiment, the memory 801 and the processor 802 are connected via a bus 804. The bus 804 is Figure 8 The connections between other components are shown in bold lines, and are not intended to be limiting. Bus 804 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0176] The memory 801 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the video recommendation method of the embodiment of the present application. The processor 802 is used to execute the above-mentioned video recommendation method.

[0177] An embodiment of the present application further provides a computer storage medium, in which computer executable instructions are stored. The computer executable instructions are used to implement the video recommendation method described in any embodiment of the present application.

[0178] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories, magnetic disks or optical disks, and other media that can store program codes.

[0179] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0180] In some possible implementations, various aspects of the video recommendation method provided by the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the video recommendation method according to various exemplary implementations of the present application described above in this specification. For example, the computer device may execute the following steps: Figure 2b Follow the steps shown in .

[0181] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0182] The program product of the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a computing device. However, the program product of the present application is not limited to this. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, device or device.

[0183] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0184] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0185] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0186] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0187] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A video recommendation method, characterized in that: The method includes: Obtaining input query information and determining at least two videos matching the query information; Based on the text features of the query information and the visual features corresponding to the at least two videos, respectively obtaining the similarity between the at least two videos and the query information, wherein the visual features of each video are at least obtained based on the dynamic features of the corresponding video, and the dynamic features are obtained based on the motion state of at least one object in continuous video frames; Sorting the at least two videos based on the similarity between each of the at least two videos and the query information to obtain a video sorting result; Based on the text features of the query information and the visual features corresponding to the at least two videos, the similarities between the at least two videos and the query information are respectively obtained, wherein when determining the similarity of each video, the following operations are performed: Performing feature extraction on the query information to obtain corresponding text features, and performing feature extraction on a video to obtain corresponding visual features; Determine a first context-related feature of the text feature according to the first attention weight of the text feature, and determine a second context-related feature of the visual feature according to the second attention weight of the visual feature; wherein the first attention weight of the text feature represents the degree of association between the text feature and each feature in a first splicing matrix, and the second attention weight of the visual feature represents the degree of association between the visual feature and each feature in the first splicing matrix, and the first splicing matrix is ​​obtained by splicing the text feature and the visual feature; The similarity is determined based on the first context-related feature and the second context-related feature.

2. The method according to claim 1, characterized in that The extracting features of the query information to obtain corresponding text features includes: Segmenting the query information to obtain an initial word sequence; Adding a classification word at the head of the word sequence and adding a segmentation word at the end of the word sequence to form a target word sequence; Performing vectorization processing on the target word sequence to obtain a word vector for each word in the target word sequence; According to the third attention weights of the respective character vectors, the text sub-features corresponding to the respective character vectors are respectively determined, and the respective text sub-features are concatenated to obtain the text feature; wherein the third attention weight of each character vector represents the degree of association between a corresponding character vector and each of the other character vectors.

3. The method according to claim 1, characterized in that The extracting features from the video to obtain corresponding visual features includes: Performing feature extraction on the video to obtain image features and the dynamic features of each video frame in the video; According to the fourth attention weight of the dynamic feature and the fifth attention weight of each image feature, the visual sub-features corresponding to the dynamic feature and each image feature are determined respectively, and the visual sub-features are spliced ​​to obtain the visual feature; wherein the fourth attention weight represents the degree of association of the dynamic feature with each feature in the second splicing matrix, and the fifth attention weight of each image feature represents the degree of association of a corresponding image feature with each feature in the second splicing matrix, and the second splicing matrix is ​​obtained by splicing the dynamic feature and each image feature.

4. The method according to claim 3, characterized in that The extracting features of the video to obtain the image features of each video frame in the video and the dynamic features includes: Input each video frame of the video into the image feature extraction sub-model, perform convolution processing on each video frame respectively, and obtain image features corresponding to each video frame; and The video is input into the feature extraction layer of the dynamic feature extraction sub-model for convolution processing, the motion state of at least one object in continuous video frames in the video is determined, and the dynamic features of the video are obtained based on the motion state of the at least one object.

5. The method according to claim 1, characterized in that Determining a first context-related feature of the text feature according to a first attention weight of the text feature includes: According to the first splicing matrix and a plurality of preset weight matrices, respectively obtaining a query vector of the first splicing matrix, a key vector of the first splicing matrix, and a value vector of the first splicing matrix, wherein the query vector of the first splicing matrix includes query vectors of each text sub-feature in the text feature; According to the obtained query vectors of the respective text sub-features and the key vectors of the first concatenation matrix, the sixth attention weights of the respective text sub-features are obtained respectively, and the sixth attention weights of the respective text sub-features are concatenated to obtain the first attention weight of the text feature; A first context-related feature of the text feature is obtained according to the obtained first attention weight of the text feature and the value vector of the first splicing matrix.

6. The method according to claim 1, characterized in that Determining a second context-related feature of the visual feature according to the second attention weight of the visual feature includes: According to the first splicing matrix and a plurality of preset weight matrices, respectively obtaining a query vector of the first splicing matrix, a key vector of the first splicing matrix, and a value vector of the first splicing matrix, wherein the query vector of the first splicing matrix includes query vectors of each visual sub-feature in the visual feature; Obtaining a seventh attention weight of each visual sub-feature according to the obtained query vector of each visual sub-feature and a key vector of the first splicing matrix, and splicing the seventh attention weights of each visual sub-feature to obtain a second attention weight of the visual feature; A second context-related feature of the visual feature is obtained according to the obtained second attention weight of the visual feature and the value vector of the first splicing matrix.

7. The method according to any one of claims 1 to 6, characterized in that: After obtaining the text feature and the visual feature, and before obtaining the first context-related feature and the second context-related feature, the method further includes: Performing subtitle recognition and speech recognition on the video to obtain corresponding subtitle information and speech-to-text information; Feature extraction is performed on the obtained subtitle information and speech-to-text information respectively to obtain video description sub-features, and each video description sub-feature is concatenated to obtain video description features; According to the eighth attention weight of the text feature, the third context-related feature of the text feature is determined, according to the ninth attention weight of the visual feature, the fourth context-related feature of the visual feature is determined, and according to the tenth attention weight of the video description feature, the fifth context-related feature of the video description feature is determined; wherein the eighth attention weight of the text feature represents the degree of association of the text feature with each feature in the third splicing matrix, the ninth attention weight of the visual feature represents the degree of association of the visual feature with each feature in the third splicing matrix, and the tenth attention weight of the video description feature represents the degree of association of the video description feature with each feature in the third splicing matrix, and the third splicing matrix is ​​obtained by splicing the text feature, the visual feature and the video description feature; The similarity is determined based on the third context-related feature, the fourth context-related feature, and the fifth context-related feature.

8. A video recommendation device, characterized in that: The device includes: A query module, used to obtain input query information and determine at least two videos matching the query information; A processing module, configured to obtain, based on the text features of the query information and the visual features corresponding to the at least two videos, the similarities between the at least two videos and the query information, respectively, wherein the visual features of each video are obtained based on at least a dynamic feature of the corresponding video, and the dynamic feature is obtained based on a motion state of at least one object in consecutive video frames; A sorting module, used to sort the at least two videos based on the similarity between each of the at least two videos and the query information, to obtain a video sorting result; When determining the similarity of each video, the processing module performs the following operations: Performing feature extraction on the query information to obtain corresponding text features, and performing feature extraction on a video to obtain corresponding visual features; Determine a first context-related feature of the text feature according to the first attention weight of the text feature, and determine a second context-related feature of the visual feature according to the second attention weight of the visual feature; wherein the first attention weight of the text feature represents the degree of association between the text feature and each feature in a first splicing matrix, and the second attention weight of the visual feature represents the degree of association between the visual feature and each feature in the first splicing matrix, and the first splicing matrix is ​​obtained by splicing the text feature and the visual feature; The similarity is determined based on the first context-related feature and the second context-related feature.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of any one of the methods in claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The method comprises a program code. When the program code is run on an electronic device, the program code is used to enable the electronic device to execute the steps of any method described in claims 1 to 7.

11. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program code, and when the program code is executed by a computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 7.

Images