A composite semantic video retrieval method, device, equipment and storage medium

Abstract

The application discloses a kind of composite semantic video retrieval method, device, equipment and storage medium, wherein the method includes steps: extracting the visual feature vector of video and the text feature vector of corresponding moment, the features include character, behavior and scene;Visual feature vector and text feature vector are fused to obtain fusion feature;The fusion feature is input into multilayer perceptron, and the character, behavior and scene semantic state vector are obtained;The character, behavior and scene semantic state vector are fused semantically, and the vector reflecting semantic score is obtained, to be used as multimodal composite semantic video retrieval result.The application effectively improves the utilization of information by introducing multimodal fusion method, and also improves the accuracy of retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to a video retrieval method and device based on composite semantics, equipment and storage medium. BACKGROUND

[0002] With the rapid development of multimedia technology and the popularity of the Internet, the scale of video data is growing rapidly. In the face of various forms and large quantities of video data, manual searching for video content is inefficient. Therefore, it is an urgent problem to efficiently find desired videos by machines, i.e., video retrieval.

[0003] In recent years, deep learning technology has been widely used in the field of video retrieval. For single modalities and single semantics, such as people, behaviors, scenes, etc., the performance of video retrieval has made great progress. With the continuous improvement of user demand, the user's query content tends to be detailed, and the retrieval needs to be positioned to a specific composite semantic composed of single semantics such as people, places, and actions. Therefore, video retrieval for composite semantics has gradually become the mainstream. Current video retrieval technology cannot directly meet the needs of composite semantic video retrieval. The main reasons are as follows: (1) videos contain multi-modal information such as images, texts, and audios, but current video retrieval is mainly based on visual features for similarity ranking. The audio and text information contained in the video is largely ignored. However, for composite semantic video retrieval, the dialogue, voiceprint, and background environment sound of the characters also contain important semantic information. Single-modal video retrieval based on vision alone cannot fully mine the video content to be retrieved. (2) Current research on composite semantic video retrieval mainly uses different technologies to retrieve single semantic instances, and then performs score fusion (such as weighted, filtered, and product fusion) according to the independent retrieval results. The problem with this strategy is that each single semantic retrieval branch is independent of each other, and the quality of composite semantic retrieval is difficult to guarantee by relying only on the scores of different retrieval branches to obtain the composite semantic retrieval results, ignoring the relevance and mutual influence between the semantics of composite retrieval.

[0004] Therefore, how to improve the precision of composite semantic retrieval is a technical problem that needs to be solved at present. SUMMARY

[0005] The main purpose of the present application is to provide a video retrieval method and device based on composite semantics, equipment and storage medium, which effectively improves the utilization rate of information and further improves the accuracy of retrieval.

[0006] In a first aspect, the present application provides a video retrieval method based on composite semantics, which comprises the following steps:

[0007] extracting visual feature vectors of the video and text feature vectors of corresponding time points, the features including characters, behaviors and scenes;

[0008] fusing the visual feature vectors and the text feature vectors to obtain fused features;

[0009] inputting the fused features into a multi-layer perception machine to obtain character, behavior and scene semantic state vectors;

[0010] performing semantic fusion on the character, behavior and scene semantic state vectors to obtain a vector reflecting semantic scores as a multi-modal composite semantic video retrieval result.

[0011] With reference to the first aspect above, as an optional implementation manner, tensor product operation is performed on the character, behavior and scene semantic state vectors two by two to obtain three combined semantic state vectors;

[0012] quantum observation is performed on the three combined semantic state vectors to obtain probability vectors;

[0013] maximum pooling layer processing is performed on the three probability vectors to obtain a vector reflecting semantic scores as a multi-modal composite semantic video retrieval result.

[0014] With reference to the first aspect above, as an optional implementation manner, the final semantic scores are calculated according to a formula: W(|φ k >)=max{P mn (|φ k >)}, wherein |ψ m > and |ψ n > represent the mth and nth semantic state vectors respectively, |ψ mn > represents a combination of the mth semantic state vector and the nth semantic state vector, represents tensor product operation, P mn (|φ k >) represents a probability that the combined state vector |ψ mn > collapses to a basic vector |φ k >, and W(|φ k >) represents a semantic score of a corresponding basic state.

[0015] With reference to the first aspect above, as an optional implementation manner, according to the obtained visual state vectors and text state vectors, point-by-point multiplication of vectors is used to obtain interference terms corresponding to the visual state vectors and the text state vectors;

[0016] the visual state vectors, the text state vectors and the corresponding interference terms are taken as modal features and input into a preset multi-modal fusion network for modal fusion to obtain fused features.

[0017] In combination with the first aspect, as an optional implementation manner, the video feature after the modal fusion is calculated according to a formula f is the visual feature, f vision is the text feature, f text is the video feature after the modal fusion, and is a point-by-point multiplication of vectors. fusion f is the visual feature, f text is the text feature, f fusion is the video feature after the modal fusion, and is a point-by-point multiplication of vectors. α, β and are hyperparameters.

[0018] In combination with the first aspect, as an optional implementation manner, the video is preprocessed, and the preprocessing includes scene segmentation, shot segmentation and key frame extraction.

[0019] The C3D model is used to extract the visual character feature vector, the behavior feature vector and the scene feature vector in the processed video, and the BERT model is used to extract the character feature vector, the behavior feature vector and the scene feature vector of the text at the corresponding moment, wherein the text includes a script and a dialogue.

[0020] In combination with the first aspect, as an optional implementation manner, the convolutional neural network is used to map the visual feature vector and the text feature vector at the corresponding moment to the corresponding d-dimensional Hilbert semantic space, and convert the visual state vector and the text state vector in the common space.

[0021] The second aspect provides a video retrieval device with composite semantics, which comprises:

[0022] An extraction unit is configured to extract a visual feature vector of a video and a text feature vector at a corresponding moment, and the features include characters, behaviors and scenes.

[0023] A fusion unit is configured to fuse the visual feature vector and the text feature vector to obtain a fusion feature.

[0024] A processing unit is configured to input the fusion feature into a multi-layer perceptron to obtain a character, behavior and scene semantic state vector.

[0025] A calculation unit is configured to perform semantic fusion on the character, behavior and scene semantic state vector to obtain a vector reflecting a semantic score, as a multi-modal composite semantic video retrieval result.

[0026] The third aspect provides an electronic device, which comprises a processor and a memory having computer readable instructions stored thereon, wherein the computer readable instructions, when executed by the processor, implement the method of any one of the first aspect.

[0027] In a fourth aspect, the present application provides a computer readable storage medium storing computer program instructions, when the computer program instructions are executed by a computer, the computer program instructions cause the computer to execute the method of any one of the first aspect.

[0028] The present application provides a composite semantic video retrieval method, device, equipment and storage medium, wherein the method comprises the steps of: extracting a visual feature vector of a video and a text feature vector of a corresponding moment, the features including a person, a behavior and a scene; fusing the visual feature vector and the text feature vector to obtain a fusion feature; inputting the fusion feature into a multi-layer perception machine to obtain a person, behavior and scene semantic state vector; and fusing the person, behavior and scene semantic state vector to obtain a vector reflecting a semantic score, as a multi-modal composite semantic video retrieval result. The present application effectively improves the information utilization rate by introducing a multi-modal fusion method, and also improves the accuracy of the retrieval.

[0029] It should be understood that the general description above and the following detailed description are only exemplary and do not limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0030] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0031] Figure 1 A composite semantic video retrieval method flowchart provided in an embodiment of the present application;

[0032] Figure 2 A composite semantic video retrieval device schematic diagram provided in an embodiment of the present application;

[0033] Figure 3 A composite semantic fusion processing schematic diagram provided in an embodiment of the present application;

[0034] Figure 4 An electronic device schematic diagram provided in an embodiment of the present application;

[0035] Figure 5 A computer readable program medium schematic diagram provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The exemplary embodiments will be described in detail herein below with reference to the drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0037] Furthermore, the accompanying drawings are merely schematic and are not necessarily drawn to scale. Some of the blocks in the block diagrams shown represent functional entities that can be realized by one or possibly more physical or logical components and do not necessarily have to correspond to physically or logically separate entities.

[0038] The embodiment of the present application provides a video retrieval method, device and equipment based on composite semantics and a storage medium, which effectively improves the information utilization rate and the retrieval accuracy by introducing a multi-modal fusion method.

[0039] To achieve the above technical effects, the general idea of the present application is as follows:

[0040] A video retrieval method based on composite semantics, the method comprising the steps of:

[0041] S101: Extracting a visual feature vector of a video and a text feature vector of a corresponding moment, wherein the features include a person, a behavior and a scene.

[0042] S102: Fusing the visual feature vector and the text feature vector to obtain a fusion feature.

[0043] S103: Inputting the fusion feature into a multi-layer perception machine to obtain a person, a behavior and a scene semantic state vector.

[0044] S104: Fusing the person, the behavior and the scene semantic state vector to obtain a vector reflecting a semantic score, as a multi-modal composite semantic video retrieval result.

[0045] The embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0046] Referring to Figure 1 , Figure 1 Fig. 1 shows a flow chart of a video retrieval method based on composite semantics provided by the present application, and Figure 1 Fig. 1 shows a flow chart of a video retrieval method based on composite semantics provided by the present application, and

[0047] Step S101: Extracting a visual feature vector of a video and a text feature vector of a corresponding moment, wherein the features include a person, a behavior and a scene.

[0048] Specifically, the collected video is preprocessed by scene segmentation, shot segmentation and key frame extraction, the visual person feature vector, the behavior feature vector and the scene feature vector in the processed video are extracted by using a C3D model, and the person feature vector, the behavior feature vector and the scene feature vector of the corresponding moment text are extracted by using a BERT model, wherein the text includes a script and a dialogue.

[0049] For ease of understanding, the video is first pre-processed to complete scene segmentation, shot segmentation and key frame extraction. On the obtained key frames, the C3D model is used to extract visual character, behavior and scene feature vectors, and the BERT model is used to extract character, behavior and scene feature vectors of the corresponding text (including scripts and dialogues, etc.).

[0050] In an embodiment, after extracting the visual feature vectors of the video and the text feature vectors of the corresponding moment, the visual feature vectors and the text feature vectors of the corresponding moment are mapped to the corresponding d-dimensional Hilbert semantic space using a convolutional neural network, and are converted into visual state vectors and text state vectors in the common space.

[0051] It can be understood that the obtained visual and text feature vectors are simultaneously mapped to the corresponding Hilbert semantic space using CNN, and three groups (character, behavior, scene) of state vectors in the common space are obtained. It should be noted that in this process, ReLU is used as the activation function to ensure that all feature values are non-negative, where m is the dimension of the visual feature and the text feature, i.e. the state vector is obtained: Finally, the state vector is normalized where CNN represents a neural network, X represents the fusion of visual vectors, text vectors and interference terms. m can take values 1, 2, 3, corresponding to semantic vectors character, behavior, scene respectively. CNN(X) represents the output obtained after passing through the network, i.e. the semantic state vector, and ReLU is a commonly used activation function in machine learning and a unique structure in neural networks.

[0052] It should be noted that a vector is a representation in a multi-dimensional space, and the tensor product operation between two vectors can obtain a new vector containing all the information of the two old vectors. In addition, there are other operations, and different operations represent new interactions between two pieces of information. By modeling the semantic space, the content in the video is projected into the space, and the information is encoded and represented by vectors.

[0053] Feature vector: can be divided into "feature" and "vector" to understand: "feature" indicates that the vector contains unique semantic information, such as a vector for character information is called a character feature vector, and a vector containing behavior information is called a behavior feature vector.

[0054] State vector: can be understood as a mathematical tool for describing the state of a quantum system in quantum computing. Only by converting the familiar mathematical representation into the unique state vector representation of quantum computing can subsequent operations be performed according to the rules of quantum computing.

[0055] Step S102: fuse the visual feature vector and the text feature vector to obtain a fused feature.

[0056] Specifically, the visual feature and the text feature are fused by using a multimodal fusion network based on a quantum interference model. It can be understood that, according to the obtained visual state vector and text state vector, a point-by-point multiplication of vectors is used to obtain interference terms corresponding to the visual state vector and the text state vector; the visual state vector, the text state vector and the corresponding interference terms are taken as modal features, and are input into a preset multimodal fusion network for modal fusion to obtain a fused feature.

[0057] It should be noted that multimodal, namely multimedia modalities, refers to multiple different media elements, such as images, audio, video, text and the like, which can carry information of a certain size. In this article, it mainly refers to visual and text modalities.

[0058] It should be further noted that the visual state vector and the text vector are taken as inputs of the network, and each is a modal feature vector. Although the interference term does not belong to any modality, it is still taken as a modal feature vector input. It has been proved that the network input containing processed high-level information (the interference term in this article) can effectively improve the model effect.

[0059] For the convenience of understanding, an example is given. Three groups of state vectors of a common space (person, behavior and scene) are obtained, interference terms corresponding to two state vectors are obtained, a group of visual state vectors, text state vectors and corresponding interference terms are taken as a kind of modal feature, and are input into a multimodal fusion network to obtain a fused feature. It should be noted that the group here is actually sequentially processed one group by one group in the three groups (according to the point-by-point multiplication of vectors).

[0060] In an embodiment, the obtained visual and text features are taken as modal inputs, and the formula is: wherein f vision is the visual feature, f text is the text feature, f fusion is the video feature after modal fusion, is the point-by-point multiplication of vectors, and α, β, are hyperparameters.

[0061] It should be noted that the interference term is a fusion of information of two different modalities (visual and text), which is obtained by specifying operation through state vectors of two modalities, and reflects common features in the two modalities. For the convenience of understanding, an example is given. For example, the visual representation and the text representation of a video, the text representation contains the dialogue information of the characters in the video, which can be mapped to the opening and closing of the mouth of the characters, the body language of the characters, etc. in the visual representation. Take the text representation and the visual representation to do certain operations to integrate them, create a new vector representation that contains both text information and visual information, and this vector is called the interference term. It can be understood as the mouth shape information of the characters when speaking in the video, which contains visual shape and can reflect the dialogue information of the characters to some extent.

[0062] Step S103: inputting the fusion feature into a multilayer perceptron to obtain a character, action and scene semantic state vector.

[0063] Specifically, the obtained fusion feature is input into a multilayer perceptron to obtain a vector containing video semantic information. The multilayer perceptron is composed of a hidden layer and an output layer, different semantics correspond to different multilayer perceptrons, and the mathematical expression is as follows:

[0064]

[0065]

[0066]

[0067]

[0068] m∈{character,action,scene}

[0069] t=1,2,3...

[0070] The label of the semantic m at the time t obtained by the model is as follows:

[0071]

[0072] In the above formula, X t is the fusion feature vector at the time t, is the hidden layer vector of the semantic m at the time t, is the output layer vector of the semantic m at the time t, is the normalized probability vector of the semantic m at the time t, are the hidden layer weight and bias of the semantic m, respectively, are the output layer weight and bias of the semantic m, respectively. is the cross-entropy loss for training the multilayer perceptron, is one-hot encoding of the sample semantics.

[0073] It should be noted that the semantic state vector: there are three kinds of semantics in total: characters, behaviors, and scenes, which are information formed by high-level abstraction processing on basic modal information, and the state vector is a mathematical representation in quantum computing.

[0074] Step S104: fuse the character, behavior, and scene semantic state vectors to obtain a vector reflecting semantic scores as a multi-modal composite semantic video retrieval result.

[0075] Specifically, perform tensor product operation on the character, behavior, and scene semantic state vectors two by two to obtain three combined semantic state vectors, perform multiple quantum observations on the three combined semantic state vectors to obtain three probability vectors, and perform maximum pooling layer processing on the three probability vectors to obtain a vector reflecting semantic scores as a multi-modal composite semantic video retrieval result.

[0076] It can be understood that the three obtained semantic state vectors are subjected to tensor product operation two by two to obtain three new combined semantic (character-behavior, character-scene, behavior-scene) state vectors, multiple quantum observations are performed on the three obtained combined semantic state vectors to obtain three probability vectors, the three obtained probability vectors are arranged into a matrix, and a final vector reflecting semantic scores is obtained through row maximum pooling layer.

[0077] For the convenience of understanding and illustration, the semantic fusion of the obtained different semantic state vectors (character, behavior, and scene semantic state vectors) first takes the three semantic state vectors two by two to perform tensor product operation defined in quantum computing to obtain three new combined semantic state vectors. Secondly, multiple quantum observation operations are performed on the three new combined semantic state vectors: according to the final possible semantic combination encoding basis vector |φ k >, respectively obtaining the probability of the combined semantic state vector collapsing to the basis vector, arranging it into a k×3 matrix; then the matrix is subjected to row maximum pooling layer to obtain a final vector reflecting semantic scores.

[0078] According to the formula: W(|φ k >) = max{P mn (|φ k >)}, calculate the final semantic score, where |ψ n > and |ψ n > represent the mth and nth semantic state vectors, respectively, and |ψ mn > represents the combination of the mth semantic state vector and the nth semantic state vector, represents tensor product operation, and P mn (|φ krepresents the combined state vector |Ψ mn collapses to the basis vector |Φ k , W(|Φ k ) represents the semantic score of the corresponding basic state, wherein (m≠n).

[0079] It should be noted that the score vector represents the semantic information corresponding to each video, and the user inputs some query statements when performing video retrieval, and the statements contain semantic information. By projecting the user query into the set semantic space, the "distance" between the user query vector and the score vector of each video is calculated, and an initial ranking result can be obtained. The main contribution is that the video representation vector contains more information and is interpretable, which can improve the final ranking result. It can be understood that by modeling a more complex semantic space to represent video content, the user's query input is projected into the space modeled, and the distance between the query input and each video vector is calculated to accurately retrieve the video required by the user.

[0080] It can be understood that, compared with the prior art, the advantages of the present application include: introducing a multi-modal fusion method for composite semantic video retrieval problems; based on the related theory of quantum computing, improvements have been made in multi-modal fusion and composite semantic fusion problems, effectively improving the utilization rate of information and further improving the retrieval accuracy.

[0081] Referring to Figure 2 , Figure 2 Fig. 1 is a schematic diagram of a composite semantic video retrieval device provided by the present application, as shown in Figure 2 , the device comprises:

[0082] The extraction unit 201 is used to extract the visual feature vector of the video and the text feature vector corresponding to the moment, and the features include characters, behaviors and scenes.

[0083] The fusion unit 202 is used to fuse the visual feature vector and the text feature vector to obtain the fusion feature.

[0084] The processing unit 203 is used to input the fusion feature into a multi-layer perception machine to obtain a character, behavior and scene semantic state vector.

[0085] The calculation unit 204 is used to perform semantic fusion on the character, behavior and scene semantic state vectors to obtain a vector reflecting the semantic score as a multi-modal composite semantic video retrieval result.

[0086] Further, in a possible implementation, the calculation unit 204 is further used to perform tensor product operation on the character, behavior and scene semantic state vectors two by two to obtain three combined semantic state vectors;

[0087] The three combined semantic state vectors are quantum observed to obtain probability vectors;

[0088] The three probability vectors are processed by a max-pooling layer to obtain a vector reflecting semantic scores, as a multimodal composite semantic video retrieval result.

[0089] Further, in a possible implementation, the computing unit 204 is further configured to calculate the final semantic score according to a formula: W(|φ k >) = max{P mn (|φ k >)}, where |ψ m > and |ψ n > represent the mth and nth semantic state vectors respectively, and |ψ mn > represents a combination of the mth semantic state vector and the nth semantic state vector, represents a tensor product operation, P mn (|φ k >) represents a probability that the combined state vector |ψ mn > collapses to the basic vector |φ k >, and W(|φ k >) represents a semantic score of the corresponding basic state.

[0090] Further, in a possible implementation, the fusion unit 202 is further configured to obtain, according to the obtained visual state vector and the text state vector, an interference term corresponding to the visual state vector and the text state vector by using point-by-point multiplication of vectors.

[0091] The visual state vector, the text state vector, and the corresponding interference term are input into a preset multimodal fusion network as modal features to perform modal fusion, to obtain a fusion feature.

[0092] Further, in a possible implementation, the computing unit 204 is further configured to calculate the video feature after modal fusion according to a formula f vision = f text ⊙ f fusion , where f vision is the visual feature, f text is the text feature, f fusion is the video feature after modal fusion, and is point-by-point multiplication of vectors, and α, β, are hyperparameters.

[0093] Further, in a possible implementation, the extraction unit 201 is further configured to perform preprocessing on the video, where the preprocessing includes scene segmentation, shot segmentation, and key frame extraction.

[0094] The visual character feature vector, the behavior feature vector and the scene feature vector in the processed video are extracted by using the C3D model, and the character feature vector, the behavior feature vector and the scene feature vector of the corresponding moment text are extracted by using the BERT model, wherein the text includes a script and a dialogue.

[0095] Further, in a possible implementation, the extraction unit 201 further includes a conversion unit configured to map the visual feature vector and the text feature vector of the corresponding moment to corresponding d-dimensional Hilbert semantic spaces by using a convolutional neural network, and convert the visual state vector and the text state vector in the common space.

[0096] Referring to Figure 3 , Figure 3 Fig. 1 shows a composite semantic fusion processing schematic provided by the present application, as shown in Figure 3

[0097] First, the video is preprocessed to complete scene segmentation, shot segmentation and key frame extraction. The visual character, behavior and scene feature vectors are extracted from the obtained key frames by using the C3D model, and the character, behavior and scene feature vectors of the corresponding moment text (including a script and a dialogue, etc.) are extracted by using the BERT model.

[0098] According to the obtained visual features and text features, the interference terms corresponding to the two state vectors are obtained by point-by-point multiplication of vectors. The obtained visual features and text features and the interference terms corresponding to the two are respectively regarded as a kind of modal feature, input into a multi-modal fusion network to obtain a fusion feature. It should be noted that the obtained visual and text feature vectors are simultaneously mapped to corresponding Hilbert semantic spaces by using a CNN to obtain three groups (characters, behaviors and scenes) of state vectors in the common space. A group of visual state vectors, text state vectors and corresponding interference terms are respectively regarded as a kind of modal feature, input into a multi-modal fusion network to obtain a fusion feature. It can be understood that one group here is actually sequentially processed one group by one group in the three groups. (According to the point-by-point multiplication of vectors, the interference terms corresponding to the two state vectors are obtained. The visual state vectors, text state vectors and corresponding interference terms are respectively regarded as a kind of modal feature, input into a multi-modal fusion network.) The obtained fusion feature is input into a multi-layer perceptron to obtain three (characters, behaviors and scenes) vectors containing video semantic information, i.e. character semantic state vectors, behavior semantic state vectors and scene semantic state vectors. The obtained character semantic state vectors, behavior semantic state vectors and scene semantic state vectors are subjected to tensor product operation two by two to obtain three new combined semantic (character-behavior, character-scene and behavior-scene) state vectors. The three new combined semantic state vectors are subjected to multiple quantum observation operations: according to the final possible semantic combination encoding basis vector |φ k ​The probabilities of the combination semantic state vectors collapsing to the base vectors are obtained respectively, and the probabilities are listed into a k*3 matrix, and then the matrix is subjected to a row max-pooling layer to obtain a vector finally reflecting the semantic score.

[0099] It can be understood that the video is preprocessed to extract visual features and text features corresponding to the segments, and a final score vector is obtained by using a multi-modal fusion model based on quantum interference and a complex semantic fusion model based on a quantum many-body system, and the final score vector is applied to retrieval to improve retrieval accuracy.

[0100] The electronic device 400 according to this embodiment of the present application will be described below with reference to Figure 4 Figure 4 The display electronic device 400 is only an example and should not limit the functions and use range of the embodiments of the present application.

[0101] As shown in Figure 4 The electronic device 400 is in the form of a general computing device. The components of the electronic device 400 can include, but are not limited to, the above-mentioned at least one processing unit 410, the above-mentioned at least one storage unit 420, and a bus 430 connecting different system components, including the storage unit 420 and the processing unit 410.

[0102] The storage unit stores program codes which can be executed by the processing unit 410, so that the processing unit 410 performs the steps according to various exemplary embodiments of the present application described in the above "embodiment method" part of the specification.

[0103] The storage unit 420 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 421 and / or a cache memory 422, and can further include a read-only memory (ROM) 423.

[0104] The storage unit 420 can further include program / utilities 424 having a set of (at least one) program modules 425, such as an operating system, one or more application programs, other program modules, and program data, each of which or some combination of which can include implementation of a network environment.

[0105] The bus 430 can represent one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0106] ​The electronic device 400 can also communicate with one or more external devices such as a keyboard or a pointing device, a Bluetooth device, etc., and can also communicate with one or more devices that enable a user to interact with the electronic device 400 and / or one or more devices that enable the electronic device 400 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface 450. Also, the electronic device 400 can communicate with one or more networks such as a local area network (LAN), a wide area network (WAN), and / or the Internet via the network adapter 460. As depicted, the network adapter 460 communicates with the other components of the electronic device 400 via the bus 430. It should be appreciated that although not shown, other hardware and / or software modules could be used in connection with the electronic device 400. These include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0107] Those skilled in the art will readily understand that the example embodiments described herein can be implemented by software and / or by software in combination with the requisite hardware. Thus, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in an non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0108] According to the solutions of the present disclosure, a computer readable storage medium is also provided, which stores the program product capable of implementing the above-mentioned methods. In some possible embodiments, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program codes for causing the terminal device to perform the steps according to various example embodiments of the present disclosure described in the above-mentioned “example method” section when the program product is run on the terminal device.

[0109] Reference Figure 5 As shown, the program product 500 for implementing the above-mentioned methods according to the embodiments of the present disclosure is described, which can adopt a portable compact disc read-only memory (CD-ROM) and includes program codes, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or apparatus.

[0110] The program product can employ any combination of one or more computer-readable media. The computer-readable media can be a computer-readable storage medium or a computer-readable signal medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] The computer-readable signal medium can include a computer-readable storage medium that is propagated as a carrier wave. The computer-readable signal medium can further be any computer-readable medium that is not a storage medium. The computer-readable signal medium can be a computer-readable storage medium that is a propagated signal on a computer-readable storage medium.

[0112] The program code embodied on the computer-readable media can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0113] The program code can be executed by one or more programmable processors, which can be implemented in one or more computer systems. In this context, a computer system generally includes a plurality of these programmable processors, which work in concert to perform a task. Additionally, the program code can be downloaded from an external source, including the internet, through a computer network, or through a broadcast medium.

[0114] Furthermore, the above-described diagrams are merely schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is readily understood that the processes shown in the above-described diagrams do not indicate or limit the time sequence of the processes. In addition, it is readily understood that the processes can be executed synchronously or asynchronously, for example, in a plurality of modules.

[0115] To sum up, the application provides a composite semantic video retrieval method, device, equipment and storage medium, wherein the method comprises the steps of extracting a visual feature vector of a video and a text feature vector of a corresponding moment, the features including a person, a behavior and a scene; fusing the visual feature vector and the text feature vector to obtain a fusion feature; inputting the fusion feature into a multilayer perception machine to obtain a person, a behavior and a scene semantic state vector; and performing semantic fusion on the person, the behavior and the scene semantic state vector to obtain a vector reflecting a semantic score, which is used as a multimodal composite semantic video retrieval result. The application effectively improves the information utilization rate and improves the retrieval accuracy by introducing a multimodal fusion method.

[0116] The above description is merely a specific implementation of the application, which enables those skilled in the art to understand or implement the application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0117] The application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a machine that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The device that implements the functions specified in one block or multiple blocks.

Claims

1. A video retrieval method with complex semantics, characterized in that, include: Extract the visual feature vector and the corresponding text feature vector of the video, wherein the features include people, behavior and scene; The fused feature is obtained by fusing visual feature vectors and text feature vectors; The fused features are input into a multilayer perceptron to obtain semantic state vectors for people, behaviors, and scenes. Semantic fusion is performed on the semantic state vectors of people, behaviors and scenes to obtain a vector reflecting the semantic score, which is used as the result of multimodal composite semantic video retrieval. Among them, tensor product operations are performed on each pair of semantic state vectors of characters, behaviors and scenes to obtain three combined semantic state vectors; Quantum observations are performed on the three combined semantic state vectors to obtain probability vectors; The three probability vectors are processed by a max pooling layer to obtain a vector reflecting the semantic score, which is used as the result of multimodal composite semantic video retrieval. According to the formula: ; Calculate the final semantic score, where These represent the semantic state vectors of class m and class n, respectively. This represents the combination of the m-th semantic state vector and the n-th semantic state vector. This represents the tensor product operation. This represents the combined state vector. Collapse to basic vectors The probability, The semantic score represents the corresponding basic state.

2. The method according to claim 1, characterized in that, The process of fusing visual feature vectors and text feature vectors to obtain fused features includes: Based on the obtained visual state vector and text state vector, the interference terms corresponding to the visual state vector and text state vector are obtained by point-by-point multiplication of the vectors. The visual state vector, text state vector, and corresponding interference terms are used as modal features and input into a preset multimodal fusion network for modal fusion to obtain fused features.

3. The method according to claim 2, characterized in that: According to the formula Calculate the video features after modal fusion, where As a visual feature, For text features, For video features after modal fusion, For point-by-point multiplication of vectors, , , It's a hyperparameter.

4. The method according to claim 1, characterized in that, The extraction of visual feature vectors from the video and text feature vectors at corresponding times includes: The video is preprocessed, including scene segmentation, shot segmentation, and keyframe extraction. The C3D model is used to extract visual character feature vectors, behavioral feature vectors, and scene feature vectors from the processed video. The BERT model is used to extract character feature vectors, behavioral feature vectors, and scene feature vectors from the text at the corresponding time point. The text includes scripts and lines.

5. The method according to claim 1, characterized in that, After extracting the visual feature vector of the video and the text feature vector at the corresponding time point, the process includes: A convolutional neural network is used to map visual feature vectors and corresponding text feature vectors at different times to the corresponding d-dimensional Hilbert semantic space, transforming them into visual state vectors and text state vectors in a common space.

6. A video retrieval device with composite semantics, characterized in that, include: An extraction unit is used to extract visual feature vectors and corresponding text feature vectors from the video, the features including people, behaviors, and scenes; The fusion unit is used to fuse visual feature vectors and text feature vectors to obtain fused features; The processing unit is used to input the fused features into a multilayer perceptron to obtain semantic state vectors of people, behaviors and scenes; The computing unit is used to semantically fuse the semantic state vectors of people, behaviors and scenes to obtain a vector reflecting the semantic score, which serves as the result of multimodal composite semantic video retrieval. Perform tensor product operations on each pair of character, behavior, and scene semantic state vectors to obtain three combined semantic state vectors; Quantum observations are performed on the three combined semantic state vectors to obtain probability vectors; The three probability vectors are processed by a max pooling layer to obtain a vector reflecting the semantic score, which is used as the result of multimodal composite semantic video retrieval. According to the formula: ; Calculate the final semantic score, where These represent the semantic state vectors of class m and class n, respectively. This represents the combination of the m-th semantic state vector and the n-th semantic state vector. This represents the tensor product operation. This represents the combined state vector. Collapse to basic vectors The probability, The semantic score represents the corresponding basic state.

7. An electronic device, characterized in that, The electronic device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores computer program instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 5.

Images