A multimedia content retrieval method, device, equipment and storage medium

Abstract

The present application discloses a multimedia content retrieval method, device, equipment and storage medium, and relates to the field of intelligent search technology. The specific implementation scheme is: obtaining a query vector of query information; determining a first category matching the query vector from a search library, wherein the search library includes multiple categories, each category includes multiple vectors, and each vector is associated with corresponding multimedia content; querying a target vector matching the query vector in the first category, and obtaining the multimedia content corresponding to the target vector. The present application improves retrieval efficiency.

Description

Technical Field

[0001] The present application relates to the field of intelligent search technology in the field of computer technology, and in particular to a multimedia content retrieval method, device, equipment and storage medium. Background Art

[0002] In some scenarios, it is often necessary to search to obtain relevant multimedia content, for example, searching for pictures to match advertisements. However, the current search process is to segment the query information (for example, the description text of advertisements and pictures) and perform preliminary screening. The multimedia content features are then extracted from the screened results to establish a feature database of multimedia content. After that, similarity calculations are performed using the query information features and multimedia content features, which is equivalent to brute force retrieval in the global space. This consumes a lot of time and results in low retrieval efficiency. Summary of the invention

[0003] The present application provides a multimedia content retrieval method, apparatus, device and storage medium to solve the problem of low retrieval efficiency.

[0004] According to a first aspect, a multimedia content retrieval method is provided, comprising:

[0005] Obtain a query vector of query information;

[0006] Determine a first category matching the query vector from a search library, wherein the search library includes a plurality of categories, each category includes a plurality of vectors, and each vector is associated with corresponding multimedia content;

[0007] A target vector matching the query vector is searched in the first category, and multimedia content corresponding to the target vector is acquired.

[0008] According to a second aspect, a multimedia content retrieval device is provided, comprising:

[0009] A first acquisition module, used to acquire a query vector of query information;

[0010] A determination module, configured to determine a first category matching the query vector from a search library, wherein the search library includes a plurality of categories, each category includes a plurality of vectors, and each vector is associated with corresponding multimedia content;

[0011] The second acquisition module is used to search the first category for a target vector matching the query vector, and acquire multimedia content corresponding to the target vector.

[0012] According to a third aspect, an electronic device is provided, characterized in that it includes:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimedia content retrieval method provided in the present application.

[0016] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the multimedia content retrieval method provided by the application.

[0017] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the method provided in the embodiment of the present application when executed by a processor.

[0018] The technical solution of the present application improves the retrieval efficiency.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0021] Figure 1 is a flowchart of a multimedia content retrieval method provided by the present application;

[0022] Figure 2 It is a schematic diagram of determining a center vector provided by the present application;

[0023] Figure 3 This is a structural diagram of a multimedia content retrieval device provided by the present application;

[0024] Figure 4 It is a block diagram of an electronic device used to implement the multimedia content retrieval method of the embodiment of the present application. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0026] See also Figure 1 , Figure 1 is a flowchart of a multimedia content retrieval method provided by this application, such as Figure 1 As shown, the following steps are included:

[0027] Step S101: Obtain a query vector of query information.

[0028] The query information may be query information input by a user, or query information automatically generated by an electronic device, and the query information may be information such as pictures, text, or voice.

[0029] The query vector may be a vector obtained by quantizing the query information, or may be a feature vector of the query information.

[0030] Step S102 determines a first category matching the query vector from a search library, wherein the search library includes a plurality of categories, each category includes a plurality of vectors, and each vector is associated with corresponding multimedia content.

[0031] The above-mentioned search library can be a local or remote search library, and the search library can be a distributed search library or a non-distributed search library.

[0032] The detection library includes multiple categories, each of which includes multiple vectors. Further, the multiple categories may be obtained by clustering multiple vectors using a clustering algorithm, for example, the multiple categories are obtained using a distributed k-means clustering (k-means) algorithm.

[0033] It should be noted that, in this application, classification can also be referred to as a coarse classifier or clustering.

[0034] The first classification matching the query vector may be one or more classifications whose vectors match the query vector, and the matching here may refer to similarity or a relatively small vector residual.

[0035] In the present application, the multimedia content associated with the vector may be that each multimedia content is associated with one or more vectors, and the vector associated with the multimedia content may be a vector obtained by quantizing the relevant information of the multimedia content, for example, a vector obtained by quantizing information such as keywords, description information, tags, picture content, names, etc. Or it may be a feature vector of the multimedia content.

[0036] In this application, multimedia content can be pictures, videos or texts, or a combination of pictures, videos and texts.

[0037] Step S103 searches the first category for a target vector that matches the query vector, and obtains multimedia content corresponding to the target vector.

[0038] The above-mentioned target vector matched with the query vector in the first category may be one or more vectors matched with the query vector in the first category, and the matching here may be similarity matching or residual matching, such as residual matching.

[0039] Preferably, the target vector may be determined in the first category using an approximate nearest neighbor search (ANN) method.

[0040] After the target vector is determined, its associated multimedia content can be determined.

[0041] The present application can realize using multimedia content association vectors as indexes of a search library, and first determine the classification and then perform the search within the classification, thereby improving the search efficiency.

[0042] In addition, the vector used in the present application can be a vector obtained by a model such as a W2V model, a BERT model or an Ernie model. Since the vector is used for detection, the similarity calculation is more accurate, further improving the recall accuracy. Furthermore, since multimedia content is retrieved through the vector, the inverted index can be used during retrieval to avoid brute force calculation and recall the final multimedia content.

[0043] It should be noted that the present application can be applied to advertisement configuration scenarios. Of course, there is no limitation to this and the present application can also be applied to other multimedia content search scenarios.

[0044] Furthermore, the above method can be applied to electronic devices, such as servers, computers, mobile phones and other electronic devices.

[0045] As an optional implementation manner, determining the first classification matching the query vector from the search library includes:

[0046] The residuals of the central point vectors of multiple categories in the search library and the query vector are calculated respectively, and the first category is determined according to the residuals, wherein each category in the search library includes a central point vector.

[0047] The above-mentioned first classification can be one or more classifications with the smallest residual among multiple classifications, or the first N classifications in the sorting according to the residual. Further, the above-mentioned residual can also be understood as the distance between the center point vector and the query vector, for example: symmetric distance (SDC) or asymmetric distance (ADC). Further, the above-mentioned first classification can be to take the first N classifications by local rotation of the residual. And the modulus of the rotation vector used for local rotation can be 1, so that when sorting using ADC, the first N classifications after rotation are taken for ADC sorting, which is equivalent to the original order.

[0048] The smallest one or more categories, or the top N categories sorted by residuals.

[0049] In this implementation, since only the residual between the query vector and the center point vector is calculated, the first classification can be determined, thereby reducing the amount of calculation to further improve the retrieval efficiency.

[0050] Optionally, the center point vector of the target classification of the retrieval library is determined based on the sum of all vectors included in the target classification and / or the number of vectors included in the target classification, wherein the target classification is any classification in the retrieval library.

[0051] Among them, the above-mentioned determination based on the sum of all vectors included in the target classification and / or the number of vectors included in the target classification can be the change of the center point vector determined according to the sum of all vectors included in the target classification and / or the number of vectors included in the target classification during the iterative process of classification, for example: the residual of the distance. For example: a center point vector is selected for each classification in advance, and then the change of the center point vector is calculated according to the sum of all vectors included in the classification and the number of vectors included in the classification during the iterative process of classification, and then a new center point vector is determined. In addition, the distance between the vector and the center point can be that each vector is divided into several segments (the center point vector also has the same operation), and then the sum of the distances between each segment and the corresponding segment of the center point vector is the distance from the vector to the center point vector. In this way, the vectors of different segments can be encoded separately to improve the retrieval efficiency.

[0052] like Figure 2As shown, the table (e.g., elf_table) stores the description information of the classification center point, including the id of the classification center point, the sum of all vectors in the classification, and the number of vectors in the classification. In a distributed system, each node independently obtains all the classification center points stored in the table, and calculates each center point (or cluster center point) by the sum of all vectors in the classification and / or the number of vectors. When calculating a new center point, the center point changes of all vectors of this node are calculated in turn. Here, only the change in the center point is calculated without recalculating the center point. First, it avoids the time overhead of vector serialization, deserialization, and merging each time the center point is recalculated. Second, it is convenient to record how many vectors are changing at the center point in each iteration, and use this as a condition to determine whether the algorithm has converged.

[0053] In addition, the center point vector of the target classification of the above retrieval library can also determine the vector located at the center of the classification based on the sum of all vectors included in the target classification and / or the number of vectors included in the target classification.

[0054] In this implementation, since the center point vector is determined based on the sum of all vectors included in the target classification and the number of vectors included in the target classification, it can be ensured that the center point vector is updated in time during iteration.

[0055] It should be noted that the present application is not limited to determining the first category by the classification center point vector. For example, the first category may also be determined based on other contents of each category, such as feature information of the category.

[0056] As an optional implementation manner, the step of searching the first category for a target vector matching the query vector and acquiring multimedia content corresponding to the target vector includes:

[0057] Calculate the residual between the query vector and multiple sub-center vectors (or sub-center point vectors) in the first category, and use N sub-center vectors of the multiple sub-center vectors as target vectors matching the query vector based on the residual, and obtain multimedia content corresponding to the target vector, where N is an integer greater than or equal to 1.

[0058] In this implementation, each category includes a center point vector and multiple sub-center point vectors, that is, one category includes two levels of center point vectors. In addition, each center point vector may correspond to one or more vectors.

[0059] Similarly, the above residual can also be understood as the distance between the secondary center point vector and the query vector, for example: symmetric distance (SDC) or asymmetric distance (ADC).

[0060] The above-mentioned taking N sub-center vectors among the multiple sub-center vectors as target vectors matching the query vector according to the residual may be, according to the residual, sorting the multiple sub-center vectors according to the residual and taking the first N sub-center vectors.

[0061] In this implementation, since the target vector is determined by the residual, the target vector can be accurately determined and the amount of calculation can be reduced.

[0062] Furthermore, when calculating the distance between the query vector and the sub-center vector (or the vector to be retrieved), the asymmetric distance (ADC) is preferably used to complete the measurement. For example, for the query vector x and the sub-center vector y, the distance between x and y is replaced by the quantized representation q(y) of x and y. Since the number of q(y) after quantization is only related to the number of sub-quantizers, x and q(y) can be represented as a lookup table to achieve a faster query speed.

[0063] It should be noted that the present application is not limited to determining the target vector by the classified sub-center point vector. For example, the sub-center vector may not be calculated, and the residual between each classified vector and the query vector may be directly calculated to determine the final target vector.

[0064] As an optional implementation, the query vector is a vector obtained by rotating and quantizing the query information vector; the vector in the retrieval library is a rotated and quantized vector, and before the rotation, the vector is subjected to vector centralization processing through the center point vector of the classification center.

[0065] The query vector is a vector obtained by rotating and quantizing the query information vector, and the query information vector may be obtained by rotating the query information vector by a rotation vector and then performing product quantization (PQ) to obtain the query vector. The rotation vector may be a vector with a modulus of 1.

[0066] The vectors in the above retrieval library can be obtained through the following process:

[0067] Vector centering, specifically, using the classification center vector to go through all vectors in turn to center each vector;

[0068] To calculate the covariance matrix, you can first calculate it in each node, add up the inner product values ​​of the vectors under the same classification, and update them to a new table in the retrieval library. Then each node gets the final covariance estimate and calculates the covariance matrix.

[0069] Perform eigenvalue decomposition on the matrix to obtain eigenvalues ​​and eigenvectors, and construct the transposed matrix R, which is the matrix used for the above rotation;

[0070] Use the transposed matrix R to project all vectors under this category into the new space, that is, multiply the vector by the transposed matrix R;

[0071] PQ quantization is performed on the vectors in the new space to obtain the vectors in the retrieval library.

[0072] Among them, the above covariance matrix can be expressed as follows

[0073]

[0074] Among them, cov represents the above covariance matrix, c is the sum of the inner products of all vectors in the classification, n is the number of vectors in the classification, and v is the center point vector of the classification.

[0075] It should be noted that different categories will not be able to calculate different covariance matrices, and thus complete different rotations. The above-mentioned use of the transposed matrix R to project all vectors under the category into the new space is the rotation of the vectors in the above-mentioned search library.

[0076] The above vector centering, calculation of covariance matrix and projection to the new space can be performed in a distributed manner among nodes in parallel (the granularity of parallelism is training data). In this way, the number of computing nodes can be increased to support the index construction of large-scale data sets, thereby completing index construction and retrieval matching in a shorter time.

[0077] Each node obtains the final covariance estimate and calculates the covariance matrix, and constructs the transposed matrix R. This can be done concurrently by calculating only a portion of the classification center point vector on each node.

[0078] It should be noted that the vectors in the above retrieval library are all pre-configured, for example: each vector is determined during the training process.

[0079] In this implementation, since the query vector is a vector obtained by rotating and quantizing the query information vector, and the vectors in the search library are rotated and quantized vectors, and the vectors are subjected to vector centering processing by the center point vector of the classification center before rotation, local optimization (or local optimization) processing of the vectors can be achieved, thereby further reducing the amount of calculation and further improving the search efficiency.

[0080] In addition, the vectors in the search library in the present application can also be directly obtained through the Local Optimal Product Quantization (LOPQ) algorithm.

[0081] In addition, the multimedia content in the retrieval library in the present application can be expanded by the user ID of the multimedia content before the retrieval library is classified, so as to obtain multimedia content of more users, for example: to obtain advertising pictures of more users, so as to optimize the multimedia content scene, and achieve the retrieval with good scalability and versatility. That is, before classification, a layer is added to obtain more multimedia content scenes. After that, classification can be carried out according to the constraints, and then a separate LOPQ recall can be performed within the classification. This changes the method of recalling first and then screening, and improves the recall rate under the constraint scenario.

[0082] The present application can realize using multimedia content association vectors as indexes of a search library, and first determine the classification and then perform the search within the classification, thereby improving the search efficiency.

[0083] See also Figure 3 , Figure 3 is a structural diagram of a multimedia content retrieval device provided by this application, such as Figure 3 As shown, the multimedia content retrieval device 300 includes:

[0084] A first acquisition module 301 is used to acquire a query vector of query information;

[0085] A determination module 302 is used to determine a first category matching the query vector from a search library, wherein the search library includes a plurality of categories, each category includes a plurality of vectors, and each vector is associated with corresponding multimedia content;

[0086] The second acquisition module 303 is configured to search the first category for a target vector that matches the query vector, and acquire multimedia content corresponding to the target vector.

[0087] Optionally, the determination module 302 is used to respectively calculate the center point vectors of multiple categories in the search library and the residuals of the query vector, and determine the first category based on the residuals, wherein each category in the search library includes a center point vector.

[0088] Optionally, the center point vector of the target classification of the retrieval library is determined based on the sum of all vectors included in the target classification and / or the number of vectors included in the target classification, wherein the target classification is any classification in the retrieval library.

[0089] Optionally, the second acquisition module 303 is used to calculate the residual between the query vector and multiple sub-center vectors in the first category, and based on the residual, use N sub-center vectors among the multiple sub-center vectors as target vectors matching the query vector, and obtain multimedia content corresponding to the target vector, where N is an integer greater than or equal to 1.

[0090] Optionally, the query vector is a vector obtained by rotating and quantizing the vector of the query information; the vector in the retrieval library is a vector that has been rotated and quantized, and has been subjected to vector centralization processing through the center point vector of the classification center before rotation.

[0091] The device provided in this embodiment can implement each process implemented in the method embodiment of the present application, and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0092] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0093] like Figure 4 , is a block diagram of an electronic device according to a multimedia content retrieval method according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0094] like Figure 4 As shown, the electronic device includes: one or more processors 401, a memory 402, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to an interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 401 is taken as an example.

[0095] The memory 402 is a non-transient computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the multimedia content retrieval method provided in the present application. The non-transient computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the multimedia content retrieval method provided in the present application.

[0096] The memory 402 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the multimedia content retrieval method in the embodiment of the present application (for example, the attached Figure 3 The processor 401 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 402, that is, the multimedia content retrieval method in the above method embodiment is implemented.

[0097] The memory 402 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the electronic device of the multimedia content retrieval method, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 402 may optionally include a memory remotely arranged relative to the processor 401, and these remote memories may be connected to the electronic device of the multimedia content retrieval method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0098] The electronic device of the multimedia content retrieval method may further include: an input device 403 and an output device 404. The processor 401, the memory 402, the input device 403 and the output device 404 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0099] The input device 403 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device of the multimedia content retrieval method, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 404 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0100] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0101] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0102] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0103] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0104] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0105] According to the technical solution of the embodiment of the present application, since the candidate topic concepts are filtered by their part-of-speech sequences, the manpower cost required for topic concept mining is reduced.

[0106] The present application provides a computer program product, including a computer program, which implements the method provided in the embodiment of the present application when executed by a processor.

[0107] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0108] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A multimedia content retrieval method, It is characterized in that include: Obtaining a query vector of query information, where the query information is text or voice; Determine a first category matching the query vector from a search library, wherein the search library includes a plurality of categories, each category includes a plurality of vectors, and each vector is associated with corresponding multimedia content; Searching the first category for a target vector matching the query vector, and acquiring multimedia content corresponding to the target vector; The query vector is a vector obtained by rotating a vector obtained by quantizing the query information by a rotation vector and then performing product quantization, and the rotation vector is a vector with a modulus of 1; The vectors in the retrieval library are obtained through the following process: Vector centralization, wherein the vector centralization refers to using the classification center vector to sequentially pass through all vectors in the classification to perform centralization processing on each vector, wherein the vectors in the classification are vectors obtained by quantizing relevant information of the multimedia content, wherein the relevant information is a keyword, description information, label or name; Calculating a covariance matrix, wherein calculating the covariance matrix refers to accumulating inner product values ​​of the vectors under the classification after the vectors are centered, and calculating the covariance matrix based on the accumulated sum of the inner product values; Performing eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors, and constructing a transposed matrix; Projecting all vectors under the classification after the vector centering to a new space using the transposed matrix; The vectors in the new space are product-quantized to obtain the vectors in the retrieval library.

2. The method according to claim 1, It is characterized in that The determining a first classification matching the query vector from a search library includes: The residuals of the central point vectors of multiple categories in the search library and the query vector are calculated respectively, and the first category is determined according to the residuals, wherein each category in the search library includes a central point vector.

3. The method according to claim 2, It is characterized in that The center point vector of the target classification of the retrieval library is determined according to the sum of all vectors included in the target classification and / or the number of vectors included in the target classification, wherein the target classification is any classification in the retrieval library.

4. The method according to claim 1, It is characterized in that The step of searching the first category for a target vector matching the query vector and acquiring multimedia content corresponding to the target vector includes: Calculate the residual between the query vector and multiple sub-center vectors in the first category, and use N sub-center vectors of the multiple sub-center vectors as target vectors matching the query vector based on the residual, and obtain multimedia content corresponding to the target vector, where N is an integer greater than or equal to 1.

5. The method according to any one of claims 1 to 4, It is characterized in that The query vector is a vector obtained by rotating and quantizing the vector of the query information; the vector in the retrieval library is a vector that has been rotated and quantized, and has been subjected to vector centralization processing through the center point vector of the classification center before rotation.

6. A multimedia content retrieval device, It is characterized in that include: A first acquisition module, used to acquire a query vector of query information, where the query information is text or voice; A determination module, configured to determine a first category matching the query vector from a search library, wherein the search library includes a plurality of categories, each category includes a plurality of vectors, and each vector is associated with corresponding multimedia content; A second acquisition module, configured to search the first category for a target vector matching the query vector, and acquire multimedia content corresponding to the target vector; The query vector is a vector obtained by rotating a vector obtained by quantizing the query information by a rotation vector and then performing product quantization, and the rotation vector is a vector with a modulus of 1; The vectors in the retrieval library are obtained through the following process: Vector centralization, wherein the vector centralization refers to using the classification center vector to sequentially pass through all vectors in the classification to perform centralization processing on each vector, wherein the vectors in the classification are vectors obtained by quantizing relevant information of the multimedia content, wherein the relevant information is a keyword, description information, label or name; Calculating a covariance matrix, wherein calculating the covariance matrix refers to accumulating inner product values ​​of the vectors under the classification after the vectors are centered, and calculating the covariance matrix based on the accumulated sum of the inner product values; Performing eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors, and constructing a transposed matrix; Projecting all vectors under the classification after the vector centering to a new space using the transposed matrix; The vectors in the new space are product-quantized to obtain the vectors in the retrieval library.

7. The device according to claim 6, It is characterized in that The determination module is used to respectively calculate the center point vectors of multiple categories in the search library and the residuals of the query vector, and determine the first category based on the residuals, wherein each category in the search library includes a center point vector.

8. The device according to claim 7, It is characterized in that The center point vector of the target classification of the retrieval library is determined according to the sum of all vectors included in the target classification and / or the number of vectors included in the target classification, wherein the target classification is any classification in the retrieval library.

9. The device according to claim 6, It is characterized in that The second acquisition module is used to calculate the residual between the query vector and multiple sub-center vectors in the first category, and based on the residual, use N sub-center vectors among the multiple sub-center vectors as target vectors matching the query vector, and obtain multimedia content corresponding to the target vector, where N is an integer greater than or equal to 1.

10. The device according to any one of claims 6 to 9, It is characterized in that The query vector is a vector obtained by rotating and quantizing the vector of the query information; the vector in the retrieval library is a vector that has been rotated and quantized, and has been subjected to vector centralization processing through the center point vector of the classification center before rotation.

11. An electronic device, It is characterized in that include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, It is characterized in that The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images