Multimedia data processing method, related equipment and storage medium

Abstract

The embodiment of the invention discloses a multimedia data processing method, a multimedia data processing device, electronic equipment, a computer readable medium and a computer program product, the method is applied to a GPU database, a feature retrieval function is preset in the GPU database, and the method comprises the following steps: receiving a multimedia data retrieval request sent by a retrieval client, the multimedia data retrieval request carries information of multimedia data to be retrieved; a retrieval statement is generated based on the multimedia data retrieval request, and the retrieval statement comprises an identifier of a specified feature retrieval function, an identifier of a first feature table and an identifier of a second feature table; and executing the retrieval statement to call a data retrieval code corresponding to the specified feature retrieval function, and performing similarity retrieval processing on the first feature table and the second feature table to obtain a corresponding retrieval result. According to the technical scheme, the retrieval efficiency of the multimedia data can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer and communication technology, and in particular to a multimedia data processing method, a multimedia data processing device, an electronic device, a computer storage medium, and a computer program product. Background Art

[0002] With the popularization of smart devices and the rapid development of the Internet, multimedia data has shown explosive growth; at the same time, the demand for multimedia data retrieval is also growing. In related technologies, corresponding index databases or search engines are usually built to store, index and retrieve multimedia data according to retrieval needs, but the retrieval efficiency is low and the cost is high. Therefore, how to improve the retrieval efficiency of multimedia data is an urgent problem to be solved. Summary of the invention

[0003] The embodiments of the present application provide a multimedia data processing method, a multimedia data processing device, an electronic device, a computer storage medium and a computer program product, which can improve the retrieval efficiency of multimedia data.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] In a first aspect, an embodiment of the present application provides a multimedia data processing method, which is applied to a GPU database, wherein a feature retrieval function is preset in the GPU database, and the method comprises:

[0006] Receiving a multimedia data retrieval request sent by a retrieval client, wherein the multimedia data retrieval request carries information of multimedia data to be retrieved;

[0007] Generate a search statement based on the multimedia data search request, wherein the search statement includes an identifier of a specified feature search function, an identifier of a first feature table in the GPU database for storing feature vectors of the multimedia data to be searched, and an identifier of a second feature table in the GPU database for storing feature vectors of reference multimedia data;

[0008] The search statement is executed to call the data search code corresponding to the specified feature search function, perform similarity search processing on the first feature table and the second feature table, and obtain corresponding search results.

[0009] In a second aspect, an embodiment of the present application provides a multimedia data processing device, wherein the multimedia data processing device is configured in a GPU database, wherein a feature retrieval function is preset in the GPU database, and the multimedia data processing device includes a receiving unit, a generating unit, and an executing unit, wherein:

[0010] The receiving unit is used to receive a multimedia data retrieval request sent by a retrieval client, wherein the multimedia data retrieval request carries information of the multimedia data to be retrieved;

[0011] The generating unit is configured to generate a search statement based on the multimedia data search request, wherein the search statement includes an identifier of a specified feature search function, an identifier of a first feature table in the GPU database for storing feature vectors of the multimedia data to be searched, and an identifier of a second feature table in the GPU database for storing feature vectors of reference multimedia data;

[0012] The execution unit is used to execute the search statement to call the data search code corresponding to the specified feature search function, perform similarity search processing on the first feature table and the second feature table, and obtain corresponding search results.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors; a storage device for storing one or more computer programs, wherein when the one or more computer programs are executed by the one or more processors, the electronic device implements the multimedia data processing method as described above.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of an electronic device, the electronic device executes the multimedia data processing method as described above.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, wherein the computer program is stored in a computer-readable storage medium, and a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the multimedia data processing method as described above.

[0016] In the technical solution provided in the embodiment of the present application, by presetting the feature retrieval function in the GPU database, the data retrieval code corresponding to the corresponding feature retrieval function can be called during the retrieval process to realize the similarity retrieval of multimedia data at the feature vector level, which is conducive to improving the accuracy of the retrieval results. In addition, since the data volume of the feature vector is smaller than the data volume of the multimedia data itself; therefore, compared with using the data table containing multimedia data in the database for retrieval processing, the GPU database in the embodiment of the present application only needs to perform corresponding processing on the feature table with smaller data volume during the retrieval process, which is conducive to improving the data processing speed, thereby improving the retrieval efficiency of multimedia data. At the same time, the GPU database can also use the underlying GPU device to greatly improve the data processing throughput during the retrieval process, thereby matching the retrieval tasks with extremely high computing power requirements, and further improving the retrieval efficiency of multimedia data.

[0017] In addition, different feature retrieval functions in the embodiments of the present application can complete different retrieval tasks; compared to the traditional need to build a corresponding index database or search engine separately for a retrieval task, the embodiments of the present application can build a universal GPU database that can be applied to multiple retrieval tasks by presetting feature retrieval functions for different retrieval tasks in the GPU database, so that the GPU database can parse richer retrieval semantics, which is beneficial to improving the versatility of the GPU database.

[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0020] Figure 1 is a structural diagram of a multimedia data processing system provided in an embodiment of the present application;

[0021] Figure 2 It is a flowchart of a multimedia data processing method provided by an embodiment of the present application;

[0022] Figure 3 It is a schematic diagram of a multimedia data retrieval framework provided in an embodiment of the present application;

[0023] Figure 4is a flowchart of another multimedia data processing method provided in an embodiment of the present application;

[0024] Figure 5 It is a flowchart of another multimedia data processing method provided in an embodiment of the present application;

[0025] Figure 6 It is a flowchart of another multimedia data processing method provided in an embodiment of the present application;

[0026] Figure 7 It is a flowchart of another multimedia data processing method provided in an embodiment of the present application;

[0027] Figure 8 is a schematic diagram of a similar multimedia data retrieval process provided by an embodiment of the present application;

[0028] Fig. 9 is a structural block diagram of a multimedia data processing device provided in an embodiment of the present application;

[0029] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all contents and operations, nor must they be executed in the order described. For example, some operations may be decomposed, while some operations may be combined or partially combined, so the actual execution order may change according to actual conditions.

[0033] It should also be noted that the "multiple" mentioned in this application refers to two or more than two. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0034] A database management system (DBMS) is a computer software system designed for managing databases. It generally has basic functions such as storage, retrieval, security, and backup. Database management systems can be classified according to the database model they support, such as relational, XML (Extensible Markup Language); or according to the type of computer they support, such as server clusters, mobile phones; or according to the query language used, such as Structured Query Language (SQL), XQuery (XML Query, a set of standards for extracting information from XML-like (a subset of Standard Generalized Markup Language) documents); or according to performance focus, such as maximum scale, maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMS can cross categories, for example, supporting multiple query languages ​​at the same time.

[0035] A database is a collection of data that is stored together in a certain way, can be shared with multiple objects, has as little redundancy as possible, and is independent of the application program. In short, a database can be seen as an electronic filing cabinet—a place where electronic files are stored, where managers can add, query, update, delete, and other operations on the data in the files.

[0036] Among them, GPU database is an important branch of database management system. Its essence is a database management system that uses graphics processing unit (GPU) to process certain data processing functions. In a broad sense, as long as a certain function of a database management system uses GPU devices, then the database management system can be called a GPU database.

[0037] Specifically, the mainstream GPU database form is to use GPU devices to execute SQL queries. The operators involved in a SQL statement can be deployed on the GPU to accelerate query processing. At the same time, the use of GPU can complete all or most database relational operations and can analyze more than 10TB of data. In addition, the GPU database can also store all data in memory to unleash the full potential of the GPU and increase the data processing speed. Specifically, the GPU can process smaller data sets of 1TB to 10TB.

[0038] Based on this, an embodiment of the present application provides a multimedia data processing solution applied to a GPU database, which presets a feature retrieval function in the GPU database. After the GPU database receives a multimedia data retrieval request sent by a retrieval client, a retrieval statement can be generated and executed based on the multimedia data retrieval request to obtain a corresponding retrieval result.

[0039] The multimedia data search request carries information about the multimedia data to be searched. Specifically, the information about the multimedia data to be searched may be the multimedia data to be searched, the data identifier of the multimedia data to be searched, or the data address of the multimedia data to be searched, or one or more other information related to the multimedia data to be searched, which is not limited here.

[0040] It should be noted that multimedia data refers to a combination of multiple media data, including multiple media forms such as text, sound, image, etc. Specifically, the multimedia data in this solution can be multimedia data containing images, such as photos, videos, etc.

[0041] In addition, the search statement includes an identifier of a specified feature search function, an identifier of a first feature table in the GPU database for storing feature vectors of multimedia data to be searched, and an identifier of a second feature table in the GPU database for storing feature vectors of reference multimedia data.

[0042] Therefore, when executing a search statement, the GPU database will call the data search code corresponding to the specified feature search function, perform similarity search processing on the first feature table and the second feature table, and thus obtain corresponding search results.

[0043] In addition, the feature retrieval function is a custom function of the GPU database, and each feature retrieval function corresponds to a corresponding data retrieval code. Specifically, although the database can complete functions such as adding, deleting, modifying and querying multimedia data, the more complex processing processes such as the calculation and comparison of feature vectors in the similarity retrieval process still need to call the corresponding data retrieval code to implement.

[0044] It is not difficult to see that this solution can call the data retrieval code corresponding to the corresponding feature retrieval function during the retrieval process by presetting the feature retrieval function in the GPU database, so as to realize the similarity retrieval of multimedia data at the feature vector level, which is beneficial to improve the accuracy of the retrieval results. In addition, since the data volume of the feature vector is smaller than the data volume of the multimedia data itself; therefore, compared with using the data table containing multimedia data in the database for retrieval processing, the GPU database in this solution only needs to perform corresponding addition, deletion, modification and query processing on the first feature table and the second feature table with smaller data volume during the retrieval process, which is beneficial to improve the data processing speed, thereby improving the retrieval efficiency of multimedia data. At the same time, the GPU database in this solution can also greatly improve the data processing throughput by using the underlying GPU device, thereby matching the retrieval tasks with extremely high computing power requirements, and further improving the retrieval efficiency of multimedia data.

[0045] Based on the above multimedia data processing solution, the present application embodiment provides a multimedia data processing system, which can be seen in Figure 1 , Figure 1 The multimedia data processing system shown may include multiple terminal devices 101 and multiple servers 102. Among them, a communication connection is established between any terminal device and any server. For example, the terminal device 101 may include any one or more of a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart car-mounted device, and a smart wearable device. Video playback clients, social clients, shopping clients, browser clients, information stream clients, etc. can be run in the terminal device 101. A GPU database is deployed in the server 102, and the server 102 can be a server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. Any terminal device 101 and any server 102 can be directly or indirectly connected in communication through wired or wireless communication, and this application is not limited here.

[0046] In some embodiments, the multimedia data processing method can be performed by Figure 1The server 102 in the multimedia data processing system shown is executed, and a GPU database is deployed in the server 102, and a feature retrieval function is preset in the GPU database. The specific execution process is: the GPU database in the server 102 can receive a multimedia data retrieval request sent by a retrieval client. Then, the GPU database in the server 102 can generate a retrieval statement based on the multimedia data retrieval request; wherein the retrieval statement includes an identifier of a specified feature retrieval function, an identifier of a first feature table in the GPU database for storing feature vectors of the multimedia data to be retrieved, and an identifier of a second feature table in the GPU database for storing feature vectors of reference multimedia data. Afterwards, the GPU database in the server 102 can execute the retrieval statement to call the data retrieval code corresponding to the specified feature retrieval function, perform similarity retrieval processing on the first feature table and the second feature table, and obtain corresponding retrieval results.

[0047] Optionally, the server 102 may also send the search result to the search client, so that the search client outputs the search result. Optionally, the above multimedia data processing method may also be performed by only Figure 1 The terminal device 101 in the multimedia data processing system shown is executed. The specific execution process can refer to the specific execution process of the server 102, which will not be repeated here.

[0048] In other embodiments, the multimedia data processing method can be run in a multimedia data processing system, which can include a terminal device and a server. Figure 1 The multimedia data processing system shown is jointly completed by the terminal device 101 and the server 102; a search client is running in the terminal device 101, and a GPU database is deployed in the server 102, and a feature search function is preset in the GPU database. The specific execution process is: the search client in the terminal device 101 can generate a multimedia data search request, and send the multimedia data search request to the server 102. After receiving the multimedia data search request, the server 102 can generate a search statement based on the multimedia data search request through the GPU database; thereafter, the GPU database in the server 102 can execute the search statement to call the data search code corresponding to the specified feature search function, perform similarity search processing on the first feature table and the second feature table, and obtain the corresponding search result.

[0049] Optionally, the server 102 may send the search result to a search client in the terminal device 101 , and the terminal device 101 may output the search result through the search client.

[0050] It should be noted that the embodiments of the present application can process various multimedia data and can be applied to various scenarios, including but not limited to cloud technology, AI (Artificial Intelligence), smart medical care, smart home, smart driving and other scenarios. It can also be used in shopping applications, social applications, video applications and other applications that can transmit or view multimedia data, without limitation.

[0051] It should be noted that in the specific implementation of this application, multimedia data is object-related. When the embodiments of this application are applied to specific products or technologies, it is necessary to obtain the permission or consent of the object, and the collection, use and processing of relevant data or information must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0052] The following is a detailed description of various implementation details of the technical solution of the embodiment of the present application:

[0053] like Figure 2 As shown, Figure 2 is a flowchart of a multimedia data processing method according to an embodiment of the present application, which can be applied to Figure 1 The multimedia data processing system shown in the figure, the method can be executed by the terminal device or the server, or can be executed by the terminal device and the server together. In the embodiment of the present application, the method is executed by the server as an example for explanation, and a GPU database is deployed in the server; wherein the multimedia data processing method may include S201 to S203, which are described in detail as follows:

[0054] S201: Receive a multimedia data retrieval request sent by a retrieval client.

[0055] In the embodiment of the present application, the multimedia data retrieval request carries information about the multimedia data to be retrieved. Specifically, the information about the multimedia data to be retrieved can be one or more information related to the multimedia data to be retrieved, such as the multimedia data to be retrieved, the data identifier of the multimedia data to be retrieved, or the data address of the multimedia data to be retrieved, which is not limited here.

[0056] In one embodiment, the information of the multimedia data to be retrieved may also include a retrieval task for the multimedia data to be retrieved; specifically, the retrieval task may be one or more of the following tasks: finding multimedia data similar to the multimedia data to be retrieved, finding multimedia data dissimilar to the multimedia data to be retrieved, detecting abnormal conditions in the data content of the multimedia data to be retrieved, and the like.

[0057] It should be noted that multimedia data refers to a combination of multiple media data, including multiple media forms such as text, sound, image, etc. Specifically, the multimedia data in this solution can be multimedia data containing images, such as photos, videos, etc.

[0058] In one embodiment, the specific method of generating a multimedia data search request by a search client may be: the search client displays a data search interface, wherein the data search interface may include a data editing option; the search client responds to a data editing operation on the data editing option to obtain information about the multimedia data to be searched; the search client generates a multimedia data search request based on the information about the multimedia data to be searched. The data editing operation initiated by the searcher on the data search interface may specifically be inputting a data identifier, a data address, etc. of the multimedia data, or directly inputting the multimedia data, which is not limited here.

[0059] Optionally, since the information of the multimedia data to be retrieved mentioned above may also include retrieval tasks, the data retrieval interface may also include task editing options; then, the specific way for the retrieval client to generate a multimedia data retrieval request may also be: the retrieval client displays the data retrieval interface, wherein the data retrieval interface may include data editing options and task editing options; the retrieval client responds to data editing operations on the data editing options and editing operations on the task editing options to obtain information about the multimedia data to be retrieved; the retrieval client generates a multimedia data retrieval request based on the information about the multimedia data to be retrieved.

[0060] S202: Generate a search statement based on the multimedia data search request.

[0061] In an embodiment of the present application, the search statement includes an identifier of a specified feature retrieval function, an identifier of a first feature table in a GPU database for storing feature vectors of multimedia data to be retrieved, and an identifier of a second feature table in the GPU database for storing feature vectors of reference multimedia data.

[0062] The number of reference multimedia data may include one or more, and the reference multimedia data may be part or all of the multimedia data stored in the GPU database. The multimedia data to be retrieved may be multimedia data originally stored in the GPU database; or multimedia data not stored in the GPU database, in which case the GPU database may obtain the multimedia data to be retrieved through the data address or data identifier of the multimedia data to be retrieved.

[0063] In one embodiment, the GPU database can establish a connection with the feature extraction module in the server so that when the multimedia data is stored in the GPU database, the GPU database can call the feature extraction module to perform feature extraction processing on the stored multimedia data, thereby obtaining a feature table of the corresponding multimedia data.

[0064] The feature extraction module may include a trained feature extraction network for extracting feature vectors of multimedia data. For example, the feature extraction network may be a convolutional neural network (CNN), a recurrent convolutional neural network (R-CNN), a fast recurrent convolutional neural network (Fast R-CNN), or any other network that can extract features after training.

[0065] Optionally, if the multimedia data to be retrieved is not stored in the GPU database, the GPU database may call the feature extraction module to perform feature extraction processing on the multimedia data to be retrieved after acquiring the multimedia data to be retrieved, so as to obtain the first feature table.

[0066] In one embodiment, a template statement may be pre-stored in the GPU database, and the positions of the identifier of the specified feature retrieval function and the identifiers of the two feature tables in the template statement are both empty. Then, the specific method of generating a retrieval statement based on a multimedia data request may be: the GPU database parses the multimedia data retrieval request to obtain information about the multimedia data to be retrieved; then, the GPU database may fill the empty positions in the template statement with the identifier of the specified feature retrieval statement, the identifier of the first feature table, and the identifier of the second feature table based on the information about the multimedia data to be retrieved, thereby obtaining a retrieval statement.

[0067] Optionally, since the information of the multimedia data to be retrieved mentioned in step S201 may include a retrieval task, in which different feature retrieval functions need to be called to complete different retrieval tasks. Therefore, the specific method of generating a retrieval statement based on a multimedia data request may be: the GPU database parses the multimedia data retrieval request to obtain the data identifier of the multimedia data to be retrieved and the retrieval task; then, the GPU database may obtain the identifier of the corresponding first feature table according to the data identifier of the multimedia data to be retrieved, and select a specified feature retrieval function that matches the retrieval task from the feature retrieval functions preset in the GPU database according to the retrieval task; thereafter, the GPU database may fill the corresponding positions in the template statement with the identifier of the specified feature retrieval statement, the identifier of the first feature table, and the identifier of the second feature table, respectively, to obtain a retrieval statement.

[0068] In a possible implementation, for some search objects that understand the database query language, such search objects can also directly enter a search statement in the search client to search for the multimedia data to be searched. Therefore, the information of the multimedia data to be searched may include the search statement for the multimedia data to be searched. Then, the specific method of generating the search statement based on the multimedia data search request may be: parsing the multimedia data search request to obtain the search statement.

[0069] S203: Execute the search statement to call the data search code corresponding to the specified feature search function, perform similarity search processing on the first feature table and the second feature table, and obtain corresponding search results.

[0070] In an embodiment of the present application, fields in the search statement regarding operations such as adding, deleting, modifying, and querying feature tables and merging feature tables can be executed through the GPU database; at the same time, more complex processing processes such as calculation and comparison of feature vectors during similarity retrieval require the GPU database to call the data retrieval code corresponding to the specified feature retrieval function to implement it.

[0071] Optionally, after obtaining the search results, the server may send the search results to the search client, and the search client outputs the search results.

[0072] For specific implementation, please refer to the attached Figure 3 , showing a schematic diagram of a multimedia data retrieval framework. The GPU database 301 establishes a communication connection with the feature extraction module 302 and the vector retrieval module 303. The feature extraction module 302 may include one or more feature extraction networks for extracting feature vectors of multimedia data; the vector retrieval module 303 may be used to store data retrieval codes corresponding to feature retrieval functions, or to generate data retrieval codes corresponding to feature retrieval functions, thereby realizing the calculation and comparison of feature vectors.

[0073] Specifically, after receiving the multimedia data search request, the GPU database 301 can call the feature extraction module 302 to perform feature extraction processing on the multimedia data to be searched, obtain the feature vector of the multimedia data to be searched, and generate a first feature table according to the feature vector of the multimedia data to be searched. Afterwards, when executing the search statement, the GPU database 301 can call the data search code corresponding to the specified feature search function stored in the vector search module 303 to perform similarity search processing on the first feature table and the second feature table to obtain the corresponding search result.

[0074] In an embodiment of the present application, by presetting a feature retrieval function in a GPU database, the data retrieval code corresponding to the corresponding feature retrieval function can be called during the retrieval process to implement similarity retrieval of multimedia data at the feature vector level, which is beneficial to improving the accuracy of the retrieval results. In addition, since the data volume of the feature vector is smaller than the data volume of the multimedia data itself; therefore, compared to using the data table containing multimedia data in the database for retrieval processing, the GPU database in the embodiment of the present application only needs to perform corresponding processing on the first feature table and the second feature table with smaller data volume during the retrieval process, which is beneficial to improving the data processing speed, thereby improving the retrieval efficiency of multimedia data. At the same time, the GPU database can also use the underlying GPU device to greatly improve the data processing throughput during the retrieval process, thereby matching retrieval tasks with extremely high computing power requirements, and further improving the retrieval efficiency of multimedia data.

[0075] In addition, different feature retrieval functions in the embodiments of the present application can complete different retrieval tasks; compared to the traditional need to build a corresponding index database or search engine separately for a retrieval task, the embodiments of the present application can build a universal GPU database that can be applied to multiple retrieval tasks by presetting feature retrieval functions for different retrieval tasks in the GPU database, so that the GPU database can parse richer retrieval semantics, which is beneficial to improving the versatility of the GPU database.

[0076] In one embodiment of the present application, another multimedia data processing method is provided. The multimedia data processing method can be applied to Figure 1 The multimedia data processing system shown in the figure, the method can be executed by a terminal device or a server, or by a terminal device and a server together. In the embodiment of the present application, the method is described by taking the server as an example, and a GPU database is deployed in the server. Figure 4 FIG. 1 is a flow chart of another multimedia data processing method. Figure 2 This paper expands on the method shown in .

[0077] Among them, S401 to S406 are described in detail as follows:

[0078] S401: Receive a multimedia data retrieval request sent by a retrieval client.

[0079] In the embodiment of the present application, the specific implementation of step S401 can refer to the specific implementation of step S201 in the above embodiment, which will not be repeated here.

[0080] S402: Generate a search statement based on the multimedia data search request.

[0081] In the embodiment of the present application, the first feature table may include a retrieval feature vector corresponding to the multimedia data to be retrieved, and the second feature table may include a reference feature vector corresponding to the reference multimedia data.

[0082] In one embodiment, the retrieval task carried by the multimedia data retrieval request may be to search for similar multimedia data, and the corresponding specified feature retrieval function in the retrieval statement may specifically include a clustering screening function, a first distance determination function, and a similarity determination function.

[0083] Optionally, when the multimedia data is a video, the retrieval feature vector corresponding to the multimedia data to be retrieved may be a feature vector obtained by performing feature extraction processing on each frame image in the multimedia data to be retrieved; the reference feature vector corresponding to the reference multimedia data may be a feature vector obtained by performing feature extraction processing on each frame image in the reference multimedia data.

[0084] Optionally, the retrieval feature vector corresponding to the retrieved multimedia data may also be a feature vector obtained by performing feature extraction processing on the key frame images in the retrieved multimedia data; and the reference feature vector corresponding to the reference multimedia data may also be a feature vector obtained by performing feature extraction processing on the key frame images in the reference multimedia data.

[0085] Specifically, an image frame with a large feature difference in the multimedia data can be selected as a key frame image. Optionally, a specific method for determining a key frame image in any multimedia data can be: obtaining a color histogram of each frame image in any multimedia data; if the histogram difference between the color histogram of any frame image and the color histogram of the adjacent frame image is greater than a preset difference, then determining any frame image as a key frame image. The preset difference can be set manually or by a server or terminal device in the above-mentioned multimedia data processing system, which is not limited here.

[0086] Optionally, the specific method for determining a key frame image in any multimedia data may also be: determining the cosine similarity between each two adjacent frames of images in any multimedia data; if the cosine similarity between any image and the two adjacent frames of images is less than or equal to a preset similarity, then determining any image as a key frame image. The preset similarity may be set manually or by a server or terminal device in the multimedia data processing system, which is not limited here.

[0087] Optionally, the key frame images in the multimedia data may be determined by other methods, which are not limited here. In addition, the specific implementation of step S402 can refer to the specific implementation of step S202 in the above embodiment, which will not be repeated here.

[0088] S403, execute the search statement to call the data search code corresponding to the specified feature search function, perform clustering processing on the search feature vector and the reference feature vector, obtain the neighbor feature vector of the search feature vector, and determine the reference multimedia data corresponding to the neighbor feature vector as the target multimedia data.

[0089] In the embodiment of the present application, the number of neighbor feature vectors of the search feature vector may include one or more. The search feature vector and the reference feature vector may be clustered by a clustering classification algorithm to obtain K neighbor feature vectors of the search feature vector.

[0090] Wherein, K is a positive integer, and K can be 1, 3, 100, etc., which is not limited here. The clustering classification algorithm can specifically be a neighboring algorithm (i.e., K-nearest neighbor classification algorithm, KNN), an approximate nearest neighbor search algorithm (Approximate nearest neighbor search, ANNs, implemented by inverted index quantization (IVFPQ), which can realize fast retrieval of high-dimensional vectors), etc., which are used to cluster and find the neighbor features of feature vectors, which is not limited here.

[0091] Specifically, the search statement may include K; K may be manually set, such as the search object in the search client may also input the value of K, so that the multimedia data search request may also carry the value of K.

[0092] In one embodiment, clustering processing may be implemented by calling the data retrieval code corresponding to the clustering screening function mentioned in step S402.

[0093] S404: Determine a feature vector distance between the search feature vector and a target feature vector corresponding to the target multimedia data to obtain a distance matrix between the multimedia data to be searched and the target multimedia data.

[0094] In the embodiment of the present application, the feature vector distance between the search feature vector and the target feature vector can be combined to obtain a distance matrix between the multimedia data to be searched and the target multimedia data. In addition, the method for determining the feature distance between two feature vectors can be to calculate the Euclidean distance, Manhattan distance, Chebyshev distance or Mahalanobis distance between the two feature vectors, etc., which is not limited here.

[0095] In one embodiment, the target multimedia data may include multiple frames of target images, and the multimedia data to be retrieved may include multiple frames of images to be retrieved. Then, the specific method of obtaining the distance matrix may be: determining the feature vector distance between the target feature vector of each frame of the target image and the retrieval feature vector of each frame of the image to be retrieved; obtaining the size of the distance matrix based on the first sequence length of the target image sequence and the second sequence length of the image sequence to be retrieved; and updating the feature vector distance to the corresponding element in the distance matrix based on the arrangement order of each image in the target image sequence and the image sequence to be retrieved.

[0096] Specifically, the length of the distance matrix may be the same as the length of the first sequence, and the width may be the same as the length of the second sequence; or the length of the distance matrix may be the same as the length of the second sequence, and the width may be the same as the length of the first sequence.

[0097] Since each feature vector distance is obtained based on the target feature vector of a frame of target image and the retrieval feature vector of a frame of image to be retrieved; therefore, one feature vector distance corresponds to a target image and a frame of image to be retrieved; and the size of the distance matrix is ​​determined by the target image sequence and the image sequence to be retrieved, therefore, an element in the distance matrix can be located by the order of a frame of target image in the target image sequence and the order of a frame of image to be retrieved in the image sequence to be retrieved.

[0098] Then, the corresponding elements in the distance matrix refer to: in the distance matrix, the order of the target image corresponding to the feature vector distance in the target image sequence, and the elements corresponding to the feature vector distance in the order of the image to be retrieved in the image sequence to be retrieved.

[0099] In one embodiment, the process of obtaining the distance matrix may be implemented by calling the data retrieval code corresponding to the first distance determination function mentioned in step 402 .

[0100] S405: Determine the data similarity between the multimedia data to be retrieved and the target multimedia data based on the distance matrix.

[0101] In the embodiment of the present application, since the feature distance can represent the degree of similarity, the average element value of all elements in the distance matrix can be calculated as the data similarity.

[0102] In one embodiment, a specific method for determining the data similarity between the multimedia data to be retrieved and the target multimedia data based on the distance matrix may also be: based on the distance matrix, generating a similarity matrix between the multimedia data to be retrieved and the target multimedia data; wherein the distance matrix and the similarity matrix have the same size; and determining the minimum element value in the last row of elements of the similarity matrix as the data similarity.

[0103] Optionally, when the multimedia data contains multiple frames of images, there is correlation between the consecutive images; therefore, the feature vector distances corresponding to the consecutive images in the target multimedia data and the feature vector distances corresponding to the consecutive images in the multimedia data to be retrieved are also correlated. Therefore, when generating a similarity matrix through a distance matrix, the feature vector distances corresponding to the previous frame images of each frame of the target image and / or the feature vector distances corresponding to the previous frame images of each frame of the image to be retrieved can be fully referenced to determine the element values ​​of the corresponding elements in the similarity matrix, so that the element values ​​in the last row of the similarity matrix are obtained after reasonably integrating the similarities between each frame of the target image and each frame of the image to be retrieved, thereby facilitating improving the accuracy of the data similarity determined by the similarity matrix.

[0104] Specifically, the distance matrix can be an M×N dimensional matrix, and any element position in the distance matrix is ​​represented as the i-th row and j-th column, where i and j are positive integers. Then, the specific method of generating the similarity matrix based on the distance matrix can be: 1) taking the characteristic distance of the 1st row and 1st column in the distance matrix as the element value of the 1st row and 1st column in the similarity matrix; 2) when i=1, and j is greater than 1 and j is less than or equal to N, based on the element value of the 1st row and j-1 column in the similarity matrix and the characteristic distance of the 1st row and j-1 column in the distance matrix, determine the element value of the 1st row and j-1 column in the similarity matrix; 3) when i is greater than 1 and less than or equal to M, and j=1, based on the corresponding The element value of the 1st column, i-1th row in the similarity matrix and the feature distance of the 1st column, i-th row in the distance matrix are used to determine the element value of the 1st column, i-th row in the similarity matrix; 4) when i is greater than 1 and less than or equal to M, and j is greater than 1 and j is less than or equal to N, based on the minimum element value in the i-1th row, j-1th column, the i-1th row, j-1th column and the i-1th row, j-1th column in the similarity matrix, and the feature distance of the i-th row, j-th column in the distance matrix, determine the element value of the i-th row, j-th column in the similarity matrix.

[0105] Among them, based on the element value of the 1st row, j-1st column in the similarity matrix and the characteristic distance of the 1st row, jth column in the distance matrix, the specific method of determining the element value of the 1st row, jth column in the similarity matrix can be: adding the element value of the 1st row, j-1st column in the similarity matrix and the characteristic distance of the 1st row, jth column in the distance matrix to obtain the element value of the 1st row, jth column in the similarity matrix. The method for determining other element values ​​in the similarity matrix is ​​similar, and will not be repeated here.

[0106] Optionally, since the feature distance can represent the degree of similarity, the specific method of generating the similarity matrix based on the distance matrix can also be: using the distance matrix as the similarity matrix.

[0107] It should be noted that if there are multiple target multimedia data, there is a distance matrix between the multimedia data to be retrieved and each target multimedia data; similarly, there is a similarity matrix between the multimedia data to be retrieved and each target multimedia data.

[0108] In one embodiment, the process of obtaining the distance matrix may be implemented by calling the data retrieval code corresponding to the similarity determination function mentioned in step S402.

[0109] S406 , sorting the target multimedia data based on data similarity, and generating a retrieval result based on the target multimedia data sequence obtained by the sorting process.

[0110] In an embodiment of the present application, the sorting process can be ascending sorting, that is, sorting from small to large according to the data similarity between each target multimedia data and the multimedia data to be retrieved; or it can be descending sorting, that is, sorting from large to small according to the data similarity between each target multimedia data and the multimedia data to be retrieved, which is not limited here.

[0111] Optionally, when the number of target multimedia data includes multiple ones, a preset number of target multimedia data can be selected from the target multimedia data before sorting, wherein the data similarity corresponding to the selected target multimedia data can be greater than the data similarities corresponding to other target multimedia data, or the data similarity corresponding to the selected target multimedia data can also be less than the data similarities corresponding to other target multimedia data; then, the selected target multimedia data are sorted based on the data similarity to obtain a target multimedia data sequence and generate a retrieval result.

[0112] In one embodiment, the specific method of generating the search result based on the target multimedia data sequence obtained by the sorting process may be: using the target multimedia data sequence as the search result. Further, when outputting the search result, each target multimedia data is output according to the order of each target multimedia data in the target multimedia data sequence.

[0113] In a specific implementation, the retrieval task carried in the multimedia data retrieval request may be searching for similar videos, the multimedia data to be retrieved may be a video, and the reference multimedia data may also be a video.

[0114] The SQL dialect for searching similar videos may be pre-designed in the GPU database. Specifically, the SQL dialect for searching similar videos may include feature retrieval functions such as a clustering screening function KNN, a first distance determination function DISTANCE_1, and a video similarity determination function VIDEO_SIMILARITY.

[0115] Then, when the GPU database obtains the video to be retrieved or the reference video, it can obtain the feature vector of the key frame image in the corresponding video through the key frame image extraction method mentioned in step S402 and the feature extraction network in the feature extraction module mentioned in step S202, and organize the obtained feature vector into the column storage format of the GPU database, thereby obtaining the first feature table and the second feature table.

[0116] Among them, the search statement for searching similar videos can be as follows:

[0117]

[0118]

[0119] Among them, the search_video table (i.e., the first feature table) can have one column, namely, the s_feature column, which records the feature vector extracted from the video to be retrieved; the dataset_video table (i.e., the second feature table) can have two columns, namely, video_name to record the video name, and d_feature to record the feature vector extracted from the reference video stored in the GPU database.

[0120] In the subquery of the search statement, a KNN join syntax is added, which draws on the existing join outer join syntax and adds the input of the KNN parameter; k=5 in the search statement, that is, the number of neighbor feature vectors is limited to 5, and the value of k can be set manually or by the server or terminal device in the above-mentioned multimedia data processing system, and is not limited here.

[0121] Specifically, KNN join retrieves k nearest neighbor feature vectors from the table on the right and connects them according to each feature vector in the table on the left. The process of retrieving the nearest neighbor feature vector is implemented by calling the data retrieval code corresponding to KNN. The DISTANCE_1 function calculates the exact L2 distance (i.e., Euclidean distance) between the two column vectors when the input is two columns. VIDEO_SIMILARITY uses the video similarity algorithm to obtain the similarity between the video to be retrieved and the reference video already in the GPU database.

[0122] When executing the search statement, the GPU database generates operators corresponding to the corresponding feature retrieval functions. Specifically, in the subquery, the kNN join generates a join operator, which retrieves the k nearest neighbor feature vectors from the right table (i.e., dataset_video table) and connects them according to each feature vector in the left table (i.e., search_video table).

[0123] When the DISTANCE_1 function is input as two feature vector columns, it generates a project operator, calculates the exact L2 distance between the feature vectors, and projects the feature vector column into the feature vector distance between the feature vectors.

[0124] The VIDEO_SIMILARITY function is an aggregate function that naturally generates a group by operator and divides the table into a set of tuples to be processed separately based on the input group by column. The tuples are then input into the VIDEO_SIMILARITY function for processing and the data similarity between the video to be retrieved and the reference video is obtained.

[0125] In order to understand the video similarity algorithm executed by the VIDEO_SIMILARITY function, we can make a specific interpretation of the above search statement. When the above search statement is executed, it first obtains the search_video and dataset_video tables and filters them according to the where condition. Since the video_name column in the where condition is related to dataset_video, and the other part of the subquery will be executed separately, the where condition is only related to the dataset_video table.

[0126] Specifically, the meaning of the where condition is to filter based on whether video_name exists in the subquery. For each feature vector in the search_video table, the subquery retrieves k neighbor feature vectors from the dataset_video table and connects them, and then obtains the video_name in the dataset_video table that has frame-level correlation with the search_video table (that is, the reference video corresponding to the neighbor feature vector) through select, thereby obtaining the filtered dataset_video table. Therefore, after filtering by the where condition, only a small number of candidate videos (that is, target multimedia data) that have frame-level correlation with the video to be retrieved are retained in the dataset_video table.

[0127] Since the database usually contains a huge amount of reference videos, the subquery avoids brute-force matching of all videos through frame-level nearest neighbor retrieval, and only retains a small number of candidate videos that have frame-level correlation with the video to be retrieved, which helps improve retrieval efficiency.

[0128] Next, the GPU database performs a join operation on the two tables, specifically the Cartesian product operation of the two tables. For any candidate video in the filtered dataset_video table. Suppose the video to be retrieved has M frames and any candidate video has N frames. After the Cartesian product operation, M×N tuples can be obtained, where both M and N are positive integers. By calling the data retrieval code corresponding to the DISTANCE_1 function, the feature vector distance between the two feature vectors in each tuple can be calculated. At this time, the M×N feature vector distances can be input into the VIDEO_SIMILARITY function.

[0129] By calling the data retrieval code corresponding to the VIDEO_SIMILARITY function, the M×N feature vector distances will be organized into a two-dimensional distance matrix of M×N (i.e., dis in the retrieval statement), and a similarity matrix (i.e., sim in the retrieval statement) will be generated based on the distance matrix.

[0130] Specifically, assume 0 <= i < M and 0 <= j < N, where both i and j are positive integers. The similarity calculation rule for generating the similarity matrix sim based on the distance matrix dis can be as follows:

[0131] When i = 1 and j = 1, sim[i][j] = dis[i][j]

[0132] When i = 1 and 1 < j <= N, sim[i][j] = sim[i][j - 1] + dis[i][j]

[0133] When 1 < i <= M and j = 1, sim[i][j] = sim[i - 1][j] + dis[i][j]

[0134] When 1 < i <= M and 1 < j <= N, sim[i][j] = min(sim[i - 1][j], sim[i][j - 1], sim[i - 1][j - 1]) + dis[i][j]

[0135] For example, assume M = 4 and N = 3. The distance matrix is shown in Table 1:

[0136] Distance Matrix (dis) 1 2 3 4 1 0.1 0.4 0.7 0.9 2 0.2 0.3 0.5 0.9 3 0.4 0.7 0.2 0.5

[0137] Table 1

[0138] Using the similarity calculation rule, the similarity matrix obtained after processing the distance matrix shown in Table 1 is shown in Table 2:

[0139]

[0140] Table 2

[0141] Finally, the minimum element value of the last row in the similarity matrix is ​​selected, that is, min(sim[M][j]), where 1≤j≤N, as the calculation result distance of the VIDEO_SIMILARITY function (that is, the data similarity between the video to be retrieved and any candidate video). The smaller the calculation result, the higher the similarity between the video to be retrieved and any candidate video. Finally, the "order by distance" in the search statement can be used to sort multiple candidate videos in ascending order by data similarity to obtain a candidate video sequence (that is, the target multimedia data sequence).

[0142] In an embodiment of the present application, by presetting a feature retrieval function for similar video retrieval in the GPU database, the data retrieval code corresponding to the corresponding feature retrieval function can be called during the video retrieval process to implement similarity retrieval of the video at the feature vector level, so that the searched video is more similar to the video to be retrieved, which is beneficial to improving the accuracy of the retrieval results. In addition, since the data volume of the feature vector is smaller than the data volume of the multimedia data itself; therefore, compared with using the video data table in the database for retrieval processing, the GPU database in the embodiment of the present application only needs to perform corresponding processing on the first feature table and the second feature table with a smaller data volume during the retrieval process, which is beneficial to improving the data processing speed, thereby improving the retrieval efficiency of similar videos. At the same time, the GPU database can also use the underlying GPU device to greatly improve the data processing throughput during the retrieval process, thereby matching the retrieval tasks with extremely high computing power requirements, and further improving the retrieval efficiency of similar videos.

[0143] In one embodiment of the present application, another multimedia data processing method is provided. The multimedia data processing method can be applied to Figure 1 The multimedia data processing system shown in the figure, the method can be executed by a terminal device or a server, or by a terminal device and a server together. In the embodiment of the present application, the method is described by taking the server as an example, and a GPU database is deployed in the server. Figure 5 FIG. 1 is a flow chart of another multimedia data processing method. Figure 2 This paper expands on the method shown in .

[0144] Among them, S501 to S506 are described in detail as follows:

[0145] S501: Receive a multimedia data retrieval request sent by a retrieval client.

[0146] In the embodiment of the present application, the specific implementation of step S501 can refer to the specific implementation of step S201 in the above embodiment, which will not be repeated here.

[0147] S502: Generate a search statement based on the multimedia data search request.

[0148] In the embodiment of the present application, the first feature table includes retrieval object features of objects contained in images in the to-be-retrieved multimedia data, and the second feature table includes reference object features of objects contained in images in the reference multimedia data.

[0149] Specifically, the specific method of obtaining the features of the search object can be: calling the object detection network to perform object recognition on the search image in the search multimedia data to determine the object in the search image; and calling the feature extraction network to perform feature extraction processing on the area containing the object in the search image to obtain the search object features. Similarly, the specific method of obtaining the reference object features can refer to the specific method of obtaining the search object features, which will not be repeated here.

[0150] Specifically, the object detection network can be Mask R-CNN (Mask Region-based Convolutional Neural Network, a machine learning model for target detection and segmentation), recurrent convolutional neural network (R-CNN), fast recurrent convolutional neural network (Fast R-CNN), or any other network that can perform object detection after training, without limitation here.

[0151] In a specific implementation, the feature extraction module mentioned in step S202 may also include an object detection network, so that the GPU can obtain the retrieval object features or the reference object features by calling the feature extraction module.

[0152] In one embodiment, the retrieval task carried by the multimedia data retrieval request may be to obtain the abnormality of an image in the multimedia data or to obtain the abnormality of the multimedia data. Then the corresponding specified feature retrieval function in the retrieval statement may specifically include a cluster screening function, a second distance determination function, and an abnormality determination function.

[0153] In addition, the specific implementation of step S502 can refer to the specific implementation of step S202 in the above embodiment, which will not be repeated here.

[0154] S503: Execute the search statement to call the data search code corresponding to the specified feature search function, perform clustering processing on the search object features and the reference object features, and obtain the neighboring object features of the search object features.

[0155] In the embodiment of the present application, the number of neighboring object features of the search object feature may include one or more. The specific implementation of the clustering process in step S503 can refer to the specific implementation of clustering the search feature vector and the reference feature vector in step S403, which will not be repeated here.

[0156] In one embodiment, clustering processing may be implemented by calling the data retrieval code corresponding to the clustering screening function mentioned in step S502.

[0157] S504: Obtain a target distance of the retrieval object feature based on the feature distance between the retrieval object feature and the neighboring object feature.

[0158] In the embodiment of the present application, when there is one neighbor object feature, the feature distance between the search object feature and the neighbor object feature can be directly used as the target distance of the search object feature. When there are multiple neighbor object features, the target distance of the search object feature can be obtained based on the feature distance between the search object feature and each neighbor object feature.

[0159] Optionally, based on the feature distances between the retrieval object feature and each neighboring object feature, a specific method for obtaining the target distance of the retrieval object feature may be: weighted addition of the feature distances between the retrieval object feature and each neighboring object feature to obtain the target distance of the retrieval object feature.

[0160] The weights of the feature distances may be the same or different, and are not limited here. For example, the weights of the feature distances may be set to 1, or the feature distances may be set to be inversely proportional to the weights, with larger feature distances having smaller weights and smaller feature distances having larger weights.

[0161] In one embodiment, the specific process of obtaining the target distance may be implemented by calling the data retrieval code corresponding to the second distance determination function mentioned in step S502.

[0162] S505: Determine the object corresponding to the retrieval object feature whose target distance is greater than the preset distance as an abnormal object, and determine the corresponding target distance as the abnormality degree of the abnormal object.

[0163] In the embodiment of the present application, the preset distance may be set manually based on one's own experience, or may be set by a terminal device or a server in the above-mentioned multimedia data processing system, which is not limited here.

[0164] Optionally, multiple training multimedia data and multiple abnormal multimedia data can be acquired in advance, wherein the abnormal multimedia data contain images containing abnormal objects; then, the training object features of each training multimedia data and the abnormal object features of each abnormal multimedia data can be extracted; thereafter, the preset distance is determined based on the feature distance between each training object feature and each abnormal object feature.

[0165] Optionally, any reference multimedia data stored in the GPU database can be used as the designated multimedia data, and then the neighboring object features of the designated object features corresponding to the designated multimedia data can be determined from the reference object features corresponding to other reference multimedia data stored in the GPU database; then, the feature distance between the designated object features and the corresponding neighboring object features can be determined, and the distance sum can be calculated; finally, the target distance sum in the multiple distance sums can be used as the preset distance. Among them, the number of first distance sums that are smaller than the target distance sum in the multiple distance sums is greater than the number of second distance sums that are larger than the target distance sum in the multiple distance sums.

[0166] S506: Determine the abnormality degree of the image in the multimedia data to be retrieved based on the abnormality degree of the abnormal object contained in the multimedia data to be retrieved, so as to generate a retrieval result based on the abnormality degree of the image in the multimedia data to be retrieved.

[0167] In the embodiment of the present application, the specific method of generating the retrieval result based on the abnormality of the image in the multimedia data to be retrieved can be: using the abnormality of the image in the multimedia data to be retrieved as the retrieval result. Optionally, since step S502 mentions that the retrieval task can also be to obtain the abnormality of the multimedia data; therefore, the specific method of generating the retrieval result based on the abnormality of the image in the multimedia data to be retrieved can also be: weighted addition of the abnormality of the image in the multimedia data to be retrieved to obtain the data abnormality of the multimedia data to be retrieved, and using the data abnormality as the retrieval result.

[0168] The weights of abnormality of different images may be the same or different, and are not limited here. For example, the weights of abnormality of each image to be retrieved in the multimedia data to be retrieved may be set to 1, or the key frame images in the multimedia data to be retrieved may be first determined, and then the weights of abnormality of the key frame images may be set to be larger than the weights of other images to be retrieved.

[0169] In one embodiment, the specific method of determining the abnormality of the image in the multimedia data to be retrieved can be: determine the objects contained in each frame of the image to be retrieved in the multimedia data to be retrieved; if any image to be retrieved contains an abnormal object, then weighted sum the abnormality of any image to be retrieved containing the abnormal object to obtain the abnormality of any image to be retrieved; if any image to be retrieved does not contain an abnormal object, then the abnormality of any image to be retrieved is determined as a specified abnormality. The specified abnormality can be set artificially based on one's own experience, or it can be set by the terminal device or server in the above-mentioned multimedia data processing system, which is not limited here. For example, the specified abnormality can be 0, 1, 10, etc.

[0170] The weights of the various abnormalities may be the same or different, and are not limited here. For example, the weights of the various abnormalities may be set to 1, or the weights of the abnormalities may be set to be positively correlated with the size of the image area of ​​the corresponding abnormal object in the corresponding image to be retrieved.

[0171] In one embodiment, the multimedia data to be retrieved may include multiple frames of images to be retrieved, and the multiple frames of images to be retrieved may form an image sequence; because the timing processing can improve the stability and accuracy of abnormality detection. Therefore, the specific method of determining the abnormality of the image in the multimedia data to be retrieved can also be: in the process of traversing the multiple frames of images to be retrieved in the multimedia data to be retrieved, based on the abnormality of the abnormal object in the current traversed image and the abnormality of the adjacent frame images of the current traversed image, determine the abnormality of the current traversed image; if the difference between the abnormality of the current traversed image and the abnormality of the previous frame image of the current traversed image is less than or equal to a preset threshold, the current traversed image is updated to the preset abnormality. Optionally, when the difference between the abnormality of the current traversed image and the abnormality of the previous frame image of the current traversed image is less than or equal to a preset threshold, the abnormality of the current traversed image and its previous frame image can also be updated to the preset abnormality.

[0172] The adjacent frame images of the currently traversed image may specifically be the previous frame image and / or the next frame image of the currently traversed image. For example, the adjacent frame images of the currently traversed image may be only the previous frame image of the currently traversed image, or only the next frame image of the currently traversed image, or may include both the previous frame image and the next frame image of the currently traversed image.

[0173] In addition, the preset abnormality degree may be set manually or by a terminal device or server in the multimedia data processing system, which is not limited here. For example, the preset abnormality degree may be 0, 5, 20, etc. At the same time, the preset threshold value may be set manually or by a terminal device or server in the multimedia data processing system, which is not limited here. For example, the preset threshold value may be 0.5, 2, etc.

[0174] Optionally, the specific method of determining the abnormality of the current traversal image may be: performing weighted addition on the abnormality of the abnormal object in the current traversal image and the abnormality of adjacent frame images of the current traversal image to obtain the abnormality of the current traversal image.

[0175] The weight of the abnormality degree of the abnormal object in the current traversal image may be the same as or different from the weight of the abnormality degree of the adjacent frame image of the current traversal image, and is not limited here. For example, all weights may be set to 1, or the weight of the abnormality degree of the abnormal object in the current traversal image may be set to be larger than the weight of the abnormality degree of the adjacent frame image of the current traversal image.

[0176] In one embodiment, step S505 and step S506 may be implemented by calling the data retrieval code corresponding to the abnormality determination function mentioned in step S502.

[0177] In a specific implementation, the retrieval task carried in the multimedia data retrieval request may be to obtain the abnormality of each frame of an image in a video, and both the multimedia data to be retrieved and the reference multimedia data may be videos.

[0178] Among them, the SQL dialect for outlier retrieval (the farther an object is from the cluster, the more abnormal it is) can be pre-designed in the GPU database. Specifically, the SQL dialect for outlier retrieval can include feature retrieval functions such as the clustering screening function KNN, the second distance determination function DISTANCE_2, and the abnormality determination function OUTLIER_SCORE.

[0179] Then, when the GPU database obtains the video to be retrieved and the reference video, it can obtain the retrieval object features and the reference object features through the object detection network and feature extraction network mentioned in step S502, and organize the obtained retrieval object features and reference object features into the column storage format of the GPU database, thereby obtaining the first feature table and the second feature table in this example.

[0180] Among them, the search statement for obtaining the abnormality of each frame of the video can be as follows:

[0181]

[0182] The train_video table (i.e., the second feature table) has one column, train_feature (reference object feature), which records the reference object feature extracted from the reference video stored in the GPU database. The test_video table (i.e., the first feature table) has three columns, object_id is the object identifier; frame_id is the image identifier, which records which frame of the image to be retrieved in the video to be retrieved each object comes from; and test_feature is the retrieval object feature.

[0183] As shown in the above search statement, in the from subquery of the search statement, the 10 nearest neighbor object features of test_feature in train_feature can be obtained by calling the data retrieval code corresponding to KNN, and the total distance sum_dis between each test_feature and the 10 nearest neighbor object features can be calculated through SUM(DISTANCE_2(test_feature,train_feature)) in the search statement.

[0184] Then, in the where subquery of the search statement, the preset threshold can be obtained through the OUTLIER_LINE function, and the sum_dis of test_feature can be filtered through the preset threshold. Only objects above the preset threshold will be retained as abnormal objects. Finally, the data retrieval code corresponding to the OUTLIER_SCORE function can be used to process the abnormal objects in the video to be retrieved and obtain the abnormal value of each frame of the image to be retrieved.

[0185] During the execution of the search statement, the preset OUTLIER_LINE function and OUTLIER_SCORE function in the GPU database will generate aggregation operators. Among them, the OUTLIER_LINE function can determine the preset threshold according to the input train_feature column. Specifically, the OUTLIER_LINE function can search for the 10 approximate nearest neighbor features of each reference object feature in the train_feature column and calculate the sum of their distances, and then sort the sum of the distances obtained for each reference object feature, and take the 90% percentile distance value as the preset threshold.

[0186] Then, the OUTLIER_SCORE function can first determine the sum_dis of the objects in the same frame of the image to be retrieved, and obtain the maximum sum_dis in the frame of the image to be retrieved as the temporary abnormality of the frame of the image to be retrieved. After obtaining the temporary abnormality of each frame of the image to be retrieved, the OUTLIER_SCORE function can also add the final abnormality of the previous frame of any frame of the image to be retrieved to the temporary abnormality of any frame of the image to be retrieved to obtain the final abnormality of any frame of the image to be retrieved; at the same time, if the final abnormality of 4 consecutive frames of images to be retrieved does not increase, the final abnormality of the last image to be retrieved in the 4 frames is set to 0.

[0187] In practical applications, the technical solution of the embodiment of the present application can be applied to intelligent traffic scenarios, such as setting the video to be retrieved to be a traffic intersection video taken by a roadside camera. The GPU database can detect the abnormality of each frame in the intersection video, thereby assisting traffic management personnel in determining whether there is any traffic violation in the intersection video.

[0188] Specifically, after receiving a multimedia data retrieval request for an intersection video, the GPU database can call the object detection network and the feature extraction network to obtain the intersection object features of the objects contained in each frame of the intersection video, thereby generating an intersection feature table; at the same time, the GPU database can obtain a locally stored traffic feature table containing traffic object features corresponding to the traffic video.

[0189] Then, the GPU database can generate a corresponding abnormality search statement based on the multimedia data retrieval request; wherein the abnormality search statement includes the identifier of the intersection feature table, the identifier of the traffic feature table, the identifier of the clustering screening function, the identifier of the second distance determination function, and the identifier of the abnormality determination function. Finally, the GPU database can obtain the abnormality of each frame image in the intersection video by executing the abnormality search statement, thereby generating a corresponding abnormality search result. Furthermore, the GPU database can send the abnormality search result to the search client used by the traffic management personnel to assist the traffic management personnel in determining whether there is any traffic violation in the intersection video.

[0190] In the embodiment of the present application, by presetting the feature retrieval function in the GPU database, the GPU database can parse the retrieval semantics for determining the abnormality of images in multimedia data, thereby completing the corresponding outlier retrieval. Compared with the traditional index database or search engine that can only complete the simple search for similar multimedia data, the GPU database in the embodiment of the present application can parse richer retrieval semantics, which is conducive to improving the versatility of the GPU database and has a great application prospect.

[0191] In addition, since the data volume of the feature vector is smaller than the data volume of the multimedia data itself, compared with directly using the multimedia data table in the database for retrieval processing, the GPU database in the embodiment of the present application only needs to perform corresponding processing on the feature table with a smaller data volume during the retrieval process, which is conducive to improving the data processing speed, thereby improving the retrieval efficiency. At the same time, the GPU database can also use the underlying GPU device to greatly improve the data processing throughput during the retrieval process, thereby matching the retrieval tasks with extremely high computing power requirements, which is conducive to further improving the retrieval efficiency.

[0192] In one embodiment of the present application, another multimedia data processing method is provided. The multimedia data processing method can be applied to Figure 1 The multimedia data processing system shown in the figure, the method can be executed by a terminal device or a server, or by a terminal device and a server together. In the embodiment of the present application, the method is described by taking the server as an example, and a GPU database is deployed in the server. Figure 6 FIG. 1 is a flow chart of another multimedia data processing method. Figure 2 This paper expands on the method shown in .

[0193] Among them, S601 to S605 are described in detail as follows:

[0194] S601: Receive a multimedia data retrieval request sent by a retrieval client.

[0195] In the embodiment of the present application, the specific implementation of step S601 can refer to the specific implementation of step S201 in the above embodiment, which will not be repeated here.

[0196] S602: Generate a search statement based on the multimedia data search request.

[0197] In an embodiment of the present application, the search statement includes an identifier of a specified feature search function, an identifier of a first feature table in a GPU database for storing feature vectors of multimedia data to be searched, and an identifier of a second feature table in the GPU database for storing feature vectors of reference multimedia data. The reference multimedia data includes multiple reference multimedia data, the first feature table includes query object features of objects included in images in the multimedia data to be searched, and the second feature table includes reference object features of objects included in images in each reference multimedia data.

[0198] In one embodiment, the retrieval task carried by the multimedia data retrieval request may be searching for non-similar multimedia data, and the corresponding specified feature retrieval function in the retrieval statement may specifically include a third distance determination function.

[0199] In addition, the specific implementation of step S602 can refer to the specific implementation of step S202 and step S502 in the above embodiments, which will not be repeated here.

[0200] S603, execute the search statement to call the data search code corresponding to the specified feature search function, calculate the vector inner product between the search object feature and each reference object feature, and determine multiple target object features among the multiple reference object features according to the vector inner product.

[0201] In the embodiment of the present application, the vector inner product between two eigenvectors may also be referred to as the dot product or scalar product between the two eigenvectors. Since the specific method of calculating the vector inner product is well known to those skilled in the art, it will not be described in detail here.

[0202] In addition, the inner product of the vector corresponding to each target object feature is greater than the inner product of the vector corresponding to other reference object features. The number of target object features can be set manually or by the server or terminal device in the multimedia data processing system, which is not limited here. For example, the number of target object features can be set to 3, and then 3 corresponding vector inner products with the largest inner products can be selected from multiple reference object features as target object features.

[0203] Specifically, since the vector inner product between features can represent the distance between the features, the multiple target object features determined according to the vector inner product can also be understood as the farthest neighbor features of the retrieved object features.

[0204] S604: Determine the object feature distance between the retrieval object feature and each target object feature.

[0205] In an embodiment of the present application, since the vector inner product between features can represent the distance between features, such as the vector inner product can be directly converted into the L2 distance between two features; therefore, the vector inner product between the retrieval object feature and each target object feature can be directly mapped to the corresponding object feature distance.

[0206] Optionally, the object feature distance between the retrieval object feature and each target object feature may be calculated by other methods, which are not limited here.

[0207] In one embodiment, the specific process of calculating the vector inner product and determining the object feature distance in steps S603 to S604 can be implemented by calling the data retrieval code corresponding to the third distance determination function mentioned in step S602; and the process of determining multiple target object features among multiple reference object features based on the vector inner product can be implemented by the GPU database by deleting and modifying the feature table, or it can be implemented by the data retrieval code corresponding to the third distance determination function, which is not limited here.

[0208] S605 , sorting the reference multimedia data corresponding to each target object feature in descending order according to the object feature distance to obtain a corresponding multimedia data sequence, so as to generate a retrieval result based on the multimedia data sequence.

[0209] In an embodiment of the present application, a specific method of generating a retrieval result based on a multimedia data sequence may be to use the multimedia data sequence as the retrieval result.

[0210] Optionally, since in real scenarios, the amount of multimedia data that can be displayed at one time in an application interface is limited, the preset number of reference multimedia data can be determined based on the number of multimedia data that can be displayed in the application interface; then, when the multimedia data sequence includes multiple reference multimedia data, the reference multimedia data whose sequence numbers are less than or equal to the preset number in the multimedia data sequence can be determined, and the determined reference multimedia data can be used as the retrieval result.

[0211] In a specific implementation, the retrieval task carried in the multimedia data retrieval request may be to search for non-similar images, and the multimedia data to be retrieved and the reference multimedia data are also images.

[0212] The SQL dialect for searching for non-similar images may be pre-designed in the GPU database. Specifically, the SQL dialect for searching for similar videos may include a third distance determination function DISTANCE_3.

[0213] Then, when the GPU database obtains the image to be retrieved or each reference image, it can respectively obtain the retrieval object features and each reference object features through the object detection network and feature extraction network mentioned in step S502, and organize the obtained retrieval object features and multiple reference object features into the column storage format of the GPU database, thereby obtaining the first feature table and the second feature table in this example.

[0214] Among them, the search statement for searching non-similar images can be as follows:

[0215]

[0216] Among them, "path" / home / szr / frames.png"" is the data address of the image to be retrieved. The GPU database can obtain the image to be retrieved from the data address through the extraction function EXTRACTION, and call the object detection network and feature extraction network in the feature extraction module to extract the retrieval object features of each object in the image to be retrieved, and generate the first feature table. In addition, f_feature refers to the second feature table.

[0217] Specifically, the function of the DISTANCE_3 function is to find the k approximate nearest neighbor features for each retrieval object feature through the approximate nearest neighbor search algorithm, but here the retrieval method is set to reverse order according to the DESC in OrderbyNode, that is, the farthest reference object feature of each retrieval object feature is obtained as the target object feature. When searching for non-similar multimedia data, the distance calculation in the DISTANCE function is realized by the vector inner product, that is, the feature vector with the largest inner product is retrieved and converted into the L2 distance, thereby realizing the retrieval of the farthest distance point.

[0218] In the process of executing the above search statement, the DISTANCE function can first generate a filter operator, use the vector inner product to complete the farthest neighbor search, obtain the feature vector column composed of the reference object features (i.e., the target object features) that are farther away from the search object features, and filter the tuples in f_feature according to the feature vector column; then generate a project operator to calculate the accurate L2 distance between the search object features and the target object features, and project the feature vector column into a distance column.

[0219] Specifically, the formula (1-1) for calculating the object feature distance between the retrieval object feature q and each target object feature x in the feature vector column is as follows:

[0220]

[0221] Among them, χ refers to the eigenvector column, x′=[-2x,||x||2 ], and q′=[q,1].

[0222] In practical applications, the technical solution of the embodiment of the present application can be applied to shopping applications; for example, the search client can be a shopping client, and the multimedia data search request for the image to be searched can be uploaded by the shopping client to the corresponding server. Sometimes the object browsing the shopping client may feel that the recommended products are too homogeneous and want to see some new products. At this time, the object can enter the image to be searched in the shopping client and then select to search for non-similar images of the image to be searched.

[0223] Specifically, after receiving the image to be retrieved, the GPU database in the server can call the object detection network and the feature extraction network to obtain the retrieval product object features of the image to be retrieved, thereby generating a retrieval product feature table; at the same time, the GPU database in the server can also obtain the local product feature table of the local product object features of the locally stored product image.

[0224] Then, the GPU database can generate a corresponding non-similar search statement based on the multimedia data retrieval request; wherein the non-similar search statement includes an identifier of the retrieved product feature table, an identifier of the local product feature table, and an identifier of the third distance determination function. Finally, the GPU database can obtain a non-similar image of the image to be retrieved by executing the non-similar search statement, thereby generating a corresponding non-similar search result. Furthermore, the server can send the non-similar search result to the shopping client, so that the shopping client displays the non-similar image of the image to be retrieved.

[0225] In the embodiment of the present application, by presetting the feature retrieval function in the GPU database, the GPU database can parse the retrieval semantics for searching non-similar multimedia data, thereby completing the corresponding non-similar retrieval. Compared with the traditional index database or search engine that can only complete the simple search for similar multimedia data, the GPU database in the embodiment of the present application can parse richer retrieval semantics, which is conducive to improving the versatility of the GPU database and has a great application prospect.

[0226] In addition, since the data volume of the feature vector is smaller than the data volume of the multimedia data itself, compared with directly using the multimedia data table in the database for retrieval processing, the GPU database in the embodiment of the present application only needs to perform corresponding processing on the feature table with a smaller data volume during the retrieval process, which is conducive to improving the data processing speed, thereby improving the retrieval efficiency. At the same time, the GPU database can also use the underlying GPU device to greatly improve the data processing throughput during the retrieval process, thereby matching the retrieval tasks with extremely high computing power requirements, which is conducive to further improving the retrieval efficiency.

[0227] In one embodiment of the present application, another multimedia data processing method is provided. The multimedia data processing method can be applied to Figure 1 The multimedia data processing system shown in the figure, the method can be executed by a terminal device or a server, or by a terminal device and a server together. In the embodiment of the present application, the method is described by taking the server as an example, and a GPU database is deployed in the server. Figure 7 FIG. 1 is a flow chart of another multimedia data processing method. Figure 2 This is an extension of the method shown in .

[0228] Among them, S701 to S705 are described in detail as follows:

[0229] S701: Receive a multimedia data retrieval request sent by a retrieval client.

[0230] In the embodiment of the present application, the specific implementation of step S701 can refer to the specific implementation of step S201 in the above embodiment, which will not be repeated here.

[0231] S702: Generate a search statement based on the multimedia data search request.

[0232] In an embodiment of the present application, the search statement includes an identifier of a specified feature search function, an identifier of a first feature table in a GPU database for storing feature vectors of multimedia data to be searched, and an identifier of a second feature table in the GPU database for storing feature vectors of reference multimedia data. The reference multimedia data includes multiple reference multimedia data, the first feature table includes search feature vectors corresponding to the multimedia data to be searched, and the second feature table includes reference feature vectors corresponding to each reference multimedia data.

[0233] In one embodiment, the retrieval task carried by the multimedia data retrieval request may be searching for similar images, and the corresponding designated feature retrieval function in the retrieval statement may specifically include a fourth distance determination function.

[0234] In addition, the specific implementation of step S702 can refer to the specific implementation of step S202 and step S502 in the above embodiments, which will not be repeated here.

[0235] S703: Execute the search statement to call the data search code corresponding to the specified feature search function, and determine a plurality of neighbor feature vectors of the search feature vector from the second feature table.

[0236] In the embodiment of the present application, the specific implementation method of determining multiple neighbor feature vectors of the retrieved feature vector can refer to the specific implementation method of determining the neighbor feature vector in step S403 in the above embodiment, and determining the neighbor object feature in step S503, which will not be repeated here.

[0237] S704: Determine the target feature distance between the retrieval feature vector and each neighbor feature vector.

[0238] In the embodiment of the present application, the specific implementation method of determining the target feature distance can refer to the specific implementation method of determining the feature vector distance in step S404 in the above embodiment, which will not be repeated here.

[0239] S705 . Sort the reference multimedia data corresponding to each neighbor feature vector in ascending order according to the target feature distance to obtain a corresponding multimedia data sequence, so as to generate a retrieval result based on the multimedia data sequence.

[0240] In an embodiment of the present application, a specific method of generating a retrieval result based on a multimedia data sequence may be to use the multimedia data sequence as the retrieval result.

[0241] Optionally, since in real scenarios, the amount of multimedia data that can be displayed on the application interface at one time is limited, a preset number of reference multimedia data can be determined; then, when the multimedia data sequence includes multiple reference multimedia data, the reference multimedia data in the multimedia data sequence whose sequence numbers are less than or equal to the preset number can be determined, and the determined reference multimedia data can be used as the retrieval result.

[0242] In a specific implementation, the retrieval task carried in the multimedia data retrieval request may be to search for similar images, and the multimedia data to be retrieved and the reference multimedia data are also images.

[0243] The SQL dialect for searching similar images may be pre-designed in the GPU database. Specifically, the SQL dialect for searching similar videos may include a fourth distance determination function DISTANCE_4.

[0244] Then, when the GPU database obtains the image to be retrieved or each reference image, it can respectively obtain the retrieval feature vector of the image to be retrieved and the reference feature vector of each reference image through the feature extraction network mentioned in step S502, and organize the obtained retrieval feature vector and multiple reference feature vectors into the column storage format of the GPU database, thereby obtaining the first feature table and the second feature table in this example.

[0245] Among them, the search statement for searching similar images can be as follows:

[0246]

[0247] Among them, the images table (i.e., the second feature table) has two columns: image_id records the id of each frame of the reference image, and image_feature records the reference feature vector extracted from the reference image. The "path"image" in the above search statement is the data address of the image to be searched. The GPU database can obtain the image to be searched from the data address through the extraction function EXTRACTION, and call the feature extraction network to extract the search feature vector of the image to be searched, and generate the first feature table.

[0248] As shown in the above search statement, the DISTANCE_4 function in the SQL dialect contains two parameters, namely the reference feature vector column from the second feature table and the search feature vector of the image to be searched. The DISTANCE function obtains k neighbor feature vectors from the reference feature vector column based on the search feature vector, where k is determined by the limit clause.

[0249] In the process of executing the search statement, the DISTANCE function will naturally generate the filter operator and the project operator, and integrate them into the existing physical execution plan. In the filter operator, according to the inverted index quantization algorithm used by the approximate nearest neighbor search algorithm, k×σ neighbor feature vectors can be extracted first, where σ is the amplification parameter, which is used to improve the accuracy of the approximate nearest neighbor search. In this example, σ can be set to 5. In the project operator, for the k×σ approximate nearest neighbor feature vectors, their exact L2 distance with the search feature vector is calculated, and the reference feature vector column is replaced. Finally, the distance is sorted in ascending order through the order by node, and the image_id and distance of the similar images to the image to be retrieved can be obtained.

[0250] For specific implementation, please refer to the attached Figure 8 , showing a schematic diagram of a similar multimedia data retrieval process. When the GPU database obtains multimedia data 801 such as pictures and videos, it can call the feature extraction module 802 to extract the feature vector of the multimedia data 801, and then establish a corresponding feature vector index (ie, the second feature table).

[0251] Afterwards, the GPU database generates a corresponding similar search statement after receiving the multimedia data search request for the image to be searched 803. At the same time, the GPU database obtains the image to be searched 803, and can call the feature extraction module 802 to extract the feature vector of the image to be searched 803, and then establishes a corresponding search index (i.e., the first feature table).

[0252] Then, the GPU database executes the similarity search statement to call the data search code corresponding to the fourth distance determination function, such as Figure 8 Six neighbor feature vectors of the feature vector 804 of the image to be retrieved 803 are determined from the feature vectors of the multiple images shown. Afterwards, the GPU database can sort the images corresponding to the six neighbor feature vectors to obtain a corresponding image sequence. Finally, the GPU database can return the image sequence as a retrieval result to the retrieval client, and the retrieval client displays the retrieval result in the retrieval interface 805.

[0253] In an embodiment of the present application, by presetting a feature retrieval function for similar image retrieval in a GPU database, the data retrieval code corresponding to the corresponding feature retrieval function can be called during the image retrieval process to implement similarity retrieval of images at the feature vector level, thereby making the searched image more similar to the image to be retrieved, which is beneficial to improving the accuracy of the retrieval results. In addition, the GPU database in the embodiment of the present application only needs to perform corresponding processing on the feature table with a smaller amount of data during the retrieval process, which is beneficial to improving the data processing speed, thereby improving the retrieval efficiency of similar images. At the same time, the GPU database can also use the underlying GPU device to greatly improve the data processing throughput during the retrieval process, thereby matching retrieval tasks with extremely high computing power requirements, and further improving the retrieval efficiency of similar images.

[0254] Here, the device embodiment of the present application is introduced, which can be used to execute the multimedia data processing method in the above embodiment of the present application. For details not disclosed in the device embodiment of the present application, please refer to the above embodiment of the multimedia data processing method of the present application.

[0255] The present application embodiment provides a multimedia data processing device, such as Fig. 9 As shown, the multimedia data processing device is configured in a GPU database, and a feature retrieval function is preset in the GPU database, including a receiving unit 901, a generating unit 902 and an executing unit 903, wherein:

[0256] The receiving unit 901 is used to receive a multimedia data retrieval request sent by a retrieval client, where the multimedia data retrieval request carries information about the multimedia data to be retrieved;

[0257] A generating unit 902 is used to generate a search statement based on the multimedia data search request, wherein the search statement includes an identifier of a specified feature search function, an identifier of a first feature table in the GPU database for storing features of the multimedia data to be searched, and an identifier of a second feature table in the GPU database for storing features of reference multimedia data;

[0258] The execution unit 903 is used to execute the search statement to call the data search code corresponding to the specified feature search function, perform similarity search processing on the first feature table and the second feature table, and obtain corresponding search results.

[0259] In one embodiment of the present application, the first feature table includes a retrieval feature vector corresponding to the multimedia data to be retrieved, and the second feature table includes a reference feature vector corresponding to the reference multimedia data; based on the aforementioned scheme, when the execution unit 903 executes the retrieval statement to call the data retrieval code corresponding to the specified feature retrieval function, performs similarity retrieval processing on the first feature table and the second feature table, and obtains the corresponding retrieval result, it can be specifically used to: cluster the retrieval feature vector and the reference feature vector to obtain the neighbor feature vector of the retrieval feature vector, and determine the reference multimedia data corresponding to the neighbor feature vector as the target multimedia data; determine the feature vector distance between the retrieval feature vector and the target feature vector corresponding to the target multimedia data to obtain the distance matrix between the multimedia data to be retrieved and the target multimedia data; determine the data similarity between the multimedia data to be retrieved and the target multimedia data based on the distance matrix; sort the target multimedia data based on the data similarity, and generate a retrieval result based on the target multimedia data sequence obtained by the sorting process.

[0260] In one embodiment of the present application, the target multimedia data includes multiple frames of target images, and the multimedia data to be retrieved includes multiple frames of images to be retrieved; based on the aforementioned scheme, when the execution unit 903 determines the feature vector distance between the retrieval feature vector and the target feature vector corresponding to the target multimedia data to obtain the distance matrix between the multimedia data to be retrieved and the target multimedia data, it can be specifically used to: determine the feature vector distance between the target feature vector of each frame of the target image and the retrieval feature vector of each frame of the image to be retrieved; based on the first sequence length of the target image sequence and the second sequence length of the image sequence to be retrieved, obtain the size of the distance matrix; based on the arrangement order of each image in the target image sequence and the image sequence to be retrieved, update the feature vector distance to the corresponding element in the distance matrix.

[0261] In one embodiment of the present application, based on the aforementioned scheme, when the execution unit 903 determines the data similarity between the multimedia data to be retrieved and the target multimedia data based on the distance matrix, it can be specifically used to: generate a similarity matrix between the multimedia data to be retrieved and the target multimedia data based on the distance matrix; wherein the distance matrix and the similarity matrix have the same size; and determine the minimum element value in the last row of elements of the similarity matrix as the data similarity.

[0262] In one embodiment of the present application, the distance matrix is ​​an M×N dimensional matrix, and any element position in the distance matrix is ​​represented as the i-th row and j-th column, where M, N, i and j are all positive integers; based on the above scheme, when the execution unit 903 generates a similarity matrix between the multimedia data to be retrieved and the target multimedia data based on the distance matrix, it can be specifically used to: use the feature distance of the 1st row and 1st column in the distance matrix as the element value of the 1st row and 1st column in the similarity matrix; when i=1, and j is greater than 1 and j is less than or equal to N, based on the element value of the 1st row and j-1th column in the similarity matrix and the 1st row and jth column in the distance matrix feature distance, determine the element value of the 1st row and jth column in the similarity matrix; when i is greater than 1 and less than or equal to M, and j=1, based on the element value of the 1st column and i-1th row in the similarity matrix and the feature distance of the 1st column and i-th row in the distance matrix, determine the element value of the 1st column and i-th row in the similarity matrix; when i is greater than 1 and less than or equal to M, and j is greater than 1 and j is less than or equal to N, based on the minimum element value in the i-1th row and j-1th column, the i-1th row and j-1th column in the similarity matrix, and the feature distance of the i-th row and j-th column in the distance matrix, determine the element value of the i-th row and j-th column in the similarity matrix.

[0263] In one embodiment of the present application, the first feature table includes retrieval object features of objects contained in images in the multimedia data to be retrieved, and the second feature table includes reference object features of objects contained in images in the reference multimedia data; based on the aforementioned scheme, when the execution unit 903 executes the retrieval statement to call the data retrieval code corresponding to the specified feature retrieval function, performs similarity retrieval processing on the first feature table and the second feature table, and obtains the corresponding retrieval results, it can be specifically used to: cluster the retrieval object features and the reference object features to obtain the neighboring object features of the retrieval object features; obtain the target distance of the retrieval object features based on the feature distance between the retrieval object features and the neighboring object features; determine the object corresponding to the retrieval object feature whose target distance is greater than the preset distance as an abnormal object, and determine the corresponding target distance as the abnormality degree of the abnormal object; determine the abnormality degree of the image in the multimedia data to be retrieved based on the abnormality degree of the abnormal object contained in the multimedia data to be retrieved, so as to generate a retrieval result based on the abnormality degree of the image in the multimedia data to be retrieved.

[0264] In one embodiment of the present application, the multimedia data to be retrieved includes multiple frames of images to be retrieved; based on the aforementioned scheme, when the execution unit 903 determines the abnormality of the image in the multimedia data to be retrieved based on the abnormality of the abnormal object contained in the multimedia data to be retrieved, it can be specifically used to: in the process of traversing multiple frames of images to be retrieved in the multimedia data to be retrieved, based on the abnormality of the abnormal object in the current traversed image, and the abnormality of the adjacent frame images of the current traversed image, determine the abnormality of the current traversed image; if the difference between the abnormality of the current traversed image and the abnormality of the previous frame image of the current traversed image is less than or equal to a preset threshold value, the abnormality of the current traversed image is updated to the preset abnormality.

[0265] In one embodiment of the present application, the reference multimedia data includes multiple, a first feature table includes query object features of objects contained in images in the multimedia data to be retrieved, and a second feature table includes reference object features of objects contained in images in each reference multimedia data; based on the aforementioned scheme, when the execution unit 903 executes the search statement to call the data retrieval code corresponding to the specified feature retrieval function, performs similarity retrieval processing on the first feature table and the second feature table, and obtains the corresponding search results, it can be specifically used to: calculate the vector inner product between the search object feature and each reference object feature, and determine multiple target object features among the multiple reference object features based on the vector inner product; wherein the vector inner product corresponding to each target object feature is greater than the vector inner product corresponding to other reference object features; determine the object feature distance between the search object feature and each target object feature; according to the object feature distance, sort the reference multimedia data corresponding to each target object feature in descending order to obtain a corresponding multimedia data sequence, so as to generate a search result based on the multimedia data sequence.

[0266] In one embodiment of the present application, the reference multimedia data includes multiple, the first feature table includes retrieval feature vectors corresponding to the multimedia data to be retrieved, and the second feature table includes reference feature vectors corresponding to each reference multimedia data; based on the aforementioned scheme, the execution unit 903 executes the retrieval statement to call the data retrieval code corresponding to the specified feature retrieval function, performs similarity retrieval processing on the first feature table and the second feature table, and obtains the corresponding retrieval results. Specifically, it can be used to: determine multiple neighbor feature vectors of the retrieval feature vector from the second feature table; determine the target feature distance between the retrieval feature vector and each neighbor feature vector; according to the target feature distance, sort the reference multimedia data corresponding to each neighbor feature vector in ascending order to obtain a corresponding multimedia data sequence, so as to generate a retrieval result based on the multimedia data sequence.

[0267] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0268] The device provided in the above embodiment can be arranged in a terminal device or in a server. The device provided in the embodiment of the present application can call the data retrieval code corresponding to the corresponding feature retrieval function during the retrieval process through the feature retrieval function preset in the GPU database, so as to realize the similarity retrieval of multimedia data at the feature vector level, which is beneficial to improve the accuracy of the retrieval results. In addition, since the data volume of the feature vector is smaller than the data volume of the multimedia data itself; therefore, compared with using the data table containing multimedia data in the database for retrieval processing, the GPU database in the embodiment of the present application only needs to perform corresponding processing on the first feature table and the second feature table with smaller data volume during the retrieval process, which is beneficial to improve the data processing speed, thereby improving the retrieval efficiency of multimedia data. At the same time, the GPU database can also use the underlying GPU device to greatly improve the data processing throughput during the retrieval process, thereby matching the retrieval tasks with extremely high computing power requirements, and further improving the retrieval efficiency of multimedia data.

[0269] An embodiment of the present application also provides an electronic device, comprising one or more processors and a storage device, wherein the storage device is used to store one or more computer programs, and when the one or more computer programs are executed by one or more processors, the electronic device implements the above multimedia data processing method.

[0270] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0271] It should be noted that Fig.10 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0272] like Fig.10As shown, the computer system 1000 includes a processor (Central Processing Unit, CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (Read-Only Memory, ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (Random Access Memory, RAM) 1003, such as executing the method in the above embodiment. In RAM 1003, various programs and data required for system operation are also stored. CPU 1001, ROM 1002 and RAM 1003 are connected to each other through bus 1004. Input / output (Input / Output, I / O) interface 1005 is also connected to bus 1004.

[0273] In some embodiments, the following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0274] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 1009, and / or installed from a removable medium 1011. When the computer program is executed by a processor (CPU) 1001, various functions defined in the system of the present application are executed.

[0275] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (Erasable Programmable Read Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0276] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and a computer program.

[0277] The units or modules involved in the embodiments described in this application may be implemented by software or hardware, and the units or modules described may also be set in a processor. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.

[0278] Another aspect of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the multimedia data processing method as described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0279] Another aspect of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the electronic device executes the multimedia data processing method as described above in the above embodiments.

[0280] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0281] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0282] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person of ordinary skill in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A multimedia data processing method, characterized in that: Applied to a GPU database, wherein a feature retrieval function is preset in the GPU database, the method comprises: Receiving a multimedia data retrieval request sent by a retrieval client, wherein the multimedia data retrieval request carries information of multimedia data to be retrieved; Generate a search statement based on the multimedia data search request, wherein the search statement includes an identifier of a specified feature search function, an identifier of a first feature table in the GPU database for storing feature vectors of the multimedia data to be searched, and an identifier of a second feature table in the GPU database for storing feature vectors of reference multimedia data; The search statement is executed to call the data search code corresponding to the specified feature search function, perform similarity search processing on the first feature table and the second feature table, and obtain corresponding search results.

2. The method according to claim 1, characterized in that The first feature table includes a search feature vector corresponding to the multimedia data to be searched, and the second feature table includes a reference feature vector corresponding to the reference multimedia data; The executing the search statement to call the data search code corresponding to the specified feature search function, performing similarity search processing on the first feature table and the second feature table, and obtaining corresponding search results, includes: Performing clustering processing on the search feature vector and the reference feature vector to obtain a neighbor feature vector of the search feature vector, and determining the reference multimedia data corresponding to the neighbor feature vector as the target multimedia data; Determining a feature vector distance between the retrieval feature vector and a target feature vector corresponding to the target multimedia data to obtain a distance matrix between the multimedia data to be retrieved and the target multimedia data; Determine the data similarity between the to-be-retrieved multimedia data and the target multimedia data based on the distance matrix; The target multimedia data is sorted based on the data similarity, and the search result is generated based on the target multimedia data sequence obtained by the sorting process.

3. The method according to claim 2, characterized in that The target multimedia data includes multiple frames of target images, and the multimedia data to be retrieved includes multiple frames of images to be retrieved; The determining of the feature vector distance between the search feature vector and the target feature vector corresponding to the target multimedia data to obtain a distance matrix between the multimedia data to be searched and the target multimedia data includes: Determine the feature vector distance between the target feature vector of each frame of the target image and the retrieval feature vector of each frame of the image to be retrieved; Based on a first sequence length of the target image sequence and a second sequence length of the image sequence to be retrieved, obtaining a size of the distance matrix; Based on the arrangement order of each image in the target image sequence and the to-be-retrieved image sequence, the feature vector distance is updated to a corresponding element in the distance matrix.

4. The method according to claim 2, characterized in that: The determining, based on the distance matrix, the data similarity between the to-be-retrieved multimedia data and the target multimedia data comprises: Based on the distance matrix, generating a similarity matrix between the multimedia data to be retrieved and the target multimedia data; wherein the distance matrix and the similarity matrix have the same size; The minimum element value in the last row of elements of the similarity matrix is ​​determined as the data similarity.

5. The method according to claim 4, characterized in that The distance matrix is ​​an M×N dimensional matrix, and the position of any element in the distance matrix is ​​represented as the i-th row and the j-th column, where M, N, i and j are all positive integers; The generating, based on the distance matrix, a similarity matrix between the to-be-retrieved multimedia data and the target multimedia data comprises: Taking the feature distance in the first row and first column of the distance matrix as the element value in the first row and first column of the similarity matrix; When i=1, and j is greater than 1 and j is less than or equal to N, determining the element value of the 1st row and jth column in the similarity matrix based on the element value of the 1st row and j-1th column in the similarity matrix and the feature distance of the 1st row and jth column in the distance matrix; When i is greater than 1 and less than or equal to M, and j=1, based on the element value of the first column and the i-1th row in the similarity matrix and the feature distance of the first column and the i-th row in the distance matrix, determine the element value of the first column and the i-th row in the similarity matrix; When i is greater than 1 and less than or equal to M, and j is greater than 1 and less than or equal to N, the element value of the i-th row and j-th column in the similarity matrix is ​​determined based on the minimum element value in the i-1th row and j-th column, the i-th row and j-1th column, and the i-1th row and j-1th column in the similarity matrix, and the feature distance of the i-th row and j-th column in the distance matrix.

6. The method according to claim 1, characterized in that The first feature table includes retrieval object features of objects included in the images in the to-be-retrieved multimedia data, and the second feature table includes reference object features of objects included in the images in the reference multimedia data; The executing the search statement to call the data search code corresponding to the specified feature search function, performing similarity search processing on the first feature table and the second feature table, and obtaining corresponding search results, includes: Performing clustering processing on the retrieval object feature and the reference object feature to obtain neighboring object features of the retrieval object feature; Based on the feature distance between the retrieval object feature and the neighboring object feature, obtaining the target distance of the retrieval object feature; Determine an object corresponding to a retrieval object feature whose target distance is greater than a preset distance as an abnormal object, and determine the corresponding target distance as the abnormality degree of the abnormal object; Based on the abnormality degree of the abnormal object contained in the multimedia data to be retrieved, the abnormality degree of the image in the multimedia data to be retrieved is determined, so as to generate the retrieval result based on the abnormality degree of the image in the multimedia data to be retrieved.

7. The method according to claim 6, characterized in that The multimedia data to be retrieved includes a plurality of frames of images to be retrieved; and determining the abnormality degree of the image in the multimedia data to be retrieved based on the abnormality degree of the abnormal object included in the multimedia data to be retrieved includes: In the process of traversing multiple frames of images to be retrieved in the multimedia data to be retrieved, determining the abnormality degree of the currently traversed image based on the abnormality degree of the abnormal object in the currently traversed image and the abnormality degree of the adjacent frame images of the currently traversed image; If the difference between the abnormality degree of the current traversal image and the abnormality degree of the previous frame image of the current traversal image is less than or equal to a preset threshold, the abnormality degree of the current traversal image is updated to the preset abnormality degree.

8. The method according to claim 1, characterized in that The reference multimedia data includes a plurality of items, the first feature table includes query object features of objects included in images in the multimedia data to be retrieved, and the second feature table includes reference object features of objects included in images in each reference multimedia data; The executing the search statement to call the data search code corresponding to the specified feature search function, performing similarity search processing on the first feature table and the second feature table, and obtaining corresponding search results, includes: Calculating the vector inner product between the retrieval object feature and each reference object feature, and determining multiple target object features among the multiple reference object features according to the vector inner product; wherein the vector inner product corresponding to each target object feature is greater than the vector inner product corresponding to other reference object features; Determining object feature distances between the retrieval object feature and each target object feature; According to the object feature distance, the reference multimedia data corresponding to each target object feature is sorted in descending order to obtain a corresponding multimedia data sequence, so as to generate the retrieval result based on the multimedia data sequence.

9. The method according to claim 1, characterized in that: The reference multimedia data includes a plurality of data, the first feature table includes search feature vectors corresponding to the multimedia data to be searched, and the second feature table includes reference feature vectors corresponding to each reference multimedia data; The executing the search statement to call the data search code corresponding to the specified feature search function, performing similarity search processing on the first feature table and the second feature table, and obtaining corresponding search results, includes: Determine a plurality of neighbor feature vectors of the search feature vector from the second feature table; Determining a target feature distance between the retrieval feature vector and each neighbor feature vector; According to the target feature distance, the reference multimedia data corresponding to each neighbor feature vector is sorted in ascending order to obtain a corresponding multimedia data sequence, so as to generate the retrieval result based on the multimedia data sequence.

10. A multimedia data processing device, characterized in that: The multimedia data processing device is configured in a GPU database, a feature retrieval function is preset in the GPU database, and the multimedia data processing device includes a receiving unit, a generating unit and an executing unit, wherein: The receiving unit is used to receive a multimedia data retrieval request sent by a retrieval client, wherein the multimedia data retrieval request carries information of the multimedia data to be retrieved; The generating unit is configured to generate a search statement based on the multimedia data search request, wherein the search statement includes an identifier of a specified feature search function, an identifier of a first feature table in the GPU database for storing feature vectors of the multimedia data to be searched, and an identifier of a second feature table in the GPU database for storing feature vectors of reference multimedia data; The execution unit is used to execute the search statement to call the data search code corresponding to the specified feature search function, perform similarity search processing on the first feature table and the second feature table, and obtain corresponding search results.

11. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimedia data processing method according to any one of claims 1 to 9 is implemented.

12. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the multimedia data processing method described in any one of claims 1 to 9.

Images