Data processing method, device and computer-readable storage medium

Abstract

The embodiments of the present application disclose a data processing method, device and computer-readable storage medium based on artificial intelligence, wherein the method includes: obtaining a first material associated with a query text; obtaining a query image style vector, and obtaining a second material associated with a query image semantic vector; determining the first material and the second material as candidate materials, obtaining a target material image style vector, a target material image semantic vector and a target material text; sorting the candidate materials based on the query text, the query image semantic vector, the query image style vector, the target material text, the target material image semantic vector and the target material image style vector, and determining a recommended material from the sorted candidate materials. By adopting the present application, the accuracy of the description of the retrieval purpose can be improved, and thus the accuracy of the recommended material can be improved. The embodiments of the present application can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and blockchain.

Description

[0001] This application is a divisional application of the Chinese patent application submitted to the China Patent Office on September 29, 2021, with application number 2021111475451 and application name "A data processing method, device and computer-readable storage medium", all of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of Internet technology, and in particular to a data processing method, device, and computer-readable storage medium. Background Art

[0003] With the popularization of multimedia technology and the development of computer Internet, image data has grown at an astonishing rate. The process of users submitting search requirements and the material database returning image or video materials that meet the requirements can be called material retrieval.

[0004] Finding desirable material within massive databases has become a major research hotspot in computer vision. Currently, material search websites rely on text for material retrieval, specifically images and videos. Image material contains not only subject and scene information, but also complex information such as subject attributes and relationships between multiple subjects. Video content contains richer information about actions and interactions, making it difficult to accurately describe the search objective using query text alone. Inaccurate search objectives can lead to inaccurate image and video content being pushed to material search websites. Summary of the Invention

[0005] The embodiments of the present application provide a data processing method, device, and computer-readable storage medium, which can improve the accuracy of the description of the retrieval purpose and thereby improve the accuracy of the recommended materials.

[0006] On the one hand, an embodiment of the present application provides a data processing method, including:

[0007] Obtaining query information including query text and query image, and obtaining a first material associated with the query text from a material database;

[0008] Obtaining a query image style vector corresponding to the query image and a query image semantic vector corresponding to the query image, and obtaining a second material associated with the query image semantic vector from a material database;

[0009] Determine the first material and the second material as candidate materials, obtain target material image style vectors corresponding to the candidate materials, target material image semantic vectors corresponding to the candidate materials, and target material text corresponding to the candidate materials;

[0010] Based on the query text, query image semantic vector, query image style vector, target material text, target material image semantic vector, and target material image style vector, the material similarity between the candidate material and the query information is determined, the candidate materials are sorted according to the material similarity, and the recommended material is determined from the sorted candidate materials.

[0011] In one aspect, an embodiment of the present application provides a data processing device, including:

[0012] A first acquisition module is configured to acquire query information including a query text and a query image, and acquire a first material associated with the query text from a material database;

[0013] A second acquisition module is configured to acquire a query image style vector corresponding to the query image and a query image semantic vector corresponding to the query image, and acquire a second material associated with the query image semantic vector from a material database;

[0014] A first determining module is configured to determine the first material and the second material as candidate materials, and obtain a target material image style vector corresponding to the candidate materials, a target material image semantic vector corresponding to the candidate materials, and a target material text corresponding to the candidate materials;

[0015] The second determination module is used to determine the material similarity between the candidate material and the query information based on the query text, the query image semantic vector, the query image style vector, the target material text, the target material image semantic vector, and the target material image style vector, sort the candidate materials according to the material similarity, and determine the recommended material from the sorted candidate materials.

[0016] The first acquisition module includes:

[0017] The first processing unit is used to segment the query text to obtain query words corresponding to the query text;

[0018] The first acquisition unit is used to acquire an index key-value pair whose index attribute is a word segmentation attribute from a material database; the index key-value pair includes an index key generated by a material identifier and an index value generated by a material word segmentation;

[0019] The first determining unit is configured to determine an index key corresponding to an index value containing a query word as a target index key, and obtain a first material corresponding to the target index key from a material database.

[0020] The first acquisition module includes:

[0021] A first input unit is used to input the query text into the text encoding model, and obtain the query text semantic vector corresponding to the query text through the text encoding model;

[0022] The second acquisition unit is used to acquire an index key-value pair whose index attribute is a text semantic attribute from the material database; the index key-value pair includes an index key generated by the material identifier and an index value generated by the material text semantic vector;

[0023] A third obtaining unit is configured to obtain a similarity between the query text semantic vector and the index value, and determine an index value having a similarity greater than a similarity threshold as a target index value;

[0024] The second determining unit is configured to determine the index key corresponding to the target index value as the target index key, and obtain the first material corresponding to the target index key from the material database.

[0025] The second acquisition module includes:

[0026] A second input unit is used to input the query image into the image classification model, and obtain image classification information of the query image through the image classification model;

[0027] a fourth acquisition unit, configured to acquire an image semantic extraction model associated with the image classification information and an image style extraction model associated with the image classification information;

[0028] A third input unit is used to input the query image into the image semantic extraction model, and obtain a query image semantic vector corresponding to the query image through the image semantic extraction model;

[0029] The third input unit is also used to input the query image into the image style extraction model, extract the image color features of the query image and the image shape features of the query image through the image style extraction model, perform feature aggregation processing on the image color features of the query image and the image shape features of the query image, and obtain a query image style vector corresponding to the query image.

[0030] Among them, the image semantic extraction model includes the image main body extraction model, the image local extraction model and the image overall extraction model;

[0031] The third input unit includes:

[0032] A first extraction subunit is configured to input a query image into an image subject extraction model, extract subject semantic features of the query image through the image subject extraction model, and obtain a query subject semantic vector;

[0033] A second extraction subunit is configured to input the query image into the image local extraction model, extract the local semantic features of the query image through the image local extraction model, and obtain a query local semantic vector;

[0034] A third extraction subunit is configured to input the query image into the overall image extraction model, extract the overall semantic features of the query image through the overall image extraction model, and obtain the query overall semantic vector;

[0035] The first determining subunit is configured to determine the query subject semantic vector, the query overall semantic vector, and the query local semantic vector as the query image semantic vector.

[0036] The second acquisition module includes:

[0037] a fifth acquiring unit, configured to acquire an index key-value pair whose index attribute is an image semantic attribute from a material database; the index key-value pair includes an index key generated by a material identifier and an index value generated by a material image semantic vector;

[0038] The sixth acquiring unit is configured to acquire, from the material database according to the query image semantic vector and the index key-value pair, a second material associated with the query image semantic vector.

[0039] Among them, the query image semantic vector includes the query subject semantic vector, the query overall semantic vector and the query local semantic vector; the material image semantic vector includes the material subject semantic vector, the material overall semantic vector and the material local semantic vector; the index key-value pair includes the index subject key-value pair, the index overall key-value pair and the index local key-value pair; the index subject key-value pair includes the index subject key generated by the material identifier and the index subject value generated by the material subject semantic vector; the index local key-value pair includes the index local key generated by the material identifier and the index local value generated by the material local semantic vector; the index overall key-value pair includes the index overall key generated by the material identifier and the index overall value generated by the material overall semantic vector;

[0040] The sixth acquisition unit includes:

[0041] a first acquisition subunit, configured to acquire a first similarity between a query subject semantic vector and an index subject value, determine an index subject value having a first similarity greater than a first similarity threshold as a target index subject value, determine an index subject key corresponding to the target index subject value as a target index subject key, and acquire a subject material corresponding to the target index subject key from a material database;

[0042] A second acquisition subunit is configured to obtain a second similarity between the query overall semantic vector and the index overall value, determine the index overall value having the second similarity greater than a second similarity threshold as the target index overall value, determine the index overall key corresponding to the target index overall value as the target index overall key, and obtain the overall material corresponding to the target index overall key from the material database;

[0043] a third acquisition subunit, configured to obtain a third similarity between the query local semantic vector and the index local value, determine the index local value having the third similarity greater than a third similarity threshold as the target index local value, determine the index local key corresponding to the target index local value as the target index local key, and obtain the local material corresponding to the target index local key from the material database;

[0044] The second determining subunit is configured to determine the main material, the partial material, and the entire material as the second material.

[0045] The first determination module includes:

[0046] a third determining unit, configured to determine a material identifier corresponding to the candidate material as a candidate material identifier;

[0047] a seventh acquisition unit, configured to acquire, from the material database, a target material image style vector stored in association with the candidate material identifier, a target material main semantic vector stored in association with the candidate material identifier, a target material overall semantic vector stored in association with the candidate material identifier, a target material local semantic vector stored in association with the candidate material identifier, and a target material text stored in association with the candidate material identifier;

[0048] The fourth determining unit is configured to determine the target material main body semantic vector, the target material overall semantic vector, and the target material local semantic vector as the target material image semantic vector.

[0049] The total number of candidate materials is at least two, and the at least two candidate materials include candidate material C d , d is a positive integer, and d is less than or equal to the total number of at least two candidate materials; the target material text includes candidate material C d Corresponding target material text E d ; The target material image semantic vector includes the candidate material C d The corresponding target material image semantic vector F d ; The target material image style vector includes the candidate material C d The corresponding target material image style vector G d ;

[0050] The second determination module includes:

[0051] The fifth determining unit is used to obtain the query text and the target material text E d The similarity between them is determined as the candidate material C d Corresponding text similarity;

[0052] The fifth determining unit is further configured to obtain the query image semantic vector and the target material image semantic vector F dThe similarity between them is determined as the candidate material C d Corresponding semantic similarity;

[0053] The fifth determining unit is further configured to obtain the query image style vector and the target material image style vector G d The similarity between them is determined as the candidate material C d Corresponding style similarity;

[0054] The second processing unit is used to perform weighted sum processing on text similarity, semantic similarity and style similarity to obtain candidate material C d The similarity between the material and the query information;

[0055] The third processing unit is configured to sort at least two candidate materials according to the material similarity corresponding to each candidate material.

[0056] The query image semantic vector includes the query subject semantic vector, the query overall semantic vector and the query local semantic vector; the target material image semantic vector F d Including the target material main semantic vector H d , the overall semantic vector of the target material I d And the target material local semantic vector J d ;

[0057] The fifth determining unit includes:

[0058] The fourth acquisition subunit is used to obtain the query subject semantic vector and the target material subject semantic vector H d The similarity between them is determined as the candidate material C d The corresponding subject semantic similarity;

[0059] The fifth acquisition subunit is used to obtain the query overall semantic vector and the target material overall semantic vector I d The similarity between them is determined as the candidate material C d The corresponding overall semantic similarity;

[0060] The sixth acquisition subunit is used to obtain the query local semantic vector and the target material local semantic vector J d The similarity between them is determined as the candidate material C d Corresponding local semantic similarity;

[0061] The third determining subunit is configured to determine the subject semantic similarity, the overall semantic similarity, and the local semantic similarity as the semantic similarity.

[0062] The data processing device further includes:

[0063] A third acquisition module is used to acquire video material, extract at least two video frames from the video material, and perform image preprocessing on the at least two video frames to obtain target video frames;

[0064] The third acquisition module is further configured to acquire a video frame identifier corresponding to the target video frame and use the video frame identifier as a material identifier;

[0065] A first storage module is configured to obtain a video frame text corresponding to a target video frame, use the video frame text as a material text, construct an index key-value pair whose index attribute is a text attribute according to the material identifier and the material text, and store the index key-value pair whose index attribute is a text attribute in a material database;

[0066] A fourth acquisition module is used to acquire a semantic vector of a material image corresponding to a target video frame, and a style vector of a material image corresponding to a target video frame;

[0067] A second storage module is configured to use the material identifier as an index value and the material image semantic vector as an index key to construct an index key-value pair whose index attribute is the image semantic attribute, and store the index key-value pair whose index attribute is the image semantic attribute in a material database;

[0068] The third storage module is used to associate and store the material image style vector and the material identifier in the material database.

[0069] Wherein, at least two video frames include video frame A b , b is a positive integer, and b is less than or equal to the total number of at least two video frames;

[0070] The third acquisition module includes:

[0071] The eighth acquisition unit is used to acquire video frame A b and the similarity between the remaining video frames; the remaining video frames include at least two video frames except video frame A b Video frames other than

[0072] The sixth determining unit is configured to: if the similarity is equal to or greater than the similarity threshold, then the video frame A is b Determine that the video frame is a duplicate video frame, and delete the duplicate video frame from at least two video frames to obtain a video frame to be trimmed;

[0073] The fourth processing unit is configured to perform region recognition processing on the video frame to be cropped to obtain a region to be cropped of the video frame to be cropped, and perform cropping processing on the region to be cropped in the video frame to be cropped to obtain a target video frame.

[0074] The first storage module includes:

[0075] a seventh determining unit, configured to obtain a video description text carried by the video material, and determine the obtained video description text as the first video frame text;

[0076] a fifth processing unit, configured to perform recognition processing on the scene information in the target video frame to obtain scene recognition text, and determine the scene recognition text as the second video frame text;

[0077] a first extraction unit, configured to extract text information from a target video frame and determine the extracted text information as text of a third video frame;

[0078] The eighth determining unit is configured to determine the first video frame text, the second video frame text, and the third video frame text as video frame texts.

[0079] Among them, text attributes include word segmentation attributes and text semantic attributes;

[0080] The first storage module includes:

[0081] The first construction unit is used to segment the material text to obtain material segmentation corresponding to the material text, use the material identifier as the index key and the material segmentation as the index value, and construct an index key-value pair whose index attribute is the segmentation attribute;

[0082] The second construction unit is used to obtain the material text semantic vector corresponding to the material text, use the material identifier as the index key and the material text semantic vector as the index value, and construct an index key-value pair whose index attribute is the text semantic attribute.

[0083] The material image semantic vector includes the material main semantic vector, the material overall semantic vector and the material local semantic vector;

[0084] The fourth acquisition module includes:

[0085] a fourth input unit, configured to input the target video frame into the image classification model, and obtain material classification information of the target video frame through the image classification model;

[0086] A second extraction unit is configured to input the target video frame into a material subject extraction model associated with the material classification information, extract the subject semantic features of the target video frame through the material subject extraction model, and obtain a material subject semantic vector;

[0087] a third extraction unit, configured to input the target video frame into a material local extraction model associated with the material classification information, extract local semantic features of the target video frame through the material local extraction model, and obtain a material local semantic vector;

[0088] a fourth extraction unit, configured to input the target video frame into a material overall extraction model associated with the material classification information, extract the overall semantic features of the target video frame through the material overall extraction model, and obtain a material overall semantic vector;

[0089] The fifth extraction unit is used to input the target video frame into a material style extraction model associated with the material classification information, extract the image color features of the target video frame and the image shape features of the target video frame through the material style extraction model, perform feature aggregation processing on the image color features of the target video frame and the image shape features of the target video frame, and obtain a material image style vector.

[0090] On one hand, the present application provides a computer device, including: a processor, a memory, and a network interface;

[0091] The above-mentioned processor is connected to the above-mentioned memory and the above-mentioned network interface, wherein the above-mentioned network interface is used to provide data communication function, the above-mentioned memory is used to store computer programs, and the above-mentioned processor is used to call the above-mentioned computer program so that the computer device executes the method in the embodiment of the present application.

[0092] On one hand, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded by a processor and executing the method in the embodiment of the present application.

[0093] On the one hand, an embodiment of the present application provides a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method in the embodiment of the present application.

[0094] In an embodiment of the present application, a material retrieval method for a joint query of text and image is proposed. First, a first material associated with the query text is obtained from a material database, and a second material associated with the query image semantic vector is obtained. It can be seen that the embodiment of the present application can obtain candidate materials (including the above-mentioned first material and second material) based on multiple dimensional information (including text dimension and image dimension); further, based on the target material image style vector, the target material image semantic vector, the target material text, the query text, the query image semantic vector and the query image style vector, the material similarity between the candidate material and the query information is determined. It can be seen that the material similarity in the embodiment of the present application is generated based on multiple dimensional information (including text dimension, image semantic dimension and image style dimension); further, the candidate materials are sorted according to the material similarity, and recommended materials are determined from the sorted candidate materials. It can be seen that the recommended material and the query information can have not only semantic relevance but also style relevance. In summary, the present application can improve the description accuracy of the retrieval purpose and thereby improve the accuracy of the recommended material by jointly inputting the query text and the query image. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0096] Figure 1 This is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0097] Figure 2 This is a schematic diagram of a data processing scenario provided by an embodiment of the present application;

[0098] Figure 3 This is a flow chart of a data processing method provided in an embodiment of the present application;

[0099] Figure 4 This is a schematic diagram of a data processing scenario provided by an embodiment of the present application;

[0100] Figure 5 This is a schematic diagram of a data processing scenario provided by an embodiment of the present application;

[0101] Figure 6 This is a flow chart of a data processing method provided in an embodiment of the present application;

[0102] Figure 7This is a schematic diagram of a data processing scenario provided by an embodiment of the present application;

[0103] Figure 8 This is a schematic diagram of a data processing scenario provided by an embodiment of the present application;

[0104] Figure 9 is a structural diagram of a data processing device provided in an embodiment of the present application;

[0105] Figure 10 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0106] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0107] To facilitate understanding, we first provide a brief explanation of some nouns:

[0108] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0109] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0110] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying, tracking, and measuring objects. It also performs image processing, transforming the computer's image into an image more suitable for human observation or transmission to instrumentation. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0111] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0112] The solutions provided in the embodiments of this application involve artificial intelligence computer vision technology and deep learning technology, which are specifically illustrated by the following embodiments.

[0113] See Figure 1 , Figure 1 This is a schematic diagram of a system architecture provided by an embodiment of the present application. Figure 1 As shown, the system may include a server 1000a and a user terminal cluster, and the user terminal cluster may include: user terminal 100a, user terminal 100b, user terminal 100c, ..., user terminal 100n. It can be understood that the above system may include one or more user terminals, and the number of user terminals will not be limited here.

[0114] Among them, there may be a communication connection between the user terminal clusters, for example, there is a communication connection between user terminal 100b and user terminal 100c, and there is a communication connection between user terminal 100b and user terminal 100n. At the same time, any user terminal in the user terminal cluster may have a communication connection with server 1000a, for example, there is a communication connection between user terminal 100b and server 1000a, and there is a communication connection between user terminal 100c and server 1000a. The above-mentioned communication connection does not limit the connection method, and can be directly or indirectly connected through a wired communication method, or directly or indirectly connected through a wireless communication method, or through other methods, and this application does not limit it here.

[0115] It should be understood that Figure 1 Each user terminal in the user terminal cluster shown in FIG. 1 may be installed with an application client. When the application client runs in each user terminal, it may be respectively connected to the above-mentioned Figure 1 The server 1000a shown in FIG. 1000a performs data exchange between the server 1000a, i.e., the communication connection described above. The application client may be a social client, a multimedia client (e.g., a video client), an entertainment client (e.g., a game client), an educational client, a live broadcast client, or other application client with an image loading function. The application client may be an independent client or an embedded sub-client integrated into a client (e.g., a social client, an educational client, a multimedia client, etc.), without limitation herein.

[0116] Server 1000a provides services to the user terminal cluster through the communication connection function. When the user terminal (which can be user terminal 100b, user terminal 100c or user terminal 100n) obtains the query text and query image input by the user and receives the material retrieval instruction triggered by the user, the user terminal can send the query information (including the above-mentioned query text and query image) to server 1000a. After receiving the query information sent by the user terminal, server 1000a obtains the first material associated with the query text from the material database. It can be understood that the material database includes image materials, data associated with the image materials (such as index key-value pairs corresponding to the image materials and material identification data), video materials, and data associated with the video materials (such as index key-value pairs corresponding to the video materials and material identification data). The embodiment of the present application does not limit the data content in the material database, and the data content can be set according to the actual application scenario. Among them, the first material can include image materials, video materials, image materials or video materials.

[0117] Server 1000a performs image classification processing on the query image based on a pre-trained image classification model to obtain image classification information corresponding to the query image. The image classification information can represent the image category of the query image, such as facial images, animal images, and plant images. Server 1000a inputs the query image into an image semantic extraction model associated with the image classification information. The image semantic extraction model extracts semantic features from the query image to obtain a query image semantic vector, i.e., an image semantic vector corresponding to the query image. Server 1000a inputs the query image into an image style extraction model associated with the image classification information. The image style extraction model extracts style features from the query image to obtain a query image style vector, i.e., an image style vector corresponding to the query image.

[0118] The server 1000a obtains the second material associated with the query image semantic vector from the material database. Similarly, the first material may include an image material, a video material, an image material, or a video material. It is understandable that the first material is a material determined based on the query text, and the server 1000a has not yet determined the association relationship between the first material and the query image. The second material is a material determined based on the query image semantic vector, and the server 1000a has not yet determined the association relationship corresponding to the query text, the query image style vector, and the second material. Therefore, the server 1000a determines the first material and the second material as candidate materials, and then obtains the target material image style vector corresponding to the candidate material, the target material image semantic vector corresponding to the candidate material, and the target material text corresponding to the candidate material, wherein the target material image style vector is used to represent the image style vector corresponding to the candidate material, the target material image semantic vector is used to represent the image semantic vector corresponding to the candidate material, and the target material text is used to represent the text corresponding to the candidate material.

[0119] Furthermore, server 1000a determines the material similarity between the candidate materials and the query information based on the query text, query image semantic vector, query image style vector, target material text, target material image semantic vector, and target material image style vector. The server 1000a then sorts the candidate materials based on the material similarity. Specifically, the server 1000a sorts the candidate materials from highest to lowest material similarity. Finally, the server 1000a determines a recommended material from the sorted candidate materials, i.e., the candidate material with the highest material similarity is determined as the recommended material. Subsequently, server 1000a can send the recommended material to the user terminal. The recommended material can include image material, video material, or image material or video material.

[0120] After the user terminal receives the recommended material sent by the server 1000a, the recommended material can be displayed on its corresponding screen. Optionally, if the user terminal locally stores the above-mentioned image classification model, image semantic extraction model, and image style extraction model, the query information can be identified locally in the user terminal, and its identification process is consistent with the process of server 1000a identifying the query information. At this time, the user terminal can send the identification information corresponding to the query information (which may include the query image style vector, the query image semantic vector, the query word corresponding to the query text, or the text semantic vector corresponding to the query text) to the server 1000a. The subsequent processing process of the server 1000a is consistent with the process described above, so it will not be repeated here.

[0121] Among them, since training the image classification model, image semantic extraction model and image style extraction model all involve a large amount of offline calculations, the image classification model, image semantic extraction model and image style extraction model local to the user terminal can be sent to the user terminal after training by server 1000a.

[0122] in, Figure 1 The server 1000a, user terminal 100a, user terminal 100b, user terminal 100c, ..., user terminal 100n may include a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, and the like.

[0123] It should be noted that the above-mentioned server 1000a, user terminal 100a, user terminal 100b, user terminal 100c,..., user terminal 100n can all be blockchain nodes in the blockchain network, and the data described in the full text (such as the above-mentioned target material text, target material image semantic vector, and target material image style vector) can be stored. The storage method can be that the blockchain node generates blocks based on the data and adds the blocks to the blockchain for storage.

[0124] Blockchain is a novel application model of computer technologies, integrating distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. It primarily organizes data in chronological order and encrypts it into a ledger, rendering it tamper-proof and forgery-proof. It also enables data verification, storage, and updates. A blockchain is essentially a decentralized database, in which each node stores an identical blockchain. A blockchain network can categorize nodes into core nodes, data nodes, and light nodes. Core nodes, data nodes, and light nodes collectively constitute blockchain nodes. Core nodes are responsible for consensus across the entire blockchain network; in other words, they serve as consensus nodes within the blockchain network. The process by which transaction data is written to the ledger in a blockchain network involves data nodes or light nodes acquiring transaction data and passing it around the network (in a relay-like fashion) until a consensus node receives the transaction data. The consensus node then packages the transaction data into a block, performs consensus on the block, and writes the transaction data to the ledger after consensus is complete. Here, the target material text, target material image semantic vector, and target material image style vector are used as examples of transaction data. After reaching a consensus on the transaction data, server 1000a (blockchain node) generates a block based on the transaction data and stores the block in the blockchain network; and for reading the transaction data (i.e., the target material text, target material image semantic vector, and target material image style vector), the blockchain node can obtain the block containing the transaction data in the blockchain network, and further obtain the transaction data in the block.

[0125] It is understandable that the data processing method provided in the embodiments of the present application can be executed by a computer device, including but not limited to a user terminal or a server. The above-mentioned server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The above-mentioned user terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, a smart TV, etc., but is not limited to this. The user terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0126] Further, see Figure 2 , Figure 2 This is a schematic diagram of a data processing scenario provided by an embodiment of the present application. Figure 2 The user terminal 20a in can be Figure 1For any user terminal in the user terminal cluster, server 20b is equivalent to Figure 1 Server 1000a in. Figure 2 As shown, user 201a can send query information to server 20b through user terminal 20a. In the embodiment of the present application, the query information includes query image 202a and query text 203a, that is, Figure 2 After receiving the query text 203a and the query image 202a sent by the user terminal 20a, the server 20b inputs the query text 203a into the text encoding model 201c, and performs text encoding processing on the query text 203a through the text encoding model 201c to obtain the encoding semantic vector corresponding to the query text 203a, that is, Figure 2 In the embodiment of the present application, the model type of the text encoding model 201c is not limited, and it can be composed of any one or more text encoding networks, such as Bag of Word (BOW) model, Vector Space Model (VSM) model, word embedding, etc.

[0127] The server 20b obtains an index key-value pair 201d whose index attribute is a text semantic attribute in the material database 20i. It can be understood that the index key-value pair 201d includes an index key and an index value, wherein the index key is a material identifier and the index value is a material text semantic vector (i.e., an encoded semantic vector corresponding to the material). The server 20b obtains the similarity corresponding to each index value (i.e., each material text semantic vector) and the query text semantic vector 201e, and compares the similarity corresponding to each index value with a preset text similarity threshold. Furthermore, the server 20b determines the index value whose similarity is greater than the text similarity threshold as the target index value, and determines the index key (i.e., the material identifier) corresponding to the target index key as the target index key. The server 20b can obtain the material corresponding to the target index key in the material database 20i, and determine it as the first material, such as Figure 2 The video material 2001a, ..., and image material 2001n are shown as examples.

[0128] The index database 20i in this embodiment can be thought of as an electronic filing cabinet—a place where electronic files (which may include index key-value pairs corresponding to various index attributes, massive amounts of material, etc.) are stored. Server 20b can perform operations such as adding, querying, updating, and deleting data in these files. A "database" is a collection of data that is stored in a specific manner, can be shared with multiple users, has minimal redundancy, and is independent of applications.

[0129] Please see again Figure 2 , the server 20b first obtains the image classification information of the query image 202a to determine what type of image the query image 202a belongs to, such as human type, animal type, and plant type. Further, the server 20b inputs the query image 202a into the image semantic extraction model 202c and the image style extraction model 203c associated with the image classification information, such as Figure 2 As shown, the semantic features of the query image 202a are extracted through the image semantic extraction model 202c to obtain the image semantic vector corresponding to the query image 202a, that is, Figure 2 The query image semantic vector 202e in the image style extraction model 203c is used to extract the style features of the query image 202a and obtain the image style vector corresponding to the query image 202a, that is, Figure 2 The query image style vector 203e in .

[0130] In this embodiment of the present application, the model type of the image semantic extraction model 202c is not limited and can be composed of any one or more neural network models, such as Conditional Random Fields (CRFs), Convolutional Neural Networks (CNNs), Fully Convolutional Networks (FCNs), etc. Similarly, in this embodiment of the present application, the model type of the image style extraction model 203c is not limited and can be set according to the actual application scenario.

[0131] In addition to obtaining the first material in the material database 20i according to the query text 203a, the present application also obtains the second material in conjunction with the query image 202a. The specific process can be: the server 20b obtains the index key-value pair 202d whose index attribute is the image semantic attribute in the material database 20i. It can be understood that the index key-value pair 202d includes an index key and an index value, wherein the index key is the material identifier and the index value is the material image semantic vector (i.e., the image semantic vector corresponding to the material image or the image semantic vector corresponding to the material video). The server 20b obtains the similarity corresponding to each index value and the query image semantic vector 202e, and compares the similarity corresponding to each index value with a preset similarity threshold (it should be understood that the similarity threshold here and the text similarity threshold mentioned above are two different similarity thresholds). Further, the server 20b determines the index value with a similarity greater than the similarity threshold as the target index value, and determines the index key corresponding to the target index value as the target index key. The server 20b can obtain the material corresponding to the target index key in the material database 20i and determine it as the second material, such as Figure 2The image materials 2002a, ..., and 2002n are shown in the example. It is understandable that in actual applications, the first material and the second material may be the same material, but the same material is determined based on different index keys, because the first material is associated with the query text 203a, and the second material is associated with the query image 202a.

[0132] In order to push highly accurate materials to user 201a, the present application comprehensively considers the semantic features corresponding to the query information and the image style corresponding to the query image 202a, and the server 20b determines the first material and the second material as candidate materials 20f, such as Figure 2 As shown, the candidate material 20f includes image material 2002a, ..., image material 2002n, video material 2001a, ..., image material 2001n. Further, the server 20b obtains the text semantic vector corresponding to each candidate material (such as Figure 2 The text semantic vector corresponding to the image material 2002a in the example is Figure 2 The target material text semantic vector 201g in the image semantic vector, each candidate material corresponds to the image semantic vector (such as Figure 2 The image semantic vector corresponding to the image material 2002a is: Figure 2 The target material image semantic vector 202g in the image is obtained, and the image style vector corresponding to each candidate material is (such as Figure 2 The image style vector corresponding to the image material 2002a in the example is Figure 2 The target material image style vector 203g).

[0133] Please see again Figure 2 The server 20b determines the material similarity between the image material 2002a and the query information based on the query text semantic vector 201e, the query image semantic vector 202e, the query image style vector 203e, the target material text semantic vector 201g, the target material image semantic vector 202g, and the target material image style vector 203g. It can be understood that the determination of other candidate materials (such as Figure 2 The material similarities between the image material 2002n, video material 2001a, and image material 2001n shown in the example and the query information are consistent with the process of determining the material similarity between the image material 2002a and the query information, so they will not be described here in detail. You can refer to the above for understanding. In addition, the specific process of determining the material similarity between the candidate material and the query information can be found below. Figure 3 The description of step S104 in the corresponding embodiment will not be expanded here.

[0134] After determining the material similarity corresponding to each candidate material, the server 20b sorts the candidate materials 20f according to the material similarity. Specifically, the candidate materials 20f are sorted in descending order of material similarity. Assuming that the material similarity corresponding to the image material 2002a is 0.98, ..., the material similarity corresponding to the image material 2002n is 0.8, the material similarity corresponding to the video material 2001a is 0.95, ..., and the material similarity corresponding to the image material 2001n is 0.9, then Figure 2 As shown, the server 20b ranks the candidate materials 20f as image material 2002a, video material 2001a, ..., image material 2001n, image material 2002n. Finally, the server 20b determines the recommended material 20h from the ranked candidate materials, as shown in FIG. Figure 2 As shown, the recommended material 20h may include image material 2002a, ..., and video material 2001a.

[0135] In summary, the input data of the embodiment of the present application includes a query image and a query text, and the output data is a video material or image material that matches the semantics (including text semantics and image semantics) and is consistent with the query image style. A user's query includes three stages: query information input, candidate material recall, and candidate material sorting. The candidate material recall stage includes the first material recall based on the query text and the second material recall based on the query image; the candidate material sorting stage sorts the recalled candidate materials, and puts candidate images with similar styles and semantics to the query image in front, and returns them to the user terminal, so as to achieve the return of recommended materials (image materials or video materials) that are consistent with the input image style and match the input semantics.

[0136] This application embodiment proposes a material retrieval method that, through a combined search of text and images, can search for semantically similar image or video materials, while maintaining a consistent style for the retrieved materials. This method can serve video production, creative advertising, search engines, and material retrieval, as well as applications in video and other multimedia-related connected vehicle scenarios or smart transportation. It is understood that by searching for both images and text, users can better characterize their search objectives and accurately obtain recommended materials; through consistent style, users can create higher-quality, more effective videos and advertising works.

[0137] Further, see Figure 3 , Figure 3 This is a flow chart of a data processing method provided by an embodiment of the present application. The data processing method can be executed by a computer device, which may include Figure 1 The user terminals and servers in Figure 3As shown, the data processing process may include the following steps.

[0138] Step S101: Acquire query information including query text and query image, and acquire first material associated with the query text from a material database.

[0139] Specifically, the query text is input into the text encoding model, and the query text semantic vector corresponding to the query text is obtained through the text encoding model; an index key-value pair whose index attribute is the text semantic attribute is obtained in the material database; the index key-value pair includes an index key generated by the material identifier, and an index value generated by the material text semantic vector; the similarity between the query text semantic vector and the index value is obtained, and the index value whose similarity is greater than the similarity threshold is determined as the target index value; the index key corresponding to the target index value is determined as the target index key, and the first material corresponding to the target index key is obtained in the material database.

[0140] Optionally, the query text is segmented to obtain query segmentation words corresponding to the query text; an index key-value pair whose index attribute is a segmentation attribute is obtained from the material database; the index key-value pair includes an index key generated by a material identifier and an index value generated by a material segmentation word; the index key corresponding to the index value containing the query segmentation word is determined as a target index key, and the first material corresponding to the target index key is obtained from the material database.

[0141] The computer device obtains query information, wherein the query information includes query text and query image. The computer device independently obtains candidate materials based on the query text and query image. There are two ways to obtain the first material associated with the query text from the material database. The first way is to obtain the first material based on the query text semantic vector corresponding to the query text. The specific process of obtaining the first material is described above. Figure 2 The description in the corresponding embodiment will not be repeated here. The second way is a calculation method based on the relevance of text core word matching, which can be specifically as follows: segmenting the query text to obtain query segmentation words corresponding to the query text, for example: dividing the query text "how marketing companies solve the communication between companies and users" into 8 segmentation words "marketing", "company", "how", "solve", "with", "user", "of", and "communication". Among them, there are two "company" segmentation words in the above query text, so there are 8 different segmentation words; further, the computer device can filter the above 8 segmentation words to obtain query segmentation words, for example, filtering the segmentation word "of" in the above example, and using the remaining segmentation words as query segmentation words.

[0142] The word segmentation level can be the word level, such as the above-mentioned word "communication", or it can be the character level, such as further dividing the word "communication" into "ditch" and "communication", or it can be the sub-word level, such as dividing "advertising" into "ad", "ver", "tis" and "ing". The embodiment of this application does not limit the word segmentation level and can be set according to the actual application scenario.

[0143] The embodiments of the present application do not limit the word segmentation algorithm, and may include a word segmentation method based on string matching, a word segmentation method based on understanding, and a word segmentation method based on statistics, etc.

[0144] The computer device obtains an index key-value pair whose index attribute is a segmentation attribute from a material database. The index key-value pair includes an index key generated by a material identifier and an index value generated by a material segmentation. Further, the computer device matches the query segmentation with the index value in the index key-value pair and determines the index value containing the query segmentation as the target index value. It can be understood that the index value containing the query segmentation means that the material segmentation includes the query segmentation, or there is a segmentation in the material segmentation that is highly similar to the query segmentation. For example, the material segmentation includes "beautiful, girl" and the query segmentation includes "pretty, girl". Although the material segmentation and the query segmentation are different, the two are highly similar in semantics, so they can be determined as target material segmentations, that is, target index values. The computer device determines the index key corresponding to the target index value as the target index key. It can be understood that the material identifier can represent the material, so the first material corresponding to the target index key can be obtained from the material database.

[0145] Step S102 : obtaining a query image style vector corresponding to the query image and a query image semantic vector corresponding to the query image, and obtaining a second material associated with the query image semantic vector from a material database.

[0146] Specifically, the query image is input into the image classification model, and the image classification information of the query image is obtained through the image classification model; an image semantic extraction model associated with the image classification information and an image style extraction model associated with the image classification information are obtained; the query image is input into the image semantic extraction model, and the query image semantic vector corresponding to the query image is obtained through the image semantic extraction model; the query image is input into the image style extraction model, and the image color features of the query image and the image shape features of the query image are extracted through the image style extraction model, and feature aggregation processing is performed on the image color features of the query image and the image shape features of the query image to obtain a query image style vector corresponding to the query image.

[0147] Specifically, an index key-value pair whose index attribute is an image semantic attribute is obtained from a material database; the index key-value pair includes an index key generated by a material identifier, and an index value generated by a material image semantic vector; based on the query image semantic vector and the index key-value pair, a second material associated with the query image semantic vector is obtained from the material database.

[0148] For image materials, it not only contains information such as the subject and scene, that is, "what is in the image", but also contains complex information such as subject attributes and the relationship between multiple subjects; for video materials, it contains richer action information and interaction information, so it is difficult to accurately describe the material with text information alone. In actual applications, when creators want to splice retrieved video clips together, they generally hope that the styles corresponding to multiple video clips are consistent or highly similar. Therefore, in an embodiment of the present application, the computer device expands the material recall capability through the image semantic vector, and maintains the similarity or consistency between the query image style vector and the material image style vector.

[0149] Optionally, in order to accurately obtain the query image style vector and query image semantic vector, the computer device first preprocesses the acquired image. The preprocessing step includes cropping irrelevant information in the image, such as cropping the black edges, white edges, lace edges, etc. of the image, and determining the cropped image as the query image. Please refer to Figure 4 , Figure 4 This is a schematic diagram of a data processing scenario provided by an embodiment of the present application. Figure 4 As shown, the computer device 40a obtains the image 40b. Obviously, the image 40b contains a white edge 401b. At this time, the computer device 40a first performs region recognition processing on the image 40b to obtain the region to be cropped of the image 40b, that is, Figure 4 The white edge 401b in the image 40b is removed, and the area to be cropped in the image 40b is cropped to obtain the query image 402b.

[0150] It is understandable that different types of images are suitable for different feature extraction models, such as faces, animals, plants and other subcategories, so the computer device needs to perform image classification processing on the query image first. Figure 4 As shown, the computer device 40a inputs the query image 402b into the image classification model 40c, and the image classification model 40c extracts the image category features to obtain the image classification information corresponding to the query image 402b, as shown in FIG. Figure 4After determining the image classification information corresponding to the query image 402b, the computer device 40a obtains the image semantic extraction model associated with the image classification information and the image style extraction model associated with the image classification information. The process of the computer device obtaining the query image semantic vector through the image semantic extraction model can be seen in the above Figure 2 Description in the corresponding embodiment.

[0151] For the same query image, different deep convolutional neural networks can be used to calculate semantic vectors with different characteristics, such as query local semantic vectors that highlight local semantics, query subject semantic vectors that highlight subject semantics, and query overall semantic vectors that highlight overall semantics. This can enhance the multi-level representation capability of semantic vectors (embedding), improve recall and retrieval effects, and meet various retrieval requirements. The description of various semantic vectors is not expanded here, please refer to the following Figure 6 The description of step S208 in the corresponding embodiment.

[0152] The computer device obtains an index key-value pair whose index attribute is an image semantic attribute from the material database, and obtains a second material associated with the query image semantic vector from the material database based on the query image semantic vector and the index key-value pair whose index attribute is an image semantic attribute. For the specific process, please refer to the above. Figure 2 The description in is not repeated here.

[0153] It is understandable that image style can be divided into two levels: color and shape. Please refer to Figure 4 , the image style extraction model may include a convolution layer and a pooling layer. The present application does not limit the size of the convolution kernel in the convolution layer, which can be set according to the actual application scenario. The present application does not limit the pooling method of the pooling layer, which can be maximum pooling, average pooling, etc. The computer device 40a inputs the query image 402b into the image style extraction model, and extracts the low-dimensional features of the query image 402b through the low-level feature extraction layer (including the convolution layer and the pooling layer). The low-dimensional features tend to be color style, so the image color features are obtained. By inputting the low-dimensional features into the high-level feature extraction layer (including the convolution layer and the pooling layer), the high-dimensional features of the query image 402b are extracted. The high-dimensional features tend to be shape (including pattern and texture) style, so the image shape features are obtained. The embodiment of the present application can perform feature aggregation processing on the image color features of the query image 402b and the image shape features of the query image to obtain the query image style vector corresponding to the query image 402b.

[0154] Optionally, in actual applications, computer device 40a may generate a query image style vector based on the mid-dimensional features output by the mid-level feature extraction layer. Alternatively, computer device 40a may generate a query image style vector based on low-dimensional features, i.e., based on color features. Alternatively, computer device 40a may generate a query image style vector based on high-dimensional features, i.e., based on shape features.

[0155] It can be understood that the low-level feature extraction layer, the middle-level feature extraction layer and the high-level feature extraction layer in the embodiments of the present application should be set according to the actual application scenario.

[0156] Optionally, in an embodiment of the present application, for the style characterization of image patterns, traditional feature point detection methods such as scale-invariant feature transform (SIFT), Oriented FAST and Rotated BRIEF (ORB, a fast feature point extraction and description algorithm) can also be used to extract style features.

[0157] Step S103 : determining the first material and the second material as candidate materials, obtaining target material image style vectors corresponding to the candidate materials, target material image semantic vectors corresponding to the candidate materials, and target material texts corresponding to the candidate materials.

[0158] Specifically, the material identifier corresponding to the candidate material is determined as the candidate material identifier; in the material database, the target material image style vector stored in association with the candidate material identifier, the target material main semantic vector stored in association with the candidate material identifier, the target material overall semantic vector stored in association with the candidate material identifier, the target material local semantic vector stored in association with the candidate material identifier, and the target material text stored in association with the candidate material identifier are obtained; the target material main semantic vector, the target material overall semantic vector, and the target material local semantic vector are determined as the target material image semantic vector.

[0159] It is understandable that no matter what the index attribute of the index key-value pair is, the index key in each index key-value pair is a material identifier.

[0160] The first material associated with the query text and the second material associated with the query image semantic vector may have different image styles from the query image. In order to obtain recommended materials that meet the user's needs, the embodiment of the present application determines the first material and the second material as candidate materials. For the sake of ease of understanding and description, the embodiment of the present application only uses Figure 2The image material 2002a in the example of this step, the processing process of other candidate materials in the candidate material can refer to this step and the description of the image material 2002a in step S104. Figure 5 , Figure 5 This is a schematic diagram of a data processing scenario provided by an embodiment of the present application. Figure 5 As shown, the computer device obtains the material identifier corresponding to the image material 2002a and determines it as a candidate material identifier, such as Figure 5 In the example of identifier 1, it can be understood that the material identifier can be any information that can be used to identify the material, such as a hash value generated by a computer device performing a hash operation on the material name through a hash algorithm.

[0161] like Figure 5 As shown, the material database 40f stores associated material identifiers and multiple vectors corresponding to the materials. The multiple vectors may include a material image style vector corresponding to the material, a material image semantic vector corresponding to the material, and a material text semantic vector corresponding to the material. For example, identifier 1 is associated with the material image style vector 203g, the material image semantic vector 202g, and the material text semantic vector 201g; identifier 2 is associated with the material image style vector 403e, the material image semantic vector 402e, and the material text semantic vector 401e;…, identifier z is associated with the material image style vector 403z, the material image semantic vector 402z, and the material text semantic vector 401z.

[0162] The computer device matches the material identifier (i.e., identifier 1) corresponding to the image material 2002a with the material identifier in the material database 40f, and can then determine that the material image style vector 203g is the target material image style vector, can determine that the material image semantic vector 202g is the target material image semantic vector, and can determine that the material text semantic vector 201g is the target material text semantic vector.

[0163] Optionally, when the material identification and the material segmentation are associated and stored in the material database, the computer device may obtain the material segmentation corresponding to the candidate material identification, and further determine the text similarity between the material segmentation corresponding to the candidate material identification and the query segmentation.

[0164] Step S104, based on the query text, query image semantic vector, query image style vector, target material text, target material image semantic vector, and target material image style vector, determine the material similarity between the candidate material and the query information, sort the candidate materials according to the material similarity, and determine the recommended material from the sorted candidate materials.

[0165] Specifically, the total number of candidate materials is at least two, and the at least two candidate materials include candidate material C d , d is a positive integer, and d is less than or equal to the total number of at least two candidate materials; the target material text includes candidate material C d Corresponding target material text E d ; The target material image semantic vector includes the candidate material C d The corresponding target material image semantic vector F d ; The target material image style vector includes the candidate material C d The corresponding target material image style vector G d . Get the query text and target material text E d The similarity between them is determined as the candidate material C d Corresponding text similarity; obtain the query image semantic vector and the target material image semantic vector F d The similarity between them is determined as the candidate material C d Corresponding semantic similarity; obtain the query image style vector and the target material image style vector G d The similarity between them is determined as the candidate material C d Corresponding style similarity; perform weighted summation of text similarity, semantic similarity and style similarity to obtain candidate material C d The material similarity between the candidate material and the query information; and sorting the at least two candidate materials according to the material similarity corresponding to each candidate material.

[0166] The query image semantic vector includes the query subject semantic vector, the query overall semantic vector and the query local semantic vector; the target material image semantic vector F d Including the target material main semantic vector H d , the overall semantic vector of the target material I d And the target material local semantic vector J d . Determine candidate material C d The specific process of the corresponding semantic similarity may include: obtaining the query subject semantic vector and the target material subject semantic vector H d The similarity between them is determined as the candidate material C d The corresponding subject semantic similarity; obtain the query overall semantic vector and the target material overall semantic vector I d The similarity between them is determined as the candidate material C d Corresponding overall semantic similarity; obtain the query local semantic vector and the target material local semantic vector J dThe similarity between them is determined as the candidate material C d Corresponding local semantic similarity; the main semantic similarity, the overall semantic similarity and the local semantic similarity are determined as semantic similarity.

[0167] Computer equipment can obtain tens of thousands of relevant results through text-based and image-based searches. Therefore, it is necessary to sort them by style and semantics to obtain high-quality results with similar style and semantics, and use high-quality results as recommendation materials. Figure 5 , the computer device obtains the target material image style vector corresponding to the image material 2002a, that is, Figure 5 The material image style vector 203g in the image material 2002a is used to obtain the target material image semantic vector corresponding to the image material 2002a, that is, Figure 5 The material image semantic vector 202g in the image material 2002a is obtained, and the target material text semantic vector corresponding to the image material 2002a, that is, the material text semantic vector 201g, is obtained. The computer device performs similarity calculation on the material image style vector 203g and the query image style vector 203e to obtain the style similarity 1 corresponding to the image material 2002a, performs similarity calculation on the material image semantic vector 202g and the query image semantic vector 202e to obtain the semantic similarity 1 corresponding to the image material 2002a, and performs similarity calculation on the material text semantic vector 201g and the query text semantic vector 201e to obtain the text similarity 1 corresponding to the image material 2002a. The embodiment of the present application does not limit the similarity algorithm, and can be set according to the actual application scenario, including but not limited to cosine similarity, Euclidean distance, and Jaccard similarity.

[0168] In the embodiment of the present application, the method for obtaining text similarity also includes the method described in step S103 above, that is, determining text similarity by querying the segmented words and the overlap between the material segmented words.

[0169] Please see again Figure 5 The computer device performs weighted summation processing on the text similarity 1, the semantic similarity 1 and the style similarity 1 to obtain the material similarity 1 between the image material 2002a and the query information.

[0170] After obtaining the material similarity corresponding to each candidate material, the candidate materials are sorted based on the material similarity. For example, the candidate materials are sorted in descending order of material similarity. Candidate materials whose material similarity is greater than a preset material similarity threshold can be determined as recommended materials. Candidate materials within a preset number of the sorted candidate materials can also be determined as recommended materials. For example, if there are 1,000 candidate materials in total, the first 20 candidate materials in the sorted candidate materials can be determined as recommended materials.

[0171] By comprehensively sorting candidate materials based on semantic relevance and style relevance, this application can identify materials with similar semantics and styles as recommended materials. In addition, through image-assisted text input, semantic information that is difficult to describe in images can be expressed more accurately. In addition, by establishing a semantic vector index for images, this application can perform retrieval based on image semantics, with stronger recall capabilities and richer recall results.

[0172] This application can solve the following problems:

[0173] 1. Image style and some semantics are difficult to describe in words. The style of many artistic images is difficult to accurately describe in words, which makes it inconvenient to search, and thus makes many materials difficult to search with words. With this application, the input image can be used as an auxiliary search.

[0174] 2. Image search alone makes it difficult to determine search intent. A picture is worth a thousand words, and images contain rich information, such as clothing textures and patterns. For example, if you input a portrait image for image search, it is unclear whether the search target is an image of the same person, clothing of the same shape, or images of the same texture or color. Using this application, by combining image and text input, it helps to more accurately describe the search purpose.

[0175] 3. How to make the retrieved videos and images consistent with the style of the input images. For example, in the process of retrieving and generating corresponding video creatives for advertising given a novel plot description, for a description such as "1. Fierce battle with the enemy, 2. Being forced to jump off a cliff due to a desperate situation, 3. Accidentally acquiring martial arts, 4. Duel with the enemy, 5. Marrying a rich girl", when splicing together 5 different retrieved materials, the creator hopes that the styles of the 5 materials are consistent. With this application, the previous video material can be used as the input for retrieving the next video, so the styles of different materials can be kept consistent. Therefore, after splicing different materials together, a better viewing experience can be produced, thereby improving the quality of the video.

[0176] In an embodiment of the present application, a material retrieval method for a joint query of text and image is proposed. First, a first material associated with the query text is obtained from a material database, and a second material associated with the query image semantic vector is obtained. It can be seen that the embodiment of the present application can obtain candidate materials (including the above-mentioned first material and second material) based on multiple dimensional information (including text dimension and image dimension); further, based on the target material image style vector, the target material image semantic vector, the target material text, the query text, the query image semantic vector and the query image style vector, the material similarity between the candidate material and the query information is determined. It can be seen that the material similarity in the embodiment of the present application is generated based on multiple dimensional information (including text dimension, image semantic dimension and image style dimension); further, the candidate materials are sorted according to the material similarity, and recommended materials are determined from the sorted candidate materials. It can be seen that the recommended material and the query information can have not only semantic relevance but also style relevance. In summary, the present application can improve the description accuracy of the retrieval purpose and thereby improve the accuracy of the recommended material by jointly inputting the query text and the query image.

[0177] Further, see Figure 6 , Figure 6 This is a flow chart of a data processing method provided by an embodiment of the present application. The data processing method can be executed by a computer device, which may include Figure 1 The user terminals and servers in Figure 6 As shown, the data processing process may include the following steps.

[0178] Step S201 : obtaining video material, extracting at least two video frames from the video material, and performing image preprocessing on the at least two video frames to obtain target video frames.

[0179] Specifically, at least two video frames include video frame A b , b is a positive integer, and b is less than or equal to the total number of at least two video frames; Get video frame A b and the similarity between the remaining video frames; the remaining video frames include at least two video frames except video frame A b If the similarity is equal to or greater than the similarity threshold, then the video frame A b Determine that the video frame is a duplicate video frame, delete the duplicate video frame from at least two video frames, and obtain a video frame to be cropped; perform region recognition processing on the video frame to be cropped to obtain a region to be cropped of the video frame to be cropped; perform cropping processing on the region to be cropped in the video frame to be cropped to obtain a target video frame.

[0180] Please also see Figure 7 , Figure 7This is a schematic diagram of a data processing scenario provided by an embodiment of the present application. Figure 7 As shown, the computer device obtains material 60a, which may include image material, such as Figure 7 The image material 601a of the example, and the video material, such as Figure 7 Example video material 602a. The computer device first pre-processes the material 60a. The pre-processing steps include video segmentation and frame extraction. This application does not limit the video segmentation method, and it can be implemented by video segmentation application software. Frame extraction mainly filters out similar video frames and retains video frames with low similarity.

[0181] It is understandable that video materials have additional video segmentation and frame processing than image materials. The processing processes corresponding to the video frames to be cropped and the image materials are consistent. Therefore, in this application, only the video frames to be cropped are taken as an example to describe the process of constructing the index key-value pairs corresponding to the materials. If index key-value pairs are constructed for image materials, please refer to the description below.

[0182] The computer device performs region recognition processing on the video frame to be cropped, mainly identifying irrelevant information such as black edges, white edges, lace, and marking areas (such as watermarks) of the video frame to be cropped, obtaining the region to be cropped, and cropping the region to be cropped in the video frame to obtain the target video frame.

[0183] Step S202: Obtain a video frame identifier corresponding to the target video frame, and use the video frame identifier as a material identifier.

[0184] Specifically, the embodiment of the present application can use a hash algorithm to perform hash calculation on information such as the name or storage address of the target video frame to obtain a unique hash value, use the hash value as the video frame identifier, and then use the video frame identifier as the material identifier.

[0185] Step S203, obtain the video frame text corresponding to the target video frame, use the video frame text as the material text, construct an index key-value pair with the index attribute as the text attribute according to the material identifier and the material text, and store the index key-value pair with the index attribute as the text attribute in the material database.

[0186] Specifically, the video description text carried by the video material is obtained, and the obtained video description text is determined as the first video frame text; the scene information in the target video frame is identified and processed to obtain scene recognition text, and the scene recognition text is determined as the second video frame text; the text information in the target video frame is extracted, and the extracted text information is determined as the third video frame text; the first video frame text, the second video frame text and the third video frame text are determined as the video frame text.

[0187] Specifically, text attributes include word segmentation attributes and text semantic attributes; the material text is segmented to obtain the material word segmentation corresponding to the material text, the material identifier is used as the index key, the material word segmentation is used as the index value, and an index key-value pair whose index attribute is the word segmentation attribute is constructed; the material text semantic vector corresponding to the material text is obtained, the material identifier is used as the index key, the material text semantic vector is used as the index value, and an index key-value pair whose index attribute is the text semantic attribute is constructed.

[0188] Please also see Figure 8 , Figure 8 This is a schematic diagram of a data processing scenario provided by an embodiment of the present application. Figure 8 As shown, when establishing an index library, the computer device can obtain the tags and descriptions carried by the video material 602a itself, that is, the video description text; the deep neural network can be used to identify and process the scene information in the target video frame 80a to obtain the scene recognition text; for materials that contain text, optical character recognition (OCR) technology can be used to extract the text in the image. For example, the text information "Shop" in the target video frame 80a can be extracted, and the extracted text information can be determined as the third video frame text. It is understandable that in actual application, any one or more texts can be obtained from the first video frame text, the second video frame text, and the third video frame text as the video frame text.

[0189] When a computer device determines a video frame text as a material text and creates an index pair for the material text, there are two ways, such as Figure 8 As shown, the material text can be segmented to obtain the material segmentation corresponding to the material text, and the material identifier is used as the index key, the material segmentation is used as the index value, and an index key-value pair whose index attribute is the segmentation attribute is constructed; the material text semantic vector corresponding to the material text can be obtained, and the material identifier is used as the index key, the material text semantic vector is used as the index value, and an index key-value pair whose index attribute is the text semantic attribute is constructed. It is understandable that in actual application, one or both of the above two methods can be selected to construct the index key-value pair, and the embodiment of the present application does not limit this.

[0190] Step S204 , obtaining a semantic vector of the material image corresponding to the target video frame and a style vector of the material image corresponding to the target video frame.

[0191] Specifically, the material image semantic vector includes the material main semantic vector, the material overall semantic vector and the material local semantic vector; the target video frame is input into the image classification model, and the material classification information of the target video frame is obtained through the image classification model; the target video frame is input into the material main extraction model associated with the material classification information, and the main semantic features of the target video frame are extracted through the material main extraction model to obtain the material main semantic vector; the target video frame is input into the material local extraction model associated with the material classification information, and the local semantic features of the target video frame are extracted through the material local extraction model to obtain the material local semantic vector; the target video frame is input into the material overall extraction model associated with the material classification information, and the overall semantic features of the target video frame are extracted through the material overall extraction model to obtain the material overall semantic vector; the target video frame is input into the material style extraction model associated with the material classification information, and the image color features and image shape features of the target video frame are extracted through the material style extraction model, and feature aggregation processing is performed on the image color features and image shape features of the target video frame to obtain the material image style vector.

[0192] The image itself also contains rich semantic information. The scene recognition information used in the image may not be perfect or there may be a probability of error, resulting in inaccurate classification. In addition, the results of discrete material text recall are likely to be significantly different from the retrieval purpose. Therefore, the material image semantic vector corresponding to the image material or target video frame can supplement the text recall results.

[0193] It is understandable that the material image semantic vector may include at least one characteristic vector. The embodiment of the present application uses the material main semantic vector, the material overall semantic vector and the material local semantic vector as examples. In actual application, it can also be a semantic vector of other characteristics. Figure 7 , the computer device inputs the target video frame into the image classification model, and obtains the material classification information of the target video frame through the image classification model; inputs the target video frame into the material main extraction model 601b associated with the material classification information, extracts the main semantic features of the target video frame through the material main extraction model 601b, and obtains the material main semantic vector 601c; the computer device inputs the target video frame into the material local extraction model 602b associated with the material classification information, extracts the local semantic features of the target video frame through the material local extraction model 602b, and obtains the material local semantic vector 602c; inputs the target video frame into the material overall extraction model 603b associated with the material classification information, extracts the overall semantic features of the target video frame through the material overall extraction model 603b, and obtains the material overall semantic vector 603c; wherein, the generation process of the material image style vector can be referred to above. Figure 3The generation process of the query image style vector in step S102 in the corresponding embodiment will not be repeated here.

[0194] Among them, the material body extraction model can be the same as the query body extraction model described below, the material overall extraction model can be the same as the query overall extraction model described below, the material local extraction model can be the same as the query local extraction model described below, and the material style extraction model can be the same as the image style extraction model described below.

[0195] It is understandable that the present application can meet a variety of retrieval needs by using material semantic vectors with multiple characteristics (such as material main body semantic vector, material overall semantic vector and material local semantic vector).

[0196] Step S205 : Using the material identifier as the index value and the material image semantic vector as the index key, an index key-value pair whose index attribute is the image semantic attribute is constructed, and the index key-value pair whose index attribute is the image semantic attribute is stored in the material database.

[0197] Specifically, the output of each feature extraction model is a multidimensional vector. It can be understood that the vector dimensions corresponding to the material main semantic vector, the vector dimensions corresponding to the material overall semantic vector, and the vector dimensions corresponding to the material local semantic vector can be the same or different.

[0198] Please see again Figure 7 In an embodiment of the present application, the computer device can use the material body semantic vector 601c as the index body value and the material identifier as the index body key to construct an index body key-value pair whose index attribute is the image body semantic attribute; it can use the material local semantic vector 602c as the index local value and the material identifier as the index local key to construct an index local key-value pair whose index attribute is the image local semantic attribute; it can use the material overall semantic vector 603c as the index overall value and the material identifier as the index overall key to construct an index overall key-value pair whose index attribute is the image overall semantic attribute.

[0199] Generally, the material main semantic vector 601c, the material local semantic vector 602c and the material overall semantic vector 603c are all high-dimensional vectors. Large-scale high-dimensional vector storage and construction of index key-value pairs will take up a lot of space, and query retrieval will also consume more time and increase costs. Therefore, optionally, the computer device performs vector compression processing on the material main semantic vector 601c, the material local semantic vector 602c and the material overall semantic vector 603c respectively. Figure 7As shown, the computer device performs vector compression processing on the material main body semantic vector 601c through a compression method, for example, reducing the 1024-dimensional material main body semantic vector 601c to a 128-dimensional material main body semantic vector 601d. Similarly, the material local semantic vector 602c is vector compressed to obtain the material local semantic vector 602d, and the material overall semantic vector 603c is vector compressed to obtain the material overall semantic vector 603d.

[0200] The present application does not limit the compression method and can be set according to the actual application scenario, including but not limited to PCA dimensionality reduction (Principal Component Analysis, a data analysis method used to reduce the dimensionality of high-dimensional data), vector 01 quantization, and product quantization (PQ). It is understood that vector compression can greatly save vector storage space.

[0201] Optionally, the computer device can use the material body semantic vector 601d as the index body value and the material identifier as the index body key to construct an index body key-value pair whose index attributes are the image body semantic attributes; it can use the material local semantic vector 602d as the index local value and the material identifier as the index local key to construct an index local key-value pair whose index attributes are the image local semantic attributes; it can use the material overall semantic vector 603d as the index overall value and the material identifier as the index overall key to construct an index overall key-value pair whose index attributes are the image overall semantic attributes.

[0202] It is understandable that after the high-dimensional semantic vector is compressed, the retrieval effect of the vector will be deteriorated. Therefore, post-processing is required to ensure that the retrieval effect does not drop significantly after the vector is compressed. This application uses the k-nearest neighbor retrieval method to search the semantic vectors after dimensionality reduction (including Figure 7 The material main semantic vector 601d, the material local semantic vector 602d, and the material overall semantic vector 603d) are post-processed, that is, the calculated semantic vector is re-encoded using the semantic vector of the k-nearest neighbor retrieval result, as shown in FIG. Figure 7 As shown, the material main body semantic vector 601d is re-encoded to obtain the material main body semantic vector 601e, the material local semantic vector 602d is re-encoded to obtain the material local semantic vector 602e, and the material overall semantic vector 603d is re-encoded to obtain the material overall semantic vector 603e. The re-encoded semantic vectors can enhance the retrieval capability.

[0203] Optionally, the computer device can use the material body semantic vector 601e as the index body value and the material identifier as the index body key to construct an index body key-value pair whose index attributes are the image body semantic attributes; it can use the material local semantic vector 602e as the index local value and the material identifier as the index local key to construct an index local key-value pair whose index attributes are the image local semantic attributes; it can use the material overall semantic vector 603e as the index overall value and the material identifier as the index overall key to construct an index overall key-value pair whose index attributes are the image overall semantic attributes.

[0204] Step S206: Associate the material image style vector and the material identifier and store them in the material database.

[0205] Specifically, the computer device may also associate and store the material main body semantic vector and the material identification in the material database, associate and store the material local semantic vector and the material identification in the material database, and associate and store the material overall semantic vector and the material identification in the material database.

[0206] Step S207: Acquire query information including the query text and the query image, and acquire the first material associated with the query text from the material database.

[0207] Specifically, the specific process of step S206-step S207 can be found in the above Figure 3 The description in the corresponding embodiment will not be repeated here.

[0208] Step S208 : obtaining a query image style vector corresponding to the query image and a query image semantic vector corresponding to the query image, and obtaining a second material associated with the query image semantic vector from a material database.

[0209] Specifically, the image semantic extraction model includes an image subject extraction model, an image local extraction model, and an image overall extraction model. A query image is input into the image subject extraction model, where the image subject extraction model extracts the main semantic features of the query image to obtain a query subject semantic vector. The query image is input into the image local extraction model, where the image local extraction model extracts the local semantic features of the query image to obtain a query local semantic vector. The query image is input into the image overall extraction model, where the image overall extraction model extracts the overall semantic features of the query image to obtain a query overall semantic vector. The query subject semantic vector, the query overall semantic vector, and the query local semantic vector are determined as the query image semantic vector.

[0210] Specifically, the query image semantic vector includes the query subject semantic vector, the query overall semantic vector and the query local semantic vector; the material image semantic vector includes the material subject semantic vector, the material overall semantic vector and the material local semantic vector; the index key-value pair includes the index subject key-value pair, the index overall key-value pair and the index local key-value pair; the index subject key-value pair includes the index subject key generated by the material identifier, and the index subject value generated by the material subject semantic vector; the index local key-value pair includes the index local key generated by the material identifier, and the index local value generated by the material local semantic vector; the index overall key-value pair includes the index overall key generated by the material identifier, and the index overall value generated by the material overall semantic vector. Obtain a first similarity between the query subject semantic vector and the index subject value, determine the index subject value whose first similarity is greater than the first similarity threshold as the target index subject value, determine the index subject key corresponding to the target index subject value as the target index subject key, and obtain the subject material corresponding to the target index subject key in the material database; obtain a second similarity between the query overall semantic vector and the index overall value, determine the index overall value whose second similarity is greater than the second similarity threshold as the target index overall value, determine the index overall key corresponding to the target index overall value as the target index overall key, and obtain the overall material corresponding to the target index overall key in the material database; obtain a third similarity between the query local semantic vector and the index local value, determine the index local value whose third similarity is greater than the third similarity threshold as the target index local value, determine the index local key corresponding to the target index local value as the target index local key, and obtain the local material corresponding to the target index local key in the material database; determine the subject material, local material, and overall material as the second material.

[0211] The embodiments of the present application use different image semantic vectors, such as semantic vectors that are biased towards overall characteristics (i.e., query overall semantic vectors), semantic vectors that are biased towards local characteristics (i.e., query local semantic vectors), semantic vectors that are biased towards subject characteristics (i.e., query subject semantic vectors), etc. for recall, thereby improving the expressive power of the query image and the diversity of the recall results, and providing multiple options.

[0212] It is understood that the process by which the computer device acquires the query overall semantic vector corresponding to the query image is identical to the process by which the material overall semantic vector corresponding to the image material is acquired. Therefore, please refer to the description of step S204 above and will not be repeated here. Similarly, the process by which the computer device acquires the query local semantic vector corresponding to the query image, as well as the process by which the computer device acquires the query main semantic vector corresponding to the query image, will be referred to the description of step S204 above and will not be repeated here.

[0213] It can be understood that the vector dimensions corresponding to the query overall semantic vector, the query local semantic vector, and the query subject semantic vector can be the same or different; the vector dimension corresponding to the query overall semantic vector is equivalent to the vector dimension corresponding to the material overall semantic vector, the vector dimension corresponding to the query local semantic vector is equivalent to the vector dimension corresponding to the material local semantic vector, and the vector dimension corresponding to the query subject semantic vector is equivalent to the vector dimension corresponding to the material subject semantic vector.

[0214] Step S209 : determining the first material and the second material as candidate materials, obtaining target material image style vectors corresponding to the candidate materials, target material image semantic vectors corresponding to the candidate materials, and target material texts corresponding to the candidate materials.

[0215] Step S210, based on the query text, the query image semantic vector, the query image style vector, the target material text, the target material image semantic vector, and the target material image style vector, determines the material similarity between the candidate material and the query information, sorts the candidate materials according to the material similarity, and determines the recommended material from the sorted candidate materials.

[0216] Specifically, the specific process of step S209-step S210 can be found in the above Figure 3 The description in the corresponding embodiment will not be repeated here.

[0217] In an embodiment of the present application, a material retrieval method for a joint query of text and image is proposed. First, a first material associated with the query text is obtained from a material database, and a second material associated with the query image semantic vector is obtained. It can be seen that the embodiment of the present application can obtain candidate materials (including the above-mentioned first material and second material) based on multiple dimensional information (including text dimension and image dimension); further, based on the target material image style vector, the target material image semantic vector, the target material text, the query text, the query image semantic vector and the query image style vector, the material similarity between the candidate material and the query information is determined. It can be seen that the material similarity in the embodiment of the present application is generated based on multiple dimensional information (including text dimension, image semantic dimension and image style dimension); further, the candidate materials are sorted according to the material similarity, and recommended materials are determined from the sorted candidate materials. It can be seen that the recommended material and the query information can have not only semantic relevance but also style relevance. In summary, the present application can improve the description accuracy of the retrieval purpose and thereby improve the accuracy of the recommended material by jointly inputting the query text and the query image.

[0218] Further, see Figure 9 , Figure 9: is a structural diagram of a data processing device provided in an embodiment of the present application. The above-mentioned data processing device can be a computer program (including program code) running on a computer device, for example, the data processing device is an application software; the device can be used to execute the corresponding steps of the method provided in an embodiment of the present application. Figure 9 As shown, the data processing device 1 may include: a first acquisition module 11 , a second acquisition module 12 , a first determination module 13 and a second determination module 14 .

[0219] A first acquisition module 11 is configured to acquire query information including a query text and a query image, and acquire a first material associated with the query text from a material database;

[0220] A second acquisition module 12 is configured to acquire a query image style vector corresponding to the query image and a query image semantic vector corresponding to the query image, and acquire a second material associated with the query image semantic vector from a material database;

[0221] A first determining module 13 is configured to determine the first material and the second material as candidate materials, and obtain a target material image style vector corresponding to the candidate materials, a target material image semantic vector corresponding to the candidate materials, and a target material text corresponding to the candidate materials;

[0222] The second determination module 14 is used to determine the material similarity between the candidate material and the query information based on the query text, the query image semantic vector, the query image style vector, the target material text, the target material image semantic vector, and the target material image style vector, sort the candidate materials according to the material similarity, and determine the recommended material from the sorted candidate materials.

[0223] The specific functional implementation of the first acquisition module 11, the second acquisition module 12, the first determination module 13 and the second determination module 14 can be found in the above Figure 3 Steps S101 to S104 in the corresponding embodiment are not described again here.

[0224] See also Figure 9 The first acquisition module 11 may include: a first processing unit 111, a first acquisition unit 112 and a first determination unit 113.

[0225] The first processing unit 111 is used to segment the query text to obtain query words corresponding to the query text;

[0226] The first acquisition unit 112 is used to acquire an index key-value pair whose index attribute is a word segmentation attribute from the material database; the index key-value pair includes an index key generated by the material identifier and an index value generated by the material word segmentation;

[0227] The first determining unit 113 is configured to determine the index key corresponding to the index value containing the query segmentation word as the target index key, and obtain the first material corresponding to the target index key from the material database.

[0228] The specific functional implementation of the first processing unit 111, the first obtaining unit 112 and the first determining unit 113 can be found in the above Figure 3 The step S101 in the corresponding embodiment will not be described again here.

[0229] See also Figure 9 The first acquisition module 11 may include: a first input unit 114 , a second acquisition unit 115 , a third acquisition unit 116 and a second determination unit 117 .

[0230] The first input unit 114 is used to input the query text into the text encoding model and obtain the query text semantic vector corresponding to the query text through the text encoding model;

[0231] The second acquisition unit 115 is used to acquire an index key-value pair whose index attribute is a text semantic attribute from the material database; the index key-value pair includes an index key generated by the material identifier and an index value generated by the material text semantic vector;

[0232] The third obtaining unit 116 is configured to obtain the similarity between the query text semantic vector and the index value, and determine the index value having a similarity greater than a similarity threshold as the target index value;

[0233] The second determining unit 117 is configured to determine the index key corresponding to the target index value as the target index key, and obtain the first material corresponding to the target index key from the material database.

[0234] The specific functional implementation of the first input unit 114, the second acquisition unit 115, the third acquisition unit 116 and the second determination unit 117 can be found in the above Figure 3 The step S101 in the corresponding embodiment will not be described again here.

[0235] See also Figure 9 The second acquisition module 12 may include: a second input unit 121 , a fourth acquisition unit 122 and a third input unit 123 .

[0236] The second input unit 121 is used to input the query image into the image classification model, and obtain image classification information of the query image through the image classification model;

[0237] A fourth acquiring unit 122 is configured to acquire an image semantic extraction model associated with the image classification information and an image style extraction model associated with the image classification information;

[0238] The third input unit 123 is used to input the query image into the image semantic extraction model, and obtain the query image semantic vector corresponding to the query image through the image semantic extraction model;

[0239] The third input unit 123 is also used to input the query image into the image style extraction model, extract the image color features of the query image and the image shape features of the query image through the image style extraction model, perform feature aggregation processing on the image color features of the query image and the image shape features of the query image, and obtain a query image style vector corresponding to the query image.

[0240] The specific functional implementation of the second input unit 121, the fourth acquisition unit 122 and the third input unit 123 can be found in the above Figure 3 The step S102 in the corresponding embodiment will not be described again here.

[0241] See also Figure 9 ,The image semantic extraction model includes the image main body extraction model, the image local extraction model and the image overall extraction model;

[0242] The third input unit 123 may include a first extraction subunit 1231 , a second extraction subunit 1232 , a third extraction subunit 1233 , and a first determination subunit 1234 .

[0243] The first extraction subunit 1231 is configured to input the query image into the image subject extraction model, extract the subject semantic features of the query image through the image subject extraction model, and obtain a query subject semantic vector;

[0244] The second extraction subunit 1232 is configured to input the query image into the image local extraction model, extract the local semantic features of the query image through the image local extraction model, and obtain a query local semantic vector;

[0245] The third extraction subunit 1233 is configured to input the query image into the overall image extraction model, extract the overall semantic features of the query image through the overall image extraction model, and obtain the query overall semantic vector;

[0246] The first determining subunit 1234 is configured to determine the query subject semantic vector, the query overall semantic vector, and the query local semantic vector as the query image semantic vector.

[0247] The specific functional implementation of the first extraction subunit 1231, the second extraction subunit 1232, the third extraction subunit 1233 and the first determination subunit 1234 can be found in the above Figure 6 Step S208 in the corresponding embodiment will not be described again here.

[0248] See also Figure 9 The second acquisition module 12 may include: a fifth acquisition unit 124 and a sixth acquisition unit 125.

[0249] The fifth acquisition unit 124 is configured to acquire an index key-value pair whose index attribute is an image semantic attribute from the material database; the index key-value pair includes an index key generated by the material identifier and an index value generated by the material image semantic vector;

[0250] The sixth acquiring unit 125 is configured to acquire, from the material database according to the query image semantic vector and the index key-value pair, a second material associated with the query image semantic vector.

[0251] The specific functional implementation of the fifth acquisition unit 124 and the sixth acquisition unit 125 can be found in the above Figure 3 The step S102 in the corresponding embodiment will not be described again here.

[0252] See also Figure 9 , the query image semantic vector includes the query subject semantic vector, the query overall semantic vector and the query local semantic vector; the material image semantic vector includes the material subject semantic vector, the material overall semantic vector and the material local semantic vector; the index key-value pair includes the index subject key-value pair, the index overall key-value pair and the index local key-value pair; the index subject key-value pair includes the index subject key generated by the material identifier and the index subject value generated by the material subject semantic vector; the index local key-value pair includes the index local key generated by the material identifier and the index local value generated by the material local semantic vector; the index overall key-value pair includes the index overall key generated by the material identifier and the index overall value generated by the material overall semantic vector;

[0253] The sixth acquiring unit 125 may include: a first acquiring subunit 1251 , a second acquiring subunit 1252 , a third acquiring subunit 1253 and a second determining subunit 1254 .

[0254] The first acquisition subunit 1251 is configured to acquire a first similarity between the query subject semantic vector and the index subject value, determine the index subject value having the first similarity greater than a first similarity threshold as a target index subject value, determine the index subject key corresponding to the target index subject value as the target index subject key, and acquire the subject material corresponding to the target index subject key from the material database;

[0255] The second acquisition subunit 1252 is configured to obtain a second similarity between the query overall semantic vector and the index overall value, determine the index overall value having the second similarity greater than a second similarity threshold as the target index overall value, determine the index overall key corresponding to the target index overall value as the target index overall key, and obtain the overall material corresponding to the target index overall key from the material database;

[0256] The third acquisition subunit 1253 is configured to obtain a third similarity between the query local semantic vector and the index local value, determine the index local value having the third similarity greater than a third similarity threshold as the target index local value, determine the index local key corresponding to the target index local value as the target index local key, and obtain the local material corresponding to the target index local key from the material database;

[0257] The second determining subunit 1254 is configured to determine the main material, the partial material, and the entire material as the second material.

[0258] The specific functional implementation of the first acquisition subunit 1251, the second acquisition subunit 1252, the third acquisition subunit 1253 and the second determination subunit 1254 can be found in the above Figure 6 Step S208 in the corresponding embodiment will not be described again here.

[0259] See also Figure 9 The first determining module 13 may include: a third determining unit 131 , a seventh acquiring unit 132 and a fourth determining unit 133 .

[0260] The third determining unit 131 is configured to determine the material identifier corresponding to the candidate material as the candidate material identifier;

[0261] The seventh acquisition unit 132 is configured to acquire, from the material database, the target material image style vector stored in association with the candidate material identifier, the target material main semantic vector stored in association with the candidate material identifier, the target material overall semantic vector stored in association with the candidate material identifier, the target material local semantic vector stored in association with the candidate material identifier, and the target material text stored in association with the candidate material identifier;

[0262] The fourth determining unit 133 is configured to determine the target material main body semantic vector, the target material overall semantic vector, and the target material local semantic vector as the target material image semantic vector.

[0263] The specific functional implementation of the third determining unit 131, the seventh obtaining unit 132 and the fourth determining unit 133 can be found in the above Figure 3 The corresponding step S103 in the embodiment will not be described again here.

[0264] See also Figure 9 The total number of candidate materials is at least two, and the at least two candidate materials include candidate material C d , d is a positive integer, and d is less than or equal to the total number of at least two candidate materials; the target material text includes candidate material C d Corresponding target material text E d ; The target material image semantic vector includes the candidate material C d The corresponding target material image semantic vector F d ; The target material image style vector includes the candidate material C d The corresponding target material image style vector G d ;

[0265] The second determining module 14 may include: a fifth determining unit 141 , a second processing unit 142 , and a third processing unit 143 .

[0266] The fifth determining unit 141 is used to obtain the query text and the target material text E d The similarity between them is determined as the candidate material C d Corresponding text similarity;

[0267] The fifth determining unit 141 is further configured to obtain the query image semantic vector and the target material image semantic vector F d The similarity between them is determined as the candidate material C d Corresponding semantic similarity;

[0268] The fifth determining unit 141 is further configured to obtain the query image style vector and the target material image style vector G d The similarity between them is determined as the candidate material C d Corresponding style similarity;

[0269] The second processing unit 142 is used to perform weighted sum processing on the text similarity, semantic similarity and style similarity to obtain the candidate material C d The similarity between the material and the query information;

[0270] The third processing unit 143 is configured to sort at least two candidate materials according to the material similarity corresponding to each candidate material.

[0271] The specific functional implementation of the fifth determining unit 141, the second processing unit 142 and the third processing unit 143 can be found in the above Figure 3 The corresponding step S104 in the embodiment will not be described again here.

[0272] See also Figure 9The query image semantic vector includes the query subject semantic vector, the query overall semantic vector and the query local semantic vector; the target material image semantic vector F d Including the target material main semantic vector H d , the overall semantic vector of the target material I d And the target material local semantic vector J d ;

[0273] The fifth determining unit 141 may include a fourth acquiring subunit 1411 , a fifth acquiring subunit 1412 , a sixth acquiring subunit 1413 and a third determining subunit 1414 .

[0274] The fourth acquisition subunit 1411 is used to acquire the query subject semantic vector and the target material subject semantic vector H d The similarity between them is determined as the candidate material C d The corresponding subject semantic similarity;

[0275] The fifth acquisition subunit 1412 is used to obtain the query overall semantic vector and the target material overall semantic vector I d The similarity between them is determined as the candidate material C d The corresponding overall semantic similarity;

[0276] The sixth acquisition subunit 1413 is used to acquire the query local semantic vector and the target material local semantic vector J d The similarity between them is determined as the candidate material C d Corresponding local semantic similarity;

[0277] The third determining subunit 1414 is configured to determine the subject semantic similarity, the overall semantic similarity, and the local semantic similarity as the semantic similarity.

[0278] The specific functional implementation of the fourth acquisition subunit 1411, the fifth acquisition subunit 1412, the sixth acquisition subunit 1413 and the third determination subunit 1414 can be found in the above Figure 3 The corresponding step S104 in the embodiment will not be described again here.

[0279] See also Figure 9 The data processing device 13 may further include: a third acquisition module 15 , a first storage module 16 , a fourth acquisition module 17 , a second storage module 18 and a third storage module 19 .

[0280] The third acquisition module 15 is used to acquire video material, extract at least two video frames from the video material, and perform image preprocessing on the at least two video frames to obtain target video frames;

[0281] The third acquisition module 15 is further configured to acquire a video frame identifier corresponding to the target video frame and use the video frame identifier as a material identifier;

[0282] The first storage module 16 is configured to obtain the video frame text corresponding to the target video frame, use the video frame text as the material text, construct an index key-value pair whose index attribute is the text attribute according to the material identifier and the material text, and store the index key-value pair whose index attribute is the text attribute in the material database;

[0283] A fourth acquisition module 17 is configured to acquire a semantic vector of a material image corresponding to a target video frame, and a style vector of a material image corresponding to the target video frame;

[0284] The second storage module 18 is configured to use the material identifier as an index value and the material image semantic vector as an index key to construct an index key-value pair whose index attribute is the image semantic attribute, and store the index key-value pair whose index attribute is the image semantic attribute in the material database;

[0285] The third storage module 19 is configured to associate the material image style vector and the material identifier and store them in the material database.

[0286] The specific functional implementation of the third acquisition module 15, the first storage module 16, the fourth acquisition module 17, the second storage module 18 and the third storage module 19 can be found in the above Figure 6 Steps S201 to S205 in the corresponding embodiment are not described again here.

[0287] See also Figure 9 , at least two video frames include video frame A b , b is a positive integer, and b is less than or equal to the total number of at least two video frames;

[0288] The third acquisition module 15 may include: an eighth acquisition unit 151 , a sixth determination unit 152 and a fourth processing unit 153 .

[0289] The eighth acquisition unit 151 is used to acquire the video frame A b and the similarity between the remaining video frames; the remaining video frames include at least two video frames except video frame A b Video frames other than

[0290] The sixth determining unit 152 is configured to: if the similarity is equal to or greater than the similarity threshold, then b Determine that the video frame is a duplicate video frame, and delete the duplicate video frame from at least two video frames to obtain a video frame to be trimmed;

[0291] The fourth processing unit 153 is configured to perform region recognition processing on the video frame to be cropped to obtain a region to be cropped of the video frame to be cropped, and perform cropping processing on the region to be cropped in the video frame to be cropped to obtain a target video frame.

[0292] The specific functional implementation of the eighth acquisition unit 151, the sixth determination unit 152 and the fourth processing unit 153 can be found in the above Figure 6 The step S201 in the corresponding embodiment will not be described again here.

[0293] See also Figure 9 The first storage module 16 may include: a seventh determining unit 161 , a fifth processing unit 162 , a first extracting unit 163 and an eighth determining unit 164 .

[0294] A seventh determining unit 161 is configured to obtain a video description text carried by the video material, and determine the obtained video description text as the first video frame text;

[0295] a fifth processing unit 162 configured to perform recognition processing on the scene information in the target video frame to obtain scene recognition text, and determine the scene recognition text as the second video frame text;

[0296] a first extraction unit 163, configured to extract text information from a target video frame and determine the extracted text information as text of a third video frame;

[0297] The eighth determining unit 164 is configured to determine the first video frame text, the second video frame text, and the third video frame text as video frame texts.

[0298] The specific functional implementation of the seventh determining unit 161, the fifth processing unit 162, the first extracting unit 163 and the eighth determining unit 164 can be found in the above Figure 6 The corresponding step S203 in the embodiment will not be described again here.

[0299] See also Figure 9 ,Text attributes include word segmentation attributes and text semantic attributes;

[0300] The first storage module 16 may include a first construction unit 165 and a second construction unit 166 .

[0301] The first construction unit 165 is configured to segment the material text to obtain material segmentation corresponding to the material text, use the material identifier as the index key, and use the material segmentation as the index value to construct an index key-value pair whose index attribute is the segmentation attribute;

[0302] The second construction unit 166 is configured to obtain a material text semantic vector corresponding to the material text, use the material identifier as an index key and the material text semantic vector as an index value, and construct an index key-value pair whose index attribute is a text semantic attribute.

[0303] The specific functional implementation of the first construction unit 165 and the second construction unit 166 can be found in the above Figure 6 The corresponding step S203 in the embodiment will not be described again here.

[0304] See also Figure 9 ,The material image semantic vector includes the material main body semantic vector, the material overall semantic vector and the material local semantic vector;

[0305] The fourth acquisition module 17 may include: a fourth input unit 171 , a second extraction unit 172 , a third extraction unit 173 , a fourth extraction unit 174 and a fifth extraction unit 175 .

[0306] The fourth input unit 171 is used to input the target video frame into the image classification model, and obtain the material classification information of the target video frame through the image classification model;

[0307] The second extraction unit 172 is configured to input the target video frame into a material subject extraction model associated with the material classification information, extract the subject semantic features of the target video frame through the material subject extraction model, and obtain a material subject semantic vector;

[0308] The third extraction unit 173 is configured to input the target video frame into a material local extraction model associated with the material classification information, extract the local semantic features of the target video frame through the material local extraction model, and obtain a material local semantic vector;

[0309] The fourth extraction unit 174 is configured to input the target video frame into a material overall extraction model associated with the material classification information, extract the overall semantic features of the target video frame through the material overall extraction model, and obtain a material overall semantic vector;

[0310] The fifth extraction unit 175 is used to input the target video frame into a material style extraction model associated with the material classification information, extract the image color features of the target video frame and the image shape features of the target video frame through the material style extraction model, perform feature aggregation processing on the image color features of the target video frame and the image shape features of the target video frame, and obtain a material image style vector.

[0311] The specific functional implementation of the fourth input unit 171, the second extraction unit 172, the third extraction unit 173, the fourth extraction unit 174 and the fifth extraction unit 175 can be found in the above Figure 6Step S208 in the corresponding embodiment will not be described again here.

[0312] In an embodiment of the present application, a material retrieval method for a joint query of text and image is proposed. First, a first material associated with the query text is obtained from a material database, and a second material associated with the query image semantic vector is obtained. It can be seen that the embodiment of the present application can obtain candidate materials (including the above-mentioned first material and second material) based on multiple dimensional information (including text dimension and image dimension); further, based on the target material image style vector, the target material image semantic vector, the target material text, the query text, the query image semantic vector and the query image style vector, the material similarity between the candidate material and the query information is determined. It can be seen that the material similarity in the embodiment of the present application is generated based on multiple dimensional information (including text dimension, image semantic dimension and image style dimension); further, the candidate materials are sorted according to the material similarity, and recommended materials are determined from the sorted candidate materials. It can be seen that the recommended material and the query information can have not only semantic relevance but also style relevance. In summary, the present application can improve the description accuracy of the retrieval purpose and thereby improve the accuracy of the recommended material by jointly inputting the query text and the query image.

[0313] Further, see Figure 10 , Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 10 As shown, the computer device 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display) and a keyboard (Keyboard), and the network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 1005 may optionally also be at least one storage device located away from the aforementioned processor 1001. As Figure 10 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application.

[0314] exist Figure 10In the computer device 1000 shown, the network interface 1004 can provide network communication functions; the user interface 1003 is mainly used to provide an interface for user input; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0315] Obtaining query information including query text and query image, and obtaining a first material associated with the query text from a material database;

[0316] Obtaining a query image style vector corresponding to the query image and a query image semantic vector corresponding to the query image, and obtaining a second material associated with the query image semantic vector from a material database;

[0317] Determine the first material and the second material as candidate materials, obtain target material image style vectors corresponding to the candidate materials, target material image semantic vectors corresponding to the candidate materials, and target material text corresponding to the candidate materials;

[0318] Based on the query text, query image semantic vector, query image style vector, target material text, target material image semantic vector, and target material image style vector, the material similarity between the candidate material and the query information is determined, the candidate materials are sorted according to the material similarity, and the recommended material is determined from the sorted candidate materials.

[0319] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 3 as well as Figure 6 The description of the data processing method in the corresponding embodiments can also be performed as described above. Figure 9 The description of the data processing device 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here either.

[0320] The present invention also provides a computer-readable storage medium that stores a computer program. The computer program includes program instructions that are executed by a processor to implement Figure 3 as well as Figure 6 The data processing methods provided in each step are detailed in the above Figure 3 as well as Figure 6 The implementation methods provided by each step will not be described in detail here. In addition, the description of the beneficial effects of adopting the same method will not be described in detail here either.

[0321] The computer-readable storage medium may be the data processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Furthermore, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0322] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, so that the computer device can execute the aforementioned Figure 3 as well as Figure 6 The description of the data processing method in the corresponding embodiments will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here either.

[0323] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device comprising a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0324] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0325] The methods and related devices provided by the embodiments of the present application are described with reference to the method flow charts and / or structural diagrams provided by the embodiments of the present application. Specifically, each process and / or block in the method flow charts and / or structural diagrams, as well as the combination of processes and / or blocks in the flow charts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 The flow or flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0326] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: include: Obtaining a video frame identifier corresponding to a target video frame, and using the video frame identifier as a material identifier; Obtaining a video frame text corresponding to the target video frame, using the video frame text as a material text, constructing an index key-value pair whose index attribute is a text attribute according to the material identifier and the material text, and storing the index key-value pair whose index attribute is a text attribute in a material database; Obtaining a material image semantic vector corresponding to the target video frame and a material image style vector corresponding to the target video frame; the material image semantic vector includes a material main semantic vector, a material overall semantic vector, and a material local semantic vector; Using the material identifier as an index key and the material image semantic vector as an index value, constructing an index key-value pair whose index attribute is the image semantic attribute, and storing the index key-value pair whose index attribute is the image semantic attribute in the material database; storing the material image style vector and the material identifier in association with each other in the material database; The method of using the material identifier as an index key and the material image semantic vector as an index value to construct an index key-value pair whose index attribute is an image semantic attribute, and storing the index key-value pair whose index attribute is an image semantic attribute in the material database includes: Using the material identifier as the index subject key and the material subject semantic vector as the index subject value, constructing an index subject key-value pair whose index attribute is the image subject semantic attribute; Using the material identifier as the index overall key and the material overall semantic vector as the index overall value, constructing an index overall key-value pair whose index attribute is the overall semantic attribute of the image; Using the material identifier as an index local key and the material local semantic vector as an index local value, constructing an index local key-value pair whose index attribute is the local semantic attribute of the image; The index main key-value pair, the index overall key value, and the index local key-value pair are all stored in the material database.

2. The method according to claim 1, characterized in that The method further comprises: Acquire video material, extract at least two video frames from the video material; the at least two video frames include video frame A b , b is a positive integer, and b is less than or equal to the total number of the at least two video frames; Get the video frame A b and the similarity between the remaining video frames; the remaining video frames include the at least two video frames except the video frame A b Video frames other than If the similarity is equal to or greater than the similarity threshold, the video frame A is b Determining that the video frame is a duplicate video frame, deleting the duplicate video frame from the at least two video frames to obtain a video frame to be trimmed; Performing region recognition processing on the video frame to be cropped to obtain a region to be cropped of the video frame to be cropped, and performing cropping processing on the region to be cropped in the video frame to be cropped to obtain the target video frame.

3. The method according to claim 1, characterized in that The text attributes include word segmentation attributes; The step of constructing an index key-value pair whose index attribute is a text attribute according to the material identifier and the material text, and storing the index key-value pair whose index attribute is a text attribute in the material database includes: Segmenting the material text to obtain material segmentation words corresponding to the material text; Using the material identifier as the index key and the material segmentation as the index value, constructing an index key-value pair whose index attribute is the segmentation attribute; The index key-value pair whose index attribute is the word segmentation attribute is stored in the material database.

4. The method according to claim 1, wherein The text attributes include text semantic attributes; The step of constructing an index key-value pair whose index attribute is a text attribute according to the material identifier and the material text, and storing the index key-value pair whose index attribute is a text attribute in the material database includes: Obtaining a material text semantic vector corresponding to the material text; Using the material identifier as the index key and the material text semantic vector as the index value, constructing an index key-value pair whose index attribute is the text semantic attribute; The index key-value pairs whose index attributes are the text semantic attributes are stored in the material database.

5. The method according to claim 1, wherein The obtaining of the semantic vector of the material image corresponding to the target video frame and the style vector of the material image corresponding to the target video frame includes: Inputting the target video frame into an image classification model, and obtaining material classification information of the target video frame through the image classification model; Inputting the target video frame into a material subject extraction model associated with the material classification information, extracting the subject semantic features of the target video frame through the material subject extraction model to obtain the material subject semantic vector; Inputting the target video frame into a material local extraction model associated with the material classification information, extracting local semantic features of the target video frame through the material local extraction model to obtain the material local semantic vector; Inputting the target video frame into a material overall extraction model associated with the material classification information, extracting overall semantic features of the target video frame through the material overall extraction model to obtain the material overall semantic vector; The target video frame is input into a material style extraction model associated with the material classification information. The image color features of the target video frame and the image shape features of the target video frame are extracted through the material style extraction model. Feature aggregation processing is performed on the image color features and the image shape features of the target video frame to obtain the material image style vector.

6. The method according to claim 1, characterized in that The method of using the material identifier as an index key and the material image semantic vector as an index value to construct an index key-value pair whose index attribute is an image semantic attribute, and storing the index key-value pair whose index attribute is an image semantic attribute in the material database includes: Performing vector compression processing on the material subject semantic vector to obtain the material subject semantic vector after the vector compression processing, using the material identifier as an index subject key, using the material subject semantic vector after the vector compression processing as an index subject value, and constructing an index subject key-value pair whose index attribute is the image subject semantic attribute; Performing vector compression processing on the overall semantic vector of the material to obtain the compressed overall semantic vector of the material, using the material identifier as the overall index key and the compressed overall semantic vector of the material as the overall index value, to construct an index overall key-value pair whose index attribute is the overall semantic attribute of the image; Performing vector compression processing on the local semantic vector of the material to obtain the compressed local semantic vector of the material, using the material identifier as an index local key and the compressed local semantic vector of the material as an index local value, to construct an index local key-value pair whose index attribute is the local semantic attribute of the image; The index main key-value pair, the index overall key value, and the index local key-value pair are all stored in the material database.

7. The method according to claim 1, characterized in that The method of using the material identifier as an index key and the material image semantic vector as an index value to construct an index key-value pair whose index attribute is an image semantic attribute, and storing the index key-value pair whose index attribute is an image semantic attribute in the material database includes: Performing vector compression processing on the material subject semantic vector to obtain a material subject semantic vector after vector compression processing, encoding the material subject semantic vector after vector compression processing to obtain an encoded material subject semantic vector, using the material identifier as an index subject key and the encoded material subject semantic vector as an index subject value, to construct an index subject key-value pair whose index attribute is an image subject semantic attribute; Performing vector compression processing on the overall semantic vector of the material to obtain the overall semantic vector of the material after vector compression processing, encoding the overall semantic vector of the material after vector compression processing to obtain the encoded overall semantic vector of the material, using the material identifier as the overall index key and the encoded overall semantic vector of the material as the overall index value, to construct an index overall key-value pair whose index attribute is the overall semantic attribute of the image; Performing vector compression processing on the material local semantic vector to obtain a material local semantic vector after vector compression processing, encoding the material local semantic vector after vector compression processing to obtain an encoded material local semantic vector, using the material identifier as an index local key and the encoded material local semantic vector as an index local value to construct an index local key-value pair whose index attribute is the image local semantic attribute; The index main key-value pair, the index overall key value, and the index local key-value pair are all stored in the material database.

8. The method according to claim 1, characterized in that The method further comprises: Acquire query information including query text and query image, and acquire first material associated with the query text from the material database; Obtaining a query image style vector corresponding to the query image and a query image semantic vector corresponding to the query image, and obtaining a second material associated with the query image semantic vector from the material database; Determine the first material and the second material as candidate materials, and obtain target material image style vectors corresponding to the candidate materials, target material image semantic vectors corresponding to the candidate materials, and target material text corresponding to the candidate materials; Based on the query text, the query image semantic vector, the query image style vector, the target material text, the target material image semantic vector, and the target material image style vector, the material similarity between the candidate material and the query information is determined, the candidate materials are sorted according to the material similarity, and recommended materials are determined from the sorted candidate materials.

9. The method according to claim 8, characterized in that The obtaining of the query image style vector corresponding to the query image and the query image semantic vector corresponding to the query image includes: Inputting the query image into an image classification model, and obtaining image classification information of the query image through the image classification model; Acquire an image semantic extraction model associated with the image classification information, and an image style extraction model associated with the image classification information; Inputting the query image into the image semantic extraction model, and obtaining the query image semantic vector corresponding to the query image through the image semantic extraction model; The query image is input into the image style extraction model, and the mid-dimensional features of the query image are extracted through the mid-dimensional feature extraction layer of the image style extraction model, and the query image style vector is obtained according to the mid-dimensional features.

10. A data processing device, characterized in that: include: A third acquisition module is used to obtain a video frame identifier corresponding to the target video frame and use the video frame identifier as a material identifier; a first storage module, configured to obtain a video frame text corresponding to the target video frame, use the video frame text as a material text, construct an index key-value pair whose index attribute is a text attribute according to the material identifier and the material text, and store the index key-value pair whose index attribute is a text attribute in a material database; A fourth acquisition module is configured to acquire a material image semantic vector corresponding to the target video frame and a material image style vector corresponding to the target video frame; the material image semantic vector includes a material main semantic vector, a material overall semantic vector, and a material local semantic vector; a second storage module, configured to use the material identifier as an index key and the material image semantic vector as an index value to construct an index key-value pair whose index attribute is the image semantic attribute, and store the index key-value pair whose index attribute is the image semantic attribute in the material database; a third storage module, configured to associate and store the material image style vector and the material identifier in the material database; The second storage module includes: Using the material identifier as the index subject key and the material subject semantic vector as the index subject value, constructing an index subject key-value pair whose index attribute is the image subject semantic attribute; The second storage module is specifically configured to use the material identifier as an index overall key and the material overall semantic vector as an index overall value to construct an index overall key-value pair whose index attribute is an overall semantic attribute of the image; Using the material identifier as an index local key and the material local semantic vector as an index local value, constructing an index local key-value pair whose index attribute is the local semantic attribute of the image; The index main key-value pair, the index overall key value, and the index local key-value pair are all stored in the material database.

11. A computer device, characterized in that: include: processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a data communication function, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 9.

13. A computer program product or a computer program, characterized in that The computer program product or computer program includes computer instructions, which are stored in a computer-readable storage medium. The computer instructions are suitable for being read and executed by a processor, so as to enable a computer device having the processor to perform the method according to any one of claims 1 to 9.

Images