Resource query method and device, equipment and medium

Abstract

The invention provides a resource query method and device, equipment and a medium, relates to the technical field of artificial intelligence, can be applied to scenes such as cloud technology, artificial intelligence, smart traffic and aided driving, and comprises the following steps: acquiring an image text of a to-be-queried image, image features of the to-be-queried image and text semantic features of the image text; based on the image text and the text semantic features, performing resource recall in a preset resource library to obtain a recalled data set; based on the image text, the text semantic feature, the image feature and the resource feature data, generating a multi-dimensional association index between each pre-stored resource corresponding to the recall data set and the to-be-queried image; screening candidate resources from pre-stored resources corresponding to the recall resource set according to the multi-dimensional association index; and based on the multi-dimensional association index of the candidate resource, the resource feature data of the candidate resource and the image text, carrying out classification identification on the candidate resource to obtain a resource classification result. According to the invention, the accuracy of resource recall can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a resource query method, device, equipment and medium. Background Art

[0002] With the development of artificial intelligence technology, image search has become an efficient means of network data query, providing convenience for resource query. In related technologies, image retrieval is usually performed through image fingerprint features, and corresponding resource links are provided. However, the retrieval accuracy of this method is poor, the number of recalled resources is large, and it is not suitable for precise query scenarios. In addition, the same resource may correspond to multiple iconic images. The above method is highly dependent on image features. If the relevant pre-stored images of the same pre-stored resource are different from the query image, it is easy to cause resource omission. Summary of the invention

[0003] The present application provides a resource query method, apparatus, device and medium, which can significantly improve the accuracy of resource recall.

[0004] On the one hand, the present application provides a resource query method, the method comprising:

[0005] Acquire image text of the image to be queried, image features of the image to be queried, and text semantic features of the image text, wherein the image text is content text carried by the image to be queried;

[0006] Based on the image text and the text semantic features, resources are recalled in a preset resource library to obtain a recalled data set, wherein the preset resource library includes resource feature data of pre-stored resources;

[0007] Based on the image text, the text semantic features, the image features and the resource feature data, a multidimensional correlation index is generated between each pre-stored resource corresponding to the recall data set and the image to be queried, wherein the multidimensional correlation index is used to indicate the degree of correlation between the image to be queried and the pre-stored resources in multiple feature dimensions;

[0008] Screening candidate resources from the pre-stored resources corresponding to the recalled resource set according to the multi-dimensional association index;

[0009] Based on the multi-dimensional association index of the candidate resources, the resource feature data of the candidate resources and the image text, the candidate resources are classified and identified to obtain a resource classification result, and the resource classification result is used to characterize the matching degree between the candidate resources and the image to be queried.

[0010] Another aspect provides a resource recall device, the device comprising:

[0011] Acquisition module: used to acquire image text of the image to be queried, image features of the image to be queried and text semantic features of the image text, wherein the image text is the content text carried by the image to be queried;

[0012] Recall module: used to recall resources in a preset resource library based on the image text and the text semantic features to obtain a recall data set, wherein the preset resource library includes resource feature data of pre-stored resources;

[0013] An index generating module: used for generating a multi-dimensional correlation index between each pre-stored resource corresponding to the recall data set and the image to be queried based on the image text, the text semantic features, the image features and the resource feature data, wherein the multi-dimensional correlation index is used for indicating the correlation degree between the image to be queried and the pre-stored resources in multiple feature dimensions;

[0014] Resource screening module: used for screening candidate resources from each pre-stored resource corresponding to the recalled resource set according to the multi-dimensional correlation index;

[0015] Resource classification module: used to classify and identify the candidate resources based on the multi-dimensional association indicators of the candidate resources, the resource feature data of the candidate resources and the image text, and obtain resource classification results, and the resource classification results are used to characterize the degree of matching between the candidate resources and the image to be queried.

[0016] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the resource query method as described above.

[0017] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the resource query method as described above.

[0018] On the other hand, a server is provided, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the resource query method as described above.

[0019] On the other hand, a terminal is provided, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the resource query method as described above.

[0020] On the other hand, a computer program product or a computer program is provided. The computer program product or the computer program comprises computer instructions. When the computer instructions are executed by a processor, the resource query method as described above is implemented.

[0021] The resource query method, apparatus, device, storage medium, server, terminal, computer program and computer program product provided by the present application have the following technical effects:

[0022] The technical solution of the present application first obtains the image text of the image to be queried, the image features of the image to be queried, and the text semantic features of the image text, wherein the image text is the content text carried by the image to be queried; based on the image text and the text semantic features, resources are recalled in a preset resource library to obtain a recall data set, so as to perform multi-dimensional recall in combination with text semantics and text content, thereby avoiding resource omissions and improving recall accuracy and recall rate; then, based on the image text, text semantic features, image features, and resource feature data, a multi-dimensional correlation index is generated between each pre-stored resource corresponding to the recall data set and the image to be queried, and then, according to the multi-dimensional correlation index, each pre-stored resource corresponding to the recalled resource set is retrieved. Candidate resources are screened from pre-stored resources, and candidate resources are classified and identified based on their multi-dimensional correlation indicators, resource feature data and image text to obtain resource classification results. In this way, the correlation between the image to be queried and the pre-stored resources is characterized by the degree of correlation on multiple feature dimensions, thereby improving the accuracy and comprehensiveness of the expression of correlation evaluation information, thereby improving the accuracy of candidate resource screening and resource classification and identification, and avoiding missed detection of resources in multi-iconic image scenarios. In addition, the multi-dimensional correlation indicators, resource feature data and image text of candidate resources are combined for classification and identification, further improving the information dimension of the identification input and realizing accurate identification of related resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application;

[0025] Figure 2 It is a flowchart of a resource query method provided in an embodiment of the present application;

[0026] Figure 3 It is a flowchart of another resource query method provided in an embodiment of the present application;

[0027] Figure 4 It is a flowchart of another resource query method provided in an embodiment of the present application;

[0028] Figure 5 It is a schematic diagram of the principle of text recognition provided by an embodiment of the present application;

[0029] Figure 6 It is another principle schematic diagram of text recognition and semantic recall provided by an embodiment of the present application;

[0030] Figure 7 It is a structural framework diagram of a resource query system provided by an embodiment of the present application;

[0031] Figure 8 It is a structural framework diagram of another resource query system provided by an embodiment of the present application;

[0032] Fig. 9 This is a resource sharing interface diagram of a client provided in an embodiment of the present application;

[0033] Fig.10 for Fig. 9 The resource link interface diagram corresponding to the book image in the figure;

[0034] Fig.11 for Fig.10 Resource details interface corresponding to the book image;

[0035] Fig.12 It is a schematic diagram of a framework of a resource recall device provided in an embodiment of the present application;

[0036] Fig.13 It is a hardware structure block diagram of an electronic device for executing a resource query method or a neural network training method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server comprising a series of steps or submodules is not necessarily limited to those steps or submodules clearly listed, but may include other steps or submodules that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0040] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0041] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0042] In recent years, with the research and progress of artificial intelligence technology, artificial intelligence technology has been widely used in many fields. The solution provided in the embodiment of the present application involves artificial intelligence machine learning / deep learning, computer vision technology, natural language processing and other technologies, which are specifically explained through the following embodiments.

[0043] See also Figure 1 , Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application, such as Figure 1As shown, the application environment may include at least a terminal 01 and a server 02. In practical applications, the terminal 01 and the server 02 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0044] The server 02 in the embodiment of the present application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0045] Specifically, cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data computing, storage, processing, and sharing. Cloud technology can be applied to various fields, such as medical cloud, cloud Internet of Things, cloud security, cloud education, cloud conferencing, artificial intelligence cloud services, cloud applications, cloud calls, and cloud social networking. Cloud technology is based on the cloud computing business model application. It distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space, and information services as needed. The network that provides resources is called a "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for by use. As a basic capability provider for cloud computing, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service)) platform will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0046] According to the logical function division, the PaaS (Platform as a Service) layer can be deployed on the IaaS layer, and the SaaS (Software as a Service) layer can be deployed on the PaaS layer, or SaaS can be directly deployed on IaaS. PaaS is a platform for software operation, such as databases, web containers, etc. SaaS is a variety of business software, such as web portals, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0047] Specifically, the server 02 mentioned above may include a physical device, which may specifically include a network communication submodule, a processor, a memory, etc., and may also include software running in the physical device, which may specifically include an application program, etc.

[0048] Specifically, terminal 01 may include physical devices such as smart phones, desktop computers, tablet computers, laptops, digital assistants, augmented reality (AR) / virtual reality (VR) devices, intelligent voice interaction devices, smart home appliances, smart wearable devices, and vehicle-mounted terminal devices, and may also include software running in physical devices, such as applications.

[0049] In the embodiment of the present application, terminal 01 can be used to obtain the image to be queried and send it to server 02, so that server 02 obtains the image text of the image to be queried, the image features of the image to be queried, and the text semantic features of the image text, and then performs corresponding resource recall, multi-dimensional correlation index generation, candidate resource screening, classification and identification of candidate resources, and other operations to obtain resource classification results. Furthermore, server 02 can also push the target resource to terminal 01 for display when determining that the candidate resource is a target resource. Server 01 can provide model training services for each link in the resource recall system, and can also be used to store preset resource libraries.

[0050] Specifically, the resource query method of the present application can be applied to image-based media resource retrieval, such as e-book retrieval, film and television resource retrieval, and other application scenarios.

[0051] Furthermore, it is understandable that Figure 1 What is shown is only an application environment of a resource query method, and the application environment may include more or fewer nodes, and this application does not impose any limitation thereto.

[0052] The application environment involved in the embodiment of the present application, or the terminal 01 and the server 02 in the application environment, etc., can be a distributed system formed by connecting a client and multiple nodes (any form of computing devices in the access network, such as a server, a user terminal) through network communication. The distributed system can be a blockchain system, which can provide the above-mentioned resource recall service, neural network training service and data storage service, etc.

[0053] The following is an introduction to the technical solution of this application based on the above application environment. The embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. Please refer to Figure 2 , Figure 2It is a flowchart of a resource query method provided by an embodiment of the present application. This specification provides method operation steps such as the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or server product is executed, it can be executed in the order of the method shown in the embodiment or the figure or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, as Figure 2 As shown, the method may include the following steps S201-S209:

[0054] S201: Obtain image text of a to-be-queried image, image features of the to-be-queried image, and text semantic features of the image text.

[0055] Specifically, the image to be queried may be an image pointed to by a resource query instruction triggered by a terminal based on a query event. Specifically, the resource query instruction may carry image address information of the image to be queried for obtaining the image to be queried. The query event trigger may include but is not limited to an image click operation, an image upload operation, and the like.

[0056] In some embodiments, S201 may include S2011-S2013:

[0057] S2011: Perform text recognition on the query image to obtain image text.

[0058] Specifically, the image text is the content text carried by the image to be queried; it can be understood that the image content in the image to be queried may include text, such as a book cover image includes text such as the book name, author, publishing information, etc., and a film and television poster image includes text such as the film and television name, director, etc. The image text is the text that appears in the image content in the image to be queried.

[0059] In some embodiments, the query image may be input into a text recognition model to obtain the image text of the query image, where the image text includes a series of text lines. Figure 5 , input the book cover image on the left into the text recognition model, and output multiple lines of text containing "XX novel title", "XX novel subtitle", "author...", "publisher...", etc.

[0060] Specifically, the text recognition model may be a pre-trained model, which is obtained by performing text recognition based on a training set of images containing texts. In one embodiment, the text recognition model may be constructed based on optical character recognition (OCR).

[0061] S2012: Input the image text into the text feature model to extract semantic features and obtain text semantic features.

[0062] Specifically, the text feature model can be a pre-trained model obtained by semantic feature extraction training based on training corpus, and the text semantic feature is obtained by contextual semantic cross extraction of each text line in the image text. Specifically, the text feature model can include but is not limited to pre-trained models such as BERT (Bidirectional Encoder Representation from Transformers, bidirectional encoding network based on compiler), ALBERT (A Little Bert model), ERNIE (Enhanced Representation through Knowledge Integration, enhanced representation network realized through knowledge integration). In one embodiment, the BERT model is used to construct a text feature model to extract lexical, syntactic and bidirectional semantic features. Specifically, feature embedding is performed on each text line mark of the image text, including word embedding (token embeddings), segment embedding (segment embeddings) and position embedding (position embeddings), and a word sequence vector, a segment sequence vector and a position sequence vector are obtained by concatenating the word sequence vector, the segment sequence vector and the position sequence vector to obtain a text embedding vector, and then the text embedding vector is input into a bidirectional transformer for feature extraction to obtain text semantic features.

[0063] S2013: Input the image to be queried into the image feature model to extract image features and obtain image features.

[0064] Specifically, the image feature model may be a model obtained by training a neural network to extract image features using a training image set, and may include but is not limited to CNN (Convolutional Neural Networks), RNN (Recurrent Neural Networks) or VGG (Visual Geometry Group), etc. The image features may include shallow features and / or deep features of the image to be queried.

[0065] By obtaining the image text, image features and text semantic features of the image to be queried, it is possible to combine the image and the text in the image to express information and form semantically rich input data, which is conducive to improving the accuracy of subsequent recall and association evaluation.

[0066] It can be understood that if the text recognition result is empty, indicating that there is no text in the image to be queried, then the image category can be determined to be other categories, and the query is terminated.

[0067] S203: Based on the image text and text semantic features, resources are recalled in a preset resource library to obtain a recalled data set, where the preset resource library includes resource feature data of pre-stored resources.

[0068] Specifically, the pre-stored resources may be media resources, including but not limited to electronic book resources, film and television resources, etc. The preset resource library is an offline constructed database including resource feature data of multiple pre-stored resources.

[0069] Specifically, the resource feature data includes multi-dimensional feature data of pre-stored resources, which is obtained by extracting multi-dimensional data features from resource images of pre-stored resources. It can be understood that a resource image is an image including content that can describe personalized information of a pre-stored resource, and different pre-stored resources can be distinguished based on the image content (including text content) of the resource image. Each pre-stored resource can correspond to multiple resource images, and the image content carried by multiple resource images of the same pre-stored resource has overlapping content. Exemplarily, the same electronic book can correspond to multiple physical book cover images, or the same film and television resource can correspond to multiple different film and television posters.

[0070] In some embodiments, resource feature data may include resource image text, resource semantic features, resource image features, and identification entity information corresponding to the pre-stored resource; similar to the aforementioned image text, text semantic features, and image features, the resource image text may be obtained by performing text recognition on the resource image, the resource semantic features may be obtained by performing semantic feature extraction on the resource image text, and the resource image features may be obtained by performing image feature extraction on the resource image. The identification entity information may be extracted from the resource image text, or the identification entity information may be extracted based on the resource description information of the pre-stored resource, and the identification entity information includes identification entity words or identification entity sentences used to describe the personalized attributes of the pre-stored resource, and different pre-stored resources can be distinguished based on the identification entity information. Exemplarily, the resource image may be, for example, a book cover image, a film and television poster, etc., and the identification entity sentence may include but is not limited to the resource name, such as a book title, a title, a film and television name, etc.; the identification entity words may include but are not limited to words related to resource ownership such as the author name and the director name, etc.

[0071] Specifically, with image text and text semantic features as targets, the pre-stored resources related to image text and pre-stored resources related to text semantic features are queried in the preset resource library to obtain a recall data set. For example, resource feature data of related e-book resources can be recalled based on book cover images.

[0072] In some embodiments, reference Figure 3 , S203 may include S2031-S2033:

[0073] S2031: Perform semantic recall of resource semantic features on a preset resource library based on text semantic features to obtain a first resource data set.

[0074] Specifically, the first resource data set includes resource semantic features that match the text semantic features. The text semantic features are used as the recall target, and the resource semantic features stored in the preset resource library are used as the recall source. The text semantic features are traversed and matched with the pre-stored resource semantic features to obtain the first resource data set. Specifically, semantic recall can be based on the feature similarity between the text semantic features and the resource semantic features. The resource semantic features with feature similarity greater than or equal to the preset similarity are used as the recalled resource data, and the pre-stored resources to which the recalled resource semantic features belong are used as the pre-stored resources to be screened. Reference Figure 6 , the text semantic features (text semantic feature 1, text semantic feature 2, ... text semantic feature n) can be input into the semantic recall module, and the first resource data set (resource semantic feature 1, resource semantic feature 2, ... resource semantic feature k) can be output; in some embodiments, the semantic recall module can be constructed using Milvus to store and retrieve resource semantic features to achieve semantic recall.

[0075] S2032: Perform entity recall based on the image text and the identification entity information to obtain a second resource data set.

[0076] Specifically, entity words or entity sentences carried in the image text are taken as recall targets, entity recall is performed on the identification entity information in the preset resource library, identification entity information matching the image text is obtained, a second resource data set is obtained, and the pre-stored resources to which the matching identification entity information belongs are determined as pre-stored resources to be screened.

[0077] S2033: Generate a recall data set according to the first resource data set and the second resource data set.

[0078] Specifically, the first resource dataset and the second resource dataset are merged and deduplicated to obtain a recall dataset. Combining image text and semantic features to perform entity recall and semantic recall respectively can improve the recall rate and avoid missed screening.

[0079] It is understandable that there is a large amount of identification entity information of pre-stored resources in the preset resource library. If each entity word in the image text is traversed and matched one by one, the resource occupation and time cost are extremely high. In view of this, in some embodiments, S2032 may include:

[0080] 1) Obtain query structure data of a preset resource library;

[0081] 2) Perform entity matching on image text based on query structure data;

[0082] 3) Generate a second resource data set based on the pre-stored resources corresponding to the matched identification entity information.

[0083] Specifically, the query structure data is obtained by constructing a data tree based on the tree structure for the identification entity information of each pre-stored resource in the preset resource library. The query structure data can be an entity prefix tree, an entity suffix tree, an entity search tree, etc. constructed based on the identification entity information, preferably an entity prefix tree. In one embodiment, the identification entity information used to construct the query structure data is the resource name, such as constructing a prefix tree according to the book name of the electronic book resource. It should be noted that the construction method of the query structure data (such as the prefix tree) adopts the existing tree structure construction method to achieve the structuring of the identification entity information, and no specific limitation is made here. Based on the data structure of the query structure data, the text characters in the image text are traversed and matched with each query structure data generated by the preset resource library to obtain the identification entity information matching the image text. If the book name related to the image text is matched, the pre-stored electronic book resource to which the book name belongs is determined as the pre-stored resource to be screened. In this way, by generating the query structure data of the tree structure for entity recall, there is no need to repeatedly traverse and match for each text word, thereby improving the efficiency of entity recall and reducing resource occupation and time cost.

[0084] S205: Based on the image text, text semantic features, image features and resource feature data, a multi-dimensional correlation index is generated between each pre-stored resource corresponding to the recall data set and the image to be queried.

[0085] Specifically, the multidimensional correlation index is used to indicate the correlation degree between the image to be queried and the pre-stored resources in multiple feature dimensions; the multiple feature dimensions may include but are not limited to semantic feature dimension, image feature dimension, text data dimension, entity information dimension and overall dimension, etc. For each feature dimension, the higher the correlation index value, the higher the correlation degree, and vice versa.

[0086] In some embodiments, the association index may be a similarity index. For each pre-stored resource, the similarity of each feature dimension is calculated in combination with the resource feature data to obtain the similarity index of each feature dimension between the pre-stored resource and the image to be queried to characterize the degree of association between the two. Accordingly, for each pre-stored resource, S205 may include S2051-S2055:

[0087] S2051: Perform similarity calculation on the resource image text and the image text of the pre-stored resource to obtain a text similarity index;

[0088] Specifically, the similarity between the resource image text and the image text can be determined by a sequence matching algorithm to obtain a text similarity index, where the higher the index value, the higher the correlation between the two, and vice versa. In one embodiment, the sequence matching algorithm used can be a longest common subsequence algorithm.

[0089] S2052: Performing a similarity operation on the resource semantic features and the text semantic features of the pre-stored resource to obtain a semantic similarity index;

[0090] Specifically, the similarity between resource semantic features and text semantic features can be calculated based on a vector similarity algorithm. The similarity algorithm includes but is not limited to a cosine similarity algorithm, a Euclidean distance algorithm, or a Manhattan distance algorithm. The higher the semantic similarity index value, the higher the correlation between the two, and vice versa.

[0091] S2053: performing similarity calculation on the resource image features and the image features of the pre-stored resources to obtain an image similarity index;

[0092] Specifically, the similarity between the resource image feature and the image feature may be calculated based on a vector similarity algorithm, and the similarity algorithm includes but is not limited to a cosine similarity algorithm, a Euclidean distance algorithm, or a Manhattan distance algorithm.

[0093] S2054: performing a similarity operation on the identification entity information of the pre-stored resource and the image text to obtain an identification entity similarity index;

[0094] Similar to the text similarity index, the similarity between the identification entity information and the image text can be determined by a sequence matching algorithm to obtain an identification entity similarity index. The higher the index value, the higher the correlation between the two, and vice versa. In one embodiment, the sequence matching algorithm used can be a longest common subsequence algorithm. Specifically, the identification entity information here can include the resource name of the pre-stored resource.

[0095] S2055: Determine a total similarity index between the pre-stored resource and the image to be queried based on the text similarity index, the semantic similarity index, the image similarity index, and the identification entity similarity index;

[0096] Specifically, the text similarity index, semantic similarity index, image similarity index and identification entity similarity index can be summed or averaged to obtain a total similarity index, which is used to characterize the total relevance between the image to be queried and the pre-stored resources. The larger the total relevance value, the higher the degree of association, and vice versa. The summation here can be a simple summation or a weighted summation, and the summation and averaging here can be a simple summation and averaging or a weighted summation and averaging. The weights involved can be determined based on empirical values, or can be determined by adjusting parameters based on the training process of the resource classification model.

[0097] Based on the image features and text semantic features on the feature vector side, similarity index calculations are performed, and combined with the overall text and identification entity information on the text side to perform similarity index calculations, the information dimension and information expression accuracy of the relevance evaluation are improved, and the information weight on the text side is improved to improve the accuracy of the relevance evaluation. In addition, it can be understood that there may be multiple iconic resource images for the same pre-stored resource, and the resource feature data in the preset resource library may be obtained based on one or part of the iconic resource images. Since the feature vector is highly dependent on the image content, if only the feature vector is used for recall and similarity evaluation, when other iconic resource images are queried as the query images, there may be resource omissions caused by feature mismatches. By increasing the information weight on the text side, the dependence on the image is reduced, and resource omissions are effectively avoided.

[0098] S207: Filter candidate resources from the pre-stored resources corresponding to the recalled resource set according to the multi-dimensional correlation index.

[0099] Specifically, the higher the correlation indicated by the multidimensional correlation index, the higher the possibility that the pre-stored resource is a candidate resource, and vice versa. In one embodiment, screening can be performed based on the total similarity index, and the pre-stored resources corresponding to the total similarity index that meets the preset screening conditions in the recalled resource set are determined as candidate resources; wherein the preset screening condition can be that the value of the overall similarity index is greater than or equal to the preset total index value, or it can be sorted based on the total similarity index of each pre-stored resource corresponding to the preset resource library. If the total similarity index is one of the TOP m in the sorting, it is determined that it meets the preset screening condition, otherwise it does not meet it, and m is greater than or equal to 1, preferably 1. It can be understood that TOP m refers to the top m with the highest total similarity index in the sorting. In this way, based on the total similarity index that can express similarity index information in multiple dimensions as a screening condition, the screening efficiency is improved while the screening accuracy is improved.

[0100] S209: Classify and identify the candidate resources based on the multi-dimensional correlation indexes of the candidate resources, the resource feature data of the candidate resources, and the image text to obtain a resource classification result.

[0101] Specifically, based on the multi-dimensional association index, the resource feature data of the candidate resource and the image text, the resource classification feature of the candidate resource is generated, and the resource classification feature is input into the resource classification model to obtain the resource classification result, which is used to characterize the degree of match between the candidate resource and the image to be queried. The resource classification result can be a binary classification result, where 1 represents a match and 0 represents a mismatch, or it can be a confidence level, where a higher confidence level represents a higher degree of match, and if the confidence level exceeds a preset confidence level, a match is determined, otherwise a mismatch.

[0102] In some embodiments, in combination Figure 4 , S209 may include S2091-S2093:

[0103] S2091: Perform entity matching on the identification entity information of the candidate resource and the image text to obtain entity matching features;

[0104] S2092: Generate resource features based on multi-dimensional association indicators and entity matching features;

[0105] S2093: Input resource features into a resource classification model for classification and identification to obtain resource classification results.

[0106] Specifically, the identification entity information includes at least one identification entity, and the identification entity can be an entity word or an entity sentence. The entity matching feature is used to characterize the matching between the identification entity and the image text, and can specifically characterize whether the image text contains the identification entity, or can also be similar to the aforementioned identification entity similarity index, which is an entity similarity index between the identification entity and the image text. Exemplarily, the identification entity information can include two identification entities, the resource name and the resource author, and the entity matching feature can include a name feature and an author feature. The name feature characterizes whether the resource name appears in the image text, and the author feature characterizes whether the author name appears in the image text. In the name feature and the author feature, yes and no are represented by different characters, such as 1 for appearance and 0 for non-appearance.

[0107] Specifically, resource features can be formed based on multidimensional correlation indicators and entity matching features to express information. Specifically, resource features can be formed based on each of the aforementioned semantic similarity indicators, image similarity indicators, text similarity indicators, identification entity similarity indicators, overall similarity indicators and entity matching features.

[0108] Specifically, the resource classification model can be a machine learning model. The resource query link of the present application is relatively long. Using a machine learning model for classification and identification can reduce resource occupation and prediction time. In some embodiments, the resource classification model can be a decision tree model, specifically a random forest model. The resource classification model can be obtained by constraining the initial classification model to identify resource categories based on a training sample set. The training sample set can include multiple sample resource images and sample resource feature data of the sample resource images. Sample resource images and sample resource feature data can be extracted from the training sample set to form a positive and negative sample group. The sample label is a representation of the resource category, specifically representing whether the sample resource image and the sample resource feature data in the sample group match. Based on the sample group, a sample resource feature corresponding to the sample resource feature data is generated, and the initial classification model is output to obtain a resource category index. The model loss is generated according to the resource category index and the sample label to perform iterative constraint training until the training end condition is met to obtain a resource classification model. It can be understood that the sample resource feature is similar to the acquisition method of the aforementioned resource feature, and will not be repeated here.

[0109] In this way, resource features are generated by combining the entity matching structure of image text and multidimensional correlation indicators, realizing multidimensional expression of correlation information between resources and images to be queried, increasing the weight of text side information, and improving the accuracy of correlation evaluation between candidate resources and images, thereby improving resource query accuracy. Based on the solution of this application, the accuracy of the output results of the resource classification model is over 97.3%.

[0110] Based on some or all of the above implementations, in the embodiment of the present application, after S209, the method further includes:

[0111] S211: If the resource classification result indicates that the candidate resource matches the image to be queried, the candidate resource is determined as the target resource corresponding to the image to be queried;

[0112] S213: If the resource classification result indicates that the candidate resource does not match the image to be queried, it is determined that the target resource does not exist in the preset resource library.

[0113] Specifically, if the candidate resource is determined to be the target resource, it is determined to be recalled and fed back. Specifically, it can be sent to the terminal for display, or the target resource can be associated with the image to be queried and stored, or updated resource feature data can be generated based on the image to be queried and stored in a pre-stored database. If there is no match, no recall is performed, indicating that there is no relevant resource for the image. Combined with the resource classification results, it is further determined whether to recall the target resource, to achieve multiple guarantees of resource accuracy and optimize the resource query product experience.

[0114] In one embodiment, combining Figure 7 The resource query system includes a text recognition module, a recall module and a rearrangement module. The text recognition module obtains the image to be queried based on the image information to be queried and performs text recognition to obtain the image text. The image information to be queried can be the image address; the image text and the image to be queried are input into the recall module to extract image features and text semantic features to obtain text semantic features and image features, perform semantic recall and image recall, and obtain a recalled resource set; the recalled resource set, image features, image text, and text semantic features are input into the rearrangement module to generate multi-dimensional correlation indicators, screen candidate resources, and classify and identify candidate resources to output the result that the target resource or the resource is empty.

[0115] Based on some or all of the above implementations, in some embodiments, after obtaining the image text of the image to be queried in S201, the method further includes S301-S307:

[0116] S301: Extract text features from image text to obtain multi-dimensional element characteristic features.

[0117] Specifically, there are differences in the element characteristics of image texts of different resource element categories. The resource element category is used to indicate the resource element to which the image content of the image to be queried belongs, and the elements of different resources may be different. For example, in e-book resources, resource elements may include but are not limited to book covers, text, abstracts, etc., and in film and television resources, resource elements may include but are not limited to film and television posters, film and television content video frames, etc. The multi-dimensional element characteristic features may include personalized features and text statistical features of the image text, etc.; wherein, the personalized features are features unique to the target resource element category, including but not limited to whether it contains identification entities, characteristic keywords or characteristic characters, etc. For example, if the cover of an e-book resource is the target resource element category, the identification entity may include but not limited to the resource name (book name), author name, etc., the characteristic keywords may include but not limited to the publisher name, publishing unit name, etc., the characteristic characters may include but not limited to page numbers, etc. Accordingly, the personalized features may include whether it contains a publisher, whether it contains a publishing unit, whether it contains the author name, whether it contains the book name, and whether it contains page numbers, etc.; the text statistical features are features that differ between different resource element categories, such as when the cover is the target resource element category, the text statistical features may include but not limited to text length, the number of punctuation marks, text direction statistics, or the number of font and size categories in the image, etc. Specifically, the text statistical feature extraction can be obtained based on the aforementioned text recognition model, and the image to be queried is input into the text recognition model to obtain the image text and generate text statistical features. Personalized features can be realized based on sequence matching and string matching, or can be obtained based on the aforementioned query structure data query, such as text query based on the corresponding prefix trees of the identifier entity, characteristic keyword or characteristic character, etc. The query structure data here can be the same as the above, obtained based on a preset resource library, or can also be obtained based on other network resource libraries.

[0118] S303: Input the multi-dimensional element characteristic features into the image classification model for classification and recognition to obtain an image classification result, which is used to characterize the resource element category of the image to be queried.

[0119] Specifically, the image classification model is obtained by constraining the classification training of the image resource element category on the preset classification model based on multiple sample images and image labels. The sample multidimensional element characteristic features of the sample image can be obtained based on a generation method similar to the aforementioned multidimensional element characteristic features, and input into the preset classification model for classification and recognition to obtain the image category confidence, and the model loss is generated by combining the image label and the image category confidence to perform iterative training of the preset classification model until the training end condition is met to obtain the image classification model. In some embodiments, the image classification model can be a decision tree model, specifically a random forest model.

[0120] S305: If the image classification result indicates that the image to be queried is of the target resource element category, a step of acquiring image features of the image to be queried and text semantic features of the image text is performed.

[0121] S307: If the image classification result indicates that the image to be queried does not belong to the target resource element category, it is determined that the candidate resource is not related to the image to be queried.

[0122] Specifically, if the image classification result indicates that the current image to be queried belongs to the target resource element category, a resource query is performed in the preset resource library. Otherwise, if it does not belong to the target resource element category, it indicates that there is no need to perform relevant resource query, and the resource query event is terminated. In this way, before performing a resource query, image classification is first performed in combination with the image text of the image to be queried, so as to screen the image to be queried from the text side, so as to avoid invalid queries and reduce resource waste.

[0123] In one embodiment, the pre-stored resources are electronic book resources, and the resource element categories may include text, book cover, or others, with the book cover being the target resource element category. If the image to be queried is identified as a book cover, the image features and text semantic features of the image to be queried are obtained to perform electronic book resource retrieval, otherwise, the query is terminated.

[0124] In one embodiment, combining Figure 8 In addition to the text recognition module, the recall module and the rearrangement module, the resource query system also includes an image classification module. The text recognition module obtains the image to be queried based on the information of the image to be queried and performs text recognition to obtain the image text; the image text is input into the image classification module to obtain the image classification result. If it is the target resource element category, the image text and the image to be queried are input into the recall module to extract image features and text semantic features to obtain text semantic features and image features, and semantic recall and image recall are performed to obtain the recalled resource set; the recalled resource set, image features, image text, and text semantic features are input into the rearrangement module to generate multi-dimensional correlation indicators, screen candidate resources, and classify and identify candidate resources to output the result that the target resource or resource is empty; otherwise, the query is terminated.

[0125] Based on some or all of the above implementation modes, in some embodiments, the resource feature data includes a resource unique identification code having a preset format corresponding to a pre-stored resource, and the resource unique identification codes of different pre-stored resources are different from each other; accordingly, after obtaining the image text of the image to be queried in S201, the method further includes S401-S405:

[0126] S401: If the image text is identified to include an identification code in a preset format, the identification code is matched with a unique resource identification code in a preset resource library;

[0127] S403: If a resource unique identification code corresponding to the identification code is matched, the pre-stored resource to which the resource unique identification code belongs is determined as a target resource corresponding to the image to be queried;

[0128] S405: If the resource unique identification code corresponding to the identification code is not matched, the above step of obtaining the image features of the image to be queried and the text semantic features of the image text is executed.

[0129] Specifically, the image text is matched with the identification code of the preset format, which can be implemented specifically by string matching. If the identification code exists in the image text, the identification code is traversed and matched with the unique identification codes of each resource in the preset resource library. If there is a consistent resource unique identification code, the pre-stored resource of the resource unique identification code is determined as the target resource, otherwise the aforementioned steps of querying resources based on image text, image features, and text semantic features are executed. Exemplarily, the resource unique identification code of an e-book resource can be an ISBN (International Standard Book Number), and the resource unique identification code of a film and television resource can be a distribution license number, etc. A unique identification query is performed based on the image text before the image features and text semantic features are extracted to improve retrieval efficiency and accuracy and avoid waste of resources.

[0130] Taking the pre-stored resources as e-book resources as an example, the resource query method of the present application can be applied to the scenario of querying e-book resources based on book cover images. The book cover images are mapped to the corresponding e-books through the above method, and the relevant information or links of the e-book resources are fed back to facilitate convenient content retrieval and learning. Fig. 9 , the figure shows the resource information sharing interface of the client, through which the user can display the image information of the resource to be shared, including the book cover image. The query initiator can click on the book cover image in the interface to trigger the resource query event and generate a resource query instruction. The resource query instruction carries the image address information of the book cover image. After receiving the resource query instruction, the server executes the aforementioned resource query method. If the e-book resource corresponding to the book cover image is found, the link information is fed back to the client for display, such as Fig.10 As shown in the figure, the target resource display interface is shown, showing the book cover image and resource link information to be queried. The query initiator clicks the resource link information to jump to the book details page (reference Fig.11 ).

[0131] In summary, the present application combines text semantics and text content for multi-dimensional recall, thereby avoiding resource omissions while improving recall accuracy and recall rate; and, characterizing the correlation between the image to be queried and the pre-stored resources through the degree of correlation in multiple feature dimensions, thereby improving the accuracy and comprehensiveness of the expression of correlation evaluation information, thereby improving the accuracy of candidate resource screening and resource classification and identification, and avoiding resource omissions in multi-label iconic image scenarios; in addition, the multi-dimensional correlation indicators, resource feature data and image text of candidate resources are combined for classification and identification, thereby further improving the information dimension of the identification input and achieving accurate identification of related resources.

[0132] The embodiment of the present application also provides a resource recall device 800, such as Fig.12 As shown, Fig.12 A schematic diagram of the structure of a resource recall device provided in an embodiment of the present application is shown, and the device may include the following modules.

[0133] Acquisition module 10: used to acquire image text of the image to be queried, image features of the image to be queried, and text semantic features of the image text, where the image text is the content text carried by the image to be queried;

[0134] Recall module 20: used to recall resources in a preset resource library based on image text and text semantic features to obtain a recalled data set, where the preset resource library includes resource feature data of pre-stored resources;

[0135] Index generation module 30: used to generate a multi-dimensional correlation index between each pre-stored resource corresponding to the recall data set and the image to be queried based on the image text, text semantic features, image features and resource feature data, the multi-dimensional correlation index is used to indicate the correlation degree between the image to be queried and the pre-stored resources in multiple feature dimensions;

[0136] Resource screening module 40: used to screen candidate resources from the pre-stored resources corresponding to the recalled resource set according to the multi-dimensional correlation index;

[0137] Resource classification module 50: used to classify and identify candidate resources based on the multi-dimensional association index of the candidate resources, the resource feature data of the candidate resources and the image text, and obtain a resource classification result, which is used to characterize the matching degree between the candidate resources and the image to be queried.

[0138] In some embodiments, the device also includes a target determination module: used to classify and identify candidate resources based on the multi-dimensional association indicators of the candidate resources, the resource feature data of the candidate resources and the image text, and after obtaining the resource classification result, if the resource classification result indicates that the candidate resource matches the image to be queried, the candidate resource is determined as the target resource corresponding to the image to be queried.

[0139] In some embodiments, the resource feature data includes resource image text, resource semantic features, resource image features and identification entity information corresponding to the pre-stored resource; the indicator generation module 30 may include:

[0140] The first similarity calculation submodule is used to perform similarity calculation on the resource image text and the image text of the pre-stored resource to obtain a text similarity index;

[0141] The second similarity calculation submodule is used to perform similarity calculation on the resource semantic features and text semantic features of the pre-stored resources to obtain a semantic similarity index;

[0142] The third similarity calculation submodule is used to perform similarity calculation on the resource image features and image features of the pre-stored resources to obtain image similarity indicators;

[0143] The fourth similarity calculation submodule is used to perform similarity calculation on the identification entity information of the pre-stored resource and the image text to obtain the identification entity similarity index;

[0144] Total similarity calculation submodule: used to determine the total similarity index between the pre-stored resources and the image to be queried based on the text similarity index, semantic similarity index, image similarity index and identification entity similarity index. The total similarity index is used to characterize the total relevance between the image to be queried and the pre-stored resources.

[0145] In some embodiments, the resource screening module 40 may include a module specifically configured to determine pre-stored resources corresponding to total similarity indicators in the recalled resource set that meet preset screening conditions as candidate resources.

[0146] In some embodiments, the resource classification module 50 may include:

[0147] Entity matching submodule: used to perform entity matching between the identification entity information of the candidate resource and the image text to obtain entity matching features;

[0148] Resource feature generation submodule: used to generate resource features based on multi-dimensional association indicators and entity matching features;

[0149] Classification and identification submodule: used to input resource features into the resource classification model for classification and identification to obtain resource classification results.

[0150] In some embodiments, the recall module 20 may include:

[0151] A first recall submodule: used to perform semantic recall of resource semantic features on a preset resource library based on text semantic features to obtain a first resource data set, wherein the first resource data set includes resource semantic features that match the text semantic features;

[0152] The second recall submodule is used to perform entity recall based on the image text and the identification entity information to obtain a second resource data set;

[0153] The data set generation submodule is used to generate a recall data set according to the first resource data set and the second resource data set.

[0154] In some embodiments, the second recall submodule may include:

[0155] Structural data acquisition unit: used to acquire query structural data of a preset resource library, where the query structural data is obtained by constructing a data tree based on the identification entity information of each pre-stored resource in the preset resource library;

[0156] Entity matching unit: used to perform entity matching on image text based on query structure data;

[0157] The second resource generating unit is used to generate a second resource data set based on the pre-stored resources corresponding to the matched identification entity information.

[0158] In some embodiments, the acquisition module 10 may include:

[0159] Text recognition submodule: used to perform text recognition on the query image and obtain the image text;

[0160] Semantic feature extraction submodule: used to input image text into the text feature model to extract semantic features and obtain text semantic features;

[0161] Image feature extraction submodule: used to input the query image into the image feature model to extract image features and obtain image features.

[0162] In some embodiments, the apparatus further comprises:

[0163] Text feature extraction module: used to extract text features from the image text after obtaining the image text to be queried, and obtain multi-dimensional element characteristic features;

[0164] Image classification module: used to input multi-dimensional element characteristics into the image classification model for classification and recognition to obtain image classification results. The image classification results are used to characterize the resource element category of the image to be queried. The element characteristics of image texts of different resource element categories are different.

[0165] The first query trigger module is used to execute the step of obtaining the image features of the image to be queried and the text semantic features of the image text if the image classification result indicates that the image to be queried is a target resource element category.

[0166] In some embodiments, the resource characteristic data includes a resource unique identification code having a preset format corresponding to a pre-stored resource, and the apparatus further includes:

[0167] The identification code matching module is used to match the identification code with the unique identification code of the resource in the preset resource library after obtaining the image text of the image to be queried, if it is recognized that the image text includes the identification code in the preset format;

[0168] The target determination module is also used to determine the pre-stored resource to which the resource unique identification code belongs as the target resource corresponding to the image to be queried if the resource unique identification code corresponding to the identification code is matched.

[0169] It should be noted that the above device embodiment and method embodiment are based on the same implementation method.

[0170] An embodiment of the present application provides a device, which may be a terminal or a server, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement a resource query method or a neural network training method as provided in the above method embodiment.

[0171] The memory can be used to store software programs and modules. The processor executes various functional applications and abnormality detection by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required for functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0172] The method embodiments provided in the embodiments of the present application can be executed in electronic devices such as mobile terminals, computer terminals, servers or similar computing devices. Fig.13 1 is a hardware structure block diagram of an electronic device for a resource query method or a neural network training method provided in an embodiment of the present application. Fig.13As shown, the electronic device 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 910 (the processor 910 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 930 for storing data, and one or more storage media 920 (such as one or more mass storage devices) for storing application programs 923 or data 922. Among them, the memory 930 and the storage medium 920 can be short-term storage or permanent storage. The program stored in the storage medium 920 may include one or more modules, each of which may include a series of instruction operations in the electronic device. Furthermore, the central processing unit 910 can be configured to communicate with the storage medium 920 and execute a series of instruction operations in the storage medium 920 on the electronic device 900. The electronic device 900 may also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input and output interfaces 940, and / or, one or more operating systems 921, such as Windows Server TM , Mac OS X TM , Unix TM , LinuxTM, FreeBSDTM, etc.

[0173] The input / output interface 940 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the electronic device 900. In one example, the input / output interface 940 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 940 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0174] It can be understood by those skilled in the art that Fig.13 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.13 More or fewer components as shown, or with Fig.13 Different configurations shown.

[0175] An embodiment of the present application also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to an anomaly detection method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the anomaly detection method provided by the above method embodiment.

[0176] Optionally, in this embodiment, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0177] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0178] The resource query method, apparatus, device, storage medium, server, terminal and program product provided by the above-mentioned application, the technical solution of the application first obtains the image text of the image to be queried, the image features of the image to be queried and the text semantic features of the image text, the image text is the content text carried by the image to be queried; based on the image text and text semantic features, resources are recalled in a preset resource library to obtain a recall data set, so as to combine text semantics and text content for multi-dimensional recall, avoid resource omissions while improving recall accuracy and recall rate; then, based on the image text, text semantic features, image features and resource feature data, a multi-dimensional correlation index between each pre-stored resource corresponding to the recall data set and the image to be queried is generated, and then According to the multi-dimensional correlation index, candidate resources are screened from the pre-stored resources corresponding to the recalled resource set, and the candidate resources are classified and identified based on the multi-dimensional correlation index of the candidate resources, the resource feature data of the candidate resources and the image text to obtain the resource classification result. In this way, the correlation between the image to be queried and the pre-stored resources is characterized by the degree of correlation on multiple feature dimensions, so as to improve the accuracy and comprehensiveness of the expression of the correlation evaluation information, thereby improving the accuracy of candidate resource screening and resource classification and identification, and avoiding resource omission in multi-label landmark image scenarios; in addition, the multi-dimensional correlation index, resource feature data and image text of the candidate resources are combined for classification and identification, so as to further improve the information dimension of the identification input and realize the accurate identification of related resources.

[0179] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0180] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0181] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0182] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A resource query method, characterized in that: The method comprises: Acquire image text of the image to be queried, image features of the image to be queried, and text semantic features of the image text, wherein the image text is content text carried by the image to be queried; Based on the image text and the text semantic features, resources are recalled in a preset resource library to obtain a recalled data set, wherein the preset resource library includes resource feature data of pre-stored resources; Based on the image text, the text semantic features, the image features and the resource feature data, a multidimensional correlation index is generated between each pre-stored resource corresponding to the recall data set and the image to be queried, wherein the multidimensional correlation index is used to indicate the degree of correlation between the image to be queried and the pre-stored resources in multiple feature dimensions; Screening candidate resources from the pre-stored resources corresponding to the recalled resource set according to the multi-dimensional association index; Based on the multi-dimensional association index of the candidate resources, the resource feature data of the candidate resources and the image text, the candidate resources are classified and identified to obtain a resource classification result, and the resource classification result is used to characterize the matching degree between the candidate resources and the image to be queried.

2. The method according to claim 1, characterized in that: After classifying and identifying the candidate resources based on the multi-dimensional association index of the candidate resources, the resource feature data of the candidate resources and the image text to obtain a resource classification result, the method further includes: If the resource classification result indicates that the candidate resource matches the image to be queried, the candidate resource is determined as the target resource corresponding to the image to be queried.

3. The method according to claim 1, characterized in that: The resource feature data includes resource image text, resource semantic features, resource image features and identification entity information corresponding to the pre-stored resource; the generating of a multi-dimensional correlation index between each pre-stored resource corresponding to the recall data set and the image to be queried based on the image text, the text semantic features, the image features and the resource feature data includes: Performing a similarity operation on the resource image text of the pre-stored resource and the image text to obtain a text similarity index; Performing a similarity operation on the resource semantic features of the pre-stored resource and the text semantic features to obtain a semantic similarity index; Performing a similarity operation on the resource image feature of the pre-stored resource and the image feature to obtain an image similarity index; Performing a similarity operation on the identification entity information of the pre-stored resource and the image text to obtain an identification entity similarity index; Based on the text similarity index, the semantic similarity index, the image similarity index and the identification entity similarity index, a total similarity index between the pre-stored resource and the image to be queried is determined, and the total similarity index is used to characterize the total relevance between the image to be queried and the pre-stored resource.

4. The method according to claim 3, characterized in that The selecting candidate resources from the pre-stored resources corresponding to the recalled resource set according to the multi-dimensional association index comprises: The pre-stored resources corresponding to the total similarity index satisfying the preset screening condition in the recalled resource set are determined as the candidate resources.

5. The method according to claim 1, characterized in that The classification and identification of the candidate resources based on the multi-dimensional association index of the candidate resources, the resource feature data of the candidate resources and the image text to obtain the resource classification result includes: Perform entity matching on the identification entity information of the candidate resource and the image text to obtain entity matching features; Generate resource features based on the multi-dimensional association index and the entity matching features; The resource characteristics are input into a resource classification model for classification and identification to obtain the resource classification result.

6. The method according to any one of claims 1 to 5, characterized in that The resource recall is performed in a preset resource library based on the image text and the text semantic features, and the obtained recall data set includes: Performing semantic recall of resource semantic features on the preset resource library based on the text semantic features to obtain a first resource data set, wherein the first resource data set includes resource semantic features matching the text semantic features; Perform entity recall based on the image text and the identification entity information to obtain a second resource data set; The recall data set is generated according to the first resource data set and the second resource data set.

7. The method according to claim 6, characterized in that The entity recall based on the image text and the identification entity information to obtain the second resource data set includes: Acquire query structure data of the preset resource library, wherein the query structure data is obtained by constructing a data tree based on the identification entity information of each pre-stored resource in the preset resource library in a tree structure; Performing entity matching on the image text based on the query structure data; The second resource data set is generated based on the pre-stored resources corresponding to the matched identification entity information.

8. The method according to any one of claims 1 to 5, characterized in that The step of acquiring the image text of the image to be queried, the image features of the image to be queried, and the text semantic features of the image text comprises: Performing text recognition on the image to be queried to obtain the image text; Inputting the image text into a text feature model to extract semantic features, thereby obtaining the text semantic features; The image to be queried is input into an image feature model to extract image features to obtain the image features.

9. The method according to any one of claims 1 to 5, characterized in that After obtaining the image text of the image to be queried, the method further includes: Extracting text features from the image text to obtain multi-dimensional element characteristic features; Inputting the multidimensional element characteristic features into an image classification model for classification and identification to obtain an image classification result, wherein the image classification result is used to characterize the resource element category of the image to be queried, and the element characteristic features of image texts of different resource element categories are different; If the image classification result indicates that the image to be queried is a target resource element category, the step of acquiring image features of the image to be queried and text semantic features of the image text is performed.

10. The method according to any one of claims 1 to 5, characterized in that The resource feature data includes a resource unique identification code having a preset format corresponding to the pre-stored resource. After acquiring the image text of the image to be queried, the method further includes: If it is recognized that the image text includes the identification code in the preset format, matching the identification code with a unique resource identification code in the preset resource library; If a resource unique identification code corresponding to the identification code is matched, the pre-stored resource to which the resource unique identification code belongs is determined as the target resource corresponding to the image to be queried.

11. A resource recall device, characterized in that: The device comprises: Acquisition module: used to acquire image text of the image to be queried, image features of the image to be queried and text semantic features of the image text, wherein the image text is the content text carried by the image to be queried; Recall module: used to recall resources in a preset resource library based on the image text and the text semantic features to obtain a recall data set, wherein the preset resource library includes resource feature data of pre-stored resources; An index generating module: used for generating a multi-dimensional correlation index between each pre-stored resource corresponding to the recall data set and the image to be queried based on the image text, the text semantic features, the image features and the resource feature data, wherein the multi-dimensional correlation index is used for indicating the correlation degree between the image to be queried and the pre-stored resources in multiple feature dimensions; Resource screening module: used for screening candidate resources from each pre-stored resource corresponding to the recalled resource set according to the multi-dimensional correlation index; Resource classification module: used to classify and identify the candidate resources based on the multi-dimensional association indicators of the candidate resources, the resource feature data of the candidate resources and the image text, and obtain resource classification results, and the resource classification results are used to characterize the degree of matching between the candidate resources and the image to be queried.

12. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the resource query method according to any one of claims 1 to 10.

13. A computer device, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the resource query method according to any one of claims 1 to 10.

Images