A search method, device, equipment and storage medium

Abstract

The present application discloses a retrieval method, device, equipment and storage medium, wherein the method comprises: based on a target database, searching for retrieval input information to obtain a first retrieval result; determining a user's browsing behavior list for the first retrieval result; determining an expression tag corresponding to each browsing behavior in the browsing behavior list; generating a user's interest point feedback coefficient based on the browsing behavior list and each of the expression tags; and reordering the first retrieval result based on the interest point feedback coefficient to obtain a second retrieval result. In this way, personalized retrieval recommendation results can be provided to users, so that the user's expected retrieval results can be quickly and accurately located from massive information.

Description

Technical Field

[0001] The present application relates to, but is not limited to, the field of computer technology, and in particular to a retrieval method, apparatus, device and storage medium. Background Art

[0002] With the rapid development of computer vision and Internet technology, various websites and terminals have brought convenience to people's lives and entertainment. However, how to quickly and accurately locate the user's expected search results from massive video and image information is a problem that technical personnel in this field need to solve. Summary of the invention

[0003] In view of this, the present application provides a retrieval method, apparatus, device and storage medium, which can combine the user's potential retrieval intention to provide the user with personalized retrieval recommendation results, so that the user's expected retrieval results can be quickly and accurately located from massive information.

[0004] The technical solution of this application is implemented as follows:

[0005] In one aspect, the present application provides a retrieval method, the method comprising:

[0006] Based on the target database, searching the search input information to obtain a first search result;

[0007] Determining a list of browsing behaviors of the user on the first search result;

[0008] Determine an expression tag corresponding to each browsing behavior in the browsing behavior list;

[0009] Generating a user's interest point feedback coefficient according to the browsing behavior list and each of the expression tags;

[0010] The first search results are reordered according to the interest point feedback coefficient to obtain a second search result.

[0011] In another aspect, the present application provides a retrieval device, the device comprising:

[0012] A search module, used to search the search input information based on the target database to obtain a first search result;

[0013] A first determining module, used to determine a list of browsing behaviors of the user on the first search result;

[0014] A second determination module is used to determine the expression tag corresponding to each browsing behavior in the browsing behavior list;

[0015] A generating module, used for generating a user's interest point feedback coefficient according to the browsing behavior list and each of the expression tags;

[0016] The reordering module is used to reorder the first search result according to the interest point feedback coefficient to obtain a second search result.

[0017] On the other hand, the present application provides a retrieval device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and wherein the processor implements the steps in the above method when executing the program.

[0018] On the other hand, the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps in the above method when executed by a processor.

[0019] In the present application, after obtaining the first search result, the user's interest point feedback coefficient is generated according to the user's browsing behavior and expression tags of the search result, and the ranking of the first search result is adjusted according to the interest point feedback coefficient. In this way, when searching, due to the combination of the first search result and the user's interest feedback, personalized search recommendation results can be provided to the user, so that the user's expected search results can be quickly and accurately located from the massive information.

[0020] In some embodiments, the user's interest point feedback coefficient can be determined in different ways according to whether there is any interactive behavior in the user's browsing behavior. In this way, the generated interest point feedback coefficient more comprehensively reflects the user's interest in each result item, thereby improving the accuracy of the retrieval recommendation results.

[0021] In some embodiments, a two-stage retrieval method may be used, firstly, candidate results related to the retrieval input information are recalled coarsely through the recall matching feature in the retrieval feature, and then the candidate results are re-ranked finely through the sorting matching feature in the retrieval feature, and the sorting strategy is continuously adjusted in combination with user interest feedback information. In this way, personalized retrieval recommendation results can be provided for individual users.

[0022] In some embodiments, the CV model and the NLP model can be integrated when extracting features from the search input information, and shallow features and high-level semantic features can be integrated into the image search feature expression. In this way, the accuracy of image retrieval can be improved, the universality of image retrieval can be ensured, and it can support receiving user input images, texts, or image-texts, and returning image retrieval results, thereby further improving the user experience.

[0023] In some embodiments, the user's interest feedback coefficient for the previous search result can be generated by using the facial expression and sight direction detected by the face, and the user's potential search intention can be effectively mined by using the interest feedback coefficient. In this way, the problem of sparse or missing user dimension information during search in the related art can be solved, and search results that meet the user's search intention can be obtained.

[0024] In some embodiments, the user's browsing behavior and corresponding expression tags for the previous search results can be obtained, and the user's interest feedback coefficient is updated through the obtained browsing behavior and corresponding expression tags, and the previous search results are re-ranked according to the user's interest feedback coefficient. In this way, the search results can be continuously updated and ranked in combination with the user's interest feedback coefficient and the historical ranking results, so as to obtain search results that better meet the user's expectations. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of an implementation flow of a search method provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of an implementation flow of a search method provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of the structure of an image retrieval and recommendation device provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of the composition structure of a fusion network model provided in an embodiment of the present application;

[0029] Figure 5 This is a schematic diagram of key points of a human face;

[0030] Figure 6 A schematic diagram of the implementation flow of a method for acquiring user browsing behavior provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of an implementation flow of a method for generating an interest score provided in an embodiment of the present application;

[0032] Fig. 8A A schematic diagram of a data flow for extracting image features from an image using a CV model provided in an embodiment of the present application;

[0033] Figure 8B A data flow diagram of extracting text features from text using an NLP model provided in an embodiment of the present application;

[0034] Fig. 9A A schematic diagram of data flow of a two-stage image retrieval method provided in an embodiment of the present application;

[0035] Fig. 9B A data flow diagram of a search recommendation ranking method combined with user interest feedback provided in an embodiment of the present application;

[0036] Fig.10 A schematic diagram of the composition structure of a retrieval device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0038] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] If similar descriptions of "first / second" appear in the application documents, the following instructions are added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0041] In order to better understand the retrieval method provided in this application, the retrieval method used in the related art is first explained.

[0042] Taking image retrieval as an example, in related technologies, in order to retrieve the images required by users, it is usually necessary to first extract the image features of the images or videos in the database, use these features to describe the features of the images or videos, and then format the image features, create an index, and generate a feature index library; when the user searches by inputting relevant images or texts, the system first extracts the features of the input images or texts, and then calculates the similarity between the features of the input images or texts and the image features in the feature index library, and outputs the search results that are identical, similar, or related to them.

[0043] The image retrieval technology in the above-mentioned related technologies is a content-based image retrieval technology, which aims to search for images that are identical, similar or related to the input content by analyzing the input content. There are two main tasks during retrieval: image representation and similarity measurement. Since the feature expression of the retrieved image is fixed and the similarity measurement method is consistent, the similar images or related images returned by the retrieval can be understood as fixed. However, there are deviations in the understanding of similar images or related images by different users, such as semantic similarity and visual similarity. Therefore, how to return results that meet user expectations is a major problem in the field of image retrieval. In addition, since the content to be retrieved input by users is diverse, such as text retrieval images (also known as "text search for images") and image retrieval images (also known as "image search for images"). Image search based on text queries and image instance search are the two main ways of image retrieval. Faced with massive Internet images with only weak supervision information describing text, it is necessary not only to solve the visual modeling problem, but also to solve the natural language understanding problem, and establish a mapping relationship between vision and natural language, which is also a major difficulty.

[0044] The present application embodiment provides a retrieval method, which can be executed by a processor of a retrieval device, such as Figure 1 As shown, the method comprises the following steps:

[0045] Step S101, searching the search input information based on the target database to obtain a first search result;

[0046] Here, the target database is a database storing resources to be retrieved, and the stored resources to be retrieved include but are not limited to texts, images, videos, etc.

[0047] The search input information is the information that needs to be searched that is input by the user, and may include but is not limited to input images, input text, etc. In implementation, the user may input the search input information through the interactive interface of the search device, or may input the search input information through other devices, and then the other devices send the search input information input by the user to the processor of the search device. Those skilled in the art may select a suitable method to obtain the search input information according to actual conditions, and the embodiments of the present application are not limited to this.

[0048] The first search result includes information in the target database that is identical, similar, or related to the search input information, and may include but is not limited to text, images, videos, and other information. Each information in the first search result can be used as a result entry. After obtaining the first search result, some or all of the result entries of the first search result can be displayed on the search result display page. In some embodiments, the first search result can be the result of sorting the information in the target database that is similar to the search input information in descending order according to the similarity. In other embodiments, the first search result can be the result of sorting the information in the target database that is related to the search input information in descending order according to the relevance.

[0049] Step S102, determining a list of browsing behaviors of the user on the first search result;

[0050] Here, after obtaining the first search result, the user will browse the result items in the first search result, and may browse only some of the result items in the first search result, or may browse all the result items in the first search result. The browsing behavior list includes the browsing behaviors generated by the user browsing the result items in the first search result, and the number of browsing behaviors in the browsing behavior list depends on the actual situation. The browsing behavior may include the browsed result items and the indicator data for browsing the result items. The indicator data may include but is not limited to browsing time, interaction time, stay time, etc. The method for determining the browsing behavior may include but is not limited to determining by the user's interaction record on the current search page or by tracking the user's line of sight. During implementation, those skilled in the art may select appropriate indicator data and the method for determining the browsing behavior according to the actual situation to determine the browsing behavior generated by the user browsing each result item, and the embodiment of the present application does not limit this.

[0051] Step S103, determining an expression tag corresponding to each browsing behavior in the browsing behavior list;

[0052] Here, the expression tag can represent the expression state of the user when browsing, and can reflect the user's interest in the currently browsed result item. In implementation, the expression tag may include but is not limited to specific values ​​or characters. Those skilled in the art can select appropriate expression tags according to actual conditions, and the embodiments of the present application are not limited to this. For example, the expression tag may include 1, 0, -1, where 1, 0, -1 respectively represent the expression state of the user when browsing the current result item as smiling, neutral, and frowning, and further can respectively reflect that the user is interested, neutral, and uninterested in the currently browsed result item.

[0053] Step S104, generating a user's interest point feedback coefficient according to the browsing behavior list and each of the expression tags;

[0054] Here, the user's interest point feedback coefficient may include the user's interest level in each result item in the first search result. For the result items that the user has browsed, the user's interest level in the result items may be determined based on the browsing behavior for the result items in the browsing behavior list and the expression tag corresponding to the browsing behavior. For the result items that the user has not browsed, an appropriate interest level may be determined for the result items based on actual conditions. For example, it may be assumed that the user is not interested in the result items that have not been browsed.

[0055] Step S105: reorder the first search results according to the interest point feedback coefficient to obtain a second search result.

[0056] Here, the ranking weight of each result item can be recalculated based on a specific ranking weight calculation model according to the user's interest level in each result item in the first search result in the interest point feedback coefficient. According to the recalculated ranking weight, each result item in the first search result can be reordered to obtain a second search result. Those skilled in the art can select a suitable ranking weight calculation model according to actual conditions during implementation, and the embodiments of the present application are not limited to this.

[0057] In some embodiments, the interest point feedback coefficient may include the user's interest score for each result item in the first search result, and the browsing behavior may include the browsed result items, interaction time, sight browsing time, and result item dwell time. Correspondingly, the above step S104 may include:

[0058] Step S1041, for each browsing behavior in the browsing behavior list, when the interaction duration is greater than 0, determining a first ratio; the first ratio is a ratio of the interaction duration to the dwell time of the result item;

[0059] Here, when the interaction duration is greater than 0, it means that the user has interactive behavior for the result item when browsing the corresponding result item. The interactive behavior can be a record of participation in any interactive scene in the result item, including but not limited to any one or more of clicking a button, dragging an icon, watching a video, etc. The interactive behavior for the same result item can be one or more times, and the interaction duration can be the duration of one interactive behavior or the total duration of multiple interactive behaviors. In implementation, the interaction duration can be determined by the user's interaction record in the interactive interface of the current result item, or by tracking and analyzing the user's line of sight, which is not limited in the embodiments of the present application.

[0060] The result entry dwell time is the time a user dwells on a browsing page when browsing the current result entry, and can be determined through operation records of the browsing page.

[0061] Step S1042, determining the interest score of the result item according to the first ratio;

[0062] Here, the first ratio can be directly determined as the interest score of the result item; or the first ratio can be normalized according to a specific normalization algorithm, and the normalized first ratio can be determined as the interest score of the result item. During implementation, those skilled in the art can select a suitable method to determine the interest score according to actual conditions, and the embodiments of the present application are not limited to this.

[0063] Step S1043: when the interaction time is equal to 0, determining a second ratio; the second ratio is the ratio of the visual browsing time to the dwell time of the result entry;

[0064] Here, when the interaction duration is equal to 0, it means that the user has not generated any interaction behavior with respect to the result item when browsing the corresponding result item.

[0065] The sight browsing time is the time that the user's sight browses the result item. In implementation, the user's sight direction can be tracked through face detection, and the sight browsing time for the result item can be determined based on the consistency of the sight direction and the time the sight stays on the browsing page of the result item.

[0066] Step S1044: determining the interest score of the result item according to the second ratio and the expression tag of the browsing behavior.

[0067] Here, the value of the second ratio can be appropriately adjusted according to different expression tags, and the adjusted value can be determined as the interest score of the result item, or the adjusted value can be normalized according to a specific normalization algorithm and determined as the interest score of the result item. During implementation, those skilled in the art can select a suitable adjustment algorithm to adjust the value of the second ratio according to actual conditions, and the embodiments of the present application are not limited to this. For example, a label experience coefficient can be set for each expression tag, and the label experience coefficient corresponding to the expression tag of each browsing behavior can be used as an adjustment coefficient, and the product of the adjustment coefficient and the second ratio of the browsing behavior can be used as the adjusted value.

[0068] The retrieval method provided in the embodiment of the present application, after obtaining the first retrieval result, generates the user's interest point feedback coefficient through the user's browsing behavior and expression tags of the retrieval result, and adjusts the ranking of the retrieval results according to the interest point feedback coefficient. In this way, since the user's interest feedback is taken into account, personalized retrieval recommendation results can be provided to the user, so that the user's expected retrieval results can be quickly and accurately located from massive information. Furthermore, the user's interest point feedback coefficient can be determined in different ways according to whether there is interactive behavior in the user's browsing behavior. The interest point feedback coefficient generated in this way more comprehensively reflects the user's interest in each result item, thereby improving the accuracy of the retrieval recommendation results.

[0069] The present application embodiment provides a retrieval method, which can be executed by a processor of a retrieval device, such as Figure 2 As shown, the method includes:

[0070] Step S201, extracting features from the search input information to obtain search features of the search input information;

[0071] Here, the retrieval feature is a feature expression of the retrieval input information, which may include but is not limited to semantic features, visual features, etc.

[0072] In some embodiments, the retrieved input information may include input images and input text. Correspondingly, the above step S201 may include:

[0073] Step S2011, using a computer vision (CV) model to extract image features from the input image to obtain a first feature vector;

[0074] Here, the first feature vector is a feature vector of the input image, and may include but is not limited to one or more of a visual feature vector, a semantic feature vector, and the like.

[0075] The CV model is used to encode the input image to obtain the semantic information of the input image and the corresponding feature vector. During implementation, a trained image processing network can be used to fine-tune the image processing network on the scene data set to obtain the fine-tuned image processing network, and the fine-tuned image processing network is used to perform target detection and positioning on the input image, and extract the image features of the input image. Those skilled in the art can select a suitable CV model for image feature extraction during implementation, and the embodiments of the present application are not limited to this. For example, a convolutional neural network (CNN) trained on a large-scale ImageNet data set can be used, and the CNN is fine-tuned on the scene data set to obtain a fast regional convolutional neural network (Faster Region-CNN, Faster-RCNN), and Faster-RCNN is used to perform target detection and positioning on the input image, and extract the FC7 layer depth features of the input image.

[0076] Step S2012, using a natural language processing (NLP) model to extract features from the input text to obtain a second feature vector;

[0077] Here, the second feature vector is a feature vector of the input text, and may include but is not limited to one or more of a visual feature vector, a semantic feature vector, and the like.

[0078] The NLP model is used to implement text information encoding, and the semantic information in the input text can be encoded into a corresponding feature vector. During implementation, those skilled in the art can select a suitable NLP model according to actual conditions, and the embodiments of the present application are not limited to this. For example, a gated recurrent unit (Gated Recurrent Unit, GRU) network and a time domain signal global maximum pooling (GlobalMaxpooling1D) layer feature can be used to implement encoding the semantic information in the input text into a corresponding feature vector.

[0079] Step S2013: adopt a specific fusion model to perform feature fusion and compression mapping on the first feature vector and the second feature vector to obtain retrieval features.

[0080] Here, the fusion model can be any suitable network model for feature encoding association between the CV model and the NLP model. Those skilled in the art can select a suitable fusion model according to actual conditions during implementation, and the embodiments of the present application are not limited to this. By performing feature fusion and compression mapping on the first feature vector and the second feature vector by the fusion model, a retrieval feature that meets the needs of subsequent retrieval can be obtained. Subsequent retrieval needs may include, but are not limited to, one or more of the vector dimensions and formats of the retrieval features.

[0081] Step S202, using the retrieval feature, performing feature matching on each image feature in the index library to obtain a first target index set; the index library includes the image feature of each image in the image database;

[0082] Here, the first target index set includes indexes corresponding to the target number of images in the image database that are identical, similar or related to the search input information. Image features may include but are not limited to image semantic features, image visual features, etc. During implementation, those skilled in the art may select a suitable algorithm for feature matching according to actual conditions, and the embodiments of the present application are not limited to this.

[0083] Step S203, using the first target index set to query the image database to obtain a first search result;

[0084] Here, the image database may be queried according to a specific index rule to obtain the first search result. Those skilled in the art may select a suitable index rule according to actual conditions, which is not limited here.

[0085] Step S204, determining a list of browsing behaviors of the user on the first search result;

[0086] Step S205, determining an expression tag corresponding to each browsing behavior in the browsing behavior list;

[0087] Step S206, generating a user's interest point feedback coefficient according to the browsing behavior list and each of the expression tags;

[0088] Step S207: reorder the first search results according to the interest point feedback coefficient to obtain a second search result.

[0089] Here, the above steps S204 to S207 correspond to the above steps S102 to S105. When implementing, the specific implementation methods of the above steps S102 to S105 can be referred to, and will not be repeated here.

[0090] In some embodiments, the retrieval feature may include a recall matching feature and a sort matching feature. Correspondingly, the above step S202 may include:

[0091] Step S2021, using the recalled matching features, perform feature matching on each image feature in the index library to determine a candidate index set;

[0092] Here, the candidate index set includes indexes corresponding to images in the image database that are identical, similar or related to the retrieval input information. The recall matching features are features used to recall images that are identical, similar or related to the retrieval input information from the image database, and may include but are not limited to image semantic features, image visual features, etc. When performing feature matching, image semantic features can be used for matching, or image visual features can be used for matching, which is not limited in the embodiments of the present application. For example, the hash code of the mapping layer (Latent Layer) H between F7 and F8 in the deep hash network can be used, and the similarity can be calculated by Hamming distance to obtain a candidate set similar to the retrieval input information as the first target index set.

[0093] Step S2022, for each index in the candidate index set, using the sorting matching feature, performing feature matching on the image feature corresponding to the index to obtain the matching degree of the index;

[0094] Here, the ranking matching feature is a feature used to match and determine the degree of match between the image corresponding to each index in the candidate set and the retrieval input information. The ranking matching feature can adopt but is not limited to one or more of deep features (such as semantic features of the image), local features (such as local visual features of the image), etc. Those skilled in the art can select appropriate ranking matching features and feature similarity calculation methods according to actual conditions during implementation, and the embodiments of the present application are not limited to this. For example, the depth features of the image can be used as ranking matching features, and the cosine similarity algorithm can be used to calculate feature similarity.

[0095] Step S2023 , selecting the target number of indexes from the candidate index set in descending order according to the matching degree, to obtain the first target index set.

[0096] Here, the target quantity may be determined according to the actual situation, may be a default value, or may be a value set by the user when performing a search, and is not limited here.

[0097] In some embodiments, the above step S207 may include:

[0098] Step S2071, taking the retrieval feature and the image feature corresponding to each index in the first target index set as input, taking the interest point feedback coefficient as a model parameter, and adopting a specific regression model to perform feature fitting to obtain an updated ranking matching feature;

[0099] Here, the image interest point feedback coefficient includes the user's interest score for the image corresponding to each index in the first target index set. The regression model is used to update the sorting matching features, and any suitable feature fitting algorithm can be used, which is not limited in the embodiments of the present application. For example, the interest score of the image corresponding to each index in the first target index set can be used as a weighting coefficient, and each corresponding image feature can be weighted and summed to obtain a feature vector based on interest feedback, and then the feature vector is summed with the retrieval feature and averaged to obtain the updated sorting matching feature.

[0100] Step S2072, for each index in the first target index set, using the updated sorting matching feature, performing feature matching on the image feature corresponding to the index, and updating the matching degree of the index;

[0101] Here, the specific implementation of feature matching for the image features corresponding to the index may refer to the implementation of the aforementioned step S2022.

[0102] Step S2073, re-sorting the indexes in the first target index set in descending order of matching degree to obtain a second target index set;

[0103] Step S2074: Query the image database using the second target index set to obtain a second search result.

[0104] The retrieval method provided in the embodiment of the present application adopts a two-stage retrieval method. First, the candidate results related to the retrieval input information are coarsely recalled through the recall matching feature in the retrieval feature, and then the candidate results are re-ranked through the sorting matching feature in the retrieval feature. At the same time, the sorting strategy is continuously adjusted in combination with the user's interest feedback information, which can provide personalized retrieval recommendation results for individual users. In addition, when extracting features from the retrieval input information, the CV model and the NLP model are integrated, and shallow features and high-level semantic features are incorporated into the image retrieval feature expression, which can improve the accuracy of image retrieval, ensure the universality of image retrieval, and support receiving user input images, texts, or image-texts, and returning image retrieval results, which can further enhance the user experience.

[0105] The present application provides a retrieval method, which can be executed by a processor of a retrieval device. The method includes:

[0106] Step S301, searching the search input information based on the target database to obtain a first search result;

[0107] Here, the above step S301 corresponds to the above step S101, and the specific implementation of the above step S101 may be referred to during implementation.

[0108] Step S302, performing face detection on the collected user face image to obtain the user's sight direction;

[0109] Here, the eye sight can be detected and tracked based on face detection to obtain the user's sight direction. In implementation, any suitable eye sight direction detection algorithm can be used, and the embodiments of the present application are not limited to this. For example, the eye sight detection and tracking can be implemented through the computer vision library OpenFace.

[0110] Step S303, determining a browsing behavior list of the user for the first search result according to the sight direction and the user's interaction record on the current search page;

[0111] Here, each browsing behavior in the browsing behavior list may include a browsing result item and index data of browsing the result item. The index data may include but is not limited to browsing time, interaction time, stay time, etc.

[0112] The user's interaction record on the current search page may include but is not limited to one or more of the user's clicks, scrolling, selections, element dragging, etc. on the current search page. According to the user's line of sight direction and the user's interaction record on the current search page, any one or more indicator data of the result items browsed by the user, the browsing start time, the browsing end time, the interaction duration, the result item stay duration, the line of sight stay duration, etc. can be determined. For example, the user's browsing start time and browsing end time for the result item can be determined by recording the time when the user clicks on the browsing result item and the time when the user clicks on the closing browsing result item, and the result item stay duration can be determined by calculating the difference between the browsing end time and the browsing start time. For another example, it can be determined whether the user is browsing the current result item by detecting the consistency of the left and right eye directions in the line of sight direction, and the user's line of sight stay interval duration for the result item can be determined by detecting the consistency of the line of sight direction of the left and right eyes in the image frame with an interval time t in the line of sight direction. In implementation, the line of sight direction consistency can be obtained by the cosine similarity of the line of sight direction vector.

[0113] Step S304, determining the user facial features corresponding to each browsing behavior in the browsing behavior list through face detection;

[0114] Here, the user's facial features are used to determine the user's current expression. The user's facial features can be obtained by performing face detection on the user's face image or face video collected within the time range of each browsing behavior. During implementation, any suitable expression detection algorithm can be used to determine the user's facial features, and the embodiments of the present application are not limited to this.

[0115] In some embodiments, the user's facial features may be the facial key point information of the current user, which may include but is not limited to key points corresponding to facial organs such as eyebrows, eyes, nose, mouth, etc. During implementation, the facial key point information may be detected through the computer vision library Dlib.

[0116] Step S305, determining an expression tag corresponding to each browsing behavior according to the user facial feature corresponding to each browsing behavior;

[0117] Here, the expression tag is used to represent the expression state of the user when browsing, and can reflect the user's interest in the currently browsed result item. Any appropriate expression recognition algorithm can be used to recognize the user's facial features to obtain the corresponding user expression. According to the obtained user expression, the expression tag corresponding to the user expression can be determined based on the correspondence between a specific expression tag and the expression.

[0118] Step S306, generating a user's interest point feedback coefficient according to the browsing behavior list and each of the expression tags;

[0119] Step S307: reorder the first search results according to the interest point feedback coefficient to obtain a second search result.

[0120] Here, the above steps S306 to S307 correspond to the above steps S104 to S105, and when implementing, reference may be made to the specific implementation of the above steps S104 to S105.

[0121] In some embodiments, the above step S305 may include:

[0122] Step S3051, determining the user's eyebrow distance and mouth features according to the user's facial features;

[0123] Here, the mouth features may include but are not limited to the distance between the upper and lower lips, the width of the circumscribed rectangle of the mouth, the aspect ratio of the circumscribed rectangle of the mouth, the curvature of the lips, and any other features that can determine whether the user is smiling.

[0124] Step S3052: determining an expression tag corresponding to the browsing behavior according to the eyebrow distance and the mouth feature.

[0125] Here, it is possible to determine whether the user is in a frowning state based on the distance between the eyebrows, and whether the user is in a smiling state based on the features of the mouth. The user's expression can be determined by combining whether the user is frowning and whether the user is smiling. During implementation, those skilled in the art can select an appropriate judgment method based on actual conditions, and the embodiments of the present application are not limited to this. For example, when the user is in a smiling state, the user's expression is determined to be a smile; when the user is in a frowning state, the user's expression is determined to be a frown; when the user is neither in a smiling state nor in a frowning state, the user's expression can be determined to be neutral.

[0126] In some embodiments, the mouth feature includes the distance between the upper and lower lips or the aspect ratio of the circumscribed rectangle of the mouth, and the expression tags include a first expression tag, a second expression tag, and a third expression tag. Correspondingly, the above step S3052 may include: when the distance between the upper and lower lips and the aspect ratio of the circumscribed rectangle of the mouth meet the first condition, determining that the browsing behavior corresponds to the first expression tag; the first condition includes: the distance between the upper and lower lips is greater than a specific lip distance threshold, or the aspect ratio of the circumscribed rectangle of the mouth is less than a specific aspect ratio coefficient threshold; when the eyebrow distance meets the first condition, determining that the browsing behavior corresponds to the second expression tag; the first condition includes that the eyebrow distance is less than a specific eyebrow distance threshold; when the distance between the upper and lower lips and the aspect ratio of the circumscribed rectangle of the mouth do not meet the first condition, and the eyebrow distance does not meet the second condition, determining that the browsing behavior corresponds to the third expression tag.

[0127] The retrieval method provided in the embodiment of the present application can generate a user's interest feedback coefficient for the previous retrieval result by using the facial expression and gaze direction detected by the face. The interest feedback coefficient can be used to effectively explore the user's potential retrieval intention, thereby solving the problem of sparse or missing user dimensional information when searching in related technologies, and obtaining retrieval results that meet the user's retrieval intention.

[0128] The present application provides a retrieval method, which can be executed by a processor of a retrieval device. The method includes:

[0129] Step S401, searching the search input information based on the target database to obtain a first search result;

[0130] Step S402, determining a list of browsing behaviors of the user on the first search result;

[0131] Step S403, determining an expression tag corresponding to each browsing behavior in the browsing behavior list;

[0132] Step S404, generating a user's interest point feedback coefficient according to the browsing behavior list and each of the expression tags;

[0133] Step S405, reordering the first search results according to the interest point feedback coefficient to obtain a second search result;

[0134] Here, the above steps S401 to S405 correspond to the above steps S101 to S105, and the specific implementation methods of the above steps S101 to S105 may be referred to during implementation.

[0135] Step S406, determining a list of browsing behaviors of the user on the second search result;

[0136] Step S407, determining the expression tag corresponding to each browsing behavior in the browsing behavior list;

[0137] Step S408, updating the user's interest point feedback coefficient according to the browsing behavior list and each of the expression tags;

[0138] Step S409: reorder the second search results according to the updated interest point feedback coefficient to obtain a third search result.

[0139] Here, the above steps S406 to S409 are similar to the description of the above steps S102 to S105, and the specific implementation of the above steps S102 to S105 may be referred to during implementation.

[0140] The retrieval method provided in the embodiment of the present application can obtain the user's browsing behavior and corresponding expression tags for the previous retrieval results, and update the user's interest feedback coefficient through the obtained browsing behavior and corresponding expression tags, and re-rank the previous retrieval results according to the user's interest feedback coefficient. In this way, the retrieval results can be continuously updated and ranked in combination with the user's interest feedback coefficient and the historical ranking results, so as to obtain retrieval results that better meet the user's expectations.

[0141] The present application provides a retrieval method, which can be executed by a processor of a retrieval device. The method includes:

[0142] Step S501, extracting features from images in the image database to obtain image features of each image;

[0143] Step S502: for each of the image features, a specific indexing rule is used to establish an index library.

[0144] Here, based on the index library, the user can query the corresponding image features through the index, and can also query the corresponding image based on the image database according to the index in the index library. The index rule can adopt any appropriate rule, which is not limited here.

[0145] Step S503, searching the search input information based on the target database to obtain a first search result;

[0146] Step S504, determining a list of browsing behaviors of the user on the first search result;

[0147] Step S505, determining the expression tag corresponding to each browsing behavior in the browsing behavior list;

[0148] Step S506, generating a user's interest point feedback coefficient according to the browsing behavior list and each of the expression tags;

[0149] Step S507, reordering the first search result according to the interest point feedback coefficient to obtain a second search result;

[0150] Here, the above steps S503 to S507 correspond to the above steps S101 to S105, and when implementing, reference may be made to the specific implementation of the above steps S101 to S105.

[0151] The present invention proposes an image retrieval recommendation device combining user interest feedback, such as Figure 3 As shown, the image retrieval recommendation device includes a human-computer interaction module 310, a server retrieval recommendation module 320, and a user interest collection feedback module 330, wherein:

[0152] The human-computer interaction module 310 includes a text or image receiving module 311 and a feature extraction module 312. The text or image receiving module 311 is used to receive the search image or text input by the user, and the feature extraction module 312 is used to extract features of the search image or text input by the user by combining the advantages of computer vision and natural language processing. When performing feature extraction, the CV model can be used to implement image information encoding, and the NLP model can be used to implement text information encoding, thereby combining the shallow features of the image (such as contour, shape, geometric relationship, etc.) and high-level semantic features to obtain the semantic features and visual feature expressions of the image, which can effectively solve the "semantic gap" problem of image semantics in text and image.

[0153] The server retrieval recommendation module 320 includes an image database feature indexing module (not shown in the figure), an image two-stage retrieval module 321, an image database module 322 and an index library module 323. The image database feature indexing module is used to extract features from the images in the image database module 322 through the above-mentioned feature extraction module 312, index the extracted image features and store them in the index library module 323. When extracting features from the images in the image database module 322, the feature extraction module 312 uses the same CV model and NLP model as when extracting features from the retrieval image or text input by the user, and at the same time, performs feature processing and compression mapping on the extracted image features in combination with the needs of the image two-stage retrieval module 321 when performing retrieval. Here, the compression mapping can use a hash map. The image two-stage retrieval module 321 is used to perform two-stage retrieval on the features of the retrieval image or text input by the user. Specifically, the image two-stage retrieval module 321 includes: a recall feature comparison module 3211, which is used to maximize the recall of images related to the retrieval image or text input by the user through coarse-grained query, thereby ensuring the recall rate; a reordering feature comparison module 3212, which is used to reorder the results returned by the recall feature comparison module 3211 through fine-grainedness, return the sorting results, and ensure that the images are similar or related.

[0154] The user interest collection feedback module 330 includes a search result acquisition module 331 and an interest point collection module 332. The interest point collection module 332 is used to: obtain the user's facial information and key point features through the face detection process to obtain the user's current expression label; obtain the user's eye sight direction, and obtain the corresponding interest image label from the consistency of the user's eye sight direction; upload the obtained user's current expression label and interest image label to the server, and combine the user's cross-behavior feedback information (including interaction time, stay time and other information) to feedback the user's interest point feedback coefficient in the current scene.

[0155] In addition, the reordering feature comparison module 3212 is also used to reorder the previous sorting result in combination with the user's current interest point feedback coefficient to generate a reordered image retrieval result.

[0156] The specific implementation processes of the human-computer interaction module, user interest feedback module and server retrieval recommendation module are described below:

[0157] 1) Human-computer interaction module:

[0158] This module receives images, texts, and image-texts input by users, and performs feature extraction on the received images, texts, and image-texts. The feature extraction adopts a fusion network model that associates the feature encoding of the Computer Vision (CV) model with the Natural Language Processing (NLP) model. Figure 4 As shown, two models are used to construct the fusion network model 400: a CV model 410 and an NLP model 420, wherein the CV model 410 includes an image receiving module 411, a data preprocessing module 412, a CNN network 413 and an FC7-4096 layer 414; the NLP model 420 includes a text receiving module 421, a data preprocessing module 422, a GRU network 423 and a GlobalMaxpooling1D layer 424. Here, the CV model is used to detect the image to obtain semantic information and corresponding feature vectors. In the CV model, a CNN network pre-trained on a large-scale ImageNet dataset is used, and the CNN network is fine-tuned on the scene dataset to obtain a fine-tuned CNN network (such as Faster-RCNN). The fine-tuned CNN network is used to detect and locate the image processed by the data preprocessing module, and the FC-7 layer is used for calculation to extract the "FC7" layer deep features of the image. The NLP model uses the GRU network and the GlobalMaxpooling1D layer features to implement text feature encoding. In addition, the fusion network model 400 also includes a linear transformation module 430 and a projection matrix module 440. The CV model and the NLP model are further integrated, and the feature encoding of the extracted image features and text features is linearly transformed by the linear transformation module 430, so that the image, text or image-text input into the fusion network model have the same feature dimension; the projection matrix module 440 obtains the projection matrix parameters through network training, and the extracted features can be compressed and mapped based on the projection matrix parameters.

[0159] 2) User interest collection and feedback module:

[0160] This module includes facial feature point detection, eye gaze direction detection, and interest point feedback coefficient generation, as follows:

[0161] Facial feature point detection is mainly to obtain the facial key point information of the current user. The computer vision library Dlib can detect 68 key points on the face, such as Figure 5 As shown, the key points of eyebrows, eyes, nose and mouth of a person's face can be represented by feature points 18 to 68.

[0162] Eye gaze direction detection is mainly used to obtain the current user's eye gaze direction. Eye gaze detection and tracking can be achieved through the computer vision library OpenFace. The user's current browsing behavior can be obtained through the interaction record between the eye gaze direction and the current page (including start time, gaze dwell time, etc.).

[0163] Figure 6 A schematic diagram of the implementation flow of a method for obtaining user browsing behavior provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the method includes:

[0164] Step S601, performing face detection on the collected face image;

[0165] Step S602, detecting and tracking the user's sight direction through face detection;

[0166] Step S603, determining the browsing direction and browsing duration of the user according to the user's line of sight.

[0167] According to the user's browsing behavior, the user's interest point feedback coefficient can be generated. The interest point feedback coefficient can be an interest score list, including the interest score for each browsing behavior. For each browsing behavior, the user's interest score for the browsing behavior can be obtained through the user's eye line of sight, the interaction record of the current browsing page, and the expression tag of the current browsing behavior. Then, according to the browsing behavior list, the browsing behavior and its interest score can be obtained. Among them, the interaction record of the current browsing page includes the start time, viewing time, and total browsing time.

[0168] Figure 7 A schematic diagram of an implementation flow of a method for generating an interest score provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the method includes:

[0169] Step S701, determining the user's current browsing behavior;

[0170] Step S702, determining the user's current sight direction and expression label;

[0171] Here, the user's facial expression can be determined through face detection, the user's mood label can be determined, and the corresponding expression label can be determined according to the user's mood label.

[0172] Among them, mood labels are divided into three categories: smile, neutral, and frown. The smile, neutral, and frown mood labels are mapped to expression labels with values ​​of 1, 0, and -1 respectively.

[0173] The generation strategy of mood labels is determined based on the position information of key points in the image coordinate system and the aspect ratio coefficient of the circumscribed rectangle of the facial organs. In implementation, mood labels can be determined using the following formula (1-1):

[0174]

[0175] Among them, Condition m is the generated mood label, 1 represents smiling, 0 represents neutral, and -1 represents frowning. Dis(P 52 ,P 58 )for Figure 5 The key points P of the upper and lower lips shown in 52 , P 58 Distance, threshold DisM The key points P of the upper and lower lips 52 , P 58 The distance threshold can be set to 16 as an empirical value during implementation. Dis(P 22 ,P 23 )for Figure 5 The eyebrow key points P shown in 22 , P 23 Distance, threshold DisE P is the eyebrow key point 22 , P 23 The distance threshold can be set to 18. M It is the ratio coefficient threshold of the mouth's circumscribed rectangle. An empirical value of 1.8 can be used for implementation.

[0176] In the image coordinate system, the Euclidean distance between two points can be calculated using the following formula (1-2):

[0177]

[0178] Among them, P1 and P2 are the coordinates of facial key points respectively.

[0179] The aspect ratio of the circumscribed rectangle of the facial organ can be calculated using the following formula (1-3):

[0180] Scale = wide / height(1-3);

[0181] Among them, wide is the length of the circumscribed rectangle of the facial organ, and height is the width of the circumscribed rectangle of the facial organ.

[0182] Step S703, determining whether there is any interactive behavior record for the current user's line of sight browsing; if there is any interactive behavior record, proceeding to step S704; if there is no interactive behavior record, proceeding to step S706;

[0183] Step S704, determining the start time of the user's current browsing behavior, the total time spent on the browsing page, and the interaction time or viewing time;

[0184] Step S705, determining the ratio of the current viewing time to the total time spent on the browsing page as the user's interest score for the current browsing behavior, or determining the ratio of the current interaction time to the total time spent on the browsing page as the user's interest score for the current browsing behavior;

[0185] Step S706, determining the start time of the user's current browsing behavior, the total time spent browsing the page, and the current user's expression tag;

[0186] Step S707, determining the user's gaze dwelling time in the current browsing behavior according to the consistency of the gaze direction and the gaze dwelling interval; normalizing the gaze browsing time with the total duration of the browsed page to obtain a normalized gaze browsing time ratio; generating the user's interest score for the current browsing behavior according to the gaze browsing time ratio and in combination with the user's current expression tag;

[0187] Here, the label experience coefficients of 2, 1, and 0.5 can be assigned to the expression labels 1, 0, and -1 respectively, and the product of the label experience coefficient and the ratio of the visual browsing time is determined as the user's interest score for the current browsing behavior. Specifically, the user's interest score for the current browsing behavior can be calculated using the following formula (1-4):

[0188]

[0189] Among them, POI represents the interest score, POI 0 means no interest, λ represents the current expression label coefficient, p i is the ratio of the viewing time, a and b are the viewing direction vectors of the left and right eyes respectively, are the sight direction vectors of the left and right eyes in two image frames with an interval of t, respectively. cos(a,b) is the cosine similarity of the two vectors a and b, which is used to reflect the directional consistency of the two sight direction vectors. Among them, the role of the expression label coefficient is to encourage the browsing behavior of users with positive feedback (smile) (the corresponding sight browsing time ratio multiplied by the expression label coefficient 2), punish the browsing behavior of users with negative feedback (frown) (the corresponding sight browsing time ratio multiplied by the expression label coefficient 0.5), and do not adjust the browsing behavior of users with no obvious feedback (neutral) (the corresponding sight browsing time ratio multiplied by the expression label coefficient 1).

[0190] p i It can be calculated according to the following formula (1-5):

[0191]

[0192] Where n is the number of browsing behaviors (such as viewing behaviors, interactions between eyes and targets, etc.); t i is the duration of the sight stay of the ith browsing behavior (such as viewing time or interaction time); U Eye is the set of user eye browsing behaviors. For example, user browsing behaviors include: browsing up, down, left, and right targets, then n = 4, t i is the viewing time or interaction time of the ith behavior (such as browsing the left target).

[0193] The cosine similarity can be calculated using the following formula (1-6):

[0194]

[0195] Step S708: adding the generated interest points to the interest points list.

[0196] 3) Server retrieval recommendation module.

[0197] This module includes image database feature indexing and image two-stage retrieval.

[0198] In the image database feature indexing stage, to create an index for images in the image database, you can first extract features from the images in the image database, use the extracted image features to describe the images, then format the image features, and finally create an index for the formatted image features. CV models and NLP models can be used to extract features from images or texts, respectively.

[0199] Fig. 8A A data flow diagram of using a CV model 410 to extract image features from an image is provided in an embodiment of the present application, such as Fig. 8A As shown in the figure, the image to be processed is input into the CV model. After being processed by the model, the image path img_path, image category cls(text), deep feature deep_feature, and hash feature hash_feature output by the model can be obtained. A deep convolutional neural network can be used in the CV model. The local visual representation can be learned through shallow calculations, while the semantic information suitable for semantic recognition can be captured through deep calculations.

[0200] Figure 8B A data flow diagram of using NLP model 420 to extract text features from text is provided in an embodiment of the present application, such as Figure 8B As shown in the figure, the text to be processed is input into the NLP model. After being processed by the model, the text feature text_feature output by the model can be obtained.

[0201] In the retrieval stage, a two-stage retrieval method is adopted - a coarse-to-fine search strategy, including a coarse-grained search stage and a fine-grained search stage, such as Fig. 9A As shown, the coarse-grained search stage can be implemented through a candidate set determination model 910 based on an image database 940 , and the coarse-grained search stage can be implemented through a re-ranking model 920 based on an index library 930 and an image database 940 .

[0202] The coarse-grained search stage uses a candidate set determination model, which recalls candidate results related to the input image or text through coarse-grained recall. It can use feature matching recall or input semantic expression matching. Here, the hash code of the mapping layer H between F7 and F8 in the deep hash network is used to calculate the Hamming distance between features or semantic expressions, and the similarity between features or semantic expressions is obtained. Finally, the candidate results with the top k similarity ranking are obtained, where k>0 is a preset value.

[0203] The re-ranking model is used in the fine-grained search stage. The purpose of the re-ranking model is to re-rank the candidate results in the candidate set. It can use deep features, local features, etc. for matching. Here, it is associated with the deep features of the image index library. By calculating the cosine similarity of the deep features, the similarity between each candidate result in the candidate set and the input image or text is obtained, and the candidate results are re-ranked according to the similarity, and the ranked retrieval results are output.

[0204] When the user enters text or images for query, the coarse-grained search maximizes the recall of images related to the input image or text to ensure the recall rate; the fine-grained search returns the top N retrieval results after reordering the retrieval results to ensure that the returned retrieval results are similar or related to the image results, where 0 <N<k。

[0205] After the search results are returned, if the user clicks to view and does not click to browse again, and closes the current search display interface or does not enter a search again, the search results are considered to meet the user's expectations; if the user frequently clicks to view, returns to the current search result display interface, or frequently interacts, the search results are considered to not meet the user's expectations. If the search results meet the user's expectations, the current search session ends; otherwise, the search results will be ranked for re-search recommendations in combination with the user interest collection feedback module. Here,

[0206] The specific process of re-searching and recommending the search results is as follows: Fig. 9BAs shown, the user interest point feedback coefficient is fed back to the re-ranking model 920 through the user interest collection feedback module 950, and the re-ranking model 920 re-ranks the current retrieval result in combination with the retrieval feature, the current retrieval result and the user interest point feedback coefficient, and returns the image retrieval recommendation result combining the user interest point feedback and the historical ranking. Here, the historical ranking is the previous ranking at the time of the current re-ranking.

[0207] The specific retrieval recommendation ranking strategy can consider the re-ranking results of the current retrieval features and the user interest point feedback coefficient. The retrieval features extracted from the input text or graphic and the top N re-ranked image features x i As input, the interest score table corresponding to the user's interest point feedback coefficient is used as the initialization parameter and sent to the regression model shown in the following formula (2-1) for fitting to obtain a new ranking matching feature:

[0208]

[0209] Among them, x input The retrieval feature corresponding to the input text or graphic, w i The interest score of the i-th reordered image in the interest score list, x i is the i-th reordered image feature, and f(x) is the new ranking matching feature. From the above formula (2-1), it can be seen that the new ranking matching feature is obtained by the following process: after weighting the image feature vector by the interest score, a weighted image feature vector is obtained; the weighted image feature vector is averaged with the input retrieval feature (summed and multiplied by a coefficient of 0.5) to obtain a new ranking matching feature.

[0210] Finally, the new ranking matching features are used to re-rank the current search results, and image retrieval recommendation results that combine user interest point feedback and historical ranking can be obtained.

[0211] During the entire search process, the output results can be in the following three situations:

[0212] 1) Search based solely on input, and the returned results are pure search results;

[0213] 2) Combine the user's interest point feedback and the adjusted historical ranking results to perform retrieval and return the retrieval recommendation results;

[0214] 3) When the user inputs too little information, the accuracy of the search results cannot be guaranteed, so recommended results will be added to the search results, such as recent popular search results, historical valid search results, etc.

[0215] Compared with the solutions in the related art, the embodiments provided in this application have the following advantages:

[0216] 1) Obtaining the user's facial expression label and gaze direction information through face detection, and generating a quantitative coefficient suitable for the user's current interest, can effectively tap the user's potential search intention and solve the problem of sparse or missing user dimension information during retrieval in related technologies;

[0217] 2) Based on the user's current facial expression features and sight direction, the user's current interest point feedback coefficient is generated, which can provide an effective user interest point quantification method and device;

[0218] 3) Incorporating shallow features and high-level semantic features into the image retrieval feature expression to obtain the semantic features and visual feature expressions of the image, thereby improving the accuracy of image retrieval and ensuring the universality of image retrieval. Universality is reflected in two aspects: (1) Input. The embodiment of the present application supports inputting text, images, and images + text descriptions for retrieval; (2) User retrieval results, which are mainly reflected in feature descriptions. In the related art, a single feature description is used for retrieval. For example, when searching with "a curved blue ribbon", the results will return crawling snakes (similar in shape), blue stripes (similar in color), etc. In the embodiment of the present application, shallow features (such as shape, color, texture, etc.) and high-level semantic features are used to describe objects. The irrelevant return results of the aforementioned search using "a curved blue ribbon" will be effectively filtered out.

[0219] 4) Image retrieval adopts a two-stage retrieval, recalling candidate results related to the input image or text through coarse-grained retrieval, re-ranking the candidate results through fine-grained retrieval, and continuously adjusting the ranking strategy based on user interest feedback information to provide personalized retrieval recommendation results for individual users;

[0220] 5) Based on the diversity of user input, it supports receiving user image, text or image-text input, returns image retrieval results, and combines the advantages of computer vision and natural language processing to represent image semantics through text description and image feature extraction. It can effectively overcome the "semantic gap" problem of image semantic representation, which is conducive to improving user experience and providing results that are more in line with user expectations for image retrieval.

[0221] Based on the foregoing embodiments, an embodiment of the present application provides a retrieval device, which includes the units included and the modules included in the units, and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0222] Fig.10 Schematic diagram of the structure of the retrieval device in the embodiment of the present application. Fig.10As shown, the search device 1000 includes: a search module 1010, a first determination module 1020, a second determination module 1030, a generation module 1040 and a reordering module 1050, wherein:

[0223] A search module 1010 is used to search the search input information based on the target database to obtain a first search result;

[0224] A first determination module 1020, used to determine a list of browsing behaviors of the user on the first search result;

[0225] The second determining module 1030 is used to determine the expression tag corresponding to each browsing behavior in the browsing behavior list;

[0226] A generating module 1040, configured to generate a user's interest point feedback coefficient according to the browsing behavior list and each of the expression tags;

[0227] The reordering module 1050 is used to reorder the first search result according to the interest point feedback coefficient to obtain a second search result.

[0228] In some embodiments, the target database is an image database. Correspondingly, the retrieval module also includes a feature extraction module, a feature matching module, and a query module, wherein:

[0229] A feature extraction module, used to extract features from the search input information to obtain search features of the search input information;

[0230] A feature matching module, used to perform feature matching on each image feature in an index library using the retrieval feature to obtain a first target index set; the index library includes the image feature of each image in the image database;

[0231] The query module is used to query the image database using the first target index set to obtain a first search result.

[0232] In some embodiments, the retrieval feature includes a recall matching feature and a sort matching feature. Correspondingly, the feature matching module is further used to: use the recall matching feature to perform feature matching on each image feature in the index library to determine a candidate index set; for each index in the candidate index set, use the sort matching feature to perform feature matching on the image feature corresponding to the index to obtain the matching degree of the index; and select the target number of indexes from the candidate index set in descending order according to the matching degree to obtain the first target index set.

[0233] In some embodiments, the reordering module 1050 is also used to: take the retrieval features and the image features corresponding to each index in the first target index set as input, take the interest point feedback coefficient as a model parameter, use a specific regression model to perform feature fitting, and obtain an updated sorting matching feature; for each index in the first target index set, use the updated sorting matching feature to perform feature matching on the image features corresponding to the index, and update the matching degree of the index; reorder the indexes in the first target index set in descending order of matching degree to obtain a second target index set; use the second target index set to query the image database to obtain a second retrieval result.

[0234] In some embodiments, the retrieval input information includes an input image and an input text. Correspondingly, the feature extraction module is further used to: use a CV model to extract image features from the input image to obtain a first feature vector; use an NLP model to extract features from the input text to obtain a second feature vector; use a specific fusion model to perform feature fusion and compression mapping on the first feature vector and the second feature vector to obtain a retrieval feature.

[0235] In some embodiments, the first determination module 1020 is further used to: perform face detection on the collected user face image to obtain the user's line of sight direction; determine the user's browsing behavior list for the first search result based on the line of sight direction and the user's interaction record on the current search page. The second determination module 1030 is further used to determine the user's facial features corresponding to each browsing behavior in the browsing behavior list through face detection; determine the expression label corresponding to each browsing behavior based on the user's facial features corresponding to each browsing behavior.

[0236] In some embodiments, the first determination module 1020 is further used to: determine the user's eyebrow distance and mouth features according to the user's facial features; and determine the expression tag corresponding to the browsing behavior according to the eyebrow distance and the mouth features.

[0237] In some embodiments, the mouth feature includes the distance between the upper and lower lips or the aspect ratio of the circumscribed rectangle of the mouth, and the expression tags include a first expression tag, a second expression tag, and a third expression tag. Correspondingly, the first determination module 1020 is also used to: when the distance between the upper and lower lips and the aspect ratio of the circumscribed rectangle of the mouth meet the first condition, determine that the browsing behavior corresponds to the first expression tag; the first condition includes: the distance between the upper and lower lips is greater than a specific lip distance threshold, or the aspect ratio of the circumscribed rectangle of the mouth is less than a specific aspect ratio coefficient threshold; when the eyebrow distance meets the first condition, determine that the browsing behavior corresponds to the second expression tag; the first condition includes that the eyebrow distance is less than a specific eyebrow distance threshold; when the distance between the upper and lower lips and the aspect ratio of the circumscribed rectangle of the mouth do not meet the first condition, and the eyebrow distance does not meet the second condition, determine that the browsing behavior corresponds to the third expression tag.

[0238] In some embodiments, the interest point feedback coefficient includes the user's interest score for each result item in the first search result, and the browsing behavior includes the browsed result items, interaction time, line of sight browsing time, and the dwell time of the result item. Correspondingly, the generation module 1040 is also used to: for each browsing behavior in the browsing behavior list, when the interaction time is greater than 0, determine a first ratio; the first ratio is the ratio of the interaction time to the dwell time of the result item; determine the interest score of the result item according to the first ratio; when the interaction time is equal to 0, determine a second ratio; the second ratio is the ratio of the line of sight browsing time to the dwell time of the result item; determine the interest score of the result item according to the second ratio and the expression tag of the browsing behavior.

[0239] In some embodiments, the first determination module 1020 is also used to determine the user's browsing behavior list for the second retrieval result; the generation module 1040 is also used to update the user's interest point feedback coefficient based on the browsing behavior list and each of the emoticon tags; the reordering module 1050 is also used to reorder the second retrieval result according to the updated interest point feedback coefficient to obtain a third retrieval result.

[0240] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the retrieval device of this application, please refer to the description of the method embodiment of this application for understanding.

[0241] It should be noted that in the embodiment of the present application, if the above-mentioned retrieval method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0242] Correspondingly, an embodiment of the present application provides a retrieval device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and is characterized in that the processor implements the steps in the above method when executing the program.

[0243] Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, characterized in that the computer program implements the steps in the above method when executed by a processor.

[0244] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0245] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0246] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0247] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0248] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0249] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0250] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0251] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0252] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A search method, characterized in that: The method comprises: Based on the target database, searching the search input information to obtain a first search result; Determining a list of browsing behaviors of the user on the first search result; Determine an expression tag corresponding to each browsing behavior in the browsing behavior list; Generating a user's interest point feedback coefficient according to the browsing behavior list and each of the expression tags; reordering the first search results according to the interest point feedback coefficient to obtain a second search result; The interest point feedback coefficient includes the user's interest score for each result item in the first search result; the browsing behavior includes the browsed result items, interaction time, sight browsing time and result item dwell time; The step of generating a user's interest point feedback coefficient according to the browsing behavior list and each of the expression tags includes: For each browsing behavior in the browsing behavior list, when the interaction time is greater than 0, a first ratio is determined; the first ratio is the ratio of the interaction time to the dwell time of the result item; determining an interest score of the result item according to the first ratio; When the interaction time is equal to 0, a second ratio is determined; the second ratio is the ratio of the visual browsing time to the dwell time of the result entry; An interest score of the result item is determined according to the second ratio and the expression tag of the browsing behavior.

2. The method according to claim 1, characterized in that The target database is an image database; the step of searching the search input information based on the target database to obtain a first search result includes: Extracting features from the search input information to obtain search features of the search input information; Using the retrieval feature, feature matching is performed on each image feature in the index library to obtain a first target index set; the index library includes the image feature of each image in the image database; The image database is queried using the first target index set to obtain a first retrieval result.

3. The method according to claim 2, characterized in that The retrieval features include recall matching features and sort matching features; The method of using the retrieval feature to perform feature matching on each image feature in the index library to obtain a first target index set includes: Using the recalled matching features, feature matching is performed on each image feature in the index library to determine a candidate index set; For each index in the candidate index set, using the sorting matching feature, perform feature matching on the image feature corresponding to the index to obtain the matching degree of the index; The indexes of the target number are selected from the candidate index set in descending order according to the matching degree to obtain the first target index set.

4. The method according to claim 3, characterized in that: The step of reordering the first search results according to the interest point feedback coefficient to obtain the second search results includes: Taking the retrieval feature and the image feature corresponding to each index in the first target index set as input, taking the interest point feedback coefficient as a model parameter, and using a regression model to perform feature fitting to obtain an updated ranking matching feature; For each index in the first target index set, using the updated sorted matching feature, perform feature matching on the image feature corresponding to the index, and update the matching degree of the index; Reorder the indexes in the first target index set in descending order of matching degree to obtain a second target index set; The image database is queried using the second target index set to obtain a second retrieval result.

5. The method according to claim 2, characterized in that: The search input information includes an input image and an input text; and extracting features from the search input information to obtain search features of the search input information includes: Using a computer vision model to extract image features from the input image to obtain a first feature vector; Using a natural language processing model to extract features from the input text to obtain a second feature vector; A fusion model is adopted to perform feature fusion and compression mapping on the first feature vector and the second feature vector to obtain a retrieval feature.

6. The method according to claim 1, characterized in that The determining of the browsing behavior list of the user for the first search result includes: Perform face detection on the collected user face images to obtain the user's line of sight direction; Determining a browsing behavior list of the user for the first search result according to the sight direction and the user's interaction record on the current search page; The step of determining the expression tag corresponding to each browsing behavior in the browsing behavior list includes: Determine the user's facial features corresponding to each browsing behavior in the browsing behavior list through face detection; According to the user facial feature corresponding to each browsing behavior, an expression tag corresponding to the browsing behavior is determined.

7. The method according to claim 6, characterized in that The step of determining the expression tag corresponding to each browsing behavior according to the user facial feature corresponding to each browsing behavior includes: Determine the user's eyebrow distance and mouth features according to the user's facial features; An expression label corresponding to the browsing behavior is determined according to the eyebrow distance and the mouth feature.

8. The method according to claim 7, characterized in that The mouth feature includes the distance between the upper and lower lips or the length-to-width ratio of the circumscribed rectangle of the mouth; the expression tags include a first expression tag, a second expression tag and a third expression tag; Correspondingly, determining the expression tag corresponding to the browsing behavior according to the eyebrow distance and the mouth feature includes: When the distance between the upper and lower lips and the aspect ratio of the circumscribed rectangle of the mouth meet a first condition, determining that the browsing behavior corresponds to a first expression tag; the first condition includes: the distance between the upper and lower lips is greater than a lip distance threshold, or the aspect ratio of the circumscribed rectangle of the mouth is less than an aspect ratio coefficient threshold; When the eyebrow distance satisfies a first condition, determining that the browsing behavior corresponds to a second expression tag; the first condition includes that the eyebrow distance is less than an eyebrow distance threshold; When the distance between the upper and lower lips and the aspect ratio of the circumscribed rectangle of the mouth do not satisfy the first condition, and the eyebrow distance does not satisfy the second condition, it is determined that the browsing behavior corresponds to a third expression tag.

9. The method according to claim 1, characterized in that: The method further comprises: Determining a list of browsing behaviors of the user on the second search result; Determine an expression tag corresponding to each browsing behavior in the browsing behavior list; According to the browsing behavior list and each of the expression tags, updating the user's interest point feedback coefficient; The second search results are reordered according to the updated interest point feedback coefficient to obtain a third search result.

10. A search device, characterized in that: include: A search module, used to search the search input information based on the target database to obtain a first search result; A first determining module, used to determine a list of browsing behaviors of the user on the first search result; A second determination module is used to determine the expression tag corresponding to each browsing behavior in the browsing behavior list; A generating module, used for generating a user's interest point feedback coefficient according to the browsing behavior list and each of the expression tags; A reordering module, used for reordering the first search result according to the interest point feedback coefficient to obtain a second search result; The interest point feedback coefficient includes the user's interest score for each result item in the first search result; the browsing behavior includes the browsed result items, interaction time, sight browsing time and result item dwell time; The generation module is also used for: For each browsing behavior in the browsing behavior list, when the interaction time is greater than 0, a first ratio is determined; the first ratio is the ratio of the interaction time to the dwell time of the result item; determining an interest score of the result item according to the first ratio; When the interaction time is equal to 0, a second ratio is determined; the second ratio is the ratio of the visual browsing time to the dwell time of the result entry; An interest score of the result item is determined according to the second ratio and the expression tag of the browsing behavior.

11. A retrieval device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps in the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 9 are implemented.

Images