Product recommendation method and device for multi-modal data, equipment and storage medium
By classifying and filtering the images and text information of target products using keywords, the problem of excessively long retrieval times caused by large-scale search data is solved, and efficient and accurate product recommendations are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN BANK CO LTD
- Filing Date
- 2022-06-22
- Publication Date
- 2026-07-03
AI Technical Summary
When searching large amounts of data, existing intelligent recommendation technologies take too long to retrieve data, resulting in performance degradation.
By acquiring image and text information of the target product, extracting and classifying keywords, and using user preference information to rank and filter keywords, the first and second target products can be determined, thereby reducing the amount of search data.
It improves search efficiency, ensures that search results match user intent, and enhances search accuracy and performance.
Smart Images
Figure CN115186091B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent recommendation technology, specifically to a product recommendation method, apparatus, device, and storage medium based on multimodal data. Background Technology
[0002] In the current technology field, intelligent recommendation technology has been widely used in various industries. For example, banks often need to recommend various products to users, such as credit cards, wealth management, funds, mobile phones, and computers.
[0003] In related technologies, intelligent recommendation typically uses search data to recommend products. However, when the amount of search data is too large, it can lead to excessively long retrieval times and reduce the performance of intelligent recommendation. Summary of the Invention
[0004] This application provides a product recommendation method, apparatus, device, and storage medium based on multimodal data. By combining user preference information to perform a partial retrieval of data, and then performing a second retrieval based on the click-through rate of the target product displayed after the retrieval, the amount of data retrieved is reduced, thereby improving retrieval efficiency.
[0005] Firstly, this application provides a product recommendation method based on multimodal data, including:
[0006] Obtain image and text information of the target product;
[0007] Extract keywords from the image information and keywords from the text information, and classify the keywords into product name keywords and product condition keywords according to their parts of speech;
[0008] The product name keywords are divided into first product name keywords and second product name keywords based on preset preferred product name words;
[0009] The first target product is determined based on the first product name keywords and the product condition keywords;
[0010] If the click-through rate of the first target product is detected to be less than the preset click-through rate threshold, then the second target product is determined based on the second product name keywords and the product condition keywords.
[0011] In one possible implementation of this application, obtaining the image information and text information of the target product includes:
[0012] Collect the location of the touch point;
[0013] If the touch point location corresponds to the product search area, then the information collection window of the target product is fed back, and the information collection window includes a text information collection area and an image information collection area;
[0014] When an information collection completion signal is detected, the image and text information of the target product collected by the information collection window are acquired, and the information collection window is closed.
[0015] In one possible implementation of this application, the step of dividing the product name keywords into first product name keywords and second product name keywords based on preset preferred product name terms includes:
[0016] Calculate the similarity between the preset preferred product name words and the product name keywords;
[0017] If the similarity is greater than the preset similarity threshold, it is considered the first product name keyword; if the similarity is less than or equal to the preset similarity threshold, it is considered the second product name keyword.
[0018] In one possible implementation of this application, determining the first target product based on the first product name keywords and the product condition keywords includes:
[0019] Calculate the relevance between the first product name keywords and the product condition keywords;
[0020] Obtain target product condition keywords with a relevance higher than a preset relevance threshold, and perform a product search in a preset product database according to the first product name keywords corresponding to the target product condition keywords to obtain several first target products;
[0021] Generate a graphic and text column that corresponds one-to-one with the first target product based on the product information of each of the first target products;
[0022] Based on the image and text bar, a corresponding image and text bar display page for the first target product is generated.
[0023] In one possible implementation of this application, after the step of generating the graphic display page corresponding to the first target product based on the graphic bar, the following steps are included:
[0024] The image and text bar display page is displayed in the display area;
[0025] Count the number of clicks on the graphic and text bar and the number of graphic and text bars displayed in the display area;
[0026] The click-through rate of the first target product is calculated based on the number of clicks on the image and text bar and the number of images and text bars displayed.
[0027] In one possible implementation of this application, determining the second target product based on the second product name keywords and the product condition keywords includes:
[0028] Based on the second product name keywords and the product condition keywords, a product search is performed in a preset product database to obtain several second target products;
[0029] Generate a graphic and text display page for the second target product based on the product information of the second target product;
[0030] The pop-up window displays the image and text display page of the second target product.
[0031] In one possible implementation of this application, after the step of displaying the graphic and text display page of the second target product based on the pop-up window, the method further includes:
[0032] If any request to display the details page of the second target product is detected in the pop-up window, the details page of the second target product is obtained;
[0033] The pop-up window is used to display the details page of the second target product, and the pop-up window is set as the main page, covering the entire display area.
[0034] Secondly, this application provides a product recommendation device based on multimodal data, the product recommendation device based on multimodal data comprising:
[0035] Acquisition module: Used to acquire image and text information of the target product;
[0036] Extraction module: used to extract keywords from the image information and keywords from the text information, and to classify the keywords into product name keywords and product condition keywords according to their parts of speech;
[0037] Classification module: used to classify the product name keywords into first product name keywords and second product name keywords based on preset preferred product name terms;
[0038] First determination module: used to determine the first target product based on the first product name keywords and the product condition keywords;
[0039] The second determining module is used to determine the second target product based on the second product name keywords and the product condition keywords if the click-through rate of the first target product is detected to be less than a preset click-through rate threshold.
[0040] Thirdly, this application provides a product recommendation device based on multimodal data, the product recommendation device based on multimodal data comprising:
[0041] One or more processors;
[0042] Memory; and
[0043] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the product recommendation method for multimodal data as described in any one of them.
[0044] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps in the product recommendation method for multimodal data as described in any one of the claims.
[0045] This application provides a product recommendation method, apparatus, device, and storage medium based on multimodal data. The method involves acquiring image and text information of a target product; extracting keywords from the image and text information; classifying the keywords into product name keywords and product condition keywords based on their parts of speech; further classifying the product name keywords into first product name keywords and second product name keywords based on preset preferred product name keywords; determining a first target product based on the first product name keywords and the product condition keywords; and determining a second target product based on the second product name keywords and the product condition keywords if the click-through rate of the first target product is less than a preset click-through rate threshold. By extracting keywords from the collected data, the system identifies the target product. It then categorizes the extracted product name keywords based on user preferences, classifying them into primary and secondary product name keywords. The system further determines the primary product based on the primary product name keywords and their associated product condition keywords. This reduces the number of product name keywords retrieved, thereby reducing the amount of data retrieved and the processing load. Furthermore, categorizing the product name keywords based on user preferences ensures that the retrieved primary target product better aligns with user preferences. When user click-through rates are low, the system identifies the secondary target product based on the categorized secondary product name keywords and the product condition keywords, guaranteeing the accuracy of the search results and improving search efficiency. Attached Figure Description
[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0047] Figure 1 This is a schematic diagram illustrating a scenario of the product recommendation method based on multimodal data provided in an embodiment of this application.
[0048] Figure 2This is a schematic flowchart of an embodiment of the product recommendation method based on multimodal data provided in this application.
[0049] Figure 3 This is a schematic flowchart of an embodiment of the product recommendation method for multimodal data provided in this application, which obtains image and text information of a target product.
[0050] Figure 4 This is a schematic flowchart of an embodiment of the product recommendation method for multimodal data provided in this application, which involves determining the first product name keyword and the second product name keyword.
[0051] Figure 5 This is a schematic flowchart of an embodiment of the product recommendation method for multimodal data provided in this application for determining the first target product;
[0052] Figure 6 This is a schematic flowchart of an embodiment of the product recommendation method for multimodal data provided in this application for determining the second target product;
[0053] Figure 7 This is a schematic diagram of an embodiment of the product recommendation device for multimodal data provided in this application.
[0054] Figure 8 This is a schematic diagram of an embodiment of the product recommendation device for multimodal data provided in this application. Detailed Implementation
[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0056] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.
[0057] In this application, the term "exemplary" is used to mean "serving as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.
[0058] This application provides a product recommendation method, apparatus, device, and computer-readable storage medium for multimodal data, which will be described in detail below.
[0059] The product recommendation method for multimodal data in this embodiment of the invention is applied to a product recommendation device for multimodal data. The product recommendation device for multimodal data is set in a product recommendation equipment for multimodal data. The product recommendation equipment for multimodal data includes one or more processors, a memory, and one or more applications. The one or more applications are stored in the memory and configured to be executed by the processor to implement the product recommendation method for multimodal data. The product recommendation equipment for multimodal data can be a terminal, such as a mobile phone or a tablet computer. The product recommendation equipment for multimodal data can also be a server or a service cluster composed of multiple servers.
[0060] like Figure 1 As shown, Figure 1This is a schematic diagram of a scenario for a product recommendation method using multimodal data according to an embodiment of this application. The product recommendation scenario using multimodal data in this embodiment includes a product recommendation device 100 (the product recommendation device 100 integrates a product recommendation apparatus for multimodal data). The product recommendation device 100 uses a computer-readable storage medium to run the product recommendation process for multimodal data, thereby executing the steps of product recommendation using multimodal data.
[0061] Understandable, Figure 1 The multimodal data product recommendation device in the scenario of the multimodal data product recommendation method, or the device included in the multimodal data product recommendation device, does not constitute a limitation on the embodiments of the present invention. That is, the number or type of device included in the scenario of the multimodal data product recommendation method, or the number or type of device included in each device, does not affect the overall implementation of the technical solution in the embodiments of the present invention, and can all be considered as equivalent substitutions or derivatives of the technical solutions claimed in the embodiments of the present invention.
[0062] In this embodiment of the invention, the product recommendation device 100 based on multimodal data is mainly used for: acquiring image information and text information of target products; extracting keywords from the image information and the text information, and classifying the keywords into product name keywords and product condition keywords according to their parts of speech; classifying the product name keywords into first product name keywords and second product name keywords according to preset preferred product name keywords; determining a first target product based on the first product name keywords and the product condition keywords; and determining a second target product based on the second product name keywords and the product condition keywords if the click-through rate of the first target product is detected to be less than a preset click-through rate threshold.
[0063] In this embodiment of the invention, the multimodal data product recommendation device 100 can be an independent multimodal data product recommendation device, or it can be a network or cluster of multimodal data product recommendation devices. For example, the multimodal data product recommendation device 100 described in this embodiment includes, but is not limited to, computers, network hosts, a single network multimodal data product recommendation device, a set of multiple network multimodal data product recommendation devices, or a cloud multimodal data product recommendation device composed of multiple multimodal data product recommendation devices. The cloud multimodal data product recommendation device is composed of a large number of computer or network multimodal data product recommendation devices based on cloud computing.
[0064] Those skilled in the art will understand that Figure 1The application environment shown is merely one application scenario of the solution in this application and does not constitute a limitation on the application scenario of the solution in this application. Other application environments may include those that are more specific to this application. Figure 1 The product recommendation device shown has more or less multimodal data, or the network connection relationship of the product recommendation device with multimodal data, for example... Figure 1 Only one multimodal data product recommendation device is shown in the diagram. It is understood that the scenario of this multimodal data product recommendation method may also include one or more other multimodal data product recommendation devices, which are not specifically limited here. The multimodal data product recommendation device 100 may also include a memory for storing data, such as preset calculation models, user information, etc.
[0065] Furthermore, in the scenario of the product recommendation method using multimodal data in this application, the product recommendation device 100 can be equipped with a display device, or the product recommendation device 100 can be connected to an external display device 200 without a display device. The display device 200 is used to output the results of the product recommendation method executed in the multimodal data product recommendation device. The product recommendation device 100 can access a background database 300 (the background database can be located in the local storage of the product recommendation device, or it can be located in the cloud). The background database 300 stores information related to product recommendation using multimodal data, such as user preference information and user behavior information.
[0066] It should be noted that, Figure 1 The schematic diagram of the product recommendation method using multimodal data shown is merely an example. The scenarios of the product recommendation method using multimodal data described in this embodiment are intended to more clearly illustrate the technical solutions of this embodiment and do not constitute a limitation on the technical solutions provided in this embodiment.
[0067] Based on the above-mentioned scenario of product recommendation methods using multimodal data, an embodiment of a product recommendation method using multimodal data is proposed.
[0068] See Figure 2 This application provides an implementation scheme for a product recommendation method based on multimodal data, including steps S201-S205.
[0069] S201. Obtain image and text information of the target product.
[0070] The image information and location information are the original retrieval data used to retrieve the target product. It is understood that the image information may include several images, that is, in some embodiments of this application, the image information may include one image, two images, three images, etc.; the text information may be one or more words, or a sentence, and this application does not make specific limitations.
[0071] Specifically, in the embodiments of this application, the product recommendation method based on multimodal data is applied to a product recommendation device based on multimodal data, and the product recommendation device based on multimodal data integrates a product recommendation apparatus based on multimodal data. The product recommendation apparatus based on multimodal data can be a mobile phone, computer, etc. The product recommendation apparatus based on multimodal data collects image information and text information through data acquisition equipment to obtain image information and text information of the target product.
[0072] For details, see Figure 3 In one embodiment of this application, obtaining the image information and text information of the target product specifically includes the following steps: S301-S303:
[0073] S301, Collect the location of the touch point.
[0074] The touch point location refers to the touch point location collected by the touchscreen display area. This touchscreen display area can be located on the product recommendation device for the multimodal data, or on an independent display device signal-connected to the product recommendation device for the multimodal data. Furthermore, when a user operates the touchscreen display area, such as by clicking or long-pressing, a corresponding touch signal is generated. The display content corresponding to the touch point is then processed based on this touch signal, such as entering a details page. Specifically, when the touchscreen receives a user operation, it detects a pressure signal and converts it into an electrical signal to determine the screen touch point location. Then, the touch point location is determined based on the correspondence between the screen resolution and the image resolution of the display area.
[0075] S302. If the touch point location corresponds to the product search area, then the information collection window of the target product is fed back. The information collection window includes a text information collection area and an image information collection area.
[0076] The product search area is one of the modules displayed in the display area. It can be understood that the multimodal data product recommendation method can utilize the product recommendation module in a software system, such as in a shopping mall in a bank's APP. When the display area shows the main page of the shopping mall, the main page may include brand areas, product search areas, fund areas, etc. When the multimodal data product recommendation device receives the touch point position of the corresponding product search area, it starts the multimodal data product recommendation method and then provides feedback to the information collection window of the target product.
[0077] It is understood that the information collection window allows users to collect image information and text information. That is, the information collection window includes a text information collection area and an image information collection area. It is understood that users can provide text information through the text information collection area and image information through the image information collection area.
[0078] S303. When an information acquisition completion signal is detected, the image and text information of the target product acquired by the information acquisition window are obtained, and the information acquisition window is closed.
[0079] The information acquisition completion signal can be fed back through the information acquisition contact in the information acquisition window, or it can be fed back through a preset acquisition time, with the timer starting when the information acquisition window pops up, and the information acquisition completion signal being fed back when the preset time is reached.
[0080] Specifically, when the multimodal data product recommendation device receives the information collection completion signal, it acquires the image and text information based on the information collection window through the signal transmission path and issues an information collection window closing command. Based on this command, the information collection window is closed. It can be understood that the opening and closing of the information collection window can be controlled by the pop-up component of the display system.
[0081] S202. Extract keywords from the image information and keywords from the text information, and classify the keywords into product name keywords and product condition keywords according to their parts of speech.
[0082] The keywords in the image information refer to the semantic keywords contained in the image. For example, if the image information includes an image containing watermelon and grapefruit, then watermelon and grapefruit can be identified as keywords in the image information through keyword extraction. If the image includes an ice pop with mung beans printed on it, then the keywords are mung bean flavor and ice pop. It can be understood that the keyword extraction of the image information can be obtained by a keyword extraction model trained by inputting the image information into the keyword extraction model to extract the keywords of the image information.
[0083] Furthermore, it is understood that the text information can be a word or a sentence. For example, the text information could be a sentence: "a black puppy." That is, keyword extraction from the text information can yield the keywords: "black," "puppy," and "dog." It is understood that keyword extraction can be performed using a trained keyword extraction model. The text information is input into the keyword extraction model to extract the keywords. It is understood that the keyword extraction model can be the same, used to extract keywords from both text and image information, or different keyword extraction models can be used for text and image information respectively.
[0084] The keywords, including adjectives, nouns, and adverbs, are used to describe and restrict product name keywords. Essentially, by defining nouns as product name keywords and non-nouns as product condition keywords, and using the example above, "dog" is a product name keyword, while "small" and "black" are product condition keywords. This means that the product condition keywords "small" and "black" restrict and describe the product name keyword "dog," thus limiting the search scope. It is understood that the descriptions of keywords extracted from text and images are merely illustrative. In practice, there is generally a correlation between the keywords in the text and images; for example, the keywords in the text might be product condition keywords, and the keywords in the images might be product name keywords. This application does not impose specific limitations on this.
[0085] S203. Divide the product name keywords into first product name keywords and second product name keywords according to preset preferred product name words.
[0086] The preset preferred product name can be set by the user or determined by analyzing the user's historical behavior data, and the preset preferred product name is stored in the storage structure corresponding to the product recommendation device with multimodal data.
[0087] For details, see Figure 4 In the product recommendation method based on multimodal data provided in this application, the determination of the first product name keyword and the second product name keyword specifically includes steps S401-S402:
[0088] S401. Calculate the similarity between the preset preferred product name words and the product name keywords.
[0089] S402. If the similarity is greater than the preset similarity threshold, it is the keyword of the first product name; if the similarity is less than or equal to the preset similarity threshold, it is the keyword of the second product name.
[0090] In this embodiment, after determining the product name keywords, the multimodal data product recommendation device acquires user information corresponding to the source of the image information and the text information. It is understood that the source of the image information and the text information can be determined through the device model of the information collection device or the login user information corresponding to the information collection, and the preset preferred product name words in the user information are determined. Semantic matching is performed between the product name keywords and the preset preferred product name words to find semantically similar preset preferred product name words and product name keywords, and the similarity between semantically similar preset preferred product name words and product name keywords is calculated. In a more specific embodiment, the preset preferred product name words and the product name keywords can be input into a trained preset model for similarity calculation, obtaining a similarity value between each preset preferred product name word and each product name keyword. If the similarity is greater than a preset similarity threshold, it is considered a first product name keyword; if the similarity is less than or equal to the preset similarity threshold, it is considered a second product name keyword.
[0091] It is understood that the preset similarity threshold is used to determine the relevance between the product name keywords and the user's preferred product name keywords, and to divide the product name keywords into first product name keywords that are more relevant to the user's preferred products and second product name related words with lower relevance based on the relevance. It is also understood that in the extraction of keywords from image information or text information, product name keywords that are not relevant to the user's search intent may be extracted, resulting in excessive retrieval data and excessive time. Multimodal data product recommendation devices predict the user's intended products by using product name keywords extracted from the image information and text information based on the user's preferred product name keywords. That is, they first determine the first product name keywords that are more relevant to the user's preferred product name keywords and retrieve them first.
[0092] S204. Determine the first target product based on the first product name keywords and the product condition keywords.
[0093] Specifically, after determining the first product name keyword based on user-preferred product name keywords, the multimodal data product recommendation device determines the first target product based on the first product name keyword and product condition keywords obtained based on image and text information. It is understood that, since there is a correlation between image information and the text information and the target product, determining the first target product based on the first product name keyword and product condition keywords obtained based on image and text information can ensure that the obtained target products are not too generic, thereby enhancing the accuracy of target product retrieval.
[0094] For details, see Figure 5 The implementation scheme for determining the first target product provided in the embodiments of this application includes steps S501-S504:
[0095] S501. Calculate the relevance between the first product name keywords and the product condition keywords.
[0096] The multimodal data product recommendation device calculates the relevance between the first product name keyword and the product condition keywords. Specifically, it calculates the relevance between each first product name keyword and every other product name keyword, obtaining the relevance of each first product name keyword to each product condition keyword. Based on this relevance, the product condition keywords corresponding to the first product name keywords are filtered. It is understood that a first product name keyword may not match a particular product condition keyword. A preset relevance threshold is used for filtering to avoid data confusion, increase the accuracy of product recommendations, and improve recommendation performance. It is understood that the preset relevance threshold can be preset according to different application systems. The calculation of the relevance between the first product name keywords and the product condition keywords can also be completed using a pre-trained self-attention network model.
[0097] S502. Obtain target product condition keywords with a relevance higher than a preset relevance threshold, and perform product search in a preset product database according to the first product name keywords corresponding to the target product condition keywords to obtain several first target products.
[0098] In this multimodal data product recommendation device, after obtaining target product condition keywords with a relevance higher than a preset relevance threshold, a word association is formed by the target product condition keywords and the first product name keywords corresponding to the target product keywords. It can be understood that each first product name keyword forms a word association with its corresponding target product condition keyword. That is, multiple word associations can be established according to the data volume of the first product name keywords. Based on the target product condition keywords and the first product name keywords of each word association, a matching search is performed in the preset product database to obtain several first target products.
[0099] S503. Generate a graphic and text column that corresponds one-to-one with the first target product based on the product information of each first target product.
[0100] The product information includes product name, product image, etc. It is understood that the image and text bar includes product image and product name. It is understood that the image and text bar can be drawn based on the display page drawing component to obtain the image and text bar component. Furthermore, it is understood that the product data can be stored in a cloud server communicating with the multimodal data product recommendation device. The multimodal data product recommendation device, after searching for products and obtaining several first target products, can send the product information of the first target products to the display component of the multimodal data product recommendation device. Based on the display component of the multimodal data product recommendation device, an image and text bar component is created. It is understood that the image and text bar can include multiple components, each corresponding to one first target product.
[0101] S504. Generate a graphic display page corresponding to the first target product based on the graphic display page.
[0102] The first target product's graphic and text display page, i.e., the display page of the display area, can be understood as being drawn by the image drawing component of the multimodal data product recommendation device. Specifically, the image drawing component first stitches the graphic and text bars together, then draws the corresponding display page by acquiring the display area size data, and finally displays the drawn graphic and text display page in the display area. It can be understood that the graphic and text display page in the display area can be flipped or scrolled up and down, and the image drawing component draws different graphic and text display pages based on operation commands.
[0103] S205. If the click-through rate of the first target product is detected to be less than the preset click-through rate threshold, then the second target product is determined based on the second product name keywords and the product condition keywords.
[0104] The preset click threshold is a pre-defined value primarily used to assess the matching degree between the recommended first target product and the user's intended target product. It can be understood that the preset click threshold can be obtained through historical data analysis and updated with newer historical data. The click-through rate (CTR) of the first target product refers to the view rate of the image and text sections on the displayed image and text page when the user views the page. It can be understood that if a user clicks on an image or text section, it indicates that the corresponding first target product may be the user's intended product. Conversely, if the CTR of the first target product is detected to be less than the preset CTR threshold, it indicates that the first target product may not be the user's intended product. In this case, the multimodal data product recommendation device performs another product search in the preset product database based on the second product name keywords and the product condition keywords to ensure the accuracy of the recommendation.
[0105] Furthermore, it can be understood that if the click-through rate of the first target product is detected to be greater than or equal to the preset click-through rate threshold, it indicates that the recommendation accuracy of the first target product is high and it matches the user's intended target product well. In this case, the second product name keywords will not be used for further retrieval to reduce the consumption of program resources.
[0106] Specifically, the method for determining the click-through rate includes, after the step of generating the corresponding image and text display page for the first target product based on the image and text bar: counting the number of clicks on the image and text bar and the number of image and text bars displayed in the display area; and calculating the click-through rate of the first target product based on the number of clicks on the image and text bar and the number of image and text bars displayed.
[0107] It is understood that the number of clicks on the graphic bar, i.e., the number of clicks on the graphic bar displayed on the graphic bar display page in the display area, can be determined by the voltage sensing of the touch screen, as described in the above implementation method for determining the touch point position, and will not be repeated here. Furthermore, the number of graphic bar displays, i.e., the number of different graphic bars displayed in the display area during the display process, can be understood to be calculated by statistically analyzing the number of clicks and the number of displays within a preset time period, and then calculating the click-through rate of the first target product based on the number of clicks and the number of displays.
[0108] Furthermore, based on the above implementation plan, see [link to relevant documentation]. Figure 6 In one embodiment of this application, the determination of the second target product includes steps S601-S603:
[0109] S601. Based on the second product name keywords and the product condition keywords, perform a product search in a preset product database to obtain several second target products;
[0110] S602. Generate a graphic and text display page for the second target product based on the product information of the second target product;
[0111] S603. Display the graphic and text display page of the second target product based on the pop-up window.
[0112] In this system, the multimodal data-based product recommendation device, upon detecting that the clicks corresponding to the first target product are below a preset click threshold, searches a preset product database based on the second product name keywords and the product condition keywords to determine the second target product. Specifically, the multimodal data-based product recommendation device calculates the relevance between the second product name keywords and the product condition keywords; that is, it calculates the relevance between every two second product name keywords and every other product name keyword, obtaining the relevance between each pair of second product name keywords and each pair of product condition keywords. Based on this relevance, the device filters the product condition keywords corresponding to the second product name keywords. It is understood that the second product name keywords may not match any of the two product condition keywords. Using a preset relevance threshold for filtering avoids data confusion, increases the accuracy of product recommendations, and improves recommendation performance. It is understood that the preset relevance threshold can be preset according to different application systems. The calculation of the relevance between the second product name keywords and the product condition keywords can also be completed using a pre-trained self-attention network model.
[0113] In this multimodal data product recommendation device, after obtaining target product condition keywords with a relevance higher than a preset relevance threshold, the device forms two-word associations based on the target product condition keywords and the corresponding second product name keywords. This means that every two second product name keywords form a two-word association with their corresponding target product condition keywords. In other words, multiple word associations can be established based on the amount of data related to the second product name keywords. Based on each pair of word associations, the target product condition keywords and second product name keywords are matched and searched in a preset product database to obtain several second target products.
[0114] Furthermore, the multimodal data product recommendation device generates a graphic and text display page for the second target product based on the product information of the second target product, and adjusts the display ratio and displays the graphic and text display page of the second target product based on the pop-up window component of the multimodal data product recommendation device.
[0115] The pop-up component displays the second target product, enhancing the diversity of data display. Furthermore, if any request to display the details page of the second target product within the pop-up is detected, the details page of the second target product is obtained; the details page of the second target product is displayed based on the pop-up, and the pop-up is set as the main page, covering the entire display area. It is understood that the details page display request can be triggered by clicking any text or image bar in the text and image bar display page of the second target product within the pop-up. That is, if a touch point is detected corresponding to any text or image bar of the second target product within the pop-up, the details page of the second target product associated with the text or image bar corresponding to the touch point is obtained, the details page of the second target product is displayed based on the pop-up, and the pop-up is set as the main page, covering the entire display area, facilitating the display of the details page of the second target product. It is understood that after exiting the details page, the pop-up can be set to shrink to its original size for display, or the pop-up size can remain unchanged; this application does not impose specific limitations on this.
[0116] This application provides a product recommendation method based on multimodal data. The method involves acquiring image and text information of a target product; extracting keywords from the image and text information; and classifying the keywords into product name keywords and product condition keywords based on their parts of speech. Then, based on preset preferred product name keywords, the product name keywords are further divided into first product name keywords and second product name keywords. A first target product is determined based on the first product name keywords and the product condition keywords. If the click-through rate of the first target product is detected to be less than a preset click-through rate threshold, a second target product is determined based on the second product name keywords and the product condition keywords. By extracting keywords from the collected data, the system identifies the target product. It then categorizes the extracted product name keywords based on user preferences, classifying them into primary and secondary product name keywords. The system further determines the primary product based on the primary product name keywords and their associated product condition keywords. This reduces the number of product name keywords retrieved, thereby reducing the amount of data retrieved and the processing load. Furthermore, categorizing the product name keywords based on user preferences ensures that the retrieved primary target product better aligns with user preferences. When user click-through rates are low, the system identifies the secondary target product based on the categorized secondary product name keywords and the product condition keywords, guaranteeing the accuracy of the search results and improving search efficiency.
[0117] To better implement the product recommendation method based on multimodal data in the embodiments of this application, this application also provides a product recommendation device based on multimodal data, such as... Figure 7 As shown, Figure 7 This is a schematic diagram of an embodiment of a product recommendation device based on multimodal data. The product recommendation device based on multimodal data includes the following modules 701-705:
[0118] Acquisition module 701: Used to acquire image and text information of the target product;
[0119] Extraction module 702: used to extract keywords from the image information and keywords from the text information, and to classify the keywords into product name keywords and product condition keywords according to their parts of speech;
[0120] Classification module 703: used to classify the product name keywords into first product name keywords and second product name keywords according to preset preferred product name words;
[0121] First determining module 704: used to determine a first target product based on the first product name keywords and the product condition keywords;
[0122] The second determining module 705 is used to determine the second target product based on the second product name keywords and the product condition keywords if the click-through rate of the first target product is detected to be less than a preset click-through rate threshold.
[0123] In some embodiments of this application, the acquisition module 701 is used to acquire image information and text information of the target product, specifically including:
[0124] Collect the location of the touch point;
[0125] If the touch point location corresponds to the product search area, then the information collection window of the target product is fed back, and the information collection window includes a text information collection area and an image information collection area;
[0126] When an information collection completion signal is detected, the image and text information of the target product collected by the information collection window are acquired, and the information collection window is closed.
[0127] In some embodiments of this application, the first determining module 704 is used to determine the first target product based on the first product name keywords and the product condition keywords, specifically including:
[0128] Calculate the relevance between the first product name keywords and the product condition keywords;
[0129] Obtain target product condition keywords with a relevance higher than a preset relevance threshold, and perform a product search in a preset product database according to the first product name keywords corresponding to the target product condition keywords to obtain several first target products;
[0130] Generate a graphic and text column that corresponds one-to-one with the first target product based on the product information of each of the first target products;
[0131] Based on the image and text bar, a corresponding image and text bar display page for the first target product is generated.
[0132] In some embodiments of this application, the classification module 703 is used to classify the product name keywords into first product name keywords and second product name keywords according to preset preferred product name terms; specifically, it includes the following functions:
[0133] Calculate the similarity between the preset preferred product name words and the product name keywords;
[0134] If the similarity is greater than the preset similarity threshold, it is considered the first product name keyword; if the similarity is less than or equal to the preset similarity threshold, it is considered the second product name keyword.
[0135] In some embodiments of this application, after the first determining module 704 generates a graphic display page corresponding to the first target product based on the graphic bar, it further includes:
[0136] The image and text bar display page is displayed in the display area;
[0137] Count the number of clicks on the graphic and text bar and the number of graphic and text bars displayed in the display area;
[0138] The click-through rate of the first target product is calculated based on the number of clicks on the image and text bar and the number of images and text bars displayed.
[0139] In some embodiments of this application, the second determining module 705 is used to determine the second target product based on the second product name keywords and the product condition keywords, specifically including:
[0140] Based on the second product name keywords and the product condition keywords, a product search is performed in a preset product database to obtain several second target products;
[0141] Generate a graphic and text display page for the second target product based on the product information of the second target product;
[0142] The pop-up window displays the image and text display page of the second target product.
[0143] In some embodiments of this application, after the second determining module 705 displays the graphic and text display page of the second target product based on the pop-up window, it further includes functions for:
[0144] If any request to display the details page of the second target product is detected in the pop-up window, the details page of the second target product is obtained;
[0145] The pop-up window is used to display the details page of the second target product, and the pop-up window is set as the main page, covering the entire display area.
[0146] This application provides a multimodal data product recommendation device, which acquires image and text information of target products; extracts keywords from the image information and keywords from the text information, and classifies the keywords into product name keywords and product condition keywords according to their parts of speech; then, based on preset preferred product name keywords, classifies the product name keywords into first product name keywords and second product name keywords; furthermore, a first target product is determined based on the first product name keywords and the product condition keywords; and if the click-through rate of the first target product is detected to be less than a preset click-through rate threshold, a second target product is determined based on the second product name keywords and the product condition keywords. By extracting keywords from the collected data, the system identifies the target product. It then categorizes the extracted product name keywords based on user preferences, classifying them into primary and secondary product name keywords. The system further determines the primary product based on the primary product name keywords and their associated product condition keywords. This reduces the number of product name keywords retrieved, thereby reducing the amount of data retrieved and the processing load. Furthermore, categorizing the product name keywords based on user preferences ensures that the retrieved primary target product better aligns with user preferences. When user click-through rates are low, the system identifies the secondary target product based on the categorized secondary product name keywords and the product condition keywords, guaranteeing the accuracy of the search results and improving search efficiency.
[0147] This invention also provides a product recommendation device based on multimodal data, such as... Figure 8 As shown, Figure 8 This is a schematic diagram of an embodiment of the product recommendation device for multimodal data provided in this application.
[0148] The multimodal data product recommendation device integrates any of the multimodal data product recommendation devices provided in the embodiments of the present invention, wherein the multimodal data product recommendation device includes:
[0149] One or more processors;
[0150] Memory; and
[0151] One or more applications, wherein the one or more applications are stored in the memory and configured by the processor to perform the steps of the product recommendation method for multimodal data as described in any of the embodiments of the product recommendation method for multimodal data described above.
[0152] Specifically, a multimodal data product recommendation device may include components such as a processor 801 with one or more processing cores, a memory 802 with one or more computer-readable storage media, a power supply 803, and an input unit 804. Those skilled in the art will understand that... Figure 8 The product recommendation device structure shown in the diagram does not constitute a limitation on the product recommendation device for multimodal data. It may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:
[0153] The processor 801 is the control center of the multimodal data product recommendation device. It connects various parts of the device via various interfaces and lines, and executes software programs and / or modules stored in the memory 802, as well as calling data stored in the memory 802, to perform various functions and process data, thereby providing overall monitoring of the device. Optionally, the processor 801 may include one or more processing cores; preferably, it may integrate an application processor and a modem processor, where the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the processor 801.
[0154] The memory 802 can be used to store software programs and modules. The processor 801 executes various functional applications and data processing by running the software programs and modules stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on product recommendations for device use based on multimodal data. In addition, the memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 802 may also include a memory controller to provide the processor 801 with access to the memory 802.
[0155] The product recommendation device for multimodal data also includes a power supply 803 that supplies power to various components. Preferably, the power supply 803 can be logically connected to the processor 801 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 803 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.
[0156] The product recommendation device for multimodal data may also include an input unit 804, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
[0157] Although not shown, the product recommendation device for multimodal data may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 801 in the product recommendation device for multimodal data loads the executable files corresponding to the processes of one or more application programs into the memory 802 according to the following instructions, and the processor 801 runs the application programs stored in the memory 802 to realize various functions, as follows:
[0158] Obtain image and text information of the target product;
[0159] Extract keywords from the image information and keywords from the text information, and classify the keywords into product name keywords and product condition keywords according to their parts of speech;
[0160] The product name keywords are divided into first product name keywords and second product name keywords based on preset preferred product name words;
[0161] The first target product is determined based on the first product name keywords and the product condition keywords;
[0162] If the click-through rate of the first target product is detected to be less than the preset click-through rate threshold, then the second target product is determined based on the second product name keywords and the product condition keywords.
[0163] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.
[0164] Therefore, embodiments of the present invention provide a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc. A computer program is stored thereon, which is loaded by a processor to execute the steps in any of the multimodal data product recommendation methods provided in the embodiments of the present invention. For example, the computer program loaded by the processor can execute the following steps:
[0165] Obtain image and text information of the target product;
[0166] Extract keywords from the image information and keywords from the text information, and classify the keywords into product name keywords and product condition keywords according to their parts of speech;
[0167] The product name keywords are divided into first product name keywords and second product name keywords based on preset preferred product name words;
[0168] The first target product is determined based on the first product name keywords and the product condition keywords;
[0169] If the click-through rate of the first target product is detected to be less than the preset click-through rate threshold, then the second target product is determined based on the second product name keywords and the product condition keywords.
[0170] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed descriptions of other embodiments above, which will not be repeated here.
[0171] In practice, each of the above units or structures can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments, which will not be repeated here.
[0172] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.
[0173] The above provides a detailed description of a product recommendation method, apparatus, device, and readable storage medium for multimodal data provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A product recommendation method based on multimodal data, characterized in that, include: Obtain image and text information of the target product; Extract keywords from the image information and keywords from the text information, and classify the keywords into product name keywords and product condition keywords according to their parts of speech; The product name keywords are divided into first product name keywords and second product name keywords based on preset preferred product name words; The first target product is determined based on the first product name keywords and the product condition keywords; If the click-through rate of the first target product is detected to be less than the preset click-through rate threshold, then the second target product is determined based on the second product name keywords and the product condition keywords.
2. The product recommendation method based on multimodal data according to claim 1, characterized in that, The acquisition of image and text information of the target product includes: Collect the location of the touch point; If the touch point location corresponds to the product search area, then the information collection window of the target product is fed back, and the information collection window includes a text information collection area and an image information collection area; When an information collection completion signal is detected, the image and text information of the target product collected by the information collection window are acquired, and the information collection window is closed.
3. The product recommendation method based on multimodal data according to claim 1, characterized in that, The step of dividing the product name keywords into first product name keywords and second product name keywords based on preset preferred product name terms includes: Calculate the similarity between the preset preferred product name words and the product name keywords; If the similarity is greater than the preset similarity threshold, it is considered the first product name keyword; if the similarity is less than or equal to the preset similarity threshold, it is considered the second product name keyword.
4. The product recommendation method based on multimodal data according to claim 1, characterized in that, The step of determining the first target product based on the first product name keywords and the product condition keywords includes: Calculate the relevance between the first product name keywords and the product condition keywords; Obtain target product condition keywords with a relevance higher than a preset relevance threshold, and perform a product search in a preset product database according to the first product name keywords corresponding to the target product condition keywords to obtain several first target products; Generate a graphic and text column that corresponds one-to-one with the first target product based on the product information of each of the first target products; Based on the image and text bar, a corresponding image and text bar display page for the first target product is generated.
5. The product recommendation method based on multimodal data according to claim 4, characterized in that, After the step of generating the graphic display page corresponding to the first target product based on the graphic bar, the following steps are included: The image and text bar display page is displayed in the display area; Count the number of clicks on the graphic and text bar and the number of graphic and text bars displayed in the display area; The click-through rate of the first target product is calculated based on the number of clicks on the image and text bar and the number of images and text bars displayed.
6. The product recommendation method based on multimodal data according to claim 1, characterized in that, The step of determining the second target product based on the second product name keywords and the product condition keywords includes: Based on the second product name keywords and the product condition keywords, a product search is performed in a preset product database to obtain several second target products; Generate a graphic and text display page for the second target product based on the product information of the second target product; The pop-up window displays the image and text display page of the second target product.
7. The product recommendation method based on multimodal data according to claim 6, characterized in that, After the step of displaying the graphic and text display page of the second target product based on the pop-up window, the method further includes: If any request to display the details page of the second target product is detected in the pop-up window, the details page of the second target product is obtained; The pop-up window is used to display the details page of the second target product, and the pop-up window is set as the main page, covering the entire display area.
8. A product recommendation device based on multimodal data, characterized in that, The product recommendation device based on multimodal data includes: Acquisition module: Used to acquire image and text information of the target product; Extraction module: used to extract keywords from the image information and keywords from the text information, and to classify the keywords into product name keywords and product condition keywords according to their parts of speech; Classification module: used to classify the product name keywords into first product name keywords and second product name keywords based on preset preferred product name terms; First determination module: used to determine the first target product based on the first product name keywords and the product condition keywords; The second determining module is used to determine the second target product based on the second product name keywords and the product condition keywords if the click-through rate of the first target product is detected to be less than a preset click-through rate threshold.
9. A product recommendation device based on multimodal data, characterized in that, The product recommendation device based on multimodal data includes: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the product recommendation method based on multimodal data as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to perform the steps of the product recommendation method based on multimodal data as described in any one of claims 1 to 7.