Quantitative analysis method for hot new products based on NLP and knowledge graph technology

Abstract

The application provides a hot new product quantification analysis method based on NLP and knowledge graph technology, comprising: generating product identification based on product image, product text, shopping behavior data and consumer portrait data, and establishing associated data set by product identification; obtaining image features and attribute terms based on product image and product text, and normalizing the attribute terms into standardized attributes; constructing a knowledge graph and a dictionary based on the standardized attributes, and mapping the standardized attributes to the knowledge graph; screening new words from incremental product text and comparing the new words with the dictionary to obtain new words, and determining new product signals in combination with the new words, image features and knowledge graph mapping results; and associating the new product signals to the shopping behavior data and consumer portrait data to obtain quantification analysis results. The application has the beneficial effects of being capable of associating multi-source heterogeneous data, multi-modal recognizing image text, automatically discovering new product signals and performing quantification analysis based on shopping behavior data and consumer portrait data.

Description

Technical Field

[0001] This invention belongs to the field of quantitative analysis, and in particular relates to a quantitative analysis method for hot new products based on NLP and knowledge graph technologies. Background Technology

[0002] Current market research and new product insights typically rely on questionnaires, traditional statistical software, web crawlers, or third-party reports. These methods mostly employ manual or semi-automated data processing workflows, and have shortcomings in areas such as multi-source data fusion, fine-grained identification of product attributes, discovery of innovative products, analysis response speed, and the reliability of intelligent analysis.

[0003] Existing data is typically scattered across different data sources such as product image systems, business databases, consumer shopping databases, user tag databases, and publicly available internet data systems. The lack of a unified data association and processing mechanism makes it difficult to establish correlations between product images, product attributes, shopping behavior, and consumer profiles. The different storage formats for data such as product images, product text, purchase volume, sales revenue, purchase channels, purchase locations, age groups, household size, and consumption preferences lead to data silos in subsequent analysis.

[0004] Existing product dictionaries typically have a coarse-grained attribute level, making it difficult to identify fine-grained aspects such as flavor, raw materials, main ingredients, variety and origin, processing methods, packaging, efficacy claims, and health concepts. For example, for strawberry-flavored products, traditional product dictionaries usually only retain the attribute "strawberry," failing to further identify specific varieties such as Red Beauty Strawberry, Light Snow Strawberry, and Dandong 99 Strawberry. Existing OCR and keyword recognition technologies are mainly used for extracting text or recognizing keywords, lacking capabilities for industry-specific semantic parsing, synonym merging, alternative name correction, ambiguity resolution, and knowledge association.

[0005] Meanwhile, traditional statistical models typically rely on predefined indicators and existing classification systems for analysis, making it difficult to automatically capture non-linear market signals such as new flavors, new ingredients, new concepts, new packaging forms, and new visual styles from the continuously growing product text, advertising text, and product images. Existing full-text search mainly depends on keyword matching, and while vector search can achieve semantic similarity search, it lacks a collaborative processing mechanism with knowledge graphs, full-text search, quantitative indicator calculation, and large language model inference.

[0006] Furthermore, existing analysis workflows often involve multiple independent systems performing data collection, data cleaning, text recognition, image recognition, retrieval, statistical analysis, and report generation separately. This results in a lengthy data processing chain, making it difficult to automate the closed loop from updating underlying data to updating analysis reports according to a pre-set schedule. When directly using large language models to generate analysis results, the analytical basis and credibility of the generated content cannot be guaranteed if the retrieved structured data and knowledge graph data are not provided to the large language model. Summary of the Invention

[0007] In view of this, the present invention aims to propose a method for quantitative analysis of hot new products based on NLP and knowledge graph technologies, in order to solve at least one of the above-mentioned technical problems.

[0008] To achieve the above objectives, the technical solution of the present invention is implemented as follows: Methods for quantitative analysis of trending new products based on NLP and knowledge graph technologies include: Acquire product images, product text, shopping behavior data, and consumer profile data; clean and standardize the encoding to generate product identifiers; and establish an associated dataset based on the product identifiers. Multimodal recognition is performed on the product image and product text to obtain image features and attribute entries, and the attribute entries are normalized into standardized attributes; A knowledge graph and dictionary are constructed based on the standardized attributes, and the standardized attributes are mapped to the knowledge graph. New words are obtained by filtering incremental product text and comparing it with the dictionary. New product signals are determined by combining the new words, image features, and knowledge graph mapping results. The new product signals are correlated with shopping behavior data and consumer profile data to calculate quantitative indicators and output quantitative analysis results.

[0009] Furthermore, the process of generating product identifiers and establishing associated datasets includes: Generate corresponding product identifiers from cleaned and standardized coded product images, product text, shopping behavior data, and consumer profile data; Establish a connection between product images, product attributes, shopping behavior, and consumer profiles based on the same product identifier.

[0010] Furthermore, the multimodal recognition includes: Perform text recognition on product images to obtain packaging text information; Target detection and image feature extraction are performed on product images to obtain packaging shape and visual features; The product image is mapped to an image vector, and similar image retrieval is performed based on the image vector to obtain similar product images; Image features and attribute entries are extracted based on the packaging text information, packaging shape, visual features, and image vectors.

[0011] Furthermore, the normalization of attribute entries into standardized attributes includes: Based on the historical product text, the point mutual information value of candidate word groups is calculated, and the candidate word groups that meet the conditions are added to the basic word segmentation dictionary; Add the product dictionary and historical review entries to the basic word segmentation dictionary to obtain the industry dictionary; The product text is segmented using the industry dictionary, and the segmentation boundaries are verified using a sequence labeling model to obtain attribute entries; The attribute entries are normalized according to the correspondence between original words, standard words, major attribute categories, and minor attribute categories to obtain standardized attributes.

[0012] Furthermore, the construction of the knowledge graph and the mapping of standardized attributes to the knowledge graph include: Establish a hierarchical ontology consisting of major attribute categories, minor attribute categories, and specific keywords; The standardized attributes are matched with specific keywords, and if no specific keyword is matched, the corresponding specific keyword is determined based on string similarity or semantic vector similarity. Based on the hierarchical ontology, the corresponding specific keywords are successively merged into attribute subcategories and attribute major categories; Based on the product category, determine the knowledge graph nodes corresponding to standardized attributes with the same word form.

[0013] Furthermore, the process of filtering new words in the incremental product text includes: Perform N-Gram segmentation on the incremental product text within a preset time window to obtain candidate word strings; Calculate the inverse term frequency score, keyword length score, term frequency-inverse document frequency score, and point mutual information score for each candidate word string; Candidate new words are determined based on each score; The candidate new words are compared with the entries in the dictionary, and the candidate new words that do not exist in the dictionary are identified as new words.

[0014] Furthermore, the signal for determining a new product includes: Associate new terms with corresponding product identifiers; Determine the co-occurrence relationships between new terms and product brand, product attribute terms, and advertising text; The co-occurrence relationship is jointly determined with the image features and knowledge graph mapping results corresponding to the product identifier to obtain the new product signal; New words in the new product signals and their knowledge graph mapping relationships are written into the dictionary and knowledge graph.

[0015] Furthermore, the calculated quantitative indicators include: Using the attribute keywords in the knowledge graph as grouping keys, the shopping behavior data associated with the attribute keywords are aggregated to obtain SKU volume, purchase volume, and sales revenue; The penetration rate is calculated based on the number of users who purchased products corresponding to the attribute keywords and the total number of users in the product category. The growth rate is calculated based on the sales figures for the current time window and the sales figures for the previous time window.

[0016] Furthermore, the calculated quantitative indicators also include: The novelty score of attribute keywords is calculated by weighting the new word tag score, the normalized growth rate value, and the keyword length weight score. Construct time series and predict trends based on the frequency of mention of attribute keywords across multiple time windows; Item sets are constructed based on the attribute keywords associated with SKUs, and frequent item sets and association rules are determined based on the item sets and sales revenue.

[0017] Furthermore, it also includes: The system receives query conditions and performs full-text and vector searches on standardized attributes, image features, and quantitative analysis results in the knowledge graph based on the query conditions to obtain search results. The search results are used as context input into a large language model to generate a market insight report. The data acquisition, multimodal recognition, knowledge graph update, new word screening, new product signal determination, and quantitative indicator calculation are re-executed according to the preset cycle, and the market insight report is updated.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. By linking product images, product text, shopping behavior data, and consumer profile data through product identification, a data foundation for linking product attributes, shopping behavior, and consumer profiles is formed; 2. By using OCR text recognition, object detection, image feature extraction, image vectorization, and text semantic parsing, we obtain packaging text, packaging shape, visual features, and fine-grained attribute entries; 3. Through industry dictionaries, sequence labeling models, semantic normalization, and hierarchical mapping of knowledge graphs, unified processing of product attributes is achieved, from raw words to standard words, attribute subcategories, and attribute major categories. 4. By filtering new words in incremental text, comparing dictionaries, analyzing image features, and mapping knowledge graphs, a dual-path new product detection mechanism is constructed that integrates textual new word discovery and visual new feature matching to automatically discover new words and new product signals appearing in the product market; 5. By linking new product signals with shopping behavior data and consumer profile data, quantitative indicators such as SKU volume, purchase volume, sales revenue, penetration rate, growth rate, and novelty score are calculated; 6. Through full-text search, vector search, and search enhancement, provide the large language model with structured data, image features, and knowledge graph data corresponding to the query conditions to generate market insight reports; 7. Through a pre-set data update mechanism, an automated process is formed for data updates, multimodal processing, knowledge graph updates, new word discovery, quantitative indicator recalculation, and report updates. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the workflow of the hot new product quantitative analysis method based on NLP and knowledge graph technology described in the embodiments of the present invention. Detailed Implementation

[0021] It should be noted that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application. In the absence of conflict, the embodiments and features in the embodiments of this invention can be combined with each other. In order to make the purpose, technical solution and advantages of this application clearer, the following describes this application in further detail with reference to the accompanying drawings and embodiments.

[0022] like Figure 1 As shown, the quantitative analysis method for trending new products based on NLP and knowledge graph technologies includes: S1. Acquire product images, product text, shopping behavior data, and consumer profile data; clean and standardize the encoded data to generate product identifiers; and establish an associated dataset based on the product identifiers. S2. Perform multimodal recognition on the product image and product text to obtain image features and attribute entries, and normalize the attribute entries into standardized attributes; S3. Construct a knowledge graph and dictionary based on the standardized attributes, and map the standardized attributes to the knowledge graph; S4. New words are selected from the incremental product text and compared with the dictionary to obtain new words. The new product signal is determined by combining the new words, image features and knowledge graph mapping results. S5. Associate the new product signal with shopping behavior data and consumer profile data, calculate quantitative indicators and output quantitative analysis results.

[0023] In some embodiments, the quantitative analysis method for trending new products can be deployed on a browser / server architecture.

[0024] The underlying raw data can be deployed on the enterprise's internal servers to store product images, product text, shopping behavior data, consumer profile data, product dictionary data, and historical review data; Application-layer metrics, insights, and retrieval indexes can be deployed on cloud servers to provide authorized users with query, visualization, report generation, and data export services.

[0025] Logically, this invention comprises a data acquisition layer, an algorithm processing layer, a data storage layer, and a platform application layer.

[0026] The data acquisition layer is used to connect to multi-source heterogeneous data interfaces to acquire product image data, product text data, consumer shopping behavior data, consumer profile data, and product dictionary data; The algorithm processing layer includes an image semantic understanding module, a semantic engineering and refined word segmentation module, a knowledge graph construction module, a new word and new product discovery algorithm module, a statistical modeling and quantitative index calculation module, an intelligent retrieval module, and a large language model analysis module; The data storage layer includes a raw database, a word segmentation result database, an image feature database, an image vector database, a knowledge graph database, a dictionary database, a key indicator calculation result database, and an insight result database; The platform application layer provides a web front-end to receive user queries, display quantitative analysis results, visualization charts and market insight reports, and control the scope of user data access according to account permissions.

[0027] The process of generating product identifiers and establishing associated datasets includes: Generate corresponding product identifiers from cleaned and standardized coded product images, product text, shopping behavior data, and consumer profile data; Establish a connection between product images, product attributes, shopping behavior, and consumer profiles based on the same product identifier.

[0028] In some embodiments, the raw data acquired by the data acquisition layer includes product images, product text, shopping behavior data, and consumer profile data.

[0029] Product images can be product packaging images, product label images, promotional poster images, or product display images; Product text can be product detail page text, product description text, advertising copy, packaging text recognition results, or complete product description text; Shopping behavior data can include purchase time, purchase quantity, number of items purchased, sales amount, purchase frequency, purchase channel, purchase location, and purchase scenario; Consumer profile data may include the age group of the main buyer, the number of family members, consumption preferences, and user tags; Product dictionary data can include product brand, product name, product category, ingredients, weight / volume, packaging form, and existing attribute entries.

[0030] In one implementation, the company collects daily consumption data from sample consumer households through an application, and collects information such as product content, purchase time, purchase channel, purchase scenario, and consumption amount corresponding to the consumption data. The daily consumption data may involve food, beverages, daily chemical products, and other consumer goods categories.

[0031] The acquired data undergoes cleaning processes, including removing garbled characters, handling null values, correcting formatting errors, standardizing date formats, standardizing units of measurement, and verifying the integrity of product fields. The cleaned data then undergoes standardized coding to ensure consistency in coding for the same product, user, channel, and product category across different data sources.

[0032] A product identifier is generated for each product record. The product identifier can be determined by the product brand, product name, product specifications, product barcode, internal product code, or a combination of these fields. Product images, product text, shopping behavior data, and consumer profile data are all associated with corresponding product identifiers, user identifiers, or transaction identifiers.

[0033] A relational dataset is established based on product identifiers. This dataset includes at least the relationships between product identifiers, product images, product text, product attributes, shopping behavior data, and consumer profile data. Through these relationships, it is possible to locate the corresponding product, purchase record, and consumer profile based on any product attribute, and also to trace the corresponding product attributes and product images based on purchase records.

[0034] The multimodal recognition includes: Perform text recognition on product images to obtain packaging text information; Target detection and image feature extraction are performed on product images to obtain packaging shape and visual features; The product image is mapped to an image vector, and similar image retrieval is performed based on the image vector to obtain similar product images; Image features and attribute entries are extracted based on the packaging text information, packaging shape, visual features, and image vectors.

[0035] In some embodiments, multimodal recognition includes two parts: image semantic understanding and text semantic engineering. Image semantic understanding obtains packaging text information by performing OCR text recognition on product images associated with product identification. Packaging text information may include product name, brand, flavor, main ingredients, ingredients, efficacy claims, health claims, specifications, and advertising text.

[0036] Object detection and image feature extraction are performed on product images to obtain packaging form and visual features. Packaging form can represent whether the product is packaged in a bottle, bag, can, box, or other packaging structure; visual features can represent color features, pattern features, layout features, font features, icon features, and arrangement features in the image.

[0037] The product image is input into the image embedding model to obtain the corresponding image vector, which is then stored in an image vector library. Subsequently, the image to be retrieved can be converted into a query image vector, and similar product images can be obtained based on the similarity between the query image vector and the image vectors in the library. The packaging form and visual features corresponding to similar product images can be used to identify packaging structures or visual styles that differ from existing products.

[0038] The image features are composed of OCR text recognition results, packaging shape, visual features, and image vectors. Among them, the packaging text information can be used as input data for the text semantic engineering module.

[0039] The text semantic engineering module receives the complete product description text. The complete product description text can consist of packaging text information, product details page text, product description text, advertising copy, product description fields in the product dictionary, and other text fields associated with product identification.

[0040] The text semantic engineering module is used to identify attribute entries such as brand, flavor, main ingredients, ingredient type, variety and origin, processing method, efficacy claims, health claims, concept claims, packaging information and packaging form from the complete product description text.

[0041] For example, for product description text containing "Dandong Jiujiu Strawberry", the text semantic engineering module will recognize "Dandong Jiujiu Strawberry" as a complete attribute term, rather than just retaining "strawberry"; For product description text containing "fragrant pomelo and yuzu", the text semantic engineering module identifies "fragrant pomelo" and "yuzu" as corresponding fine-grained attribute terms, rather than just retaining "pomelo".

[0042] The process of normalizing attribute entries into standardized attributes includes: Based on the historical product text, the point mutual information value of candidate word groups is calculated, and the candidate word groups that meet the conditions are added to the basic word segmentation dictionary; Add the product dictionary and historical review entries to the basic word segmentation dictionary to obtain the industry dictionary; The product text is segmented using the industry dictionary, and the segmentation boundaries are verified using a sequence labeling model to obtain attribute entries; The attribute entries are normalized according to the correspondence between original words, standard words, major attribute categories, and minor attribute categories to obtain standardized attributes.

[0043] In some embodiments, the present invention uses a combination of cold start, vertical domain enhancement and intelligent verification to construct a professional word segmentation model; Using historical product description text as the initial corpus, N-Gram segmentation is performed on the initial corpus to obtain multiple candidate word groups. The point mutual information value (PMI) between characters or words in the candidate word groups is calculated. The point mutual information value is used to characterize the co-occurrence density of characters or words within the candidate word group. Candidate word groups whose point mutual information values ​​meet a preset threshold are filtered to obtain high-cohesion candidate word groups. The high-cohesion candidate word groups are added to the general Chinese dictionary of the basic word segmenter to form an initial word list. The basic word segmenter can be Jieba word segmenter or other word segmenters with Chinese word segmentation functions. For example, "freeze-dried coffee", which frequently appears in historical product description text and whose internal co-occurrence degree meets the threshold, can be written into the initial word list as a candidate word group.

[0044] Based on the initial terminology, an industry dictionary is constructed by injecting the company's own product dictionary and historically reviewed terms. Terms in the company's own product dictionary can include brand terms, category terms, ingredient and blend terms, flavor terms, variety terms, place of origin terms, packaging terms, processing terms, efficacy terms, and health concept terms. Historically reviewed terms are those confirmed during manual review, including corrected terms, standardized terms, or newly added terms. Custom industry terms in the industry dictionary have a higher matching priority than general terms in the general Chinese dictionary, thus allowing fine-grained product attributes to be recognized as complete terms first. For example, when the industry dictionary contains "Dandong Jiujiu Strawberry," the word segmenter will output "Dandong Jiujiu Strawberry" in the product description as a complete term, instead of outputting the three terms "Dandong," "Jiujiu," and "Strawberry."

[0045] The initial segmentation results are obtained by performing preliminary word segmentation on the complete product description text using an industry dictionary. These initial results are then used as training or fine-tuning data, and a sequence labeling model consisting of a pre-trained language model and a conditional random field model is employed to verify the segmentation boundaries. The pre-trained language model can be a BERT model, and the conditional random field model is used to determine the boundaries of attribute terms based on contextual relationships. The sequence labeling model outputs the recognition confidence score of each attribute term. When the recognition confidence score of an attribute term is lower than a preset threshold, the product text corresponding to that attribute term is pushed to a manual review queue. The preset threshold can be set to 0.85. The manual review results are written into a historical review term library and used to update the industry dictionary and the sequence labeling model, thus forming a data feedback mechanism for dictionary and model updates.

[0046] Semantic normalization is performed on the attribute entries obtained from word segmentation to establish a four-level mapping relationship between original words, standard words, attribute categories, and attribute subcategories. Semantic normalization includes synonym merging, alternative name correction, unification of different expressions, and fine-grained attribute preservation. For example, "zero calories" and "0 calories" can be normalized to the standard word "zero calories"; "fat house happy water" can be normalized to the standard word "carbonated beverage"; "red strawberry" and "light snow strawberry" can be preserved as fine-grained standard words and mapped to the attribute subcategory or attribute level corresponding to "strawberry"; "fragrant coconut juice" can be normalized and associated with the specific keyword corresponding to "fragrant coconut". The result after normalization is formed into standardized attributes. Standardized attributes include at least product identifier, original word, standard word, attribute category, attribute subcategory, specific keyword, identification source, and identification confidence. Identification source can be used to identify that standardized attributes come from packaging text, product detail page, product description, advertising copy, product dictionary, or image recognition results.

[0047] The construction of the knowledge graph and the mapping of standardized attributes to the knowledge graph include: Establish a hierarchical ontology consisting of major attribute categories, minor attribute categories, and specific keywords; The standardized attributes are matched with specific keywords, and if no specific keyword is matched, the corresponding specific keyword is determined based on string similarity or semantic vector similarity. Based on the hierarchical ontology, the corresponding specific keywords are successively merged into attribute subcategories and attribute major categories; Based on the product category, determine the knowledge graph nodes corresponding to standardized attributes with the same word form.

[0048] In some embodiments, The knowledge graph module pre-constructs a standardized, multi-level ontology system. The ontology system includes three levels: major attribute categories, minor attribute categories, and specific keywords.

[0049] The major attribute categories can include flavor, main ingredients, ingredient type, variety and origin, processing method, efficacy claims, health claims, concept claims, packaging form, visual style, target audience and shopping scenario; Attribute subcategories are further classifications under attribute major categories; The specific keywords are the standardized attribute terms corresponding to the attribute subcategories.

[0050] For example, under the major category of flavor, you can set fruit flavor as a subcategory, and under the fruit flavor subcategory, you can set specific keywords such as strawberry, fragrant coconut, and white peach. For example, under the broad category of sweetness, you can set subcategories or specific keywords such as sucrose sweetness, artificial sweetness, and fructose sweetness.

[0051] For the standardized attributes input, the knowledge graph module performs the following automated classification process.

[0052] First, the specific keywords in the standardized attributes are precisely matched with the existing specific keywords in the ontology system.

[0053] When no specific keyword is matched, calculate the string similarity or semantic vector similarity between the input term and the existing node, attach the input term to the existing node that meets the similarity requirement, or add a specific keyword node under the corresponding attribute subcategory.

[0054] For example, the input term "fragrant coconut juice" can be attached to the "fragrant coconut" node through string similarity or semantic vector similarity.

[0055] Then, based on the predefined hierarchical relationships in the ontology system, specific keywords are merged level by level. For example, "fragrant coconut juice" can be merged into the nodes "fragrant coconut", "coconut flavor", and "flavor" in sequence.

[0056] Furthermore, ambiguity resolution for the same word form is achieved by leveraging the product category. Specifically, when the standardized attribute is "apple," if the product category is beverage, then "apple" is mapped to the "apple flavor" node under flavor; if the product category is mobile phone, then "apple" is mapped to the brand node under brand attribute.

[0057] The nodes in the knowledge graph include product nodes, brand nodes, category nodes, major attribute category nodes, minor attribute category nodes, specific keyword nodes, packaging form nodes, visual style nodes, user group nodes, and shopping scenario nodes.

[0058] The relationships in a knowledge graph include: a product belongs to a category; a product belongs to a brand; a product has attributes; a specific keyword belongs to a subcategory of attributes; a subcategory of attributes belongs to a major category of attributes; the product corresponds to a packaging form; the product corresponds to a visual style; the product is purchased by a user; and the product corresponds to a shopping scenario. It also includes the integration relationships between attributes, such as the relationship between single flavor and compound flavor.

[0059] Using the above graph structure, "coconut water" and "fragrant coconut juice" can be merged into the attribute level corresponding to "coconut flavor", while retaining "coconut" and "fragrant coconut" as different specific keywords to achieve fine-grained statistics and upper-level attribute aggregation statistics.

[0060] The process of filtering new words in incremental product text includes: Perform N-Gram segmentation on the incremental product text within a preset time window to obtain candidate word strings; Calculate the inverse term frequency score, keyword length score, term frequency-inverse document frequency score, and point mutual information score for each candidate word string; Candidate new words are determined based on each score; The candidate new words are compared with the entries in the dictionary, and the candidate new words that do not exist in the dictionary are identified as new words.

[0061] The signals for identifying new products include: Associate new terms with corresponding product identifiers; Determine the co-occurrence relationships between new terms and product brand, product attribute terms, and advertising text; The co-occurrence relationship is jointly determined with the image features corresponding to the product identifier, the knowledge graph mapping results, and the multimodal large model inference results to obtain the new product signal; New words in the new product signals and their knowledge graph mapping relationships are written into the dictionary and knowledge graph.

[0062] In some embodiments, the new word and new product discovery module performs new word discovery and new product signal determination on incremental product text.

[0063] Retrieve incremental product text according to a preset time window. The preset time window can be the last two to four weeks, or it can be configured according to the product launch frequency and data update cycle.

[0064] Incremental product text includes product detail page text, product description text, advertising copy, packaging text recognition results, and other text data associated with product identification that are added or updated within a preset time window.

[0065] Perform N-Gram segmentation on the incremental product text to generate candidate word strings.

[0066] For each candidate word string, calculate the term frequency-inverse document frequency score (TF-IDF), point mutual information score (PMI), inverse term frequency score, and keyword length score.

[0067] TF-IDF scores are used to characterize the importance of candidate word strings in the current incremental text; PMI scores are used to characterize the degree of cohesion of characters or words within a candidate word string; The inverse word frequency score is the reciprocal of the frequency of occurrence of candidate word strings, which is used to reduce the impact of common words that appear frequently in each time window on the selection of new words; Keyword length score is used to assign weights based on the character length of candidate word strings. Candidate word strings with a length of four characters or more are given higher weights to identify attribute entries containing place of origin, variety, or detailed descriptions.

[0068] A comprehensive score is generated based on TF-IDF score, PMI score, inverse term frequency score, and keyword length score. The comprehensive score can be calculated using a weighted summation method; that is, the comprehensive score of a candidate word string is obtained by weighted summation of the four scores: term frequency-inverse document frequency, point mutual information, inverse term frequency, and keyword length.

[0069] Candidate word strings that meet the preset conditions in terms of comprehensive score are identified as candidate new words.

[0070] The candidate new words are compared with existing entries in the dictionary.

[0071] If a candidate new word does not exist in the dictionary, mark the candidate new word as a new word; If a candidate new word already exists in the dictionary, it will not be written into the dictionary as a new word.

[0072] Establish a relationship between the new word and its corresponding product identifier, time of appearance, text source, product category, product brand, and co-occurrence attribute entries.

[0073] For newly tagged words, determine the co-occurrence relationships between the new words and product brands, product attribute terms, and advertising text.

[0074] Specifically, it can be determined whether a new word appears together with the brand name, category name, product name, flavor term, ingredient term, process term, or advertising buzzword in the product text corresponding to the same product identifier.

[0075] Simultaneously, the packaging form, visual features, and image vector retrieval results corresponding to the product identifier are obtained. The co-occurrence relationship of new words, image features, knowledge graph mapping results, and multimodal large model inference results are jointly determined to obtain the new product signal.

[0076] When a new word meets the co-occurrence condition with a specific brand, product name, category name, or advertising hot word, and the knowledge graph node corresponding to the new word is a newly added node or a subdivided node, the corresponding product identifier can be marked as having a new product signal.

[0077] When the image vector retrieval results show that the similarity between the product image and existing product images is lower than a preset threshold, or when the product packaging form and visual features differ from existing products, image features can be used as a supplementary basis for determining new product signals.

[0078] Once a new product signal is identified, the new words, standardized attributes, product identifiers, image features, knowledge graph mapping relationships, and time information from the new product signal are written into the dictionary and knowledge graph databases. This new product signal can also trigger alarm information for review by operations or analysis personnel.

[0079] The calculated quantitative indicators include: Using the attribute keywords in the knowledge graph as grouping keys, the shopping behavior data associated with the attribute keywords are aggregated to obtain SKU volume, purchase volume, and sales revenue; The penetration rate is calculated based on the number of users who purchased products corresponding to the attribute keywords and the total number of users in the product category. The growth rate is calculated based on the sales figures for the current time window and the sales figures for the previous time window.

[0080] The calculated quantitative indicators also include: The novelty score of attribute keywords is calculated by weighting the new word tag score, the normalized growth rate value, and the keyword length weight score. Construct time series and predict trends based on the frequency of mention of attribute keywords across multiple time windows; Item sets are constructed based on the attribute keywords associated with SKUs, and frequent item sets and association rules are determined based on the item sets and sales revenue.

[0081] In some embodiments, the quantitative indicator calculation module associates and aggregates product attributes with shopping behavior data and consumer profile data based on attribute keywords in the knowledge graph.

[0082] Using attribute keywords in the knowledge graph as grouping keys, shopping behavior data associated with those attribute keywords is aggregated.

[0083] SKU volume is the number of unique SKUs with this attribute keyword; purchase volume is the sum of the number of units purchased for all SKUs with this attribute keyword; sales revenue is the sum of the sales amount for all SKUs with this attribute keyword; penetration rate is the ratio between the number of users who purchase products corresponding to a certain attribute keyword and the total number of users in the product category; growth rate is calculated based on the sales revenue of the current time window and the sales revenue of the previous time window. The above indicators can be cross-aggregated according to time, category, brand, price range, channel, purchase location, purchase scenario, user age group, household size, and consumption preferences.

[0084] For each attribute keyword, a novelty score is calculated based on the new word tag score, the normalized growth rate value, and the keyword length weight score.

[0085] Time series are constructed based on the frequency of mention of attribute keywords in multiple historical time windows.

[0086] The time window can be set at a weekly granularity. For each attribute keyword, the frequency of mentions in each week is counted to obtain the weekly granular time series corresponding to the attribute keyword.

[0087] The Prophet model or the Long Short-Term Memory (LSTM) network model is used to train and predict time series data, thereby obtaining the expected mention frequency trend and confidence interval within the future time window.

[0088] The Prophet model can be used to predict the frequency of mentions of attribute keywords with trending or seasonal characteristics. LSTM models can be used to predict the frequency of mentions of attribute keywords with complex nonlinear variation characteristics.

[0089] In one implementation, the forecast time range is set to the next four to the next eight weeks.

[0090] For each SKU, construct an itemset from all the attribute keywords corresponding to the SKU.

[0091] High-volume SKUs are identified based on itemsets and sales revenue, and the Apriori algorithm is used to mine frequent itemsets and association rules.

[0092] Set minimum support, minimum confidence, and lift thresholds to filter association rules that meet the conditions.

[0093] In one implementation, the minimum support is set to 1%, the minimum confidence is set to 60%, and association rules with a lift greater than 1 are retained.

[0094] Association rules are used to identify combinations of attribute keywords that have a statistical association with high-selling SKUs.

[0095] The quantitative indicator calculation module writes SKU volume, purchase volume, sales revenue, penetration rate, growth rate, novelty score, trend prediction results, and association rule results into the key indicator calculation result library and insight result library.

[0096] Quantitative results can be output in the form of tables, bar charts, column charts, line trend charts, word clouds, or other visualizations.

[0097] Word clouds can determine the font size and display style of attribute keywords based on SKU volume, purchase volume, sales revenue, growth rate, or novelty score.

[0098] Also includes: The system receives query conditions and performs full-text and vector searches on standardized attributes, image features, and quantitative analysis results in the knowledge graph based on the query conditions to obtain search results. The search results are used as context input into a large language model to generate a market insight report. The data acquisition, multimodal recognition, knowledge graph update, new word screening, new product signal determination, and quantitative indicator calculation are re-executed according to the preset cycle, and the market insight report is updated.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for quantitative analysis of trending new products based on NLP and knowledge graph technologies, characterized in that: include: Acquire product images, product text, shopping behavior data, and consumer profile data; clean and standardize the encoding to generate product identifiers; and establish an associated dataset based on the product identifiers. Multimodal recognition is performed on the product image and product text to obtain image features and attribute entries, and the attribute entries are normalized into standardized attributes; A knowledge graph and dictionary are constructed based on the standardized attributes, and the standardized attributes are mapped to the knowledge graph. New words are obtained by filtering incremental product text and comparing it with the dictionary. New product signals are determined by combining the new words, image features, and knowledge graph mapping results. The new product signals are correlated with shopping behavior data and consumer profile data to calculate quantitative indicators and output quantitative analysis results.

2. The method for quantitative analysis of hot new products based on NLP and knowledge graph technology according to claim 1, characterized in that, The process of generating product identifiers and establishing associated datasets includes: Generate corresponding product identifiers from cleaned and standardized coded product images, product text, shopping behavior data, and consumer profile data; Establish a connection between product images, product attributes, shopping behavior, and consumer profiles based on the same product identifier.

3. The method for quantitative analysis of hot new products based on NLP and knowledge graph technology according to claim 1, characterized in that, The multimodal recognition includes: Perform text recognition on product images to obtain packaging text information; Target detection and image feature extraction are performed on product images to obtain packaging shape and visual features; The product image is mapped to an image vector, and similar image retrieval is performed based on the image vector to obtain similar product images; Image features and attribute entries are extracted based on the packaging text information, packaging shape, visual features, and image vectors.

4. The method for quantitative analysis of hot new products based on NLP and knowledge graph technology according to claim 1, characterized in that, The process of normalizing attribute entries into standardized attributes includes: Based on the historical product text, the point mutual information value of candidate word groups is calculated, and the candidate word groups that meet the conditions are added to the basic word segmentation dictionary; Add the product dictionary and historical review entries to the basic word segmentation dictionary to obtain the industry dictionary; The product text is segmented using the industry dictionary, and the segmentation boundaries are verified using a sequence labeling model to obtain attribute entries; The attribute entries are normalized according to the correspondence between original words, standard words, major attribute categories, and minor attribute categories to obtain standardized attributes.

5. The method for quantitative analysis of hot new products based on NLP and knowledge graph technology according to claim 1, characterized in that, The construction of the knowledge graph and the mapping of standardized attributes to the knowledge graph include: Establish a hierarchical ontology consisting of major attribute categories, minor attribute categories, and specific keywords; The standardized attributes are matched with specific keywords, and if no specific keyword is matched, the corresponding specific keyword is determined based on string similarity or semantic vector similarity. Based on the hierarchical ontology, the corresponding specific keywords are successively merged into attribute subcategories and attribute major categories; Based on the product category, determine the knowledge graph nodes corresponding to standardized attributes with the same word form.

6. The method for quantitative analysis of hot new products based on NLP and knowledge graph technology according to claim 1, characterized in that, The process of filtering new words in incremental product text includes: Perform N-Gram segmentation on the incremental product text within a preset time window to obtain candidate word strings; Calculate the inverse term frequency score, keyword length score, term frequency-inverse document frequency score, and point mutual information score for each candidate word string; Candidate new words are determined based on each score; The candidate new words are compared with the entries in the dictionary, and the candidate new words that do not exist in the dictionary are identified as new words.

7. The method for quantitative analysis of trending new products based on NLP and knowledge graph technology according to claim 6, characterized in that, The signals for identifying new products include: Associate new words with their corresponding product identifiers; Determine the co-occurrence relationships between new terms and product brand, product attribute terms, and advertising text; The co-occurrence relationship is jointly determined with the image features and knowledge graph mapping results corresponding to the product identifier to obtain the new product signal; New words in the new product signals and their knowledge graph mapping relationships are written into the dictionary and knowledge graph.

8. The method for quantitative analysis of hot new products based on NLP and knowledge graph technology according to claim 1, characterized in that, The calculated quantitative indicators include: Using the attribute keywords in the knowledge graph as grouping keys, the shopping behavior data associated with the attribute keywords are aggregated to obtain SKU volume, purchase volume, and sales revenue; The penetration rate is calculated based on the number of users who purchased products corresponding to the attribute keywords and the total number of users in the product category. The growth rate is calculated based on the sales figures for the current time window and the sales figures for the previous time window.

9. The method for quantitative analysis of hot new products based on NLP and knowledge graph technology according to claim 8, characterized in that, The calculated quantitative indicators also include: The novelty score of attribute keywords is calculated by weighting the new word tag score, the normalized growth rate value, and the keyword length weight score. Construct time series and predict trends based on the frequency of mention of attribute keywords across multiple time windows; Item sets are constructed based on the attribute keywords associated with SKUs, and frequent item sets and association rules are determined based on the item sets and sales revenue.

10. The method for quantitative analysis of hot new products based on NLP and knowledge graph technology according to claim 1, characterized in that, Also includes: The system receives query conditions and performs full-text and vector searches on standardized attributes, image features, and quantitative analysis results in the knowledge graph based on the query conditions to obtain search results. The search results are used as context input into a large language model to generate a market insight report. The data acquisition, multimodal recognition, knowledge graph update, new word screening, new product signal determination, and quantitative indicator calculation are re-executed according to the preset cycle, and the market insight report is updated.

Images