A furniture commodity identification method and system based on multi-modal heterogeneous data processing
By using a multimodal heterogeneous data processing method, global textual and visual features of furniture product images are extracted. Entity alignment is then performed by combining a modal fusion model and a knowledge graph model, solving the misjudgment problem in furniture product recognition and achieving accurate recognition and rapid matching.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING CITY MANAGEMENT COLLEGE
- Filing Date
- 2026-05-12
- Publication Date
- 2026-07-10
AI Technical Summary
Existing technologies struggle to effectively integrate heterogeneous image and text data, resulting in a high misjudgment rate in furniture product recognition and failing to meet the demands for accurate identification and rapid matching.
A multimodal heterogeneous data processing method is adopted. By extracting global text features, target text features, and visual features from the target product image, a modal fusion model is used for processing, and a knowledge graph model is used for entity alignment to achieve accurate identification of furniture products.
It improves the accuracy and speed of furniture product recognition, especially in maintaining high recognition accuracy when dealing with unpopular styles and small sample furniture, and enhances the generalization ability of the model.
Smart Images
Figure CN122368639A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image recognition technology, and in particular to a furniture product recognition method and system based on multimodal heterogeneous data processing. Background Technology
[0002] With the rapid development of home furnishing e-commerce, whole-house customization, and new retail furniture industries, users increasingly demand to search for the same furniture through real-life photos, social media shared images, and e-commerce screenshots. Furniture product recognition has become a core technology in online shopping guides, inventory matching, infringement comparison, and intelligent design. Currently, furniture product recognition mainly relies on single visual feature matching or text keyword retrieval, which has significant shortcomings in practical applications.
[0003] Multimodal heterogeneous data is difficult to integrate. User-uploaded images contain unstructured text and visual information, representing typical image-text heterogeneous data. Traditional methods, using only visual or text models, cannot effectively correlate textual semantics with visual details, easily leading to misjudgments such as visual similarities but different product categories, or text matching but discrepancies between the actual product and the product. This makes it difficult to meet the practical needs of accurate identification and rapid matching of furniture products. Summary of the Invention
[0004] This invention provides a furniture product identification method and system based on multimodal heterogeneous data processing, which can achieve accurate identification and rapid matching of furniture products.
[0005] In a first aspect, the present invention provides a furniture product identification method based on multimodal heterogeneous data processing, the furniture product identification method based on multimodal heterogeneous data processing comprising:
[0006] Obtain a target product image, wherein the target product image includes at least one product to be identified;
[0007] The target product image is preprocessed to obtain the global text features, target text features, and visual features of the target product image;
[0008] The global text features, target text features, and visual features are processed using a modal fusion model to obtain the classification result of the target product image;
[0009] Based on the classification result of the target product image, a candidate image set is searched in a pre-built database; wherein, the candidate image set includes at least one candidate image that is most similar to the classification result;
[0010] The target product image and the candidate image set are input into a knowledge graph model, and the knowledge graph model is used to perform entity alignment on the target product image and the candidate images;
[0011] Based on the entity alignment result between the target product image and the candidate images, the recognition result of the product to be identified in the target product image is output.
[0012] As a further aspect of the present invention: the preprocessing of the target product image to obtain the global text features, target text features, and visual features of the target product image includes:
[0013] Extract text information and the corresponding positional parameter information from the target product image to obtain the global text features of the target product image;
[0014] Extract the location parameters of the target product from the target product image, and generate the target text features based on the location parameters and the global text features;
[0015] Visual extraction is performed on the target product image to obtain its visual features.
[0016] As a further aspect of the present invention: the process of using a modal fusion model to process the global text features, target text features, and visual features to obtain the classification result of the target product image includes:
[0017] A linear transformation is performed on the global text features, the target text features, and the visual features to obtain an attention matrix of the global text features, the target text features, and the visual features; the attention matrix includes a query vector matrix, a key vector matrix, and a value vector matrix.
[0018] Two features are randomly selected from the global text features, the target text features, and the visual features. An attention mechanism is used to perform weighted fusion on the attention matrices of the two selected features to obtain multiple weighted fused features.
[0019] Multiple weighted fusion features are concatenated, and a classification function is used to process the concatenated features to obtain the classification result of the target product image.
[0020] As a further aspect of the present invention: Two features are arbitrarily selected from the global text features, the target text features, and the visual features; an attention mechanism is used to perform weighted fusion on the attention matrices of the two selected features to obtain multiple weighted fused features, including:
[0021] The attention matrix of the global text features, the target text features, and the visual features is divided into multiple sub-matrices using multiple attention heads;
[0022] In an attention head, two features are arbitrarily selected, and the similarity between the query vector matrix of one selected feature and the key vector matrix of the other selected feature is calculated to generate cross-modal attention weights. The value vector matrix of the latter is then weighted and aggregated using the cross-modal attention weights to obtain a weighted fusion feature.
[0023] Iterate through all attention heads and repeat the weighted fusion step to obtain multiple weighted fused features.
[0024] As a further aspect of the present invention: the calculation formula for the weighted fusion feature is as follows:
[0025] ;
[0026] In the formula, This indicates a selected feature in the i-th attention head. With another selected feature Weighted fusion characteristics; Represents the features in the i-th attention head The query vector matrix; Represents the features in the i-th attention head The transpose of the key vector matrix; Represents the features in the i-th attention head The value vector matrix; Represents a scalar.
[0027] As a further aspect of the present invention: the step of inputting the target product image and the candidate image set into a knowledge graph model, and using the knowledge graph model to perform entity alignment between the target product image and the candidate images, includes:
[0028] The target product image and all candidate images in the candidate image set are converted into graph structure text information; wherein, the graph structure text information includes entity sets, relation sets, triple sets, and image sets;
[0029] Multimodal features of the target product image and the candidate images are generated based on the graph structure text information;
[0030] For the target product image and all candidate images in the candidate image set, calculate the cosine similarity between the multimodal features of the target product image and the multimodal features of each candidate image;
[0031] Based on the calculation result of the cosine similarity, a similarity mapping table is generated between the target product image and the candidate image. The similarity mapping table is the entity alignment result between the target product image and the candidate image.
[0032] As a further aspect of the present invention: the step of generating multimodal features of the target product image and the candidate image based on the graph structure text information includes:
[0033] Select all entity nodes in the entity set of the candidate images and the target product images in the candidate image set one by one; take the currently selected entity node as the current processing entity;
[0034] Perform a graph structure text information to text structure information conversion on the currently processed entity to obtain the text information of the currently processed entity;
[0035] Determine whether the number of connection edges of the currently processed entity is less than a preset threshold. If the determination result is yes, input the text information of the currently processed entity into the large language model, use the large language model to supplement the knowledge of the text information of the currently processed entity to obtain semantically enhanced text information, and use the semantically enhanced text information to update the text information before enhancement.
[0036] The text information of the currently processed entity is input into the language model of the pre-suspended chain, and the specific tags of the language model are output. The specific tags are weighted and fused to obtain the text embedding vector of the currently processed entity.
[0037] The visual encoder is used to extract the basic visual features of the image corresponding to the current entity being processed. The basic visual features are then input into a visual mapping network, which is used to transform the basic visual features into a visual prefix vector with a known dimension of the text embedding vector.
[0038] The visual prefix vector is concatenated with the text embedding vector to obtain the multimodal features of the currently processed entity.
[0039] As a further aspect of the present invention: the number of connection edges of the currently processed entity is the connection relationship between the currently processed entity and other entities in its corresponding image.
[0040] As a further aspect of the present invention: the step of searching for a candidate image set in a pre-built database based on the classification result of the target product image includes:
[0041] Based on the classification results, retrieve the top-k most similar candidate images from a pre-built database;
[0042] Calculate the cosine similarity between the target image and the Top-k most similar candidate images;
[0043] The candidate image set is obtained by combining visual feature distance and text semantic similarity in a weighted sorting process.
[0044] Secondly, the present invention also provides a furniture product recognition system based on multimodal heterogeneous data processing, the furniture product recognition system based on multimodal heterogeneous data processing comprising:
[0045] The data acquisition module is used to acquire images of the target product to be identified.
[0046] The data preprocessing module is used to preprocess the target product image to obtain the global text features, target text features, and visual features of the target product image.
[0047] The classification module is used to process the global text features, target text features, and visual features using a modality fusion model to obtain the classification result of the target product image;
[0048] The data recognition module is used to search for a candidate image set in a pre-built database based on the classification result of the target product image; wherein, the candidate image set includes at least one candidate image that is most similar to the classification result; and,
[0049] The target product image and the candidate image set are input into a knowledge graph model, and the knowledge graph model is used to perform entity alignment on the target product image and the candidate images;
[0050] The recognition result output module is used to output the recognition result of the target product image based on the entity alignment result of the target product image and the candidate images.
[0051] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the furniture product identification method based on multimodal heterogeneous data processing as described above.
[0052] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the furniture product identification method based on multimodal heterogeneous data processing as described above.
[0053] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the furniture product identification method based on multimodal heterogeneous data processing as described above.
[0054] According to the specific embodiments provided in this application, the following technical effects are disclosed:
[0055] This invention provides a furniture product recognition method and system based on multimodal heterogeneous data processing. First, feature extraction is performed on the target product image to obtain global text features, target text features, and visual features. Second, a modal fusion model is used to process these features, utilizing complementary verification between modal feature data to improve the fine-grained classification results of the image. Next, candidate images are aligned with the target product image to transform the image content into a graph structure of "entity-relationship-attribute." A large language model is used to supplement the knowledge of sparse entities, explicitly modeling the deep logical relationships between furniture components and attributes. This achieves high recognition accuracy even for unpopular styles, new categories, and small sample furniture, improving the model's generalization ability and enabling accurate furniture product recognition and rapid matching. Attached Figure Description
[0056] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0057] Figure 1 This is a flowchart illustrating the furniture product identification method based on multimodal heterogeneous data processing provided by the present invention. Detailed Implementation
[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0059] In one exemplary embodiment, such as Figure 1 As shown, this application provides a furniture product identification method based on multimodal heterogeneous data processing. This method is executed by a computer device, specifically by a terminal or server alone, or by both. In this embodiment, the furniture product identification method based on multimodal heterogeneous data processing includes the following steps S101 to S106. Wherein:
[0060] Step S101: Obtain the target product image, which includes at least one product to be identified. In one application scenario, the target product image is a product image in the furniture industry, specifically including screenshots of online sales channels (such as product detail pages on e-commerce platforms or images displayed on the official websites of home furnishing brands).
[0061] Images disseminated through social media (such as real-life photos of furniture shared by users on WeChat Moments and Xiaohongshu); images provided by users themselves (such as photos taken at offline stores and real-life photos of personal home environments); these images are usually accompanied by unstructured text information (such as product tags, promotional copy, user captions, etc.), providing a data foundation for subsequent text feature extraction.
[0062] Step S102: Preprocess the target product image to obtain the global text features, target text features, and visual features of the target product image.
[0063] Step S103: Use a modal fusion model to process the global text features, target text features, and visual features to obtain the classification result of the target product image.
[0064] Step S104: Based on the classification result of the target product image, search for a candidate image set in the pre-constructed database; wherein, the candidate image set includes at least one candidate image that is closest to the classification result. It should be noted that the target product image and the candidate images in the database are multimodal heterogeneous data. Specifically, the target product image has high randomness and unstructured characteristics, and its sources are wide-ranging (such as e-commerce screenshots, user snapshots, and social media shared images); the candidate images in the database belong to highly structured and standardized enterprise-level data. The two differ significantly in data source, representation form, semantic density, and quality dimensions, constituting a challenge for cross-modal heterogeneous recognition. Based on this, this application breaks down the barriers between heterogeneous data through multimodal feature extraction and knowledge graph entity alignment mechanisms, achieving accurate mapping from unstructured user images to structured ERP inventory products.
[0065] Step S105: Input the target product image and the candidate image set into the knowledge graph model, and use the knowledge graph model to perform entity alignment on the target product image and the candidate images.
[0066] Step S106: Output the recognition result of the product to be identified in the target product image based on the entity alignment result of the target product image and the candidate images; the recognition result is the candidate image with the highest similarity to the target product image in the candidate image set.
[0067] In summary, by implementing steps S101 to S106, this application first extracts features from the target product image to obtain global text features, target text features, and visual features. Secondly, it uses a modal fusion model to process these features, leveraging complementary verification between modal feature data to improve the fine-grained classification results of the image. Next, it aligns the candidate image with the target product image, transforming the image content into a graph structure of "entity-relationship-attribute." A large language model is used to supplement the sparse entities with knowledge, explicitly modeling the deep logical relationships between furniture components and attributes. This achieves high recognition accuracy even for unpopular styles, new categories, and small sample furniture, improving the model's generalization ability and enabling accurate identification and rapid matching of furniture products.
[0068] As one embodiment of this application, step S102, which involves preprocessing the target product image to obtain its global text features, target text features, and visual features, includes:
[0069] Extract text information and the corresponding positional parameter information from the target product image to obtain the global text features of the target product image;
[0070] Extract the location parameters of the target product from the target product image, and generate the target text features based on the location parameters and the global text features;
[0071] Visual extraction is performed on the target product image to obtain its visual features.
[0072] It should be noted that the global text features can be extracted by using an OCR (Optical Character Recognition) model (such as PP-OCRv4, EasyOCR) to detect text regions in the image, simultaneously outputting the text content and bounding box coordinates to form a "text-position" mapping relationship. The target text features can be extracted by using an object detection model (such as YOLOv8, DETR) to locate the main furniture area and obtain its bounding box coordinates; based on the positional parameters of the global text features, text fragments (such as model labels, specifications) located within the furniture area are selected; finally, the target text features are generated by combining positional weights (e.g., higher weight for text in the central area). The visual features can be extracted using the ViT model or the ConvNeXt model.
[0073] As one embodiment of this application, step S103, which involves processing the global text features, target text features, and visual features using a modal fusion model to obtain the classification result of the target product image, includes the following steps S201 to S203. Wherein:
[0074] Step S201: Perform a linear transformation on the global text features, the target text features, and the visual features to obtain an attention matrix of the global text features, the target text features, and the visual features; the attention matrix includes a query vector matrix, a key vector matrix, and a value vector matrix;
[0075] Step S202: Randomly select two features from the global text features, the target text features, and the visual features, and use an attention mechanism to perform weighted fusion on the attention matrices of the two selected features to obtain multiple weighted fused features;
[0076] Step S203: Concatenate multiple weighted fusion features, and process the concatenated features using a classification function to obtain the classification result of the target product image.
[0077] Furthermore, in step S202 of this application embodiment, the step of arbitrarily selecting two features from the global text features, the target text features, and the visual features, and using an attention mechanism to perform weighted fusion on the attention matrices of the selected two features to obtain multiple weighted fused features includes the following steps S301 to S303. Wherein:
[0078] Step S301: Use multiple attention heads to segment the attention matrix of the global text features, the target text features, and the visual features into multiple sub-matrices;
[0079] Step S302: In an attention head, arbitrarily select two features, calculate the similarity between the query vector matrix of one selected feature and the key vector matrix of the other selected feature to generate cross-modal attention weights, and use the cross-modal attention weights to perform weighted aggregation on the latter's value vector matrix to obtain a weighted fusion feature; wherein, the calculation formula of the weighted fusion feature is:
[0080] ;
[0081] In the formula, This indicates a selected feature in the i-th attention head. With another selected feature Weighted fusion characteristics; Represents the features in the i-th attention head The query vector matrix; Represents the features in the i-th attention head The transpose of the key vector matrix; Represents the features in the i-th attention head The value vector matrix; Represents a scalar.
[0082] Step S303: Traverse all attention heads and repeat the weighted fusion step to obtain multiple weighted fusion features.
[0083] In summary, by implementing steps S201 to S203, this application employs a cross-modal multi-head attention mechanism to fuse three types of features. First, text features are used to compensate for visual blind spots, while visual features are used to verify the authenticity of the text description, achieving complementary verification between modalities. Second, a cross-modal attention weight mechanism is used to strengthen key discrimination regions, accurately establishing a deep association between visual details and text features, significantly improving fine-grained classification and discrimination capabilities. Finally, the robustness of the model is enhanced; when the input data lacks information from a certain modality, the system can automatically rely on other modalities to provide redundant support, thereby effectively avoiding classification failure.
[0084] As one embodiment of this application, step S104, which involves searching for a candidate image set in a pre-built database based on the classification result of the target product image, includes:
[0085] First, based on the classification results, the top-k most similar candidate images (k≥1) are retrieved from a pre-constructed database. Second, the cosine similarity between the target image and the top-k most similar candidate images is calculated. Finally, a weighted ranking is achieved by combining visual feature distance (L2 distance) and text semantic similarity (BERT-score) to obtain the candidate image set. The database is a detailed inventory database of goods already entered into the merchant's ERP system, containing structured fields such as SKU code, product name, specifications, official images, and inbound batch number, ensuring that all candidate images represent actual products currently on sale.
[0086] As one embodiment of this application, step S105, which involves inputting the target product image and the candidate image set into a knowledge graph model and using the knowledge graph model to perform entity alignment between the target product image and the candidate images, includes steps S401 to S404. Wherein:
[0087] Step S401: Convert the target product image and all candidate images in the candidate image set into graph structure text information; wherein, the graph structure text information includes entity set, relation set, triple set and image set;
[0088] Step S402: Generate multimodal features of the target product image and the candidate image based on the graph structure text information;
[0089] Step S403: For the target product image and all candidate images in the candidate image set, calculate the cosine similarity between the multimodal features of the target product image and the multimodal features of each candidate image;
[0090] Step S404: Generate a similarity mapping table between the target product image and the candidate image based on the calculation result of the cosine similarity. The similarity mapping table is the entity alignment result between the target product image and the candidate image.
[0091] As a further embodiment of this application, in step S402, the generation of multimodal features of the target product image and the candidate image based on the graph structure text information includes the following steps S501 to S506. Wherein:
[0092] Step S501: Select all entity nodes in the entity set of the candidate images and the target product images in the candidate image set one by one; take the currently selected entity node as the current processing entity;
[0093] Step S502: Perform a conversion from graph structure text information to text structure information on the current processing entity to obtain the text information of the current processing entity;
[0094] Step S503: Determine whether the number of connection edges of the currently processed entity is less than a preset threshold. If the determination result is yes, input the text information of the currently processed entity into the large language model, use the large language model to supplement the text information of the currently processed entity with knowledge to obtain semantically enhanced text information, and use the semantically enhanced text information to update the text information before enhancement; wherein, the number of connection edges of the currently processed entity is the connection relationship between the currently processed entity and other entities in its corresponding image.
[0095] Step S504: Input the text information of the current processing entity into the language model of the pre-suspended chain, output the specific tags of the language model, and perform weighted fusion of the specific tags to obtain the text embedding vector of the current processing entity.
[0096] Step S505: Use a visual encoder to extract the basic visual features of the image corresponding to the currently processed entity, input the basic visual features into a visual mapping network, and use the visual mapping network to convert the basic visual features into a visual prefix vector with a known dimension of the text embedding vector.
[0097] Step S506: Concatenate the visual prefix vector with the text embedding vector to obtain the multimodal features of the currently processed entity.
[0098] In summary, by implementing steps S401 to S404, this application employs a knowledge graph model for entity alignment. First, through structured semantic enhancement, the image content is transformed into a graph structure of "entity-relationship-attribute," thereby explicitly modeling the deep logical relationships between furniture components and attributes. Second, by injecting industry prior knowledge using a large language model (such as an LLM model), the problem of small-sample generalization caused by sparse connection edges for new or niche furniture is effectively solved. Finally, by fusing text embeddings and visual prefix vectors, cross-modal consistency is strengthened, successfully eliminating the mismatch phenomenon that is easily caused by "same style, different color" or simple "angle difference" in traditional pure visual retrieval.
[0099] Based on the same inventive concept, the following describes a furniture product recognition system based on multimodal heterogeneous data processing provided by the present invention. The furniture product recognition system based on multimodal heterogeneous data processing described below can be referred to in correspondence with the furniture product recognition method based on multimodal heterogeneous data processing described above.
[0100] A furniture product recognition system based on multimodal heterogeneous data processing is disclosed. The system includes a data acquisition module, a data preprocessing module, a classification module, a data recognition module, and a recognition result output module. Wherein:
[0101] The data acquisition module is used to acquire images of the target product to be identified.
[0102] The data preprocessing module is used to preprocess the target product image to obtain the global text features, target text features, and visual features of the target product image.
[0103] The classification module is used to process the global text features, target text features, and visual features using a modality fusion model to obtain the classification result of the target product image;
[0104] The data recognition module is used to search for a candidate image set in a pre-built database based on the classification result of the target product image; wherein, the candidate image set includes at least one candidate image that is most similar to the classification result; and,
[0105] The target product image and the candidate image set are input into a knowledge graph model, and the knowledge graph model is used to perform entity alignment on the target product image and the candidate images;
[0106] The recognition result output module is used to output the recognition result of the target product image based on the entity alignment result of the target product image and the candidate images.
[0107] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.
[0108] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0109] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0110] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0111] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).
[0112] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0113] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0114] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A furniture product recognition method based on multimodal heterogeneous data processing, characterized in that, The furniture product identification method based on multimodal heterogeneous data processing includes: Obtain a target product image, wherein the target product image includes at least one product to be identified; The target product image is preprocessed to obtain the global text features, target text features, and visual features of the target product image; The global text features, target text features, and visual features are processed using a modal fusion model to obtain the classification result of the target product image; Based on the classification result of the target product image, a candidate image set is searched in a pre-built database; wherein, the candidate image set includes at least one candidate image that is most similar to the classification result; The target product image and the candidate image set are input into a knowledge graph model, and the knowledge graph model is used to perform entity alignment on the target product image and the candidate images; Based on the entity alignment result between the target product image and the candidate images, the recognition result of the product to be identified in the target product image is output.
2. The furniture product identification method based on multimodal heterogeneous data processing according to claim 1, characterized in that, The preprocessing of the target product image to obtain its global text features, target text features, and visual features includes: Extract text information and the corresponding positional parameter information from the target product image to obtain the global text features of the target product image; Extract the location parameters of the target product from the target product image, and generate the target text features based on the location parameters and the global text features; Visual extraction is performed on the target product image to obtain its visual features.
3. The furniture product identification method based on multimodal heterogeneous data processing according to claim 1, characterized in that, The process of using a modal fusion model to process the global text features, target text features, and visual features to obtain the classification result of the target product image includes: A linear transformation is performed on the global text features, the target text features, and the visual features to obtain an attention matrix of the global text features, the target text features, and the visual features; the attention matrix includes a query vector matrix, a key vector matrix, and a value vector matrix. Two features are randomly selected from the global text features, the target text features, and the visual features. An attention mechanism is used to perform weighted fusion on the attention matrices of the two selected features to obtain multiple weighted fused features. Multiple weighted fusion features are concatenated, and a classification function is used to process the concatenated features to obtain the classification result of the target product image.
4. The furniture product identification method based on multimodal heterogeneous data processing according to claim 3, characterized in that, The step involves arbitrarily selecting two features from the global text features, the target text features, and the visual features, and then using an attention mechanism to perform weighted fusion on the attention matrices of the two selected features to obtain multiple weighted fused features, including: The attention matrix of the global text features, the target text features, and the visual features is divided into multiple sub-matrices using multiple attention heads; In an attention head, two features are arbitrarily selected, and the similarity between the query vector matrix of one selected feature and the key vector matrix of the other selected feature is calculated to generate cross-modal attention weights. The value vector matrix of the latter is then weighted and aggregated using the cross-modal attention weights to obtain a weighted fusion feature. Iterate through all attention heads and repeat the weighted fusion step to obtain multiple weighted fused features.
5. The furniture product identification method based on multimodal heterogeneous data processing according to claim 4, characterized in that, The formula for calculating the weighted fusion feature is as follows: ； In the formula, This indicates a selected feature in the i-th attention head. With another selected feature The weighted fusion characteristics; Represents the features in the i-th attention head The query vector matrix; Represents the features in the i-th attention head The transpose of the key vector matrix; Represents the features in the i-th attention head The value vector matrix; Represents a scalar.
6. The furniture product identification method based on multimodal heterogeneous data processing according to claim 1, characterized in that, The step of inputting the target product image and the candidate image set into a knowledge graph model, and using the knowledge graph model to perform entity alignment between the target product image and the candidate images, includes: The target product image and all candidate images in the candidate image set are converted into graph structure text information; wherein, the graph structure text information includes entity sets, relation sets, triple sets, and image sets; Multimodal features of the target product image and the candidate images are generated based on the graph structure text information; For the target product image and all candidate images in the candidate image set, calculate the cosine similarity between the multimodal features of the target product image and the multimodal features of each candidate image; Based on the calculation result of the cosine similarity, a similarity mapping table is generated between the target product image and the candidate image. The similarity mapping table is the entity alignment result between the target product image and the candidate image.
7. The furniture product identification method based on multimodal heterogeneous data processing according to claim 6, characterized in that, The step of generating multimodal features for the target product image and the candidate images based on the graph structure text information includes: Select all entity nodes in the entity set of the candidate images and the target product images in the candidate image set one by one; take the currently selected entity node as the current processing entity; Perform a graph structure text information to text structure information conversion on the currently processed entity to obtain the text information of the currently processed entity; Determine whether the number of connection edges of the currently processed entity is less than a preset threshold. If the determination result is yes, input the text information of the currently processed entity into the large language model, use the large language model to supplement the knowledge of the text information of the currently processed entity to obtain semantically enhanced text information, and use the semantically enhanced text information to update the text information before enhancement. The text information of the currently processed entity is input into the language model of the pre-suspended chain, and the specific tags of the language model are output. The specific tags are weighted and fused to obtain the text embedding vector of the currently processed entity. The visual encoder is used to extract the basic visual features of the image corresponding to the current entity being processed. The basic visual features are then input into a visual mapping network, which is used to transform the basic visual features into a visual prefix vector with a known dimension of the text embedding vector. The visual prefix vector is concatenated with the text embedding vector to obtain the multimodal features of the currently processed entity.
8. The furniture product identification method based on multimodal heterogeneous data processing according to claim 7, characterized in that, The number of connection edges of the currently processed entity is the connection relationship between the currently processed entity and other entities in its corresponding image.
9. The furniture product identification method based on multimodal heterogeneous data processing according to any one of claims 1-8, characterized in that, The step of searching for a candidate image set in a pre-built database based on the classification results of the target product image includes: Based on the classification results, retrieve the top-k most similar candidate images from a pre-built database; Calculate the cosine similarity between the target image and the Top-k most similar candidate images; The candidate image set is obtained by combining visual feature distance and text semantic similarity in a weighted sorting process.
10. A furniture product recognition system based on multimodal heterogeneous data processing, characterized in that, The furniture product recognition system based on multimodal heterogeneous data processing includes: The data acquisition module is used to acquire images of the target product to be identified. The data preprocessing module is used to preprocess the target product image to obtain the global text features, target text features, and visual features of the target product image. The classification module is used to process the global text features, target text features, and visual features using a modality fusion model to obtain the classification result of the target product image; The data recognition module is used to search for a candidate image set in a pre-built database based on the classification result of the target product image; wherein, the candidate image set includes at least one candidate image that is most similar to the classification result; and, The target product image and the candidate image set are input into a knowledge graph model, and the knowledge graph model is used to perform entity alignment on the target product image and the candidate images; The recognition result output module is used to output the recognition result of the target product image based on the entity alignment result of the target product image and the candidate images.