A commodity information processing and query method and system

Abstract

The present invention proposes a method and system for processing and querying commodity information. It belongs to the field of information processing technology; the method includes: collecting the text information of commodities, preprocessing the text information of commodities through natural language processing information, and constructing a structured commodity text database; based on the high-definition image of the commodity, extracting the key features of the commodity through machine vision, and associating with the information in the text database to form a multimodal commodity information representation; using the Transformer-based BERT model to encode the commodity text and obtain the deep semantic representation of the text; using the pre-trained convolutional neural network to extract features of the commodity image and obtain the high-level visual features of the image; through natural language processing and machine vision technology, the text information and image information of the commodity are structured and multimodally fused, which greatly improves the processing efficiency of commodity information.

Description

Technical Field

[0001] The invention proposes a commodity information processing and query method and system, belonging to the technical field of information processing. Background Art

[0002] In existing e-commerce systems, users mainly search for products through text keywords. However, this method has problems such as inaccurate search results and incomplete coverage when facing complex, novel or difficult to accurately describe products. In particular, when users find it difficult to describe the specific characteristics of the product (such as material, color, style, etc.) with accurate keywords, traditional search methods seem to be unable to cope with it. In addition, the information gap between product images and text also limits the improvement of query efficiency. Summary of the invention

[0003] The present invention provides a commodity information processing and query method and system to solve the problems mentioned in the above background technology:

[0004] The present invention provides a commodity information processing and query method, the method comprising:

[0005] S1. Collect the text information of the products, pre-process the text information of the products through natural language processing information, and build a structured product text database;

[0006] S2, based on the high-definition images of the goods, extract the key features of the goods through machine vision, and associate them with the information in the text database to form a multimodal product information representation;

[0007] S3. Use the Transformer-based BERT model to encode product text and obtain the deep semantic representation of the text; use the pre-trained convolutional neural network to extract features from product images and obtain high-level visual features of the images;

[0008] S4. Through the cross-modal fusion network based on the attention mechanism, the text encoding and the image encoding are fused to generate a unified product representation vector;

[0009] S5. The user inputs a query statement or uploads a product image. The system analyzes the query intent through natural language processing or image recognition technology, extracts keywords or image features, calculates the similarity between the query analysis result and the product representation vector in the multimodal product information database, and uses cosine similarity to evaluate the matching degree.

[0010] S6. Sort the query results according to the similarity scores, filter and re-sort them according to user preferences, and present the results to the user in a list or grid format.

[0011] The present invention provides a commodity information processing and query system, the system comprising:

[0012] Information collection module: collects the text information of commodities, pre-processes the text information of commodities through natural language processing information, and builds a structured commodity text database;

[0013] Representation formation module: Based on the high-definition image of the product, the key features of the product are extracted through machine vision and associated with the information in the text database to form a multimodal product information representation;

[0014] Feature acquisition module: The Transformer-based BERT model is used to encode product text and obtain the deep semantic representation of the text; the pre-trained convolutional neural network is used to extract features from product images and obtain high-level visual features of the images;

[0015] Vector generation module: Through a cross-modal fusion network based on the attention mechanism, text encoding and image encoding are fused to generate a unified product representation vector;

[0016] Degree matching module: When a user inputs a query statement or uploads a product image, the system analyzes the query intent through natural language processing or image recognition technology, extracts keywords or image features, calculates the similarity between the query analysis result and the product representation vector in the multimodal product information database, and uses cosine similarity to evaluate the matching degree;

[0017] Result display module: Sorts query results according to similarity scores, filters and reorders them based on user preferences, and displays the results to users in a list or grid format.

[0018] The beneficial effects of the present invention are as follows: through natural language processing and machine vision technology, the text information and image information of the commodity are structured and multimodally fused, which greatly improves the processing efficiency of the commodity information; the BERT model and convolutional neural network are used to extract deep features of text and image respectively, and the cross-modal fusion network combined with the attention mechanism can more accurately understand and match the user's query intention; the results are sorted and filtered according to user preferences and behavior data, the display interface is dynamically adjusted, and personalized recommendations are provided, which significantly improves the user's query experience; commodity information is collected from multiple channels to ensure the comprehensiveness and diversity of data, which helps to more accurately reflect the characteristics of the commodity; by using technologies such as deep learning and reinforcement learning, the system can continuously learn and optimize to adapt to the needs of different users and market changes; through cross-modal fusion technology, the effective combination of text and image information is realized, and the comprehensiveness and accuracy of information retrieval are improved; the system can dynamically adjust the query strategy and sorting results according to real-time data and user feedback, ensuring the real-time and dynamic nature of the service; the technical solution has good scalability, and can easily access new data sources and integrate new technologies to adapt to changing market needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a step diagram of the method of the present invention;

[0020] Figure 2 This is a system module diagram of the present invention. DETAILED DESCRIPTION

[0021] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0022] One embodiment of the present invention, as Figure 1 As shown, a commodity information processing and query method, the method comprising:

[0023] S1. Collecting the text information of commodities, wherein the text information of commodities includes commodity titles, descriptions and specifications, preprocessing the text information of commodities through natural language processing information, wherein the preprocessing includes word segmentation, part-of-speech tagging and entity recognition; and constructing a structured commodity text database;

[0024] S2, based on the high-definition images of the goods, extract the key features of the goods through machine vision, and associate them with the information in the text database to form a multimodal product information representation;

[0025] S3. Use the Transformer-based BERT model to encode product text and obtain the deep semantic representation of the text; use pre-trained convolutional neural networks (such as ResNet and EfficientNet) to extract features from product images and obtain high-level visual features of the images;

[0026] S4. Through the cross-modal fusion network based on the attention mechanism, the text encoding and the image encoding are fused to generate a unified product representation vector;

[0027] S5. The user inputs a query statement or uploads a product image. The system analyzes the query intent through natural language processing or image recognition technology, extracts keywords or image features, calculates the similarity between the query analysis result and the product representation vector in the multimodal product information database, and uses cosine similarity to evaluate the matching degree.

[0028] S6. Sort the query results according to the similarity score, filter and re-sort them based on user preferences (such as price, brand, rating, etc.), and present the results to the user in a list or grid format.

[0029] The working principle of the above technical solution is as follows: first, collect text information such as product titles, descriptions and specifications from various sources (such as e-commerce platforms, product details pages, etc.); perform word segmentation on the collected text information and decompose sentences into vocabulary units; perform part-of-speech tagging to identify the part of speech of each word (such as nouns, verbs, etc.); and entity recognition to identify key entities in the text (such as brand names, models, etc.). These preprocessing steps are helpful for subsequent structured storage and indexing; the preprocessed text information is organized into a structured form and stored in the database to facilitate subsequent query and retrieval; use machine vision technology to extract key features such as shape, color, texture, etc. from the high-definition images of the product; associate the extracted image features with the information in the text database to form a multimodal product information representation. This representation method not only includes the text description of the product, but also includes visual intuitive information, making the product description more comprehensive and accurate; the Transformer-based BERT model is used to encode the product text. The BERT model can obtain the deep semantic representation of text by deeply understanding the contextual information of the text, which is crucial for understanding the user's query intention and matching related products. It uses pre-trained convolutional neural networks (such as ResNet and EfficientNet) to extract features from product images. These networks can capture high-level visual features in images, such as edges, textures, shapes, etc., providing a basis for subsequent cross-modal fusion. Through the attention mechanism, the text encoding and image encoding are fused to generate a unified product representation vector. The attention mechanism can dynamically adjust the weights of text and image in the fusion process according to the context of the query and the user's preferences, so that the generated product representation vector is more in line with the user's query needs. Users can initiate queries by entering query statements or uploading product images. The system parses the user's query intention through NLP or image recognition technology and extracts keywords or image features. The query parsing results are calculated for similarity with the product representation vector in the multimodal product information database, and the cosine similarity is used to evaluate the degree of matching. Cosine similarity can measure the degree of proximity of two vectors in direction, thereby determining the correlation between the query and the product; sorting the query results according to the similarity score so that the products most relevant to the user's query are at the front; filtering and re-sorting are performed based on the sorting in combination with the user's preferences (such as price, brand, rating, etc.) to meet the user's personalized needs; the sorted results are displayed to the user in a list or grid format for easy browsing and selection.

[0030] The effects of the above technical solution are as follows: through natural language processing and machine vision technology, the text and image information of the product are preprocessed and feature extracted respectively, and a multimodal product information representation is constructed. This multimodal representation method can capture the characteristics of the product more comprehensively, thereby improving the accuracy of the query; using the BERT model to obtain the deep semantic representation of the text, and the convolutional neural network to extract the advanced visual features of the image, further enhances the richness and depth of the information, making the query results more accurate; users can initiate queries by entering query statements or uploading product pictures, and the system can intelligently analyze the user's query intentions, whether it is a text query or an image query, it can get a satisfactory response, greatly improving the user experience; when displaying the results, not only are they sorted according to the similarity score, but they are also filtered and re-sorted in combination with user preferences to ensure that the user first sees the products that best meet their needs and preferences, further enhancing user satisfaction; building a structured product text database makes information storage and retrieval more efficient. Through indexing and query optimization technology, product information related to user queries can be quickly located; the use of cross-modal fusion networks enables seamless integration of text and image information, avoiding the limitations that may be brought about by single-modal information, thereby improving the overall information retrieval efficiency; filtering and reordering based on user preferences make query results more personalized. The system can intelligently recommend products that meet the user's interests and needs based on historical behavior, purchase records and other information, thereby improving conversion rates and user stickiness; this technical solution has strong scalability and flexibility. With the continuous emergence of new technologies and new methods, new models or algorithms can be easily integrated into existing systems to further improve the performance and functionality of the system; at the same time, due to the use of modular design ideas, each module is relatively independent and can be customized and modified according to actual needs to meet application requirements in different scenarios.

[0031] In one embodiment of the present invention, the S1 includes:

[0032] S11. Collecting text information of commodities from multiple channels, wherein the multiple channels include e-commerce platforms, commodity manufacturers or third-party data sources; the text information of the commodities includes commodity titles, descriptions and specifications;

[0033] S12. Use natural language processing libraries (such as Jieba and NLTK) to segment the product text, divide the continuous text into vocabulary units with independent meanings, and mark the part of speech (such as noun, verb, adjective, etc.) for each word after segmentation;

[0034] S13. Identify the key entities in the text. The key entities include brand names, product models, and materials, and remove the noise information in the text. The noise information includes meaningless words (such as "de", "le", etc.), advertising words, and repeated words.

[0035] S14. Store the preprocessed commodity text information in a unified data structure (such as JSON, XML) to build a structured commodity text database.

[0036] The working principle of the above technical solution is as follows: Collect the text information of commodities from multiple channels such as e-commerce platforms, commodity manufacturers, or third-party data sources. These channels provide rich commodity data, including key information such as commodity titles, descriptions, and specifications. During the collection process, the system will perform preliminary data verification to ensure the integrity, accuracy, and timeliness of the data. For missing or ambiguous information, the system will mark it and may trigger a manual review process or request additional information from the data source. This step is crucial for subsequent data processing and analysis because it ensures the basis of data quality. Use natural language processing libraries (such as jieba, NLTK, etc.) to perform word segmentation on the collected commodity text. Word segmentation is the process of splitting continuous text into lexical units with independent meanings, and it is one of the basic tasks in natural language processing. Through word segmentation, the system can transform complex text information into a more easily processed sequence of words. Based on word segmentation, the system annotates the part-of-speech of each word (such as noun, verb, adjective, etc.). Part-of-speech tagging helps the system understand the grammatical role and semantic function of words in the text and provides important clues for subsequent information extraction and text analysis. The system identifies the key entities in the text, and these key entities include brand names, product models, and materials, etc. Entity recognition is an important task in natural language processing, which can help the system extract information units with specific meanings from the text. At the same time as entity recognition, the system also removes the noise information in the text, including meaningless words (such as auxiliary words and function words like "de", "le", etc.), advertising words, and repeated words. Noise removal helps reduce data redundancy and interference and improve the efficiency and accuracy of subsequent information processing. Store the preprocessed commodity text information in a unified data structure (such as JSON, XML, etc.). The unified data structure helps to standardize and normalize the data, facilitating subsequent data management and query. Based on the preprocessed commodity text information, build a structured commodity text database. The structured database can clearly represent the relationships and attributes between commodities, supporting efficient query and retrieval operations. At the same time, the database also provides the function of data persistence to ensure the security and reliability of the data.

[0037] The effects of the above technical solution are: by collecting commodity text information from multiple channels and ensuring the integrity, accuracy and timeliness of the data, the problem of missing or outdated information is effectively avoided. Manual review or request for supplement of missing or ambiguous information further improves the data quality and provides a reliable basis for subsequent data processing and analysis; using the natural language processing library to segment the commodity text and mark each word with its part of speech helps the system to understand the text content more deeply. Word segmentation and part of speech tagging are basic tasks in natural language processing, which provide important support for subsequent information extraction, text classification and other tasks; identifying key entities in the text and removing noise information, such as meaningless words, advertising words and repeated words. This process helps to extract the core information in the text, reduce data redundancy and interference, and improve the efficiency and accuracy of information processing; storing the preprocessed commodity text information in a unified data structure (such as JSON, XML) to achieve data standardization and normalization. The unified data structure facilitates data sharing, exchange and management, and reduces the complexity and cost of data processing; building a structured commodity text database can clearly represent the relationship and attributes between commodities and support efficient query and retrieval operations. The structured database also provides data persistence, ensuring data security and reliability; the preprocessed and structured storage of product text information provides high-quality data support for subsequent product information query, recommendation, analysis and other tasks. These tasks can achieve more accurate and efficient processing and decision-making based on accurate and complete product text information; through automated preprocessing and structured storage processes, the need for manual intervention is reduced, reducing labor costs and time costs. At the same time, the automated processing process also reduces the possibility of human errors and improves overall work efficiency and quality; based on a high-quality, structured product text database, the system can more accurately understand the user's query intent and return product information that is highly matched with user needs. This helps to improve the user's query experience and increase user satisfaction and loyalty; combined with the user's historical behavior and preference information, the system can make personalized recommendations based on the structured product text database. By recommending products that meet the user's interests and needs, the system can further improve the user's shopping experience and satisfaction.

[0038] In one embodiment of the present invention, the S2 includes:

[0039] S21. Download high-definition images of products from product detail pages or resources provided by merchants, and preprocess the images, including resizing, denoising, contrast enhancement, etc., and use pre-trained machine vision models (such as ResNet and EfficientNet) to extract features from the pre-processed product images; focus on visual features such as shape, texture, and color of the products;

[0040] S22, identifying the main body of the product in the image through an object detection algorithm, and eliminating background interference;

[0041] S23: Associating the extracted image features with the information in the text database, and establishing a corresponding relationship through a unique identifier (such as a product ID).

[0042] S24, integrating text and image features to form a multimodal product information representation, which includes both the text description of the product and its visual features.

[0043] The working principle of the above technical solution is as follows: download high-definition images of products from product detail pages or resources provided by merchants; perform a series of preprocessing operations on the downloaded images to improve image quality and the accuracy of subsequent processing. These preprocessing steps include resizing (ensuring that the image meets the requirements of the processing model), denoising (reducing noise interference in the image), enhancing contrast (improving the clarity and detail expression of the image), etc.; use pre-trained machine vision models (such as ResNet, EfficientNet, etc.) to extract features from the pre-processed product images. These models are trained with a large amount of image data and can automatically identify and extract key features in images, such as visual features such as the shape, texture, and color of the product; identify the main body of the product in the image through object detection algorithms (such as target detection models based on deep learning). These algorithms can automatically locate the product area in the image and separate it from the background; on the basis of identifying the main body of the product, further process the image to eliminate interference factors in the background, such as cluttered background objects, text labels, etc., to ensure that the extracted product image features are purer and more accurate; associate the extracted image features with the information in the text database through a unique identifier (such as the product ID). This association method ensures the accuracy and consistency between image and text information, and provides a basis for subsequent multimodal information fusion; the extracted image features are deeply integrated with the information in the text database to form a multimodal product information representation. This representation method not only includes the text description of the product (such as title, description, specifications, etc.), but also includes the visual features of the product (such as shape, texture, color, etc.), thereby achieving a comprehensive and accurate expression of product information; multimodal product information representation can be applied to a variety of scenarios, such as product recommendation, search ranking, user experience optimization, etc. By integrating text and image features, the system can understand user needs more accurately and provide a more personalized and intelligent service experience.

[0044] The effects of the above technical solution are: by downloading high-definition product images and preprocessing them, the image quality is ensured, providing a reliable basis for subsequent feature extraction and recognition; by using pre-trained machine vision models for feature extraction, the key visual features of the product such as shape, texture, and color can be accurately captured, making the expression of product information more comprehensive and accurate; the application of object detection algorithms enables the system to automatically identify the main body of the product in the image and effectively eliminate background interference. This greatly improves the accuracy of product recognition and reduces the possibility of misidentification and missed recognition; by establishing a correspondence between image features and text information through a unique identifier (such as product ID), the accuracy and consistency of product information are ensured; the extracted image features are integrated with the information in the text database to form a multimodal product information representation. This representation method not only includes the text description of the product, but also integrates visual features, making the expression of product information richer and more three-dimensional; multimodal information fusion helps to improve the system's understanding and analysis capabilities of product information, and provides more accurate and intelligent support for subsequent tasks such as product recommendation and search sorting; when browsing products, users can fully understand the product information through the text description and high-definition images displayed at the same time, which improves the convenience and satisfaction of shopping; accurate product identification and recommendation systems can provide more personalized and accurate shopping suggestions based on user preferences and needs, further improving user experience; for merchants, multimodal product information representation helps them to have a more comprehensive understanding of the market performance and consumer needs of products, so as to formulate more scientific and reasonable sales strategies and marketing plans; at the same time, based on the analysis of multimodal information, merchants can also discover potential product associations and consumer behavior patterns, providing strong support for product optimization and inventory management.

[0045] In one embodiment of the present invention, S3 includes:

[0046] S31. Based on the Transformer architecture, select a pre-trained model for product text processing (such as XLNet, ELECTRA, etc.); use a specific corpus in the product field to perform domain adaptation training on the pre-trained model;

[0047] S32, through the multi-task learning framework, the model is trained to perform multiple related tasks (such as named entity recognition, sentiment analysis, summary generation) at the same time, using a hierarchical Transformer structure, with the bottom layer Transformer processing word-level information, the middle layer processing phrase or clause-level information, and the high layer capturing the global semantics of the entire text;

[0048] S33. In each layer of Transformer, a self-attention mechanism is used to capture the dependencies within the text. For cross-modal applications, a cross-attention mechanism is used to make the text encoding pay attention to the image features associated with it.

[0049] S34. Integrate structured information from external knowledge bases (such as commodity knowledge graphs) into text encoding, transform knowledge into embeddable vectors through knowledge representation learning techniques (such as TransE and RotatE), and integrate them with text encoding;

[0050] S35. Use adversarial training or noise enhancement techniques to perturb the text encoding, and introduce dropout or dropout variants (such as ScheduledDropPath) to prevent overfitting.

[0051] S36. Through the deep convolutional network architecture (such as EfficientNetV2, RegNet, etc.), the feature pyramid network (FPN) or multi-scale feature fusion mechanism is used to extract information from feature maps of different scales.

[0052] The working principle of the above technical solution is as follows: select pre-trained models suitable for processing commodity texts (such as XLNet, ELECTRA, etc.), which are based on the Transformer architecture and have strong text representation capabilities. Then, use the specific corpus of the commodity field to perform domain adaptation training on the pre-trained model. This step enables the model to understand the commodity text more accurately by allowing it to learn commodity-related terms, contexts, and semantic relationships; through a multi-task learning framework, the model is trained to perform multiple tasks related to commodity text processing (such as named entity recognition, sentiment analysis, and summary generation) at the same time. This training method helps to enhance the generalization ability of the model and the depth of text representation. A hierarchical Transformer structure is adopted, in which the bottom layer Transformer processes information at the word level, the middle layer processes information at the phrase or clause level, and the high layer captures the global semantics of the entire text. This hierarchical structure enables the model to gradually build a deep representation of the text; in each layer of Transformer, a self-attention mechanism is used to capture the dependencies within the text. The self-attention mechanism allows the model to consider information at all positions in the text while processing the text, so as to understand the context of the text more accurately. For cross-modal applications (such as the combination of product images and text), the cross-attention mechanism enables the text encoding to pay attention to the image features associated with it. This mechanism helps to achieve effective fusion between text and image; integrate structured information in external knowledge bases (such as product knowledge graphs) into text encoding. Through knowledge representation learning techniques (such as TransE, RotatE), knowledge is converted into embeddable vectors and fused with text encoding. This step enables the model to use external knowledge to enrich text representation and improve the ability to understand product text; use adversarial training or noise enhancement technology to perturb the text encoding to improve the model's robustness to noise and generalization ability. Adversarial training trains the model by introducing noise samples with different distributions from the training data, so that it can maintain stable performance when facing unknown or noisy data. At the same time, dropout or dropout variants (such as ScheduledDropPath) are introduced to prevent overfitting. The dropout technology reduces the model's dependence on specific features by randomly discarding the output of some network nodes during training, thereby improving the model's generalization ability; although it mainly focuses on text processing, step S36 mentions processing images or image and text fusion tasks (if applicable) through deep convolutional network architectures (such as EfficientNetV2, RegNet, etc.) and multi-scale feature fusion mechanisms. These network architectures use feature pyramid networks (FPN) or multi-scale feature fusion mechanisms to extract information from feature maps of different scales. In cross-modal applications, this helps to effectively fuse key information in the image with text encoding, thereby further improving the overall performance of the model.

[0053] The effects of the above technical solution are as follows: by using a specific corpus in the commodity field to perform domain adaptation training on the pre-trained model based on the Transformer architecture (such as XLNet, ELECTRA), the model can better understand the terms, contexts and semantic relationships related to the commodity; this significantly improves the accuracy and professionalism of the model when processing commodity texts, making the model perform better in scenarios such as commodity recommendation and search optimization; through a multi-task learning framework, the model is trained to perform multiple related tasks (such as named entity recognition, sentiment analysis, summary generation) at the same time, and a hierarchical Transformer structure is used to process text information at different levels; this training method not only enhances the generalization ability of the model, but also enables the model to understand the text more deeply and capture richer semantic information. At the same time, multi-task learning helps to improve the overall performance and efficiency of the model and reduce the risk of overfitting; the self-attention mechanism is used in each layer of Transformer to capture the dependencies within the text, and the cross-attention mechanism is used to achieve the fusion of text and image features in cross-modal applications; this helps the model to better understand the logical relationships and semantic structures within the text, while achieving effective fusion of multimodal information such as text and images, providing strong support for cross-modal applications; the structured information in the external knowledge base (such as the product knowledge graph) is integrated into the text encoding, and the knowledge is converted into embeddable vectors and fused with the text encoding through knowledge representation learning technology; this further enriches the text representation ability of the model, enabling the model to use external knowledge to enhance its understanding of the text. At the same time, the application of knowledge representation learning technology also helps to improve the accuracy and robustness of the model; adversarial training or noise enhancement technology is used to perturb the text encoding to improve the robustness and generalization ability of the model to noise, and dropout or dropout variants (such as ScheduledDropPath) are introduced to prevent overfitting; adversarial training helps the model maintain stable performance when facing unknown or noisy data, while the application of dropout technology helps to reduce the model's dependence on specific features and improve the model's generalization ability. These measures work together to improve the stability and reliability of the model; although the focus is mainly on text processing, the application of deep convolutional network architectures (such as EfficientNetV2, RegNet, etc.) and feature pyramid networks (FPN) or multi-scale feature fusion mechanisms (in cross-modal scenarios) helps to extract information from feature maps of different scales; this helps to achieve deep fusion of image and text information in cross-modal applications and improve the overall performance of the model. At the same time, the application of deep convolutional networks also helps to capture key features in images and provide richer contextual information for text processing.

[0054] In one embodiment of the present invention, the S4 includes:

[0055] S41. In the feature extraction stage of text and image, preliminary cross-modal interaction is carried out through the cross-attention mechanism, so that the two can influence each other at the feature level;

[0056] S42. After feature extraction is completed, by constructing a multimodal interaction module (such as TensorFusionNetwork, Low-rankMultimodalFusion, etc.), the features of text and image are fused at a deeper level to generate a more comprehensive cross-modal feature representation;

[0057] S43, performing fusion at the decision layer or output layer, combining the feature vectors of the text and the image (such as weighted sum, concatenation, bilinear pooling, etc.) according to specific task requirements to form a final unified vector;

[0058] S44. Use complementarity detection algorithms (such as mutual information analysis, inter-modal consistency assessment, etc.) to automatically identify complementary and redundant information in text and images, and give different weights in the fusion process;

[0059] S45. The cross-modal information is stored in a high-dimensional vector space, and a structured representation method (such as knowledge graph embedding, graph neural network, etc.) is used to embed the cross-modal information into the vector in the form of a graph structure or a tree structure.

[0060] S46, using a calibration algorithm (such as an alignment algorithm, a normalization technique, etc.) to calibrate and optimize the fused vector, and based on a multimodal loss function, comprehensively consider performance indicators between text and image, wherein the performance indicators include semantic consistency, visual similarity, and performance indicators of tasks such as cross-modal retrieval or classification, to supervise and optimize the generation process of the fused vector;

[0061] S47. In the process of generating unified vectors, a context-aware mechanism is introduced to enable the model to dynamically adjust the fusion mode and weight distribution of text and image features according to specific query or task requirements;

[0062] S48. Use reinforcement learning or meta-learning methods to continuously optimize the fusion strategy and vector generation process based on historical data and real-time feedback.

[0063] The working principle of the above technical solution is as follows: in the feature extraction stage of text and image, preliminary cross-modal interaction is achieved through the cross-attention mechanism. This step aims to make text and image interact with each other at the feature level, laying the foundation for subsequent deep fusion. The cross-attention mechanism can capture the correlation information between text and image and enhance the correlation between the two; after the feature extraction is completed, a multimodal interaction module (such as TensorFusionNetwork, Low-rankMultimodalFusion, etc.) is constructed to fuse the features of text and image at a deeper level. These modules can handle complex features from different modalities and generate more comprehensive cross-modal feature representations through specific fusion strategies (such as tensor fusion, low-rank fusion, etc.); at the decision layer or output layer, the feature vectors of text and image are combined according to specific task requirements. The combination method can be weighted sum, splicing, bilinear pooling, etc., the purpose is to form a final unified vector that can simultaneously reflect the information of text and image; using complementary detection algorithms (such as mutual information analysis, inter-modal consistency evaluation, etc.), the complementary and redundant information in text and image is automatically identified. For the detected complementary features, reinforcement learning or attention enhancement technology is used to give them more attention and reinforcement in the fusion process. This step helps to improve the comprehensiveness and accuracy of the fusion vector; the cross-modal information is stored through a high-dimensional vector space, and the dimension and structure of the vector are carefully designed to fully express the complex semantic and visual features of the text and image. Using structured representation methods (such as knowledge graph embedding, graph neural network, etc.), the cross-modal information is embedded in the vector in the form of a graph structure or a tree structure to enhance the expressiveness and interpretability of the vector; the fused vector is calibrated and optimized using calibration algorithms (such as alignment algorithms, normalization techniques, etc.). This step aims to eliminate the deviation and noise caused by modality differences or errors in the feature extraction process. At the same time, based on the multimodal loss function, the performance indicators between text and image (such as semantic consistency, visual similarity, performance of cross-modal retrieval or classification tasks, etc.) are comprehensively considered to supervise and optimize the generation process of the fusion vector; in the process of generating the unified vector, a context-aware mechanism is introduced. This mechanism enables the model to dynamically adjust the fusion mode and weight distribution of text and image features according to specific query or task requirements. This dynamic adjustment capability helps the model better adapt to different application scenarios and task requirements; using reinforcement learning or meta-learning methods, the fusion strategy and vector generation process are continuously optimized based on historical data and real-time feedback. This step is an iterative optimization process, which enables the model to achieve a higher performance level in cross-modal information processing through continuous learning and adjustment.

[0064] The effect of the above technical solution is: preliminary cross-modal interaction is carried out in the feature extraction stage through the cross-attention mechanism, so that text and image can influence each other at the feature level, which helps to capture richer cross-modal correlation information. This early interaction helps the model better understand the intrinsic connection between different modalities; multimodal interaction modules such as TensorFusionNetwork (TFN) and Low-rankMultimodalFusion (LMF) are used to perform feature fusion at a deeper level. These modules can comprehensively consider inter-modal and intra-modal features to generate more comprehensive and comprehensive cross-modal feature representations and improve the representation ability of the model; according to specific task requirements, the feature vectors of text and images are combined in various ways such as weighted sum, splicing, and bilinear pooling to form the final unified vector. This flexibility enables the model to adapt to different application scenarios and task requirements; the complementary and redundant information in text and images is automatically identified through the complementarity detection algorithm, and different weights are given during the fusion process. For the detected complementary features, reinforcement learning or attention enhancement technology is used to enhance them, which helps to improve the comprehensiveness and accuracy of the fusion vector; high-dimensional vector space is used to store cross-modal information, and the dimensions and structure of the vector are carefully designed to fully express the complex semantics and visual features of text and images. Using structured representation methods such as knowledge graph embedding and graph neural network, cross-modal information is embedded in the vector in the form of graph structure or tree structure, which helps to improve the richness and interpretability of information; the fused vector is calibrated and optimized through the calibration algorithm to eliminate the deviation and noise caused by modal differences or errors in the feature extraction process. At the same time, based on the multimodal loss function, the performance indicators between text and image (such as semantic consistency, visual similarity, cross-modal retrieval or classification task performance, etc.) are comprehensively considered, and the generation process of the fusion vector is supervised and optimized to ensure the high quality of the fusion vector; the introduction of context-aware mechanism enables the model to dynamically adjust the fusion mode and weight distribution of text and image features according to specific query or task requirements. This mechanism helps improve the flexibility and adaptability of the model, enabling it to perform well in different scenarios; reinforcement learning or meta-learning methods are used to continuously optimize the fusion strategy and vector generation process based on historical data and real-time feedback. This continuous optimization mechanism helps the model continuously adapt to new data and task requirements, maintaining its competitiveness and advancement.

[0065] In one embodiment of the present invention, the S45 includes:

[0066] S451. According to the feature complexity of text and image and the task requirements, the dimension of the vector is dynamically adjusted, and the principal component analysis (PCA) is used to reduce the dimension of the initially expanded vector space;

[0067] S452. Design a multi-level vector structure, where each level corresponds to a different semantic or visual level, such as word level, sentence level, image region level, etc., and dynamically allocate different dimensions of the vector according to the importance of different elements in the text and image;

[0068] S453. Build a knowledge graph based on text and image content, including entities, relationships, attributes and other elements. Use graph embedding algorithms such as TransE and RotatE to map the elements in the knowledge graph into a high-dimensional vector space and maintain the semantic and relationship information in the graph structure.

[0069] S454, establish cross-modal links between the knowledge graphs of text and images, represent the features of text and images as nodes in the graph structure, connect related nodes through edges, and form a complex graph network;

[0070] S455. Utilize the message passing mechanism in GNN to aggregate the information of neighbor nodes and update the representation of the current node to perform deep fusion of cross-modal information. Introduce the attention mechanism in GNN to perform weighted processing on the information of different nodes and edges.

[0071] The working principle of the above technical solution is as follows: dynamically adjust the dimension of the vector according to the feature complexity of the text and image and the task requirements. This ensures that the vector space has sufficient capacity to capture rich cross-modal information. Then, principal component analysis (PCA) is used to reduce the dimension of the initially expanded vector space. PCA optimizes the representation efficiency of the vector space by identifying the principal components in the data (i.e., the direction of maximum variance), removing redundant information, and retaining key features; designing a multi-level vector structure, each layer corresponds to a different semantic or visual level (such as word level, sentence level, image area level, etc.). Dynamically allocate different dimensions of the vector according to the importance of different elements in the text and image. This design ensures that important information has sufficient representation space, while improving the hierarchy and interpretability of the vector representation; construct a knowledge graph based on the text and image content, including elements such as entities, relationships, and attributes. Then, graph embedding algorithms such as TransE and RotatE are used to map the elements in the knowledge graph into a high-dimensional vector space. These algorithms preserve the semantic and relational information in the graph structure so that the vector representation can reflect the complex relationship between entities. They establish cross-modal links between the knowledge graphs of text and images to achieve the correspondence between text entities and image regions. This link enhances the fusion of cross-modal information and enables the features of text and images to be represented in a unified graph structure. The features of text and images are represented as nodes in the graph structure, and related nodes are connected by edges to form a complex graph network. This graph network structure helps capture the complex associations and interactions between cross-modalities. The message passing mechanism in the graph neural network (GNN) is used to aggregate the information of neighboring nodes and update the representation of the current node. This process achieves deep fusion of cross-modal information. The introduction of the attention mechanism in GNN can weight the information of different nodes and edges, so that important information is given more attention in the fusion process. By updating the node representation multiple times, GNN can capture the global information in the graph structure and generate a more comprehensive and accurate cross-modal feature representation.

[0072] The effects of the above technical solution are as follows: dynamically adjusting the dimension of the vector according to the feature complexity of the text and image and the task requirements, ensuring that the vector space can capture rich cross-modal information without introducing excessive computational complexity and storage costs due to excessive dimensionality; using principal component analysis (PCA) to reduce the dimensionality of the initially expanded vector space, effectively removing redundant information, retaining key features, and improving the efficiency and accuracy of vector representation; designing a multi-level vector structure, each layer corresponding to a different semantic or visual level (such as word level, sentence level, image area level, etc.). This hierarchical representation method enables the vector to better reflect the complexity and hierarchy of text and image content; according to the text and image The importance of different elements in the text is dynamically allocated to different dimensions of the vector, ensuring that important information has sufficient representation space, which improves the pertinence and effectiveness of the vector representation; by constructing a knowledge graph and using graph embedding algorithms such as TransE and RotatE to map the elements in the knowledge graph to a high-dimensional vector space, the semantic and relational information in the graph structure is effectively maintained, making the vector representation more semantically meaningful; the graph embedding algorithm enables the information in text and images to be represented in a unified vector space, providing a basis for subsequent cross-modal information fusion; a cross-modal link is established between the knowledge graphs of text and images, realizing the correspondence between text entities and image regions, and enhancing the fusion of cross-modal information. This fusion helps capture the deep association and interaction between text and images; the features of text and images are represented as nodes in the graph structure, and related nodes are connected by edges to form a complex graph network. This graph network structure can more comprehensively reflect the complex relationship and information interaction between text and images; the message passing mechanism in the graph neural network (GNN) is used to aggregate the information of neighbor nodes and update the representation of the current node, realizing the deep fusion of cross-modal information. This deep fusion helps to explore the potential associations and complementary information between text and images; the introduction of the attention mechanism in GNN weights the information of different nodes and edges, so that important information gets more attention in the fusion process, improving the accuracy and effectiveness of the fusion results.

[0073] In one embodiment of the present invention, the S455 includes:

[0074] For text nodes, their word embedding, sentence embedding or paragraph embedding is extracted; for image nodes, their regional features or global features are extracted; the extracted features are used as the initial representation of the node and stored in a high-dimensional vector;

[0075] Clarify the connection relationship between nodes in the graph, including the semantic relationship between text nodes, the spatial relationship between image nodes, and the cross-modal link between text and image, and aggregate the information from neighbor nodes through information aggregation functions such as summation, averaging, and maximum pooling;

[0076] According to the adjacency relationship and aggregation rules, the information of neighbor nodes is passed to the current node to form a new node representation; based on nonlinear activation functions (such as ReLU, Tanh, etc.), the aggregated information is nonlinearly transformed;

[0077] The result of the nonlinear transformation is fused with the original representation of the current node, and the state of the node is updated to reflect the new information. The attention weight of each neighbor node to the current node is calculated by the attention score function, considering factors such as the similarity and importance between nodes; the attention weight is calculated by the following formula:

[0078]

[0079] Among them, h i is the representation of the current node, h j is the representation of neighbor nodes, W is the learnable weight matrix, b is the bias term, α ij is the attention weight of the current node i to the neighboring node j, where α is a very small constant, N(i) represents the set of neighboring nodes of node i; LeakyReLU is a nonlinear activation function, LeakyReLU(x)=max(αx,x);

[0080] The softmax function is used to normalize the attention scores. According to the normalized attention weights, the information of neighbor nodes is weighted and summed to form a weighted information aggregation result.

[0081] According to task requirements and data characteristics, the parameters of the attention mechanism, such as similarity measurement method and weight calculation function, are dynamically adjusted to adapt to different cross-modal fusion scenarios; the weighted information aggregation results are fused with the original representation of the current node to form an attention-enhanced node representation;

[0082] In the process of multiple iterations, the attention mechanism is continuously used to update and optimize the node representation, and multiple rounds of information propagation are set up. In each round, information is exchanged and updated between nodes, gradually deepening the degree of cross-modal information fusion;

[0083] By monitoring the change in node representation or the convergence of the loss function, we can determine whether the information propagation process has reached a stable state. Based on the cross-modal consistency loss function, we encourage the distance between text nodes and corresponding image nodes in the representation space to be as close as possible.

[0084] During the training process, the parameters and node representations in the graph network are optimized by minimizing the consistency loss function. The fusion results are comprehensively evaluated through evaluation indicators such as semantic consistency, visual similarity, and cross-modal retrieval accuracy. Based on the evaluation results and real-time user feedback, the parameters of the fusion strategy and attention mechanism are dynamically adjusted to continuously optimize the deep fusion process of cross-modal information.

[0085] The working principle of the above technical solution is: extract word embedding, sentence embedding or paragraph embedding, which are usually obtained through pre-trained models (such as Word2Vec, BERT, etc.) and can capture the semantic information of the text; extract regional features or global features, which can be automatically learned from image data through models such as convolutional neural networks (CNN) and can reflect the visual content of the image. The extracted features are stored in high-dimensional vectors as the initial representation of the nodes to prepare for subsequent information transfer; first clarify the connection relationship between the nodes in the graph, including the semantic relationship between text nodes, the spatial relationship between image nodes, and the cross-modal link between text and image; aggregate the information from neighbor nodes through information aggregation functions (such as summation, average, maximum pooling, etc.). This step is one of the core operations in GNN, which enables each node to perceive and integrate the information of its neighbor nodes; based on the similarity and importance between nodes, the attention weight of each neighbor node to the current node is calculated through the attention score function; the softmax function is used to normalize the attention score to ensure that the sum of the weights of all neighbor nodes is 1; the information of neighbor nodes is weighted and summed according to the normalized attention weight to form a weighted information aggregation result. This step allows important neighbor nodes to obtain greater weights in the process of information transmission; the aggregated information is nonlinearly transformed (such as through nonlinear activation functions such as ReLU and Tanh) to enhance the expression ability of the model; the result of the nonlinear transformation is fused with the original representation of the current node, and the state of the node is updated to reflect the new information; multiple rounds of information propagation process are set, and each round exchanges and updates information between nodes, gradually deepening the degree of cross-modal information fusion; by monitoring the change in node representation or the convergence of the loss function, it is judged whether the information propagation process has reached a stable state; based on the cross-modal consistency loss function, the distance between text nodes and corresponding image nodes in the representation space is encouraged to be as close as possible. The parameters and node representations in the graph network are optimized by minimizing the consistency loss function to enhance the consistency of cross-modal information. The fusion results are comprehensively evaluated through evaluation indicators such as semantic consistency, visual similarity, and cross-modal retrieval accuracy. The parameters of the fusion strategy and attention mechanism are dynamically adjusted according to the evaluation results and real-time user feedback to continuously optimize the deep fusion process of cross-modal information.

[0086] The effects of the above technical solution are as follows: through multiple rounds of information propagation and node representation update, the degree of cross-modal information fusion is gradually deepened, so that the information between text and image can be fully interacted and integrated; the attention mechanism is introduced, so that important neighbor nodes obtain greater weights in the process of information transmission, thereby improving the accuracy and pertinence of the fusion results; the aggregated information is transformed based on the nonlinear activation function, the nonlinear expression ability of the model is enhanced, so that it can better capture complex cross-modal relationships; by extracting different levels of features of text and image (such as word embedding, sentence embedding, regional features, etc.) as the initial representation of the node, and constructing a multi-level vector structure, the model can more comprehensively represent and understand cross-modal information; dynamically adjust the parameters of the attention mechanism (such as similarity measurement method, weight calculation function, etc.) according to task requirements and data characteristics, and The fusion strategy and node representation method enable the model to adapt to different cross-modal fusion scenarios; the model is continuously optimized in combination with real-time user feedback, which improves the practicality of the model and user satisfaction; by monitoring the change in node representation or the convergence of the loss function, it can accurately determine whether the information propagation process has reached a stable state, thereby avoiding the problem of overfitting or underfitting; training based on the cross-modal consistency loss function can encourage the distance between text nodes and corresponding image nodes in the representation space to be as close as possible, thereby enhancing the consistency of cross-modal information; the fusion results are comprehensively evaluated through evaluation indicators such as semantic consistency, visual similarity, and cross-modal retrieval accuracy to ensure that the model performs well in many aspects; the parameters of the fusion strategy and attention mechanism are continuously adjusted and optimized according to the evaluation results and real-time user feedback to ensure that the model can continue to progress and improve. The above formula calculates the attention weights between nodes, so that the model can identify and emphasize more important relationships in the graph structure, which helps to improve the richness and accuracy of node representation; the attention mechanism allows the model to dynamically adjust the weights according to the representation of the current node and its neighboring nodes, which means that the model can learn adaptively according to different parts of the graph and different information propagation stages; in the graph structure, some neighboring nodes may be more important than other nodes. Attention weight calculation helps the model highlight these key nodes, so that it pays more attention to them during the information aggregation process. In graph data, there may be noise or irrelevant information. The attention mechanism can help the model reduce the impact of this information and focus on useful information by giving higher weights to important neighbor nodes. Attention weight can be interpreted as a measure of the importance of the relationship between nodes.This provides an intuitive basis for understanding model decisions and increases the interpretability of the model. For graphs of different sizes, the attention mechanism can automatically adjust the weights to adapt to the number of neighbors of the node, so that the model can process graph data of different sizes. In cross-modal graph structures, the attention mechanism helps to effectively integrate information from different modalities such as text and images, and enhances the interaction of cross-modal information by calculating the attention weights between nodes of different modalities. Compared with directly learning weights for each node pair, the attention mechanism calculates the attention weights by sharing the weight matrix, which reduces the number of model parameters and helps prevent overfitting. The attention mechanism can reduce unnecessary calculations in some cases by focusing only on important neighbor nodes, thereby improving the computational efficiency of the model. In many tasks, graph neural network models that introduce attention mechanisms can usually achieve better performance than traditional graph neural networks, especially in tasks that need to handle complex relationships and dependencies.

[0087] In one embodiment of the present invention, S5 includes:

[0088] S51. Use natural language processing technology (such as dependency parsing and semantic role labeling) to deeply analyze query statements and extract their semantic structure and key information. Use machine learning models (such as BERT, RoBERTa and other pre-trained models) to identify the intent and context of the query and distinguish different types of needs such as information retrieval, question answering, and recommendation.

[0089] S52. For complex queries involving multiple modalities such as text, images, and voice, multimodal fusion technology is used to unify information from different modalities, identify the associations and dependencies between them, and automatically expand related words, phrases, or concepts based on the semantics and context of the query;

[0090] S53, identifying and removing noise, irrelevant or redundant information in the query, and optimizing the query statement according to the index structure and query performance characteristics of the database or search engine;

[0091] S54. Use multiple text similarity calculation methods such as cosine similarity, Jaccard similarity, BM25 algorithm, etc. to evaluate the text content similarity between the query and the document or candidate items, use deep learning models (such as convolutional neural networks, twin networks) to extract image features, and calculate the Euclidean distance and cosine similarity between feature vectors to evaluate the similarity of image content; among them, the Euclidean distance is calculated by the following formula:

[0092]

[0093] Where D(A,B) is the Euclidean distance between vector A and vector B, A ξ and B ξare the values ​​of vector A and vector B in the ξth dimension, respectively, and n is the number of dimensions of the vector;

[0094] The cosine similarity is obtained by the following formula:

[0095]

[0096] Among them, S(A,B) is the cosine similarity between vector A and vector B, ranging from [-1, 1], 1 means the vectors are in exactly the same direction, -1 means they are in completely opposite directions, and 0 means they intersect, that is, they are unrelated. ξ and B ξ are the values ​​of vector A and vector B in the ξth dimension, and are the sum of the squares of the values ​​of each dimension of vector A and vector B respectively.

[0097] S55. Through the cross-modal similarity measurement method, the similarity calculation results of text and image are effectively integrated, and the information of both is comprehensively considered to evaluate the overall similarity. Based on the context information of the query and candidate items (such as time, location, user preference, etc.), the weight distribution in the similarity calculation is dynamically adjusted;

[0098] S56, based on user feedback, query history or real-time data, dynamically adjust the weight parameters in the similarity calculation using reinforcement learning or online learning algorithms;

[0099] S57. Effectively integrate the multi-dimensional similarity calculation results, and use weighted average, voting mechanism, ranking learning and other methods to generate the final similarity score.

[0100] The working principle of the above technical solution is: through natural language processing technology (such as dependency syntax analysis, semantic role labeling), the input query statement is deeply analyzed to extract its semantic structure and key information. Then, the machine learning model (such as BERT, RoBERTa and other pre-trained models) is used to identify the intent and contextual meaning of the query, and distinguish whether the query belongs to different types of needs such as information retrieval, question answering, recommendation, etc.; for complex queries containing multiple modalities such as text, image, voice, etc., multimodal fusion technology is used to unify the information of different modalities and identify the association and dependency between them. Based on the semantics and context of the query, relevant words, phrases or concepts are automatically expanded to enrich the semantic representation of the query; noise, irrelevant or redundant information in the query is identified and removed to reduce the impact of these factors on the query results. At the same time, according to the index structure and query performance characteristics of the database or search engine, the query statement is optimized to improve the query efficiency and accuracy; a variety of text similarity calculation methods (such as cosine similarity, Jaccard similarity, BM25 algorithm) are used to evaluate the text content similarity between the query and the document or candidate. For image content, deep learning models (such as convolutional neural networks and twin networks) are used to extract image features, and the Euclidean distance and cosine similarity between feature vectors are calculated to evaluate the similarity of image content; the similarity calculation results of text and image are effectively integrated through cross-modal similarity measurement methods, and the overall similarity is evaluated by comprehensively considering the information of both. At the same time, based on the contextual information of the query and candidate items (such as time, location, user preferences, etc.), the weight distribution in the similarity calculation is dynamically adjusted to more accurately reflect the actual needs of users; based on user feedback, query history or real-time data, the weight parameters in the similarity calculation are dynamically adjusted using reinforcement learning or online learning algorithms. This mechanism enables the system to continuously learn and optimize to adapt to the query needs of different users and different scenarios; the multi-dimensional similarity calculation results are effectively integrated, and weighted average, voting mechanism, ranking learning and other methods can be used to generate the final similarity score. This score will serve as an important basis for sorting and recommending candidate results, providing users with the most relevant and valuable query results.

[0101] The effect of the above technical solution is: through natural language processing technology and machine learning models, query statements are deeply analyzed and intent is identified. This can accurately understand the user's query needs and distinguish different types of query intentions (such as information retrieval, question and answer, recommendation, etc.), thereby improving the pertinence and accuracy of the query; using multimodal fusion technology, information in multiple modes such as text, image, and voice are uniformly represented, and the associations and dependencies between them are identified. This helps to more comprehensively understand the user's query needs and automatically expand related words, phrases or concepts, thereby increasing the relevance and richness of the query results; identifying and removing noise, irrelevant or redundant information in the query can reduce the impact of this information on the query results. By optimizing query statements, query efficiency can be improved and unnecessary computing resource consumption can be reduced; a variety of text similarity calculation methods (such as cosine similarity, Jaccard similarity, BM25 algorithm) and deep learning models (such as convolutional neural networks, twin networks) are used to evaluate the similarity between queries and documents or candidates. These methods can more accurately measure the similarity of text and image content, thereby improving the accuracy and relevance of query results; through cross-modal similarity measurement methods, the similarity calculation results of text and image are effectively integrated. This method comprehensively considers the information of text and image, and can more comprehensively evaluate the overall similarity between query and candidate items, thereby further improving the accuracy and relevance of query results; based on user feedback, query history or real-time data, the weight parameters in the similarity calculation are dynamically adjusted using reinforcement learning or online learning algorithms. This method can continuously optimize the query model according to user behavior and real-time data to make it more in line with the actual needs and preferences of users; the multi-dimensional similarity calculation results are effectively integrated to generate the final similarity score. Through weighted average, voting mechanism, ranking learning and other methods, the similarity information of different dimensions can be comprehensively considered to generate more comprehensive and accurate query results. The above Euclidean distance calculation formula can accurately quantify the straight-line distance between two vectors in space, which is very useful for judging the degree of difference between vectors. In fields such as image processing and speech recognition, it can be used to identify the distance between different features for classification or clustering; cosine similarity can accurately quantify the directional similarity of two vectors without being affected by the length of the vector. This is particularly useful in text analysis and recommendation systems because it can measure the directional consistency between documents or user preferences. The calculation results of Euclidean distance are intuitive and easy to understand. The smaller the distance, the more similar the two vectors are. The value range of cosine similarity is between [-1,1], which makes the interpretation of similarity more intuitive. Close to 1 means high similarity, and close to -1 means high dissimilarity. Both formulas are universal metrics that can be applied to vector spaces of any dimension and are not limited to specific data types or application scenarios.These two metrics can be extended to more complex models as needed. For example, in multimodal fusion technology, the similarity calculation results of text and image can be effectively fused. The calculation of Euclidean distance is simple and efficient, and is suitable for large-scale data processing. Cosine similarity can more accurately reflect the similarity of vectors when processing high-dimensional data, especially when the lengths of feature vectors vary greatly, because it is not affected by the length of the vector. In information retrieval, these two metrics can help determine the similarity between queries and documents, thereby improving the accuracy and efficiency of retrieval. In recommendation systems, they can be used to measure the similarity between users or items, and then make personalized recommendations. Because these metrics are based on the mathematical properties of vectors, they are robust to noise and outliers in the data.

[0102] In one embodiment of the present invention, S6 includes:

[0103] S61. Comprehensively sort the query results based on multiple factors, including:

[0104] Relevance sorting: Performs a preliminary sort based on the similarity score between the query and the results, ensuring that the most relevant results are listed first.

[0105] Importance sorting: Consider the importance, authority, freshness and other factors of the results and adjust the sorting results. For example, give priority to news released by authoritative media and recently updated product information.

[0106] User behavior ranking: Analyze user preferences and interests based on historical click, browse, purchase and other behavior data, and make personalized adjustments to the ranking results. Use ranking learning algorithms (such as LambdaRank, Learning to Rank, etc.) and a large amount of training data to combine different factors (such as relevance, importance, user behavior, etc.) to optimize the ranking results. And through model fusion technology (such as Stacking, Blending, etc.), the prediction results of multiple ranking models are integrated;

[0107] S62, monitor system performance and user behavior data in real time, handle abnormal situations, and use online learning algorithms to dynamically adjust the parameters and strategies of the sorting model based on real-time data feedback;

[0108] S63. Filter irrelevant results based on keywords or phrases in the query, and use natural language processing technology and semantic analysis to identify and filter out content that does not match the query intent;

[0109] S64, filtering out results that are not relevant to the user's area based on the user's geographic location information, and filtering out outdated or future content based on the time attribute of the query (such as news timeliness, event date, etc.);

[0110] S65. Build user profiles, including user interests, preferences, purchase history and other information, to provide a basis for personalized filtering. Use collaborative filtering technology to find other users or groups with similar interests to the user and recommend content they like.

[0111] S66. Dynamically adjust the layout and style of the display interface according to the user's device type (such as mobile phone, tablet, computer) and screen size, and adjust the elements of the interface according to the user's interest preferences and current query topic, wherein the elements include color, font, and icon;

[0112] S67. Recommend content that users may be interested in based on their real-time behaviors (such as clicks, browsing, dwell time, etc.), use machine learning models to predict users' future needs and behaviors, and recommend relevant content in advance.

[0113] The working principle of the above technical solution is as follows: through relevance sorting, the similarity between the query results and the query is used as the preliminary sorting basis to ensure that the most relevant results are displayed first. Subsequently, the importance sorting considers factors such as the importance, authority and freshness of the results, and further adjusts the sorting results to ensure that users see information that is both valuable and timely. Finally, the user behavior sorting uses the user's historical behavior data, and through the sorting learning algorithm and model fusion technology, it combines factors such as relevance, importance and user behavior to optimize the sorting results and achieve personalized sorting; the system monitors the performance and user behavior data in real time, and immediately handles any abnormalities (such as model performance degradation, user behavior abnormalities, etc.). At the same time, the online learning algorithm is used to dynamically adjust the parameters and strategies of the sorting model based on real-time data feedback to ensure that the model can continuously adapt to changing data and user behavior and maintain optimal performance; the system uses natural language processing technology and semantic analysis based on the keywords or phrases in the query to identify and filter out content that does not match the query intent. This step helps to reduce noise interference and improve the accuracy of query results and user satisfaction; the system further filters the results based on the user's geographic location information and the time attribute of the query. Geographical filtering ensures that users only see content related to their area; time filtering filters out outdated or future content based on factors such as the timeliness of news and the date of activities, ensuring that users see the latest and most relevant information; the system builds user portraits, including information such as user interests, preferences, and purchase history, to provide a basis for personalized filtering. At the same time, collaborative filtering technology is used to discover other users or groups with similar interests to users and recommend content they like. This recommendation method based on the similarity of user groups helps to discover users' potential points of interest and improve the accuracy and diversity of recommendations; the system dynamically adjusts the layout and style of the display interface according to the user's device type and screen size to ensure a good user experience on different devices. At the same time, the color, font, icon and other elements of the interface are adjusted according to the user's interest preferences and the current query topic to enhance the attractiveness of the interface and interactivity with users; the system predicts the content that users may be interested in based on their real-time behavior (such as clicks, browsing, stay time, etc.) and recommends relevant content in advance. This process uses machine learning models to learn and analyze users' historical behavior data to predict their future needs and behaviors. Real-time recommendations help improve user engagement and satisfaction and promote further dissemination and sharing of content.

[0114] The effects of the above technical solutions are as follows: by sorting multiple factors such as relevance, importance and user behavior, it is ensured that users first see the most relevant, important and interesting content, which significantly improves the accuracy and efficiency of queries; by building user portraits and using collaborative filtering technology, it can accurately recommend content that meets user interests, enhance user stickiness and satisfaction; by real-time monitoring of system performance and user behavior data, it can promptly discover and handle abnormal situations, ensure stable operation of the system, and improve query response speed; by using online learning algorithms, the parameters and strategies of the sorting model are dynamically adjusted according to real-time data feedback, and the sorting results are continuously optimized to make them more in line with user needs; by using natural language processing technology and semantic analysis, content that does not match the query intent is effectively filtered out, and the purity and usefulness of the query results are improved; according to the user's geographic location and query time attributes, irrelevant or outdated content is filtered out, ensuring that the information obtained by the user is timely and geographically targeted; the layout and style of the display interface are dynamically adjusted according to the user's device type and screen size, and the interface elements are adjusted according to the user's interest preferences and query topics, providing a more comfortable and personalized browsing experience; using machine learning models to predict users' future needs and behaviors, and recommending relevant content in advance, further enhancing the system's intelligence level and user experience.

[0115] One embodiment of the present invention, as Figure 2 As shown, a commodity information processing and query system, the system comprises:

[0116] Information collection module: collects the text information of commodities, including commodity titles, descriptions and specifications, pre-processes the commodity text information through natural language processing information, including word segmentation, part-of-speech tagging and entity recognition; and builds a structured commodity text database;

[0117] Representation formation module: Based on the high-definition image of the product, the key features of the product are extracted through machine vision and associated with the information in the text database to form a multimodal product information representation;

[0118] Feature acquisition module: Use the Transformer-based BERT model to encode product text and obtain the deep semantic representation of the text; use pre-trained convolutional neural networks (such as ResNet and EfficientNet) to extract features from product images and obtain high-level visual features of the images;

[0119] Vector generation module: Through a cross-modal fusion network based on the attention mechanism, text encoding and image encoding are fused to generate a unified product representation vector;

[0120] Degree matching module: When a user inputs a query statement or uploads a product image, the system analyzes the query intent through natural language processing or image recognition technology, extracts keywords or image features, calculates the similarity between the query analysis result and the product representation vector in the multimodal product information database, and uses cosine similarity to evaluate the matching degree;

[0121] Result display module: Sort the query results according to the similarity score, filter and re-sort them based on user preferences (such as price, brand, rating, etc.), and display the results to the user in a list or grid format.

[0122] The working principle of the above technical solution is as follows: first, collect text information such as product titles, descriptions and specifications from various sources (such as e-commerce platforms, product details pages, etc.); perform word segmentation on the collected text information and decompose sentences into vocabulary units; perform part-of-speech tagging to identify the part of speech of each word (such as nouns, verbs, etc.); and entity recognition to identify key entities in the text (such as brand names, models, etc.). These preprocessing steps are helpful for subsequent structured storage and indexing; the preprocessed text information is organized into a structured form and stored in the database to facilitate subsequent query and retrieval; use machine vision technology to extract key features such as shape, color, texture, etc. from the high-definition images of the product; associate the extracted image features with the information in the text database to form a multimodal product information representation. This representation method not only includes the text description of the product, but also includes visual intuitive information, making the product description more comprehensive and accurate; the Transformer-based BERT model is used to encode the product text. The BERT model can obtain the deep semantic representation of text by deeply understanding the contextual information of the text, which is crucial for understanding the user's query intention and matching related products. It uses pre-trained convolutional neural networks (such as ResNet and EfficientNet) to extract features from product images. These networks can capture high-level visual features in images, such as edges, textures, shapes, etc., providing a basis for subsequent cross-modal fusion. Through the attention mechanism, the text encoding and image encoding are fused to generate a unified product representation vector. The attention mechanism can dynamically adjust the weights of text and image in the fusion process according to the context of the query and the user's preferences, so that the generated product representation vector is more in line with the user's query needs. Users can initiate queries by entering query statements or uploading product images. The system parses the user's query intention through NLP or image recognition technology and extracts keywords or image features. The query parsing results are calculated for similarity with the product representation vector in the multimodal product information database, and the cosine similarity is used to evaluate the degree of matching. Cosine similarity can measure the degree of proximity of two vectors in direction, thereby determining the correlation between the query and the product; sorting the query results according to the similarity score so that the products most relevant to the user's query are at the front; filtering and re-sorting are performed based on the sorting in combination with the user's preferences (such as price, brand, rating, etc.) to meet the user's personalized needs; the sorted results are displayed to the user in a list or grid format for easy browsing and selection.

[0123] The effects of the above technical solution are as follows: through natural language processing and machine vision technology, the text and image information of the product are preprocessed and feature extracted respectively, and a multimodal product information representation is constructed. This multimodal representation method can capture the characteristics of the product more comprehensively, thereby improving the accuracy of the query; using the BERT model to obtain the deep semantic representation of the text, and the convolutional neural network to extract the advanced visual features of the image, further enhances the richness and depth of the information, making the query results more accurate; users can initiate queries by entering query statements or uploading product pictures, and the system can intelligently analyze the user's query intentions, whether it is a text query or an image query, it can get a satisfactory response, greatly improving the user experience; when displaying the results, not only are they sorted according to the similarity score, but they are also filtered and re-sorted in combination with user preferences to ensure that the user first sees the products that best meet their needs and preferences, further enhancing user satisfaction; building a structured product text database makes information storage and retrieval more efficient. Through indexing and query optimization technology, product information related to user queries can be quickly located; the use of cross-modal fusion networks enables seamless integration of text and image information, avoiding the limitations that may be brought about by single-modal information, thereby improving the overall information retrieval efficiency; filtering and reordering based on user preferences make query results more personalized. The system can intelligently recommend products that meet the user's interests and needs based on historical behavior, purchase records and other information, thereby improving conversion rates and user stickiness; this technical solution has strong scalability and flexibility. With the continuous emergence of new technologies and new methods, new models or algorithms can be easily integrated into existing systems to further improve the performance and functionality of the system; at the same time, due to the use of modular design ideas, each module is relatively independent and can be customized and modified according to actual needs to meet application requirements in different scenarios.

[0124] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A commodity information processing and query method, characterized in that: The method comprises: S1. Collect the text information of the products, pre-process the text information of the products through natural language processing information, and build a structured product text database; S2, based on the high-definition images of the goods, extract the key features of the goods through machine vision, and associate them with the information in the text database to form a multimodal product information representation; S3. Use the Transformer-based BERT model to encode product text and obtain the deep semantic representation of the text; use the pre-trained convolutional neural network to extract features from product images and obtain high-level visual features of the images; S4. Through the cross-modal fusion network based on the attention mechanism, the text encoding and the image encoding are fused to generate a unified product representation vector; S5. The user inputs a query statement or uploads a product image. The system analyzes the query intent through natural language processing or image recognition technology, extracts keywords or image features, calculates the similarity between the query analysis result and the product representation vector in the multimodal product information database, and uses cosine similarity to evaluate the matching degree. S6. Sort the query results according to the similarity scores, filter and re-sort them according to the user's preferences, and present the results to the user in a list or grid format; The S4 comprises: S41. In the feature extraction stage of text and image, preliminary cross-modal interaction is carried out through the cross-attention mechanism, so that the two can influence each other at the feature level; S42. After feature extraction is completed, the features of text and image are integrated at a deeper level by building a multimodal interaction module to generate a more comprehensive cross-modal feature representation; S43, performing fusion at the decision layer or output layer, combining the feature vectors of the text and the image according to the specific task requirements to form a final unified vector; S44, using the complementarity detection algorithm, automatically identify the complementary and redundant information in the text and image, and give different weights in the fusion process; S45, storing the cross-modal information in a high-dimensional vector space, and using a structured representation method to embed the cross-modal information into the vector in the form of a graph structure or a tree structure; S46, using a calibration algorithm to calibrate and optimize the fused vector, and based on a multimodal loss function, comprehensively considering the performance indicators between text and image, to supervise and optimize the generation process of the fused vector; S47. In the process of generating unified vectors, a context-aware mechanism is introduced to enable the model to dynamically adjust the fusion mode and weight distribution of text and image features according to specific query or task requirements; S48. Use reinforcement learning or meta-learning methods to continuously optimize the fusion strategy and vector generation process based on historical data and real-time feedback; The S45 comprises: S451. According to the feature complexity of text and image and the task requirements, the dimension of the vector is dynamically adjusted, and the principal component analysis is used to reduce the dimension of the initially expanded vector space; S452. Design a multi-level vector structure, where each level corresponds to a different semantic or visual level, and dynamically allocate different dimensions of the vector according to the importance of different elements in the text and image; S453. Build a knowledge graph based on text and image content, use a graph embedding algorithm to map the elements in the knowledge graph into a high-dimensional vector space, and maintain the semantic and relational information in the graph structure; S454, establish cross-modal links between the knowledge graphs of text and images, represent the features of text and images as nodes in the graph structure, connect related nodes through edges, and form a complex graph network; S455. Utilize the message passing mechanism in GNN to aggregate the information of neighbor nodes and update the representation of the current node to perform deep fusion of cross-modal information. Introduce the attention mechanism in GNN to perform weighted processing on the information of different nodes and edges.

2. The commodity information processing and query method according to claim 1, characterized in that: Said S1 comprises: S11. Collecting text information of commodities from multiple channels, wherein the multiple channels include e-commerce platforms, commodity manufacturers or third-party data sources; the text information of the commodities includes commodity titles, descriptions and specifications; S12. Use the natural language processing library to perform word segmentation on the product text, divide the continuous text into vocabulary units with independent meanings, and mark the part of speech for each word after word segmentation; S13, identifying key entities in the text, wherein the key entities include brand name, product model, and material, and removing noise information in the text; S14: storing the preprocessed commodity text information according to a unified data structure to construct a structured commodity text database.

3. The commodity information processing and query method according to claim 1, characterized in that: The S2 comprises: S21, downloading a high-definition image of the product from the product details page or resources provided by the merchant, preprocessing the image, and extracting features from the preprocessed product image using a pre-trained machine vision model; S22, identifying the main body of the product in the image through an object detection algorithm, and eliminating background interference; S23, associating the extracted image features with the information in the text database, and establishing a corresponding relationship through a unique identifier; S24. Integrate text and image features to form a multimodal product information representation.

4. The commodity information processing and query method according to claim 1, characterized in that: The S3 includes: S31. Based on the Transformer architecture, select a pre-trained model for product text processing; use a specific corpus in the product field to perform domain adaptation training on the pre-trained model; S32, through the multi-task learning framework, the model is trained to perform multiple related tasks at the same time, using a hierarchical Transformer structure, where the bottom layer Transformer processes word-level information, the middle layer processes phrase or clause-level information, and the high layer captures the global semantics of the entire text; S33. In each layer of Transformer, a self-attention mechanism is used to capture the dependencies within the text. For cross-modal applications, a cross-attention mechanism is used to make the text encoding pay attention to the image features associated with it. S34, Integrate the structured information in the external knowledge base into the text encoding, transform the knowledge into an embeddable vector through knowledge representation learning technology, and integrate it with the text encoding; S35. Use adversarial training or noise enhancement techniques to perturb the text encoding, and introduce dropout or dropout variants to prevent overfitting. S36. Through the deep convolutional network architecture, feature pyramid network or multi-scale feature fusion mechanism is adopted to extract information from feature maps of different scales.

5. The commodity information processing and query method according to claim 1, characterized in that: The S455 comprises: For text nodes, their word embedding, sentence embedding or paragraph embedding is extracted; for image nodes, their regional features or global features are extracted; the extracted features are used as the initial representation of the node and stored in a high-dimensional vector; Clarify the connection relationship between nodes in the graph, and aggregate information from neighboring nodes through information aggregation function; According to the adjacency relationship and aggregation rules, the information of neighbor nodes is passed to the current node to form a new node representation; based on the nonlinear activation function, the aggregated information is nonlinearly transformed; The result of the nonlinear transformation is merged with the original representation of the current node, the node state is updated, and the attention weight of each neighbor node to the current node is calculated through the attention score function; The softmax function is used to normalize the attention scores. According to the normalized attention weights, the information of neighbor nodes is weighted and summed to form a weighted information aggregation result. According to the task requirements and data characteristics, the parameters of the attention mechanism are dynamically adjusted to fuse the weighted information aggregation results with the original representation of the current node to form an attention-enhanced node representation; In the process of multiple iterations, the attention mechanism is continuously used to update and optimize the node representation, and multiple rounds of information propagation are set up. In each round, information is exchanged and updated between nodes, gradually deepening the degree of cross-modal information fusion; By monitoring the change in node representation or the convergence of the loss function, we can determine whether the information propagation process has reached a stable state. Based on the cross-modal consistency loss function, we encourage the distance between text nodes and corresponding image nodes in the representation space to be as close as possible. During the training process, the parameters and node representations in the graph network are optimized by minimizing the consistency loss function. The fusion results are comprehensively evaluated through evaluation indicators. According to the evaluation results and real-time user feedback, the parameters of the fusion strategy and attention mechanism are dynamically adjusted to continuously optimize the deep fusion process of cross-modal information.

6. The commodity information processing and query method according to claim 1, characterized in that: The S5 comprises: S51. Use natural language processing technology to deeply analyze query statements and extract their semantic structure and key information; use machine learning models to identify the intent and contextual meaning of the query; S52. Use multimodal fusion technology to unify information from different modalities, identify the associations and dependencies between them, and automatically expand related words, phrases or concepts based on the semantics and context of the query; S53, identifying and removing noise, irrelevant or redundant information in the query, and optimizing the query statement according to the index structure and query performance characteristics of the database or search engine; S54, using a variety of text similarity calculation methods to evaluate the text content similarity between the query and the document or candidate items, using a deep learning model to extract image features, and calculating the Euclidean distance and cosine similarity between feature vectors to evaluate the similarity of image content; S55. By using a cross-modal similarity measurement method, the similarity calculation results of text and image are effectively integrated, and the weight distribution in the similarity calculation is dynamically adjusted based on the context information of the query and the candidate items; S56, based on user feedback, query history or real-time data, dynamically adjust the weight parameters in the similarity calculation using reinforcement learning or online learning algorithms; S57, effectively integrating the multi-dimensional similarity calculation results to generate a final similarity score.

7. The commodity information processing and query method according to claim 1, characterized in that: The S6 comprises: S61. Comprehensively sort the query results by multiple factors, use the sorting learning algorithm, combine a large amount of training data, combine different factors, optimize the sorting results, and use the model fusion technology to integrate the prediction results of multiple sorting models; S62, monitor system performance and user behavior data in real time, handle abnormal situations, and use online learning algorithms to dynamically adjust the parameters and strategies of the sorting model based on real-time data feedback; S63. Filter irrelevant results based on keywords or phrases in the query, and use natural language processing technology and semantic analysis to identify and filter out content that does not match the query intent; S64, filtering out results that are not relevant to the user's area based on the user's geographic location information, and filtering out outdated or future content based on the time attribute of the query; S65. Build user profiles and use collaborative filtering technology to find other users or groups with similar interests to the user and recommend content they like; S66. Dynamically adjust the layout and style of the display interface according to the user's device type and screen size, and adjust the elements of the interface according to the user's interest preferences and current query topic; S67. Recommend content that users are interested in based on their real-time behavior, use machine learning models to predict users' future needs and behaviors, and recommend relevant content in advance.

8. A system for implementing the commodity information processing and query method according to claim 1, characterized in that: The system comprises: Information collection module: collects the text information of commodities, pre-processes the text information of commodities through natural language processing information, and builds a structured commodity text database; Representation formation module: Based on the high-definition image of the product, the key features of the product are extracted through machine vision and associated with the information in the text database to form a multimodal product information representation; Feature acquisition module: The Transformer-based BERT model is used to encode product text and obtain the deep semantic representation of the text; the pre-trained convolutional neural network is used to extract features from product images and obtain high-level visual features of the images; Vector generation module: Through a cross-modal fusion network based on the attention mechanism, text encoding and image encoding are fused to generate a unified product representation vector; Degree matching module: When a user enters a query statement or uploads a product image, the system analyzes the query intent through natural language processing or image recognition technology, extracts keywords or image features, calculates the similarity between the query analysis result and the product representation vector in the multimodal product information database, and uses cosine similarity to evaluate the matching degree; Result display module: Sorts query results according to similarity scores, filters and reorders them based on user preferences, and displays the results to users in a list or grid format.

Images