Customs transaction big data multi-dimensional processing and intelligent search recall system and real-time analysis method
Through large language model and embedding technology, customs transaction data is mapped to vector space, and customs data attribute comparison is combined with milvus database, which solves the multi-dimensional accuracy problem of customs transaction data recall and achieves high-precision recall effect.
Patent Information
- Application Number
- CN202510612781.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-13
- Publication Date
- 2025-08-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, the single dimension based on the traditional recall method leads to low accuracy. The method based on representation learning requires a large amount of interactive data and has poor training effect in small samples. The pure large language model indicates insufficient learning effect in short text tasks, resulting in inaccurate recall of customs transaction data.
The product description of customs transaction data is expanded through a large language model, combined with embedding characterization technology and milvus database, customs product data is mapped to vector space, vector cosine similarity is calculated, and customs data attribute comparison is carried out to improve recall accuracy.
It realizes multi-dimensional high-precision recall of customs transaction data, solves the problem of single dimensions of traditional recall methods and insufficient training of small samples representing learning methods, and improves the accuracy and matching accuracy of recalls.
Smart Images
Figure CN120509900A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of multi-dimensional processing and intelligent search and recall of big data, and specifically to a multi-dimensional processing and intelligent search and recall system for customs transaction big data and a real-time analysis method. Background Art
[0002] In recent years, with the development of deep learning (DNN) technology, the recall technology in recommendation systems has also been significantly improved. Recall technology has continuously evolved from traditional recall methods (content-base, user-item) to representation learning methods (embedding recall), including DNN methods (Youtube DNN, DSSM, airbnb embedding), graph representation learning methods (deepwalk, EGES, SDNE, GraphSAGE), and multi-interest modeling methods (Mind, SDE). Traditional recall methods use user attributes to match item information to achieve recall under the same attributes. Recall methods based on representation learning use user feedback data and can convert different forms of data into vector features in the same space, thereby using vector similarity matching for effective expansion and data recall. The rapid development of multimodal large language models (LLMs) has provided new support for vector construction and feature engineering. The training corpus of LLMs covers a variety of data, with general knowledge and domain knowledge of various professions, thereby expanding the richness of the recall corpus and improving the recall effect;
[0003] The existing technical solutions have the following problems:
[0004] 1. Traditional recall methods use a single dimension and are difficult to match. Traditional recall methods match user attributes with product information and recall a single attribute. Although this method yields more recall results, the accuracy is low when the dimension is single. Traditional recall methods use attributes from multiple dimensions, but the more attributes a method uses, the fewer recall results it produces, which cannot meet the user data volume requirements.
[0005] 2. Representation learning-based methods require a large amount of user interaction data for training. Representation learning methods are trained based on the interaction between users and items. Deep learning model training requires a large amount of dynamic interaction data. Representation learning has poor training effects on small samples with less exchange between users and items.
[0006] 3. Pure large language models have poor representation learning performance for short-text tasks. While LLMs can embed customs product data, they train on general knowledge from long texts and are insufficient for vector representation of short texts. When using LLMs directly for recall, input overload (long text / multiple fields) introduces significant noise, and the model tends to rely on statistical commonalities rather than personalized business needs, resulting in recommendation results that deviate from actual business scenarios. Summary of the Invention
[0007] In order to solve the above technical problems, the present invention provides a customs transaction big data multi-dimensional processing and intelligent search and recall system and real-time analysis method. By expanding product descriptions through a large language model using user needs and customs data, the model training corpus is enriched. At the same time, the system combines embedding representation technology and the Milvus database. By pre-training and mapping customs product data into a vector space, when recalling user needs, the user needs are also mapped into the vector space. By calculating the cosine similarity of the vectors, it is possible to efficiently provide high-similarity options. Furthermore, by further comparing customs data attributes, the accuracy of search and recall is improved, thereby solving the above problems.
[0008] A customs transaction big data multi-dimensional processing and intelligent search and recall system and real-time analysis method, including: a data collection module, a product core word expansion module, a text representation index module and a data post-processing module;
[0009] The data collection module includes a data access unit and a data preparation unit. The data access unit receives the product demand words input by the user by acquiring customs transaction data. The data preparation unit extracts the required data by classifying the transaction data and user demand.
[0010] The product core word expansion module is connected to the data collection module and includes a core word extraction unit and a scenario expansion unit. The core word extraction unit extracts product core words from the data by using a large model, and the scenario expansion unit obtains product extensibility descriptions by inputting product core words into the large model.
[0011] The text representation index module is connected to the product core word expansion module data, including a vectorization unit and an indexing unit. The vectorization unit obtains low-dimensional vector data by embedding the product extensibility description, and the indexing unit calls the milvus database to process the low-dimensional vector to obtain accurate keywords.
[0012] The data post-processing module is connected to the text representation index module data, and improves the matching accuracy by further querying other attributes of customs transaction commodities and matching them with user needs.
[0013] Preferably, the data access unit obtains the commodity description, brand and batch, transaction date and transaction company information in the transaction data by accessing the customs transaction data, and also receives the product demand words given by the user, including product name, product application scenario and product type. The information includes data format, data type and data collection time.
[0014] Preferably, the data preparation unit standardizes the product description information by removing redundant information, unifying the format and data segmentation, and deletes the product description information by using regular expressions, removes modifiers in the product description, retains the core keywords, and divides the core keywords into the first category of core words and the second category of subdivided words according to the complexity of the information to build a core keyword library.
[0015] Preferably, the core word extraction unit parses the core keyword library by using a lightweight model, and extracts the core words of the product from the core keyword library by building a customized task combined with an existing corpus in the customs field, such as a commodity classification rule table. The core words of the product include the product name, product purpose and product code.
[0016] Preferably, the scenario expansion unit expands the product description by acquiring product core words and combining them with the large model, and expands the product description based on the product words required by the user. The product description information includes the product demand scenario, demand industry, usage method, and product use purpose. By inputting the core words into a model that excels at understanding long texts, optimizing the model input prompt words, and allowing the large model to generate corresponding answers based on the product demand scenario, demand industry, usage method, and product use purpose, the answers generated by the large model are further questioned based on the specific product type and the corresponding product corpus, so that the product description information output by the model is consistent with the product core words.
[0017] Preferably, the vectorization unit uses a word segmentation tool to segment the commodity demand description of the paragraph text generated by the large model into word combinations, adds domain labels to the generated short word combinations, removes redundant information, and generates vector samples by traversing the text by setting a sampling window. The vector generation samples are converted into vector space using the CBOW model. First, surrounding words are extracted from the vector generation samples by setting a fixed sliding window, and unique IDs are assigned to the surrounding words. These surrounding words are organized to generate a vocabulary, and each surrounding word is represented as a sparse vector equal to the vocabulary size V, with only the corresponding index position being 1. The surrounding word codes are input into the model, and each code is converted into a dense vector through a weight matrix. The surrounding word vectors are averaged to obtain the hidden layer output h, and the probability distribution of the central word is predicted by the weight matrix. The semantic relationship of known semantic words is verified to improve the accuracy of vector generation, so that the vectors generated by similar product description attributes are close in distance in the vector space.
[0018] Preferably, the indexing unit stores the embedding vector generated by the vectorization unit in the database, converts the product demand input by the user into a demand vector, uses the approximate nearest neighbor algorithm supported in the database to perform cosine similarity quantization indexing on the product embedding vector in the database based on the IVF_FLAT type, calculates the cosine value of the product vector in the database and the user demand vector, finds the customs product vector data with the highest similarity, outputs the vector ranking with the highest correlation, thereby achieving a high similarity match between the user's product demand and the customs product data, and further achieving accurate search of the user's product demand information.
[0019] Preferably, the data post-processing module restores the vector data queried by the index unit to obtain a matching product list, queries the restored customs commodity information in the customs transaction data, and matches the commodity import and export area, quantity, and import and export date data obtained by the query with the user input information, thereby achieving a secondary match of the user demand information. If the user demand information matches, it is directly output to the user, otherwise it is returned to the index unit for re-indexing.
[0020] Compared with the prior art, the present invention has the following beneficial effects:
[0021] 1. The present invention optimizes the accuracy of customs transaction descriptions by training customs transaction data through a large model, embeds customs transaction data into vectorized representation, vectorizes the customs transaction data, stores the vectorized data in a vector database, and searches for the most matching customs transaction data by performing cosine similarity matching between user demand vectors and vectors in the vector database. The accuracy of the query data is improved by performing secondary matching on the queried customs transaction data.
[0022] 2. This invention improves the richness of product demand information by inputting core user demand words and customs commodity core words into a large language model and optimizing model prompt words in combination with a professional field corpus. It also solves the problem of insufficient interactive data in the vector representation process and improves the accuracy of model training.
[0023] 3. This invention uses embedding technology to convert customs product information and user demand information into vector information. By streamlining the input information and improving the semantic richness of the input information, it can map both short and long texts into the vector space for matching, thereby improving the accuracy of recommendation results during recall. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 It is a module connection diagram of the present invention.
[0025] Figure 2 It is a flowchart of the present invention. DETAILED DESCRIPTION
[0026] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.
[0027] The present invention provides a customs transaction big data multi-dimensional processing and intelligent search and recall system and real-time analysis method, including: a data collection module, a product core word expansion module, a text representation index module and a data post-processing module;
[0028] The data collection module includes a data access unit and a data preparation unit. The data access unit receives the product demand words input by the user by acquiring customs transaction data, and the data preparation unit extracts the required transaction data by classifying and processing the transaction data.
[0029] The product core word expansion module is data-connected to the data collection module and includes a core word extraction unit and a scenario expansion unit. The core word extraction unit uses a large model to analyze and extract product core words from transaction data. The scenario expansion unit obtains product extensibility descriptions by inputting product core words into the large model.
[0030] The text representation index module is connected to the product core word expansion module data, including a vectorization unit and an indexing unit. The vectorization unit obtains low-dimensional vector data by performing embedded vector representation on the product extensibility description, and the indexing unit calls the milvus database to process the low-dimensional vector data to obtain accurate keywords.
[0031] The data post-processing module is connected to the text representation index module data, and matches other attributes of customs transaction commodities with user needs by querying them to improve matching accuracy.
[0032] Example:
[0033] like Figure 1 - Figure 2 As shown, in this embodiment, a company needs to search and recall the user-input product requirements based on the customs transaction data, but the traditional recall method can only perform single-attribute recall based on the product attributes entered by the user. When responding to multi-attribute recall requirements, the recall accuracy is not high due to technical limitations.
[0034] By applying this method, user needs and customs data can be used to expand product descriptions through a large language model, enriching the model training corpus. At the same time, combined with embedding representation technology and the Milvus database, by mapping customs product data into a vector space through pre-training, when recalling user needs, by also mapping user needs into the vector space and calculating the cosine similarity of the vectors, it can efficiently provide highly similar options and improve the accuracy of search recall through further comparison of customs data attributes.
[0035] First, the data access unit receives customs transaction product data and user demand product data respectively, obtains the commodity description, brand and batch, transaction date and transaction company information in the product data, and also receives the demand product data given by the user, including product name, product application scenario and product type. The information includes data format, data type and data collection time.
[0036] Secondly, the data preparation unit receives the data and further processes it. It standardizes the product description information by removing redundant information, unifying the format and data segmentation. It uses regular expressions to delete the product description information, remove modifiers in the product description, retain the core keywords, and divide the core keywords into the first category of core words and the second category of subdivided words according to the complexity of the information, to build a customs product core keyword library and a user demand core keyword library.
[0037] Then, the core word extraction unit parses the core keyword library using a lightweight model, and extracts the core words of the product from the core keyword library by building a customized task combined with the existing corpus in the customs field, such as the commodity classification rule table. The core words of the product include the product name, product purpose and product code.
[0038] Next, the scenario expansion unit expands the product description by acquiring product core words and combining them with the large model. It also expands the product description based on user-required product words. The product description information includes the product demand scenario, demand industry, usage method, and product use purpose. By inputting the core words into a model that excels at understanding long texts, the model input prompt words are optimized, and the large model generates corresponding answers based on the product demand scenario, demand industry, usage method, and product use purpose. At the same time, the answers generated by the large model are further questioned based on the specific product type and the corresponding product corpus, so that the product description information output by the model is consistent with the product core words.
[0039] Next, the vectorization unit uses the word segmentation tool to split the paragraph text product demand description generated by the large model into word combinations, adds domain labels to the generated short word combinations, removes redundant information, and traverses the text to generate vector samples by setting a sampling window. The vector generation samples are converted into vector space using the CBOW model. First, surrounding words are extracted from the vector generation samples by setting a fixed sliding window, and unique IDs are assigned to the surrounding words. These surrounding words are organized to generate a vocabulary, and each surrounding word is represented as a sparse vector equal to the vocabulary size V, with only the corresponding index position being 1. The surrounding word encoding is input into the model, and each encoding is converted into a dense vector through the weight matrix. The surrounding word vectors are averaged to obtain the hidden layer output h. The probability distribution of the central word is predicted by the weight matrix. The semantic relationship of known semantic words is verified to improve the accuracy of vector generation, so that the vectors generated by similar product description attributes are close in distance in the vector space. The indexing unit stores the embedding vector generated by the vectorization unit into the database, converts the product demand input by the user into a demand vector, and uses the approximate nearest neighbor algorithm supported in the database to perform cosine similarity quantitative indexing on the product embedding vector in the database based on the IVF_FLAT type. By calculating the cosine value of the product vector in the database and the user demand vector, the customs product vector data with the highest similarity is found, and the vector ranking with the highest relevance is output, thereby achieving a high similarity match between the user's product demand and the customs product data, and further realizing accurate search of the user's product demand information.
[0040] Finally, the data post-processing module restores the vector data queried by the index unit to obtain a list of matching products. By querying the restored customs commodity information in the customs transaction data, the import and export regions, quantities, and import and export dates of the commodities obtained by the query are matched with the user input information, thereby achieving a secondary match of the user demand information. If the user demand information matches, it is directly output to the user; otherwise, it is returned to the index unit for re-indexing.
[0041] The embodiments of the present invention are provided for the purpose of illustration and description. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present invention. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present invention.
Claims
1. A customs transaction big data multi-dimensional processing and intelligent search and recall system and real-time analysis method, characterized by: include: Data collection module, product core word expansion module, text representation index module and data post-processing module; The data collection module includes a data access unit and a data preparation unit. The data access unit receives the product demand words input by the user by acquiring customs transaction data. The data preparation unit extracts the required data by classifying the transaction data and user demand. The product core word expansion module is data-connected to the data collection module and includes a core word extraction unit and a scenario expansion unit. The core word extraction unit extracts product core words from the data by using a large model, and the scenario expansion unit obtains product extensibility descriptions by inputting product core words into the large model. The text representation index module is connected to the product core word expansion module data, including a vectorization unit and an indexing unit. The vectorization unit obtains low-dimensional vector data by embedding the product extensibility description, and the indexing unit calls the milvus database to process the low-dimensional vector to obtain accurate keywords. The data post-processing module is connected to the text representation index module data, and improves the matching accuracy by further querying other attributes of customs transaction commodities and matching them with user needs.
2. The customs transaction big data multi-dimensional processing and intelligent search and recall system and real-time analysis method according to claim 1, characterized in that: The data access unit accesses customs transaction data to obtain commodity descriptions, brands and batches, transaction dates and transaction company information in the transaction data. It also receives the product demand words given by the user, including product names, product application scenarios and product types. The information includes data format, data type and data collection time.
3. The customs transaction big data multi-dimensional processing and intelligent search and recall system and real-time analysis method as claimed in claim 2, characterized in that: The data preparation unit standardizes the product description information by removing redundant information, unifying the format and processing the product description information through data segmentation, and deletes the product description information by using regular expressions, removes modifiers in the product description, retains the core keywords, and divides the core keywords into a first-class core word and a second-class subdivision word according to the complexity of the information to build a core keyword library.
4. The customs transaction big data multi-dimensional processing and intelligent search and recall system and real-time analysis method according to claim 1, characterized in that: The core word extraction unit parses the core keyword corpus by using a lightweight model, and extracts the core words of commodities from the core keyword corpus by building a customized task combined with an existing corpus in the customs field, such as a commodity classification rule table. The core words of commodities include commodity name, commodity use and commodity code.
5. The customs transaction big data multi-dimensional processing and intelligent search and recall system and real-time analysis method as claimed in claim 4, characterized in that: The scenario expansion unit expands the product description by acquiring the core words of the product and combining it with the big model, and expands the product words according to the user's needs. The product words include product demand scenarios, demand industries, usage methods and product usage purposes; by inputting the core words into the model that is good at understanding long texts, optimizing the model input prompt words, and allowing the big model to generate corresponding answers from the aspects of product demand scenarios, demand industries, usage methods and product usage purposes. At the same time, according to the specific type of the product and the corresponding product corpus, the answers generated by the big model are asked again, so that the product description information output by the model fits the core words of the product.
6. The customs transaction big data multi-dimensional processing and intelligent search and recall system and real-time analysis method as claimed in claim 5, characterized in that: The vectorization unit uses a word segmentation tool to segment the commodity demand description in the text generated by the large model into word combinations, adds domain labels to the generated short word combinations, removes redundant information, and generates vector samples by traversing the text by setting a sampling window. The vector generation samples are converted into a vector space using a CBOW model. First, surrounding words are extracted from the vector generation samples by setting a fixed sliding window, and unique IDs are assigned to the surrounding words. These surrounding words are organized to generate a vocabulary, and each surrounding word is represented as a sparse vector equal to the vocabulary size V, with only the corresponding index position being 1. The surrounding word codes are input into the model, and each code is converted into a dense vector through a weight matrix. The surrounding word vectors are averaged to obtain the hidden layer output h, and the probability distribution of the central word is predicted by the weight matrix. The accuracy of vector generation is improved by verifying the semantic relationship of known semantic words, so that the vectors generated by similar product description attributes are close in the vector space.
7. The customs transaction big data multi-dimensional processing and intelligent search and recall system and real-time analysis method according to claim 6, characterized in that: The indexing unit stores the embedding vector generated by the vectorization unit in a database, converts the product demand input by the user into a demand vector, uses the approximate nearest neighbor algorithm supported in the database to perform cosine similarity quantization indexing on the product embedding vector in the database based on the IVF_FLAT type, calculates the cosine value of the product vector in the database and the user demand vector, finds the customs product vector data with the highest similarity, and outputs the vector ranking with the highest correlation, thereby achieving a high similarity match between the user's product demand and the customs product data, and further achieving accurate search of the user's product demand information.
8. The customs transaction big data multi-dimensional processing and intelligent search and recall system and real-time analysis method as claimed in claim 7, characterized in that: The data post-processing module restores the vector data queried by the index unit to obtain a matching product list, queries the restored customs commodity information in the customs transaction data, and matches the queried commodity import and export region, quantity, and import and export date data with the user input information to achieve a secondary match of the user's required information. If the user's required information matches, it is directly output to the user; otherwise, it is returned to the index unit for re-indexing.