Data processing method and system

Abstract

The invention provides a data processing method and system, and the method comprises the steps: obtaining multi-modal data according to business demand data for each business, converting the multi-modal data into structured data through a specific rule, solving a management problem of unstructured data, further constructing a label system comprising static and dynamic labels, and improving the management efficiency. Multi-dimensional high-precision and high-efficiency processing of data is realized by establishing a data mart and combining generative artificial intelligence with a data bin covering granularity, dimensionality, fact tables and themes, and finally, accurate user group customized recommendation is realized by adopting customized data recommendation rules for different target crowds by constructing the data mart and combining with the generative artificial intelligence.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method and system. Background Art

[0002] In the field of clothing, there is a large amount of unstructured data, which is of great value for design innovation, market trend analysis, supply chain management, etc. However, due to its diverse formats, complex structures, and frequent updates, effectively processing and managing this data has become an urgent problem to be solved. Summary of the invention

[0003] The embodiments of the present application provide a data processing method and system.

[0004] According to a first aspect of the present application, a data processing method is provided, the method comprising:

[0005] Acquire a target service set, wherein the target service set includes at least one service and service demand data corresponding to each service;

[0006] According to the business demand data corresponding to each business, obtain multimodal data for each business;

[0007] For each mode of data in the multimodal data of each business, the corresponding data processing rules are used to perform data structuring processing to obtain structured data of each business, and the structured data is saved in the database;

[0008] According to the business demand data and structured data of each business, a data label system for each business is constructed, wherein the data label system includes a plurality of static feature labels and a plurality of dynamic feature labels;

[0009] Build a data storage layer and synchronize the structured data of each business from the database to the data storage layer;

[0010] Based on the structured data and data label system of each business in the data storage layer, a data warehouse for the target business set is constructed, wherein the data warehouse includes granular data, dimension data, fact data and subject data of each business;

[0011] According to at least one target population corresponding to the target business set, construct a data mart of the data warehouse, wherein the data mart includes data recommendation rules for each target population;

[0012] When a data recommendation request is received from a user through generative artificial intelligence, recommended data for the user is extracted from the data warehouse according to the data recommendation rules corresponding to the target population to which the user belongs, and recommended to the user through generative artificial intelligence, wherein the data recommendation rules include data analysis rules and data extraction rules.

[0013] According to an embodiment of the present application, the multimodal data includes image modal data, graphic modal data and text modal data; accordingly, the data of each modality in the multimodal data for each business is structured using corresponding data processing rules to obtain structured data for each business, including:

[0014] For the image modality data of each service, the image data processing rules are used to perform data structuring processing, and the image processing rules include: using the inverse algorithm of the large model to analyze and annotate the image modality data to obtain substructured data including the first image label data;

[0015] For the graphic and text modal data of each business, the graphic and text data processing rules are used to perform data structuring processing, and the graphic and text processing rules include: extracting image data and text data from the graphic and text modal data, analyzing and annotating the image data using the inverse algorithm of the large model, and obtaining substructured data including the second image label data; performing text analysis on the text data using natural language processing technology, and obtaining substructured data including the first text data;

[0016] For the text modality data of each business, data structuring processing is performed using a text data processing rule, wherein the text data processing rule includes: performing text analysis on the text modality data using a natural language processing technique to obtain substructured data including second text data;

[0017] Among them, all sub-structured data of each business constitute the structured data of each business.

[0018] According to an embodiment of the present application, the step of saving the structured data to a database includes:

[0019] Perform data verification on the structured data of each business, and save the structured data of each business after data verification to the database.

[0020] According to an implementation manner of the present application, the data label system for each business is constructed according to the business demand data and structured data of each business, including:

[0021] Input the business demand data and structured data of each business into the tag mining algorithm or natural language model to obtain multiple tags for each business and description information of each tag;

[0022] Classify all tags according to the business demand data of each business and the description information of each tag to obtain multiple static feature tags and multiple dynamic feature tags;

[0023] Among them, the multiple static feature tags and the multiple dynamic feature tags constitute the data tag system.

[0024] According to an embodiment of the present application, the data warehouse for the target business set is constructed based on the structured data of each business in the data storage layer, and the data warehouse includes granular data, dimension data, fact data and subject data of each business, including:

[0025] According to the business demand data of each business, cluster analysis is performed on the structured data of each business to obtain granular data of each business, wherein the granular data includes multiple business description granularities;

[0026] According to the business demand data of each business, determine the required dimensions of each business, and extract the dimensions of the structured data of each business according to the required dimensions to obtain the dimensional data of each business;

[0027] Design a fact table for each business based on the dimensional data of each business, and load the structured data of each business into its corresponding fact table to obtain fact data;

[0028] Based on keyword extraction technology, the structured data of each business is extracted to obtain the subject data of each business;

[0029] Among them, the granular data, dimension data, fact data and subject data of each business constitute the data warehouse.

[0030] According to one embodiment of the present application, the method further includes:

[0031] Receive the data to be processed sent by the generative artificial intelligence, and based on the data type of the data to be processed, use the data warehouse to process the data to be processed according to the data processing rules corresponding to the data type of the data to be processed.

[0032] According to one embodiment of the present application, the data analysis rule is to analyze the data according to a specific analysis algorithm, and the data extraction rule is to extract data according to a predetermined data extraction requirement.

[0033] According to a second aspect of the present application, a data processing system is provided, the system comprising:

[0034] A service acquisition module, used to acquire a target service set, wherein the target service set includes at least one service and service demand data corresponding to each service;

[0035] A data source access module, comprising a plurality of data receiving interfaces, for acquiring multimodal data for each business based on the plurality of data receiving interfaces and according to business demand data corresponding to each business;

[0036] The multimodal data processing module includes a plurality of data processing interfaces, and is used to perform data structuring processing based on the plurality of data processing interfaces for each modality of the multimodal data of each service using corresponding data processing rules to obtain structured data of each service, and save the structured data to a database;

[0037] A label system construction module is used to construct a data label system for each business according to the business demand data and structured data of each business, wherein the data label system includes a plurality of static feature labels and a plurality of dynamic feature labels;

[0038] A data synchronization module is used to build a data storage layer and synchronize the structured data of each business from the database to the data storage layer;

[0039] A data warehouse construction module, used to construct a data warehouse for a target business set based on the structured data and data label system of each business in the data storage layer, wherein the data warehouse includes granular data, dimension data, fact data and subject data of each business;

[0040] A data mart construction module, used to construct a data mart of the data warehouse according to at least one target population corresponding to the target business set, wherein the data mart includes a data recommendation rule for each target population;

[0041] The data providing module includes multiple data access interfaces, which are used to extract recommended data for the user from the data warehouse according to the data recommendation rules corresponding to the target population to which the user belongs based on the multiple data access interfaces, when receiving a data recommendation request sent by the user through generative artificial intelligence, and recommend it to the user through generative artificial intelligence. The data recommendation rules include data analysis rules and data extraction rules.

[0042] According to one embodiment of the present application, the multimodal data processing module is also used to inspect and maintain multiple data processing interfaces.

[0043] According to an embodiment of the present application, the data source access module is further used to maintain the multiple data receiving interfaces, and the maintenance includes stabilizing the data source access of each data receiving interface.

[0044] The method of the embodiment of the present application obtains multimodal data according to business demand data for each business, and converts it into structured data through specific rules, thereby solving the management problem of unstructured data, and further constructs a label system including static and dynamic labels, as well as a data warehouse covering granularity, dimensions, fact tables and topics, thereby realizing multi-dimensional high-precision and high-efficiency processing of data. Finally, by constructing a data mart and combining it with generative artificial intelligence, customized data recommendation rules are adopted for different target groups to achieve accurate customized recommendations for user groups.

[0045] It should be understood that the teachings of the present application are not required to achieve all of the beneficial effects described above, but specific technical solutions can achieve specific technical effects, and other embodiments of the present application can also achieve beneficial effects not mentioned above. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become readily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:

[0047] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0048] Figure 1 A schematic diagram of the implementation flow of the data processing method provided in the embodiment of the present application is shown;

[0049] Figure 2 A schematic diagram showing the implementation flow of a multi-modal data processing operation of a data processing method provided in an embodiment of the present application is shown;

[0050] Figure 3 A schematic diagram showing the implementation process of the label system construction operation of the data processing method provided by the present application is shown;

[0051] Figure 4 A schematic diagram of the implementation flow of the data warehouse construction operation of the data processing method provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0052] In order to make the purpose, features, and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0053] Figure 1A schematic diagram of the implementation flow of the data processing method provided in an embodiment of the present application is shown.

[0054] refer to Figure 1 , an embodiment of the present application provides a data processing method, the method comprising:

[0055] Operation 101 : acquiring a target service set, where the target service set includes at least one service and service requirement data corresponding to each service.

[0056] Obtain the target business set supported by data processing. Each target business set includes multiple businesses, and each business has corresponding business demand data. The business demand data may include various business demand indicators and business processes. The target business set mainly includes businesses in the clothing field, such as clothing modification and design.

[0057] Operation 102 : acquiring multimodal data for each service according to service demand data corresponding to each service.

[0058] For each business, it is necessary to obtain the data it needs. According to the business demand data of the business, multimodal data for the business can be obtained from different data sources. Among them, multimodal data can include data in multiple modes such as pictures, graphics and text. As an example, multimodal data can include ERP (Enterprise Resource Planning) system data: management information such as production, supply chain, inventory, etc., e-commerce platform data: transaction behavior data such as orders, browsing, and clicks, CRM (Customer Relationship Management) system data: user information such as user profiles, consumption habits, and after-sales services, image and design data: product information such as product pictures, design drafts, and patterns.

[0059] Operation 103 : For each modal data in the multimodal data of each service, a corresponding data processing rule is used to perform data structuring processing to obtain structured data of each service, and the structured data is saved in a database.

[0060] Different data processing rules are required for processing data of different modes, and the data processing rules are pre-set. After the data is processed, it needs to be saved to the database for storage for subsequent use.

[0061] In one embodiment of the present application, the structured data is saved in a database, including: performing data verification on the structured data of each business, and saving the structured data of each business after the data verification in the database.

[0062] After completing the structured processing of multimodal data, data error correction (verification) is required to ensure the accuracy and consistency of the data. There are two ways to correct data errors:

[0063] 1) Manual verification: By receiving correction information on structured data from professionals, the accuracy of annotation content, classification results, etc. is ensured.

[0064] 2) Automated program verification: Use automated tools or algorithms to check structured data and identify potential errors or abnormal data.

[0065] The verification results (whether manual or automated) will also be synchronized to the database to ensure that the final stored data is accurate and reliable.

[0066] Operation 104 , constructing a data label system for each business according to the business demand data and structured data of each business, wherein the data label system includes a plurality of static feature labels and a plurality of dynamic feature labels.

[0067] The data labeling system is the basis for structured classification of data. Labels usually need to be considered based on multiple dimensions to ensure the comprehensiveness, hierarchy and scalability of labels. Among them, data labels mainly include static feature labels and dynamic feature labels. Static feature labels are mainly used to describe the basic characteristics of data, and dynamic feature labels are mainly used to describe the changing characteristics of data.

[0068] Operation 105 : construct a data storage layer, and synchronize the structured data of each business from the database to the data storage layer.

[0069] ODS (Operational Data Store) supports fast access to data, as well as data integration, cleaning, and conversion to meet analysis and other business needs. Therefore, it is also necessary to build a data storage layer and synchronize the structured data in the database from the database to the data storage layer so that the data storage layer can unify the data and store it for subsequent data access and data warehouse construction.

[0070] Among them, the data unification at the data storage layer can include preliminary format conversion, cleaning and standardization processing.

[0071] Operation 106 , based on the structured data and data label system of each business in the data storage layer, build a data warehouse for the target business set, the data warehouse includes granular data, dimension data, fact data and subject data of each business.

[0072] The data warehouse is the core of data processing. When designing a data warehouse, it is necessary to consider multiple dimensions such as granularity, dimension, fact table and subject, so as to build a data warehouse that includes granular data, dimensional data, fact data and subject data for each business.

[0073] Operation 107 : constructing a data mart of the data warehouse according to at least one target population corresponding to the target business set, wherein the data mart includes a data recommendation rule for each target population.

[0074] In order to meet the needs of different target groups, it is also necessary to build a data mart and arrange data recommendation rules for each target group in the data mart so that when users access the data, recommendations can be made according to different data recommendation rules.

[0075] Operation 108, when a data recommendation request is received from a user through generative artificial intelligence, recommended data for the user is extracted from the data warehouse according to the data recommendation rules corresponding to the target population to which the user belongs, and recommended to the user through generative artificial intelligence, wherein the data recommendation rules include data analysis rules and data extraction rules.

[0076] In the field of clothing, after the data warehouse and data mart are built, they usually need to be used in conjunction with artificial intelligence generated content (AIGC) to recommend the data required by users. Users can send data recommendation requests through AIGC. After receiving the data recommendation request, the corresponding data recommendation rules are used to obtain the required data from the data warehouse according to the target population to which the user belongs, and then fed back to the user through AIGC.

[0077] In one embodiment of the present application, the data analysis rule is to analyze the data according to a specific analysis algorithm, and the data extraction rule is to extract data according to predetermined data extraction requirements.

[0078] The data recommendation rules are configured to include data analysis rules and data extraction rules. The data extraction rules are mainly for extracting data based on the predefined data extraction requirements of each business, and the data analysis rules are mainly for analyzing data according to specific data analysis algorithms.

[0079] In order to meet the data needs of multiple target groups, corresponding specific data analysis algorithms and data pre-extraction requirements are configured in advance for each target group. When receiving a user's data recommendation request, data recommendation data can be generated based on the specific data analysis algorithm and data pre-extraction requirements corresponding to the target group to which the user belongs. After the recommended data is generated, it is fed back to the user through AIGC to complete the data recommendation.

[0080] In one embodiment of the present application, the target population may include model trainers, marketing departments, designers, operation personnel, data analysts, and clothing merchants. Accordingly, the data recommendation rules for model trainers may include data extraction rules for extracting data from the original data set and data analysis rules for annotating the original data set, and the original data set may be any data in the data warehouse. The data recommendation rules for the marketing department may include data extraction rules for sales data and data analysis rules such as sales trend analysis, market demand analysis, and boutique comparison of sales data. The data recommendation rules for designers may include data extraction rules for extracting design data and user feedback data and data analysis rules for analyzing design style trends of design data. The data recommendation rules for operation personnel may include data extraction rules and data analysis rules for product inventory, logistics, supply chain, and other information. The data recommendation rules for data analysts may include data extraction rules for sales data and user behavior data and data analysis rules for deep mining of sales data and user behavior data. The data recommendation rules for clothing merchants may include data extraction rules and data analysis rules for sales data and user behavior data.

[0081] The embodiment of the present application solves the management problem of unstructured data by acquiring multimodal data according to business demand data for each business and converting it into structured data through specific rules, and further constructs a label system including static and dynamic labels, as well as a data warehouse covering granularity, dimensions, fact tables and topics, thereby achieving multi-dimensional high-precision and high-efficiency processing of data. Finally, by constructing a data mart and adopting customized data recommendation rules for different target groups, accurate customized recommendations for user groups are achieved.

[0082] Figure 2 A schematic diagram of the implementation flow of the multimodal data processing operation of the data processing method provided in an embodiment of the present application is shown.

[0083] refer to Figure 2 In one embodiment of the present application, the multimodal data includes image modal data, graphic modal data and text modal data; accordingly, the above operation 103 uses the corresponding data processing rules to perform data structuring processing on the data of each modality in the multimodal data of each business, and obtains the structured data of each business, including:

[0084] Operation 201, for each service's image modality data, use image data processing rules to perform data structuring processing, the image processing rules including: using a large model's inverse algorithm to analyze and annotate the image modality data to obtain substructured data including first image label data;

[0085] Operation 202, for the image-text modal data of each service, data structuring processing is performed using an image-text data processing rule, where the image-text processing rule includes: extracting image data and text data from the image-text modal data, analyzing and annotating the image data using a large model inverse algorithm to obtain substructured data including second image tag data; and performing text analysis on the text data using a natural language processing technique to obtain substructured data including first text data;

[0086] Operation 203, for the text modal data of each business, data structuring processing is performed using text data processing rules, and the text data processing rules include: using natural language processing technology to perform text analysis on the text modal data to obtain substructured data including second text data; wherein all substructured data of each business constitute the structured data of each business.

[0087] There are different data processing rules for data of different modalities. For image modality data, there are image data processing rules, for image-text modality data, there are image-text data processing rules, and for text modality data, there are text data processing rules.

[0088] The image data processing rule is mainly to process the data according to the inverse algorithm of the large model. Specifically, for image modality data, the large model inverse algorithm is used to analyze and annotate the image, and the image content is converted into structured label information (such as clothing category, color, style, etc.), that is, the first image label data. Among them, the large model can be a CLIP (Contrastive Language–Image Pre-training, a method of multimodal vision and text learning) model.

[0089] The graphic data processing rules include: classifying graphic data into image data and text data, then using the large model inversion algorithm to process the image data, and using natural language processing technology to process the text data. Specifically, extract text and image data from the graphic modal data, annotate the image data using the large model algorithm to ensure that the image can be processed in the same way as the image modal data, and perform text analysis on the extracted text. Among them, the graphic modal data is mainly PDF data.

[0090] The text data processing rule is to process the text modal data based on natural language processing technology. Specifically, natural language processing technology is applied to the text data to perform semantic segmentation and extract clothing-related keywords (such as material, design style, brand, etc.) as annotation content, that is, the second text data.

[0091] Figure 3 A schematic diagram of the implementation flow of the label system construction operation of the data processing method provided in the present application is shown.

[0092] refer to Figure 3 In one embodiment of the present application, the above operation 104 constructs a data label system for each business according to the business demand data and structured data of each business, including:

[0093] Operation 301, inputting the business demand data and structured data of each business into a tag mining algorithm or a natural language model to obtain multiple tags of each business and description information of each tag;

[0094] Operation 302 , classify all tags according to the business demand data of each business and the description information of each tag to obtain a plurality of static feature tags and a plurality of dynamic feature tags; wherein the plurality of static feature tags and the plurality of dynamic feature tags constitute a data tag system.

[0095] The extraction of tags can be considered as keyword extraction and corresponding description extraction. Most existing tag mining algorithms and natural language models have the ability to extract keywords. After inputting the business demand data and structured data of each business into the existing tag mining algorithm or natural language model, multiple keywords (i.e. tags) for each business and the description information of each tag can be obtained.

[0096] After obtaining multiple tags and the description information of each tag, the tag is determined to be a static description or a dynamic description according to the description information of the tag, and the tags can be divided into static feature tags and dynamic feature tags. Among them, the static description mainly refers to the direct description of the data itself or the direct description of the basic features, and the dynamic description refers to the relevant description obtained by analyzing the preference or change analysis of the data or features through analysis and other means.

[0097] In one embodiment of the present application, the static feature tags mainly include the basic attribute tags of the business field and the supply chain and production tags corresponding to the business, and the dynamic feature tags mainly include user behavior tags and market and sales tags.

[0098] Among them, when the business field is the clothing field, the basic attribute labels of the business field are also called clothing attribute labels, which are mainly used to describe the basic attributes and appearance characteristics of clothing. The specific dimensions include: category labels: such as tops, pants, skirts, jackets, etc.; style labels: such as dresses, T-shirts, sweatshirts, casual pants, etc.; material labels: such as cotton, wool, polyester, silk, etc.; color labels: for example, various color names, blue, red, light pink, dark green, etc.; pattern labels: such as stripes, prints, solid colors, etc.; style labels: such as casual, formal, sports, retro, street style, etc.; size labels: such as S, M, L, XL, one size, etc.; seasonal labels: such as spring and summer, autumn and winter, four seasons, etc.

[0099] Supply chain and production labels are used to describe the relevant data of clothing in the production and supply chain links, mainly including: production date: the production and manufacturing time of the product; inventory status: the current inventory quantity and its changing trend; supplier information: related suppliers, manufacturers, etc.; logistics status: full-chain tracking from production to warehousing and transportation.

[0100] User behavior tags are mainly tags obtained based on the analysis of users' purchase and use behavior of clothing products, and are used for more precise marketing and recommendations. Specific dimensions include: purchase frequency: such as high-frequency and low-frequency buyers; preference style: based on users' historical purchase records, infer users' preferred clothing styles; price sensitivity: such as users who prefer high-priced or discounted goods; channel preferences: such as online purchases, offline purchases, etc.; browsing habits: such as frequently browsed categories or brands.

[0101] The market and sales labels are obtained based on the analysis of the performance of clothing in the market, and mainly include: sales channels: such as e-commerce, self-operated stores, specialty stores, etc.; market performance: such as hot-selling, slow-selling and other status; discount intensity: price discount information of goods; user evaluation: comprehensive evaluation labels formed based on ratings and comments.

[0102] Figure 4 A schematic diagram of the implementation flow of the data warehouse construction operation of the data processing method provided in an embodiment of the present application is shown.

[0103] refer to Figure 4 In one embodiment of the present application, the above operation 106 builds a data warehouse for the target business set based on the structured data of each business in the data storage layer, and the data warehouse includes granular data, dimension data, fact data and subject data of each business, including:

[0104] Operation 401 , cluster analysis is performed on structured data of each service according to service demand data of each service to obtain granular data of each service, where the granular data includes a plurality of service description granularities.

[0105] The construction of a data warehouse includes multiple steps such as granularity selection, dimensional model design, fact table design, and subject selection.

[0106] First, the granularity needs to be selected. Based on the business demand data of each business, the structured data of each business is clustered to obtain granular data for the business. Among them, granular data includes multiple business description granularities, which are mainly used to describe related data from different levels. For example, in the field of clothing, there is order granularity for order data in structured data, user granularity for user data, and image granularity for image data such as design.

[0107] Taking the image granularity as an example, the image granularity can include the following levels:

[0108] 1) Single garment level: Each garment is the most granular level, suitable for tracking all relevant patterns, materials, colors, and design elements of each garment. This is very valuable for designers, product development, and personalized recommendations;

[0109] 2) Collection / Batch Level: For collections with similar design language, styles or patterns, a coarser granularity can be chosen, which is suitable for analyzing trends, collection sales data and inventory management.

[0110] 3) Pattern and element level: Granularity selection is performed for specific patterns or graphic elements, such as the type and style of each pattern, graphic, color combination, etc. This granularity helps to understand the popularity, market demand and consumer preferences of specific patterns.

[0111] Operation 402 : determining the required dimensions of each business according to the business demand data of each business, and performing dimension extraction on the structured data of each business according to the required dimensions to obtain the dimension data of each business.

[0112] After selecting the granularity, you also need to design a dimensional model. For different businesses, the corresponding required dimensions are stored in advance. When you need to design a dimensional model for a certain business, you only need to extract the corresponding dimensional data from the structured data based on the dimensions corresponding to the business.

[0113] In one embodiment of the present application, for the clothing field, the dimensions of each business may include:

[0114] Product dimension: includes attributes such as product ID, category, material, brand, color, style, size, etc.

[0115] Time dimension: divided into days, weeks, months, quarters, and years to support sales and operations analysis;

[0116] User dimensions: user's age, gender, region, consumption level, preferences, etc.

[0117] Channel dimension: such as online, offline, self-operated, e-commerce platforms, social media and other channel information;

[0118] Geographic dimension: detailed location information such as country, province, city, store, etc.

[0119] Supply chain dimension: involves information such as manufacturers, suppliers, and logistics.

[0120] Operation 403 : design a fact table for each business according to the dimension data of each business, and load the structured data of each business into its corresponding fact table to obtain fact data.

[0121] After obtaining the dimensional data of each business, a fact table is designed based on the dimensional data to store business activity data.

[0122] Furthermore, the fact data may include multiple fact tables, which may specifically include:

[0123] Sales fact table: records each sales transaction, including product ID, user ID, time, quantity, amount, etc.;

[0124] Inventory fact table: records inventory status, including product ID, warehouse ID, inventory quantity, inventory change time, etc.;

[0125] Browsing and behavior fact table: records users’ browsing, clicking and other behaviors, and analyzes users’ preferences and behavior trajectories;

[0126] Evaluation fact table: records users' evaluation of products to help analyze user satisfaction and product quality.

[0127] Operation 404 , based on the keyword extraction technology, subject extraction is performed on the structured data of each business to obtain subject data of each business; wherein the granular data, dimension data, fact data and subject data of each business constitute a data warehouse.

[0128] Topic selection is the core of building a data warehouse. For structured data of different businesses, multiple topics can be extracted based on business demand data using keyword extraction technology to provide personalized recommendations for different user groups through different topics.

[0129] In one embodiment of the present application, the following topics can be extracted from the image data in the structured data by using the keyword extraction technology:

[0130] 1) Pattern type theme: Patterns can be classified according to their style, such as geometric patterns, floral patterns, abstract patterns, animal patterns, etc. Each pattern category is a theme that can help understand market preferences and trends;

[0131] 2) Color matching theme: Color is a key part of clothing design. By determining the theme through the combination of patterns and colors, we can capture the impact of different color combinations on sales and help optimize design decisions;

[0132] 3) Design element themes: Such as the combination of textures, materials and patterns, create themes about different design elements. For example, a specific pattern on a certain fabric may be more popular in certain seasons.

[0133] 4) Clothing functional themes: Themes can be determined based on the usage scenarios or functions of the clothing. For example, whether the demand for clothing patterns in outdoor, leisure, and business occasions is different. Such themes can support customized recommendations for specific consumer groups.

[0134] In one embodiment of the present application, inventory topics, sales topics, logistics topics, user browsing topics, and evaluation topics, etc. can be extracted for transaction data in structured data.

[0135] After completing multiple steps such as granularity selection, dimensional model design, fact table design, and subject selection, the construction of the data warehouse is completed.

[0136] In one embodiment of the present application, the data to be processed sent by the AIGC is also received, and the data to be processed is processed using corresponding data processing rules according to the data type of the data to be processed.

[0137] Specifically, the data warehouse also needs to process data from multiple modalities (such as images, text, graphics and text, videos, etc.). Specifically, use image data processing rules to process images, use text data processing rules to process text, use graphics and text data processing rules to process graphics and text, and use video analysis technology to process videos.

[0138] Based on the above data processing method, the embodiment of the present application further provides a data processing system, which includes:

[0139] A service acquisition module, used to acquire a target service set, wherein the target service set includes at least one service and service demand data corresponding to each service;

[0140] A data source access module, including a plurality of data receiving interfaces, for acquiring multimodal data for each business based on the plurality of data receiving interfaces and according to the business demand data corresponding to each business;

[0141] The multimodal data processing module includes a plurality of data processing interfaces, and is used to perform data structuring processing based on the plurality of data processing interfaces for each modality of the multimodal data of each business using corresponding data processing rules to obtain structured data of each business, and save the structured data to a database;

[0142] The label system construction module is used to construct a data label system for each business based on the business demand data and structured data of each business. The constructed data label system includes multiple static feature labels and multiple dynamic feature labels.

[0143] The data synchronization module is used to build a data storage layer and synchronize the structured data of each business from the database to the data storage layer;

[0144] The data warehouse construction module is used to build a data warehouse for the target business set based on the structured data and data label system of each business in the data storage layer. The data warehouse includes granular data, dimension data, fact table and subject data of each business;

[0145] A data mart construction module, used to construct a data mart of the data warehouse according to at least one target population corresponding to the target business set, the data mart including data recommendation rules for each target population;

[0146] The data providing module includes multiple data access interfaces, which are used to extract recommended data for the user from the data warehouse based on the multiple data access interfaces, according to the data recommendation rules corresponding to the target population to which the user belongs, and recommend it to the user through generative artificial intelligence. The data recommendation rules include data analysis rules and data acquisition rules.

[0147] In one embodiment of the present application, the multimodal data processing module is also used to inspect and maintain multiple data processing interfaces.

[0148] The data processing system needs to regularly check the data processing interface of multimodal data to ensure the normal operation of the processing of multimodal data such as images, documents, and text. For example, whether interfaces such as image recognition, large model reasoning, and text processing can receive and process data normally. Through this mechanism, the stability and continuity of the data processing process can be guaranteed, and data backlog or loss caused by interface failure can be avoided.

[0149] In one embodiment of the present application, the data source access module is also used to maintain multiple data receiving interfaces, and the maintenance includes maintaining stable access to the data source of each data receiving interface.

[0150] During data access, different data sources may come from multiple business systems or external platforms. In order to ensure continuous and stable data access, the system needs to regularly check the connection status of these multiple data sources (data source access stability) to ensure their availability and the normal operation of data synchronization. Once the data source connection is found to be invalid or unstable, the system will promptly issue an alarm or try to reconnect.

[0151] It should be noted that the description of the system in the embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figures 1 to 4 The present invention can be understood by referring to the description of any one of the accompanying drawings.

[0152] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0153] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire a target service set, wherein the target service set includes at least one service and service demand data corresponding to each service; According to the business demand data corresponding to each business, obtain multimodal data for each business; For each mode of data in the multimodal data of each business, the corresponding data processing rules are used to perform data structuring processing to obtain structured data of each business, and the structured data is saved in the database; According to the business demand data and structured data of each business, a data label system for each business is constructed, wherein the data label system includes a plurality of static feature labels and a plurality of dynamic feature labels; Build a data storage layer and synchronize the structured data of each business from the database to the data storage layer; Based on the structured data and data label system of each business in the data storage layer, a data warehouse for the target business set is constructed, wherein the data warehouse includes granular data, dimension data, fact data and subject data of each business; According to at least one target population corresponding to the target business set, construct a data mart of the data warehouse, wherein the data mart includes data recommendation rules for each target population; When a data recommendation request is received from a user through generative artificial intelligence, recommended data for the user is extracted from the data warehouse according to the data recommendation rules corresponding to the target population to which the user belongs, and recommended to the user through generative artificial intelligence, wherein the data recommendation rules include data analysis rules and data extraction rules.

2. The method according to claim 1, characterized in that The multimodal data includes image modal data, graphic modal data and text modal data; accordingly, the data of each modality in the multimodal data for each business is structured using corresponding data processing rules to obtain structured data for each business, including: For the image modality data of each service, the image data processing rules are used to perform data structuring processing, and the image processing rules include: using the inverse algorithm of the large model to analyze and annotate the image modality data to obtain substructured data including the first image label data; For the graphic and text modal data of each business, the graphic and text data processing rules are used to perform data structuring processing, and the graphic and text processing rules include: extracting image data and text data from the graphic and text modal data, analyzing and annotating the image data using the inverse algorithm of the large model, and obtaining substructured data including the second image label data; performing text analysis on the text data using natural language processing technology, and obtaining substructured data including the first text data; For the text modality data of each business, data structuring processing is performed using a text data processing rule, wherein the text data processing rule includes: performing text analysis on the text modality data using a natural language processing technique to obtain substructured data including second text data; Among them, all sub-structured data of each business constitute the structured data of each business.

3. The method according to claim 1, characterized in that The step of saving the structured data to a database comprises: Perform data verification on the structured data of each business, and save the structured data of each business after data verification to the database.

4. The method according to claim 1, characterized in that: According to the business demand data and structured data of each business, a data label system for each business is constructed, including: Input the business demand data and structured data of each business into the tag mining algorithm or natural language model to obtain multiple tags for each business and description information of each tag; Classify all tags according to the business demand data of each business and the description information of each tag to obtain multiple static feature tags and multiple dynamic feature tags; Among them, the multiple static feature tags and the multiple dynamic feature tags constitute the data tag system.

5. The method according to claim 1, characterized in that The data warehouse for the target business set is constructed based on the structured data of each business in the data storage layer, and the data warehouse includes granular data, dimension data, fact data and subject data of each business, including: According to the business demand data of each business, cluster analysis is performed on the structured data of each business to obtain granular data of each business, wherein the granular data includes multiple business description granularities; According to the business demand data of each business, determine the required dimensions of each business, and extract the dimensions of the structured data of each business according to the required dimensions to obtain the dimensional data of each business; Design a fact table for each business based on the dimensional data of each business, and load the structured data of each business into its corresponding fact table to obtain fact data; Based on keyword extraction technology, the structured data of each business is extracted to obtain the subject data of each business; Among them, the granular data, dimension data, fact data and subject data of each business constitute the data warehouse.

6. The method according to claim 1, characterized in that The method further comprises: Receive the data to be processed sent by the generative artificial intelligence, and based on the data type of the data to be processed, use the data warehouse to process the data to be processed according to the data processing rules corresponding to the data type of the data to be processed.

7. The method according to claim 1, characterized in that The data analysis rule is to analyze the data according to a specific analysis algorithm, and the data extraction rule is to extract data according to a predetermined data extraction requirement.

8. A data processing system, characterized in that: The system comprises: A service acquisition module, used to acquire a target service set, wherein the target service set includes at least one service and service demand data corresponding to each service; A data source access module, comprising a plurality of data receiving interfaces, for acquiring multimodal data for each business based on the plurality of data receiving interfaces and according to business demand data corresponding to each business; The multimodal data processing module includes a plurality of data processing interfaces, and is used to perform data structuring processing based on the plurality of data processing interfaces for each modality of the multimodal data of each service using corresponding data processing rules to obtain structured data of each service, and save the structured data to a database; A label system construction module is used to construct a data label system for each business according to the business demand data and structured data of each business, wherein the data label system includes a plurality of static feature labels and a plurality of dynamic feature labels; A data synchronization module is used to build a data storage layer and synchronize the structured data of each business from the database to the data storage layer; A data warehouse construction module, used to construct a data warehouse for a target business set based on the structured data and data label system of each business in the data storage layer, wherein the data warehouse includes granular data, dimension data, fact data and subject data of each business; A data mart construction module, used to construct a data mart of the data warehouse according to at least one target population corresponding to the target business set, wherein the data mart includes a data recommendation rule for each target population; The data providing module includes multiple data access interfaces, which are used to extract recommended data for the user from the data warehouse according to the data recommendation rules corresponding to the target population to which the user belongs based on the multiple data access interfaces, when receiving a data recommendation request sent by the user through generative artificial intelligence, and recommend it to the user through generative artificial intelligence. The data recommendation rules include data analysis rules and data extraction rules.

9. The system according to claim 8, characterized in that The multimodal data processing module is also used to inspect and maintain multiple data processing interfaces.

10. The system according to claim 8, characterized in that The data source access module is also used to maintain the multiple data receiving interfaces, and the maintenance includes maintaining the data source access stability of each data receiving interface.

Images