An AI-based multimodal data processing method

Abstract

The present application provides an AI-based multimodal data processing method, including: obtaining multimodal data of a target industry field in a target area, wherein the target area is the country or region in which the enterprise intends to expand its market, and the target industry field is the industry field to which the products provided by the enterprise for market expansion belong, and the multimodal data includes text data and graphic data; after pre-processing the multimodal data, feature extraction and pre-classification are performed to determine feature data, wherein each feature data is a feature vector containing pre-classification information; and data integration is performed on the feature data to form an analysis data source. This solution can construct feature data of graphic data (including item labels and feature vectors), which maintains the same dimension as the feature data of plain text data (including text labels and feature vectors). The pre-classification information at the label level can perform a preliminary division of the feature data to form a well-structured and reliable analysis data source for enterprise market expansion analysis.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular, to an AI-based multimodal data processing method. Background Art

[0002] Traditional decision support systems for market expansion often rely on a single data source or simple data analysis methods. These systems struggle to comprehensively and timely process massive amounts of diverse information, and therefore fail to provide accurate and dynamic decision support for market expansion. In this era of information explosion, a technology is needed that can comprehensively process multimodal data (such as plain text, web pages, and other graphic data) to facilitate subsequent in-depth AI analysis and provide strong support for market expansion decisions.

[0003] Currently, there are many solutions for processing multimodal data, such as CN117893861A, which provides a method and apparatus for processing images and text, but they do not address the processing of web page data. For enterprise market expansion analysis, many data processing solutions are difficult to apply because key data sources are typically public documents, industry reports, and various public web page data. Public web page data is the most widely sourced and difficult to process. Because public web page data has diverse layout patterns, simply extracting text information can easily miss image information, potentially resulting in a large amount of information mismatch, reducing data reliability, and hindering enterprise market expansion analysis of product dimensions (such as current market conditions, product sales, product prices, product quality, competitive advantages, etc.). Therefore, web page-based image and text data processing solutions are key to processing enterprise market expansion data. Summary of the Invention

[0004] The purpose of the embodiment of the present application is to provide an AI-based multimodal data processing method, which obtains multimodal data in the target industry field of the target area, performs feature extraction and pre-classification, and forms an analysis data source after data integration. In the graphic data analysis, the image data is subjected to object identification to obtain object labels, and an association relationship between the object labels and the text content is established. The text content is then translated and feature extraction is performed to construct feature data containing object labels (derived from image data) and feature vectors (derived from text content). The dimensionality is unified with the feature data of the pure text data (including text labels and feature vectors), and the label-level pre-classification information can perform a preliminary division of the feature data to form a well-structured and reliable analysis data source for subsequent enterprise expansion market analysis.

[0005] In order to achieve the above objectives, the embodiments of the present application are implemented in the following manner:

[0006] In a first aspect, an embodiment of the present application provides an AI-based multimodal data processing method, comprising: obtaining multimodal data of a target industry field in a target area, wherein the target area is the country or region in which the enterprise intends to expand its market, and the target industry field is the industry field to which the products provided by the enterprise to expand its market belong, and the multimodal data includes text data and graphic data; after preprocessing the multimodal data, feature extraction and pre-classification are performed to determine feature data, wherein each feature data is a feature vector containing pre-classification information, and when the graphic data is public web page data, an association relationship between the item label of the image data in the public web page data and the text content is constructed based on spatial overlap and semantic similarity to form feature data corresponding to the text content in the public web page data; and data integration is performed on the feature data to form an analysis data source.

[0007] In combination with the first aspect, in a first possible implementation method of the first aspect, after preprocessing the multimodal data, feature extraction and pre-classification are performed to determine feature data, including: preprocessing text data and image data; for each text data: determining a text label from the text data, calling a multilingual translation model to translate the text data, using a pre-trained language model to perform feature extraction, obtaining a feature vector of the text, and then determining the feature data corresponding to the text data based on the text label and feature vector of the text data; for each image data: determining image data and text content from the image data, performing object identification on each image data to obtain an object label, and establishing an association between the object label and the text content, and then calling a multilingual translation model to translate the text content, using a pre-trained language model to perform feature extraction, obtaining a feature vector of the text content, and determining the feature data corresponding to the text content based on the object label and the feature vector of the text content.

[0008] In combination with the first possible implementation method of the first aspect, in the second possible implementation method of the first aspect, the graphic data is public web page data, image data and text content are determined from the graphic data, object identification is performed on each image data, object labels are obtained, and an association relationship between the object labels and the text content is established, including: parsing the public web page data through the web page DOM tree to obtain the spatial coordinates of the image data and the text content; using the object detection model to identify objects in the image data, generate object labels, and filter out target image data belonging to the target industry field based on the object labels; for each target image data: based on the spatial coordinates of the target image data and the spatial coordinates of each text content, determine the spatial overlap between the target image data and each text content; perform shared semantic space mapping on each target image data and each text content to determine the semantic similarity between each target image data and each text content; for each target image data: based on the spatial overlap and semantic similarity between the target image data and each text content, determine the target text content that matches the target image data; and establish an association relationship between the object label of the target image data and each target text content.

[0009] In combination with the second possible implementation method of the first aspect, in a third possible implementation method of the first aspect, the public web page data is parsed through the web page DOM tree to obtain the spatial coordinates of the image data and text content, including: parsing the public web page data through the web page DOM tree to determine the CSS box model of each image element and the CSS box model of each text element; parsing the CSS box model of each image element and the CSS box model of each text element, and calculating the absolute position of each CSS box model relative to the viewport; using Selenium to obtain the precise position of each CSS box model, thereby determining the spatial coordinates of the image data and text content.

[0010] In combination with the second possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, determining the spatial overlap between the target image data and each text content based on the spatial coordinates of the target image data and the spatial coordinates of each text content includes: determining the image area and the image influence area based on the spatial coordinates of the target image data, where the image influence area is a peripheral area determined based on the image area; determining the text area based on the spatial coordinates of the text content; and calculating the spatial overlap between the target image data and the text content using an improved IoU formula:

[0011] ,

[0012] in, Represents the target image data With text content The spatial overlap between Indicates the Target image data, Indicates the Text content, and are weights, all of which are positive, and , Represents the target image data The image area, Represents the target image data The image influence area, Represents the target image data The image area and text content The intersection of the text area, Represents the target image data Image influence area and text content The intersection of the text areas.

[0013] In combination with the second possible implementation method of the first aspect, in a fifth possible implementation method of the first aspect, a shared semantic space mapping is performed on each target image data and each text content to determine the semantic similarity between each target image data and each text content, including: calling an image encoder to process the target image data to obtain an image semantic vector corresponding to each target image data; calling a multilingual text encoder to process the text content to obtain a text semantic vector corresponding to each text content; mapping the image semantic vectors corresponding to all target image data and the text semantic vectors corresponding to all text contents to a shared semantic space, performing matrix multiplication calculations, and obtaining a similarity matrix containing the semantic similarity between each target image data and each text content.

[0014] In conjunction with the fifth possible implementation manner of the first aspect, in a sixth possible implementation manner of the first aspect, determining target text content that matches the target image data based on the spatial overlap and semantic similarity between the target image data and each text content includes: normalizing the spatial overlap between the target image data and the text content; and calculating the comprehensive matching degree between the target image data and the text content using the following formula:

[0015] ,

[0016] in, Represents the target image data With text content The comprehensive matching degree between Represents the standardized target image data With text content The spatial overlap between Represents the target image data With text content The semantic similarity between Indicates the Target image data, Indicates the Text content, and are weights, all of which are positive, and ; Based on the comprehensive matching degree between the target image data and the text content, determine the target text content that matches the target image data.

[0017] In combination with the sixth possible implementation manner of the first aspect, in the seventh possible implementation manner of the first aspect, before standardizing the spatial overlap between the target image data and the text content, the method also includes: performing statistical feature extraction on the spatial overlap between the target image data and the text content, and determining the target arrangement pattern between the target image data and the text content, wherein the target arrangement pattern includes an overlapping arrangement pattern, a mixed arrangement pattern, and a separated arrangement pattern; based on the target arrangement pattern between the target image data and the text content, determining the weight distribution of the comprehensive matching degree calculation formula.

[0018] In combination with the seventh possible implementation of the first aspect, in the eighth possible implementation of the first aspect, when the target arrangement mode is a mixed arrangement mode, the weight and The following relationship is satisfied: count the maximum spatial overlap between each target image data and the text content, and calculate the ratio of the number of target image data corresponding to the maximum spatial overlap exceeding the set value to the total amount of target image data; use the following formula to calculate the weight and :

[0019] ,

[0020] ,

[0021] in, It is the ratio of the number of target image data whose maximum spatial overlap exceeds the set value to the total amount of target image data.

[0022] In combination with the first possible implementation method of the first aspect, in the ninth possible implementation method of the first aspect, based on the text label and feature vector of the text data, the feature data corresponding to the text data is determined, including: for each feature vector of the text data, adding a text label before the feature vector, wherein the text label is converted into a feature quantity of a set length; correspondingly, based on the item label and the feature vector of the text content, the feature data corresponding to the text content is determined, including: for the feature vector of the text content, adding an item label before the feature vector, wherein the item label is converted into a feature quantity of a set length.

[0023] Beneficial Effects: This method obtains multimodal data (text and graphic data) from target industries (industries to which the products offered by the company for market expansion belong) in target regions (countries or regions to which the company intends to expand its market). Preprocessing the multimodal data, feature extraction, and pre-classification are performed to determine feature data (feature vectors containing pre-classification information). Specifically, text labels are determined from the text data. A multilingual translation model is used to translate the text data, and feature extraction is performed using a pre-trained language model to obtain feature vectors for the text. The corresponding feature data is determined by combining the text labels and feature vectors for the text data. Furthermore, image data and text content are determined from the graphic data. Objects are identified for each image data item to obtain object labels, and a correlation is established between the object labels and the text content. The multilingual translation model is then used to translate the text content, and feature extraction is performed using a pre-trained language model to obtain feature vectors for the text content. Based on the object labels and the feature vectors for the text content, feature data corresponding to the text content is determined. This feature data can then be integrated to form an analysis data source. This solution adopts differentiated feature extraction methods for text data and graphic data to build a unified data structure, and adds label information to the feature data for pre-classification. In the enterprise expansion market analysis, different data analysis can be performed based on different pre-classification information (for example, product feature data is used to build relevant market-type knowledge graphs, which are used to conduct market expansion analysis of enterprises in different dimensions, such as risk analysis, market competition analysis, etc.). In addition, when processing graphic data (mainly public web page data), this solution can establish an accurate connection between image data and text content, effectively solving the information mismatch problem, greatly improving the reliability of public web page data, and providing more accurate data support for subsequent enterprise expansion market analysis.

[0024] For publicly available web data, the spatial coordinates of image data and text content are obtained by parsing the web page's DOM tree. Specifically, the DOM tree is used to parse the publicly available web data, determining the CSS box model for each image element and the CSS box model for each text element. The CSS box model for each image element and the CSS box model for each text element are then parsed and the absolute position of each CSS box model relative to the viewport is calculated. Selenium is then used to obtain the precise position of each CSS box model, thereby determining the spatial coordinates of the image data and text content. This method efficiently and accurately obtains the coordinates of each element (image data or text content) in the publicly available web data. This solution is adaptable to various web page layout modes (e.g., separated, hybrid, and overlapping), automatically adapts to layout changes across screen sizes, and exhibits robustness against layout drift. Object detection models (e.g., YoLov8) are used to identify objects in the image data, generate object labels, and filter out target images within the target industry based on the object labels, thus eliminating most irrelevant images (e.g., buttons, icons, and dynamic advertisements). For each target image data: based on the spatial coordinates of the target image data and the spatial coordinates of each text content, determine the spatial overlap between the target image data and each text content; perform shared semantic space mapping on each target image data and each text content to determine the semantic similarity between each target image data and each text content; for each target image data: based on the spatial overlap and semantic similarity between the target image data and each text content, determine the target text content that matches the target image data; establish an association between the object label of the target image data and each target text content. In this way, by mapping data of different modalities into a unified shared semantic space (for example, based on the CLIP model), cross-modal alignment of multimodal data can be achieved, making it possible to measure the semantic similarity of image data and text content in public web page data. Different from traditional spatial association schemes, this scheme considers both spatial overlap and semantic similarity when establishing the association between image data and text content, thereby improving the accuracy of the association and matching accurate item labels with text content in public web page data, thereby effectively reducing the problem of information mismatch. Item labels can provide pre-classification information for subsequent data analysis, which is conducive to providing more reliable data support for enterprises' market expansion decisions.

[0025] Considering the diverse layout patterns of public web data across different regions, and even within the same region, this solution specifically improves the traditional spatial overlap calculation method. This method considers both the image location and the image's influence area, adapting to different web layout patterns (separate, hybrid, and overlapping). The improved IoU formula calculates the spatial overlap between the target image data and text content, assigning higher weights to the core area (where the image resides) and relatively lower weights to the peripheral areas (where the image influences the image). For overlapping web layout patterns, high spatial overlap typically indicates matching image data and text content. For hybrid and separate web layout patterns, however, the inclusion of text content outside the image data (e.g., below or to the right) is required. Based on this, an improved IoU formula is designed to calculate the spatial overlap between image data and text content. The numerical distribution of spatial overlap also reflects the web layout pattern, allowing for a comprehensive, adaptive matching calculation scheme to be developed for different layout patterns, improving the accuracy of the association between image data and text content.

[0026] When calculating the comprehensive matching degree between the target image data and the text content, it's important to consider the distribution of spatial overlap and semantic similarity under different web page layout modes (overlapping, mixed, and separated). For overlapping layouts, a higher spatial overlap indicates a stronger correlation between the image data and the text content. For separated layouts, semantic similarity plays a greater role, but spatial overlap is also important. However, for the mixed arrangement mode, in the same public web page data, there are both overlapping image data and text content arrangements, and separated image data and text content arrangements. This requires consideration of more complex issues, because the text content appearing above the image data and the text content appearing below the image data cannot be distinguished solely by spatial overlap. A higher weight should be assigned to semantic similarity, but it also depends on the situation. Because some public web page data only have a small number of overlapping arrangement areas, semantic similarity can be used as the main factor and spatial overlap as an auxiliary factor. For the case where there are a large number of overlapping arrangement areas in the public web page data, it is necessary to rely more on spatial overlap and reduce the proportion of semantic similarity. Based on this, this scheme has been tested and designed a suitable comprehensive matching calculation scheme and a weight distribution method under the mixed arrangement mode (the other two arrangement modes can use the set weights without the need for adaptive adjustment). After calculating the comprehensive matching degree, the target text content corresponding to the target image data can be determined as the one with the highest comprehensive matching degree. Overall, in order to achieve accurate association between image data and text content in graphic data (public web page data), this solution has designed a complete set of practical cross-modal matching solutions. It can effectively solve the information mismatch problem caused by extracting only text content from public web page data (for example, the text content lacks product information, while the product information is expressed in the image data, which is mainly a common problem in graphic data obtained from shopping websites). It achieves accurate association between image data and text content, improves the quality of feature data extracted based on public web page data, and provides a reliable data analysis foundation for subsequent corporate market expansion analysis.

[0027] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0029] Figure 1A flowchart of an AI-based multimodal data processing method provided in an embodiment of the present application.

[0030] Figure 2 Schematic diagram of the image area and image influence area of ​​image data.

[0031] Figure 3 A schematic diagram of the text area where the text content is located. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0033] See also Figure 1 , Figure 1 Flowchart of a multimodal data processing method based on AI provided in an embodiment of the present application. In this embodiment, the multimodal data processing method based on AI may include steps S10, S20, and S30.

[0034] Step S10: Acquire multimodal data of the target industry field of the target area, wherein the target area is the country or region where the enterprise intends to expand its market, and the target industry field is the industry field to which the products provided by the enterprise for market expansion belong. The multimodal data includes text data and graphic data.

[0035] In this embodiment, data collection nodes can be deployed in the target area to collect data from multiple channels such as news media, public data, think tank reports, e-commerce platforms, industry forums, etc. in real time.

[0036] For enterprise market expansion scenarios, enterprises provide products to enter the market in the target area. The information that enterprises need to know mainly includes relevant information about the products provided by the enterprise to expand the market, such as the market environment, competitive products, etc. The main data used are public data, think tank reports, e-commerce platforms, etc. Web crawler technology can be used to capture web page text information, and image data of specific scenes can be obtained through image acquisition equipment.

[0037] For news media websites, you can use web crawlers written in Python to regularly capture text data such as news articles and comments according to pre-defined rules. For public data platforms, you can apply for data interface permissions and use API calls to obtain relevant statistical data. For e-commerce platforms, you can collaborate with the platforms to obtain multimodal data such as product information, sales data, and user reviews. For example, you can sign data cooperation agreements with platforms like Amazon and eBay to obtain relevant data according to the specified interfaces and data formats. However, due to the difficulty and high cost of cooperation, in most cases it is impossible to obtain structured data through cooperation agreements. Therefore, it is necessary to utilize the platform's publicly available product list and detail pages, and, while complying with the platform's anti-scraping rules and terms, use compliant web crawler technology to collect publicly visible product information, prices, user reviews, and other data.

[0038] For this embodiment, the main targets are text data (in this embodiment, it mainly refers to industry analysis reports, market research reports, etc., which only require the collection of text data. This does not mean that these data only contain text, but we only need text) and graphic data (the main source is public web page data, which contains image data and text content, and usually has a corresponding relationship, such as graphic data legally collected from e-commerce platforms and shopping websites).

[0039] Of course, in future development plans, audio and video data can be further collected to obtain more comprehensive data, but this is not considered in this embodiment.

[0040] Based on this, multimodal data can be obtained for the target industry (the industry to which the products offered by the company are being marketed) in the target region (the country or region the company intends to expand into, such as the United States or the European Union). This multimodal data includes both textual and graphical data. After obtaining the multimodal data, step S20 can be executed.

[0041] Step S20: After preprocessing the multimodal data, feature extraction and preclassification are performed to determine feature data, wherein each feature data is a feature vector containing preclassification information.

[0042] In this embodiment, text data and graphic data can be preprocessed. For example, text data can be denoised (deleting or replacing non-text characters such as special symbols, garbled characters, emoticons, and residual HTML / XML tags), normalized (such as line breaks), and case-converted. For graphic data, primarily public web page data, noise content such as navigation bars, footers, advertisements, sidebars, and comments can be removed.

[0043] Then, for each text data: the text label can be determined from the text data, the multilingual translation model can be called to translate the text data, and the pre-trained language model can be used for feature extraction to obtain the feature vector of the text. Then, based on the text label and feature vector of this text data, the feature data corresponding to this text data can be determined.

[0044] Taking the target area as the United States, whose language is English, as an example, text labels can be determined from the text data (for example, for industry reports, etc., text labels can be extracted from the translated topics. The number of labels is based on the set number, for example, 2, one core topic plus one type label, or other numbers, for example, 5, four core topics plus one type label. Insufficient core topics can be made up by filling in zeros. This embodiment takes "one core topic plus one type label" as an example). Afterwards, the multilingual translation model is called to translate the text data (for major languages, it can be directly translated into Chinese, such as English, Japanese, Korean, etc.; for minor languages, it can be translated into English first and then into Chinese), and a pre-trained language model (such as a natural language processing model based on BERT) is used for feature extraction to obtain a feature vector of the text data. Of course, one text data can obtain many feature vectors, because one text data usually corresponds to multiple paragraphs, sentences, etc.

[0045] Each feature vector extracted from the text data is combined with the text label to obtain the corresponding feature data. Specifically, a text label can be added to each feature vector of the text data to convert it into a feature of a set length. For example, a core topic plus a type label is an x-dimensional feature.

[0046] For each piece of image and text data: the image data and text content can be determined from the image and text data, and the object recognition is performed on each image data to obtain the object label, and an association relationship is established between the object label and the text content. The multilingual translation model is then called to translate the text content, and the pre-trained language model is used for feature extraction to obtain the feature vector of the text content. Based on the object label and the feature vector of this text content, the feature data corresponding to this text content is determined.

[0047] In this embodiment, the image and text data are public web page data. This solution first parses the public web page data through the web page DOM tree to obtain the spatial coordinates of the image data and text content.

[0048] For example, the publicly available web page data is parsed through the webpage DOM tree to determine the CSS box model of each image element (a rectangular CSS box model) and the CSS box model of each text element (a rectangular CSS box model). The CSS box model of each image element and the CSS box model of each text element are then parsed, and the absolute position of each CSS box model relative to the viewport is calculated. Selenium is then used to obtain the precise position of each CSS box model, thereby determining the spatial coordinates of the image data and text content.

[0049] This allows for efficient and accurate acquisition of the coordinates of each element (image data or text content) in public web page data. This solution can adapt to a variety of different web page layout modes (such as separate, hybrid, overlapping, etc.), automatically adapt to layout changes at different screen sizes, and also has good resistance to layout shifts.

[0050] Afterwards, an object detection model can be used to identify objects in the image data, generate object labels, and filter out target image data belonging to the target industry based on the object labels. This embodiment uses the YoLov8 model (other embodiments may use other object recognition models) and uses images of objects in the product fields provided by enterprises for market expansion to perform domain adaptation training to improve the accuracy of product recognition in that field. Based on this, objects in the image data are identified and object labels are generated. To maintain a consistent structure with the text labels corresponding to the text data, in this embodiment, the object labels are designed to contain feature quantities of two pieces of information: "item type" and "product field", and the length is also x.

[0051] Afterwards, target image data belonging to the target industry field can be filtered out based on the item tags. This process is mainly to exclude irrelevant image data (such as some button icons, advertising images embedded in web pages, etc.).

[0052] Accordingly, for each target image data: the spatial overlap between the target image data and each text content can be determined based on the spatial coordinates of the target image data and the spatial coordinates of each text content.

[0053] For example, the image area and the image influence area can be determined based on the spatial coordinates of the target image data. The image influence area is a peripheral area determined based on the image area (e.g., each side is extended outward by 0.2 times the length of the corresponding side. For example, if the image area is 200*200 in size, then the image influence area is 280*280 minus 200*200 of the image area). Figure 2 As shown (specific products have been coded). In addition, the area where the text is located can be determined based on the spatial coordinates of the text content, such as Figure 3 As shown (the specific text content has been coded), Figure 3 The diagram below shows the area where the text is located. In practice, there may be more complex situations because some web pages may contain multiple independent text elements in the same text content. Some text elements may also have associated links for users to click and jump to. We will not go into details here.

[0054] Taking into account the diversity of the layout patterns of public web page data in different regions or even in the same region, this solution specifically improves the traditional method of calculating spatial overlap. This embodiment can use the improved IoU formula to calculate the spatial overlap between the target image data and the text content:

[0055] , (1)

[0056] in, Represents the target image data With text content The spatial overlap between Indicates the Target image data, Indicates the Text content, and are weights, all of which are positive, and (In this embodiment, , , and The values ​​of should preferably differ by at least one order of magnitude, that is, at least 10 to the zth power, where z is the number of orders of magnitude difference and is a positive integer, which is convenient for distinguishing different web page layout modes later). Represents the target image data The image area, Represents the target image data The image influence area, Represents the target image data The image area and text content The intersection of the text area, Represents the target image data Image influence area and text content The intersection of the text areas.

[0057] The improved IoU formula considers both the image area and the image's influence area, adapting to different web page layouts (separate, hybrid, and overlapping). This improved IoU formula calculates the spatial overlap between the target image data and text content, assigning a higher weight to the core area (where the image resides) and a relatively lower weight to the peripheral area (where the image influences the image). For overlapping web page layouts, high spatial overlap typically indicates matching image data and text content. For hybrid and separate web page layouts, however, text content appearing outside the image data (e.g., below or to the right) needs to be considered. Based on this, an improved IoU formula was designed to calculate the spatial overlap between image data and text content. The numerical distribution of spatial overlap also reflects the web page layout, allowing for a comprehensive, adaptive matching calculation scheme to be developed for different layouts, improving the accuracy of the association between image data and text content.

[0058] After calculating the spatial overlap between each target image data and each text content, a shared semantic space mapping can be performed on each target image data and each text content to determine the semantic similarity between each target image data and each text content.

[0059] Specifically, this embodiment uses the CLIP (Contrastive Language-Image Pretraining, a multimodal pretraining neural network that is an extension of the CLIP architecture proposed by OpenAI for multilingual scenarios) model to expand visual and text understanding capabilities to dozens or even hundreds of languages, achieving cross-language image-text alignment. Accordingly, this embodiment calls an image encoder to process target image data to obtain an image semantic vector corresponding to each target image data, and calls a multilingual text encoder to process text content to obtain a text semantic vector corresponding to each text content. The image semantic vectors corresponding to all target image data (from the same public webpage data) and the text semantic vectors corresponding to all text content (from the same public webpage data) are then mapped to a shared semantic space (e.g., a shared embedding space, essentially a hypersphere space). Matrix multiplication is then performed to obtain a similarity matrix containing the semantic similarity between each target image data and each text content (each element represents a similarity. For example, if there are m target images and n text contents, the similarity matrix is ​​an m×n matrix, and the element value in the mth row and nth column represents the semantic similarity between the mth target image data and the nth text content). This solution can quickly and batch-calculate the semantic similarity of all target image data and text data in a public webpage data set, and has excellent cross-modal data similarity measurement capabilities.

[0060] This solution can map data from different modalities into a unified shared semantic space (e.g., based on the CLIP model), achieving cross-modal alignment of multimodal data and making it possible to measure semantic similarity between image data and text content in public web page data. Unlike traditional spatial association solutions, this solution considers both spatial overlap and semantic similarity when establishing the association between image data and text content, thereby improving the accuracy of the association and matching accurate item labels to text content in public web page data, effectively reducing information mismatch issues. Item labels can serve as pre-classification information for subsequent data analysis, providing more reliable data support for companies' market expansion decisions.

[0061] After obtaining the spatial overlap and semantic similarity between the target image data and each text content, the target text content matching the target image data can be determined based on the spatial overlap and semantic similarity between the target image data and each text content.

[0062] When calculating the comprehensive matching degree between the target image data and the text content, it's important to consider the distribution of spatial overlap and semantic similarity under different web page layout modes (overlapping, mixed, and separated). For overlapping layouts, a higher spatial overlap indicates a stronger correlation between the image data and the text content. For separated layouts, semantic similarity plays a greater role, but spatial overlap is also important. However, for mixed layout patterns, in the same public web page data, there are both overlapping image data and text content layouts and separated image data and text content layouts. This requires consideration of more complex issues, because the text content appearing above the image data and the text content appearing below the image data cannot be distinguished solely by spatial overlap. A higher weight should be assigned to semantic similarity, but it also depends on the situation. Because some public web page data only have a small number of overlapping layout areas, semantic similarity can be used as the main factor and spatial overlap as an auxiliary factor. For the case where there are a large number of overlapping layout areas in the public web page data, it is necessary to rely more on spatial overlap and reduce the proportion of semantic similarity.

[0063] Based on this, this solution has been tested and designed a suitable comprehensive matching degree calculation scheme and weight distribution method under the mixed arrangement mode (the set weights can be used in the other two arrangement modes without the need for adaptive adjustment).

[0064] First, for the calculation of comprehensive matching degree, the spatial overlap between the target image data and the text content needs to be standardized (for example, the spatial overlap is divided by the maximum value of all spatial overlaps, or other standardized calculation methods) to balance the differences in spatial overlap values ​​under different web page layout modes.

[0065] The comprehensive matching degree between the target image data and the text content is calculated as follows:

[0066] , (2)

[0067] in, Represents the target image data With text content The comprehensive matching degree between Represents the standardized target image data With text content The spatial overlap between Represents the target image data With text content The semantic similarity between Indicates the Target image data, Indicates the Text content, and are weights, all of which are positive, and .

[0068] As for the setting of weight distribution in the comprehensive matching degree calculation process: statistical feature extraction can be performed on the spatial overlap between the target image data and the text content (the spatial overlap before normalization needs to be used) to determine the target arrangement pattern between the target image data and the text content, where the target arrangement pattern includes overlapping arrangement pattern, mixed arrangement pattern and separated arrangement pattern.

[0069] Specifically, the maximum value of the spatial overlap between each target image data and the text content can be counted, and the average value of the maximum value of the spatial overlap corresponding to these target image data can be calculated (that is, the average value of the maximum value of the spatial overlap corresponding to each target image data is calculated). , The spatial overlap calculated in the case of maximum and minimum values ​​is not less than 0.2, which can be considered as overlapping arrangement mode; the maximum and minimum values ​​are less than 0.02, which can be considered as separated arrangement mode; and the maximum and minimum values ​​are less than 0.2 but not less than 0.02, which can be considered as mixed arrangement mode.

[0070] Since the overlapping arrangement mode and the separated arrangement mode are relatively stable arrangement modes, when calculating the comprehensive matching degree between the target image data and the text content, the weights of the overlapping arrangement mode and the separated arrangement mode are and You can set the value based on experience. For example, in the overlapped arrangement mode, Take 0.8 (or 0.7), Take 0.2 (or 0.3); in the separated arrangement mode, Take 0.4 (or 0.35), Take 0.6 (or 0.65).

[0071] The mixed arrangement mode is relatively complex and requires a more appropriate dynamic weight allocation scheme. Based on this, this embodiment can count the maximum spatial overlap between each target image data and the text content (the spatial overlap before normalization), and calculate the ratio of the number of target image data corresponding to the maximum spatial overlap exceeding the set value (for example, 0.4) to the total amount of target image data. Then, the weight is calculated using the following formula and :

[0072] , (3)

[0073] , (4)

[0074] in, It is the ratio of the number of target image data whose maximum spatial overlap exceeds the set value to the total amount of target image data.

[0075] After calculating the comprehensive matching degree between each target image data and each text content, the target text content that matches the target image data can be determined based on the comprehensive matching degree between the target image data and the text content. For example, the target text content with the highest comprehensive matching degree can be determined as the target text content that matches the target image data.

[0076] Overall, in order to achieve accurate association between image data and text content in graphic data (public web page data), this solution has designed a complete set of practical cross-modal matching solutions. It can effectively solve the information mismatch problem caused by extracting only text content from public web page data (for example, the text content lacks product information, while the product information is expressed in the image data, which is mainly a common problem in graphic data obtained from shopping websites). It achieves accurate association between image data and text content, improves the quality of feature data extracted based on public web page data, and provides a reliable data analysis foundation for subsequent corporate market expansion analysis.

[0077] After determining the target text content that matches the target image data, an association can be established between the target image data's item tags and each target text content. The item tags are converted into feature quantities of a set length. For example, an item tag, maintaining the same structure and dimensions as the text tag, is designed to contain two pieces of information: "item type" and "product category," and its length is also x.

[0078] After establishing the association between the target image data and the target text content, a multilingual translation model can be used to translate the text content. A pre-trained language model can be used to extract features and obtain a feature vector for the text content. For example, if the target region is the United States and the language is English, a multilingual translation model can be used to translate the text content (for major languages ​​such as English, Japanese, and Korean, this can be translated directly into Chinese. For lesser-known languages, English can be translated first, then into Chinese). A pre-trained language model (such as a natural language processing model based on BERT) can be used to extract features and obtain a feature vector for the text content. Similarly, a single piece of text content can generate multiple feature vectors, as a single piece of text content can correspond to multiple sentences, but this number is typically much smaller than the number of feature vectors corresponding to the text data.

[0079] After obtaining the feature vector of the text content, an item label can be added to the text content's feature vector. This ensures the structural consistency and dimensional uniformity of the feature vectors extracted from text and graphic data, and each feature data element carries pre-classification information (text label or item label). By employing differentiated feature extraction methods for text and graphic data, a unified data structure is constructed, and label information is added to the feature data for pre-classification. In enterprise market expansion analysis, different data analyses can be performed based on different pre-classification information (for example, product feature data is used to construct relevant market-based knowledge graphs, which are used to conduct market expansion analysis on enterprises in different dimensions, such as risk analysis and market competition analysis). Furthermore, when processing graphic data (primarily public web page data), this solution can establish an accurate connection between image data and text content, effectively resolving information mismatch issues and significantly improving the reliability of public web page data, providing more accurate data support for subsequent enterprise market expansion analysis.

[0080] After the characteristic data is determined, step S30 may be executed.

[0081] Step S30: integrating the feature data to form an analysis data source.

[0082] In this embodiment, feature data can be integrated, and further data subset integration can be performed based on pre-classified information for more detailed data analysis. The data analysis in this embodiment is primarily used for comprehensive market expansion analysis of enterprises. The dimensions of comprehensive market expansion analysis can include market environment dimensions, enterprise competition dimensions, etc., which are not detailed here. Of course, the integrated analysis data source can also be used for other purposes, which are not limited here.

[0083] In summary, the present embodiment provides an AI-based multimodal data processing method. This method obtains multimodal data (text and graphic data) from a target industry (the industry to which the products offered by the company for market expansion belong) in a target region (the country or region to which the company intends to expand its market). The method then preprocesses the multimodal data, extracts features, and preclassifies them to determine feature data (feature vectors containing preclassification information). Specifically, the method determines text labels from the text data, preprocesses the text data, translates the text data using a multilingual translation model, and then uses a pretrained language model to extract features to obtain text feature vectors. The method then combines the text labels and feature vectors of the text data to determine corresponding feature data. Furthermore, the method determines image data and text content from the graphic data. Object identification is performed on each image data to obtain an object label, and an association is established between the object label and the text content. The method then calls the multilingual translation model to translate the text content, and uses the pretrained language model to extract features to obtain a feature vector for the text content. Based on the object labels and the feature vectors of the text content, the method determines the feature data corresponding to the text content. The feature data can then be integrated to form an analysis data source. This solution adopts differentiated feature extraction methods for text data and graphic data to build a unified data structure, and adds label information to the feature data for pre-classification. In the enterprise expansion market analysis, different data analysis can be performed based on different pre-classification information (for example, product feature data is used to build relevant market-type knowledge graphs, which are used to conduct market expansion analysis of enterprises in different dimensions, such as risk analysis, market competition analysis, etc.). In addition, when processing graphic data (mainly public web page data), this solution can establish an accurate connection between image data and text content, effectively solving the information mismatch problem, greatly improving the reliability of public web page data, and providing more accurate data support for subsequent enterprise expansion market analysis.

[0084] For publicly available web data, the spatial coordinates of image data and text content are obtained by parsing the web page's DOM tree. Specifically, the DOM tree is used to parse the publicly available web data, determining the CSS box model for each image element and the CSS box model for each text element. The CSS box model for each image element and the CSS box model for each text element are then parsed and the absolute position of each CSS box model relative to the viewport is calculated. Selenium is then used to obtain the precise position of each CSS box model, thereby determining the spatial coordinates of the image data and text content. This method efficiently and accurately obtains the coordinates of each element (image data or text content) in the publicly available web data. This solution is adaptable to various web page layout modes (e.g., separated, hybrid, and overlapping), automatically adapts to layout changes across screen sizes, and exhibits robustness against layout drift. Object detection models (e.g., YoLov8) are used to identify objects in the image data, generate object labels, and filter out target images within the target industry based on the object labels, thus eliminating most irrelevant images (e.g., buttons, icons, and dynamic advertisements). For each target image data: based on the spatial coordinates of the target image data and the spatial coordinates of each text content, determine the spatial overlap between the target image data and each text content; perform shared semantic space mapping on each target image data and each text content to determine the semantic similarity between each target image data and each text content; for each target image data: based on the spatial overlap and semantic similarity between the target image data and each text content, determine the target text content that matches the target image data; establish an association between the object label of the target image data and each target text content. In this way, by mapping data of different modalities into a unified shared semantic space (for example, based on the CLIP model), cross-modal alignment of multimodal data can be achieved, making it possible to measure the semantic similarity of image data and text content in public web page data. Different from traditional spatial association schemes, this scheme considers both spatial overlap and semantic similarity when establishing the association between image data and text content, thereby improving the accuracy of the association and matching accurate item labels with text content in public web page data, thereby effectively reducing the problem of information mismatch. Item labels can provide pre-classification information for subsequent data analysis, which is conducive to providing more reliable data support for enterprises' market expansion decisions.

[0085] Considering the diverse layout patterns of public web data across different regions, and even within the same region, this solution specifically improves the traditional spatial overlap calculation method. This method considers both the image location and the image's influence area, adapting to different web layout patterns (separate, hybrid, and overlapping). The improved IoU formula calculates the spatial overlap between the target image data and text content, assigning higher weights to the core area (where the image resides) and relatively lower weights to the peripheral areas (where the image influences the image). For overlapping web layout patterns, high spatial overlap typically indicates matching image data and text content. For hybrid and separate web layout patterns, however, the inclusion of text content outside the image data (e.g., below or to the right) is required. Based on this, an improved IoU formula is designed to calculate the spatial overlap between image data and text content. The numerical distribution of spatial overlap also reflects the web layout pattern, allowing for a comprehensive, adaptive matching calculation scheme to be developed for different layout patterns, improving the accuracy of the association between image data and text content.

[0086] When calculating the comprehensive matching degree between the target image data and the text content, it's important to consider the distribution of spatial overlap and semantic similarity under different web page layout modes (overlapping, mixed, and separated). For overlapping layouts, a higher spatial overlap indicates a stronger correlation between the image data and the text content. For separated layouts, semantic similarity plays a greater role, but spatial overlap is also important. However, for the mixed arrangement mode, in the same public web page data, there are both overlapping image data and text content arrangements, and separated image data and text content arrangements. This requires consideration of more complex issues, because the text content appearing above the image data and the text content appearing below the image data cannot be distinguished solely by spatial overlap. A higher weight should be assigned to semantic similarity, but it also depends on the situation. Because some public web page data only have a small number of overlapping arrangement areas, semantic similarity can be used as the main factor and spatial overlap as an auxiliary factor. For the case where there are a large number of overlapping arrangement areas in the public web page data, it is necessary to rely more on spatial overlap and reduce the proportion of semantic similarity. Based on this, this scheme has been tested and designed a suitable comprehensive matching calculation scheme and a weight distribution method under the mixed arrangement mode (the other two arrangement modes can use the set weights without the need for adaptive adjustment). After calculating the comprehensive matching degree, the target text content corresponding to the target image data can be determined as the one with the highest comprehensive matching degree. Overall, in order to achieve accurate association between image data and text content in graphic data (public web page data), this solution has designed a complete set of practical cross-modal matching solutions. It can effectively solve the information mismatch problem caused by extracting only text content from public web page data (for example, the text content lacks product information, while the product information is expressed in the image data, which is mainly a common problem in graphic data obtained from shopping websites). It achieves accurate association between image data and text content, improves the quality of feature data extracted based on public web page data, and provides a reliable data analysis foundation for subsequent corporate market expansion analysis.

[0087] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0088] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A multimodal data processing method based on AI, characterized in that: include: Obtain multimodal data on target industries in target regions, where the target regions are the countries or regions in which the enterprise intends to expand its market, and the target industries are the industries to which the products provided by the enterprise for market expansion belong. Multimodal data includes text data and graphic data. After preprocessing the multimodal data, feature extraction and pre-classification are performed to determine feature data, where each feature data is a feature vector containing pre-classification information. When the image and text data is public web data, the association relationship between the object label of the image data and the text content in the public web data is constructed based on spatial overlap and semantic similarity to form feature data corresponding to the text content in the public web data; Integrate feature data to form an analysis data source; The method uses a method based on spatial overlap and semantic similarity to construct an association relationship between the item labels of image data and the text content in the public web page data, thereby forming feature data corresponding to the text content in the public web page data. The method includes: determining the image data and text content from the image and text data, performing item recognition on each image data to obtain the item label, and establishing an association relationship between the item label and the text content; Determine the image data and text content from the image and text data, perform object recognition on each image data, obtain the object label, and establish the association between the object label and the text content, specifically including: Parse public web page data through the web page DOM tree to obtain the spatial coordinates of image data and text content; use the object detection model to identify objects in the image data, generate object labels, and filter out target image data belonging to the target industry field based on the object labels; for each target image data: based on the spatial coordinates of the target image data and the spatial coordinates of each text content, determine the spatial overlap between the target image data and each text content; perform shared semantic space mapping on each target image data and each text content, and determine the semantic similarity between each target image data and each text content; for each target image data: based on the spatial overlap and semantic similarity between the target image data and each text content, determine the target text content that matches the target image data; establish an association relationship between the object label of the target image data and each target text content; Determining a spatial overlap between the target image data and each text content based on the spatial coordinates of the target image data and the spatial coordinates of each text content includes: Based on the spatial coordinates of the target image data, the image area and the image influence area are determined. The image influence area is a peripheral area determined based on the image area. Based on the spatial coordinates of the text content, the text area is determined. The improved IoU formula is used to calculate the spatial overlap between the target image data and the text content: ， in, Represents the target image data With text content The spatial overlap between Indicates the Target image data, Indicates the Text content, and are weights, all of which are positive, and , Represents the target image data The image area, Represents the target image data The image influence area, Represents the target image data The image area and text content The intersection of the text area, Represents the target image data Image influence area and text content The intersection of the text areas; Performing shared semantic space mapping on each target image data and each text content to determine the semantic similarity between each target image data and each text content, including: An image encoder is called to process target image data to obtain an image semantic vector corresponding to each target image data; a multilingual text encoder is called to process text content to obtain a text semantic vector corresponding to each text content; the image semantic vectors corresponding to all target image data and the text semantic vectors corresponding to all text contents are mapped to a shared semantic space, and matrix multiplication is performed to obtain a similarity matrix containing the semantic similarity between each target image data and each text content.

2. The AI-based multimodal data processing method according to claim 1, characterized in that: After preprocessing the multimodal data, feature extraction and pre-classification are performed to determine the feature data, including: Preprocess text data and graphic data; For each piece of text data: determine the text label from the text data, call the multilingual translation model to translate the text data, use the pre-trained language model to perform feature extraction to obtain the text feature vector, and then determine the feature data corresponding to the text data based on the text label and feature vector of the text data; For each piece of image and text data: determine the image data and text content from the image and text data, perform object recognition on each image data, obtain the object label, and establish an association between the object label and the text content. Then call the multilingual translation model to translate the text content, use the pre-trained language model for feature extraction, and obtain the feature vector of the text content. Based on the object label and the feature vector of this text content, determine the feature data corresponding to this text content.

3. The AI-based multimodal data processing method according to claim 1, characterized in that: Parse the public web page data through the web page DOM tree to obtain the spatial coordinates of image data and text content, including: Parse the public web page data through the web page DOM tree to determine the CSS box model of each image element and the CSS box model of each text element; Parse the CSS box model of each image element and the CSS box model of each text element, and calculate the absolute position of each CSS box model relative to the viewport; Use Selenium to get the exact position of each CSS box model to determine the spatial coordinates of image data and text content.

4. The AI-based multimodal data processing method according to claim 1, characterized in that: Based on the spatial overlap and semantic similarity between the target image data and each text content, the target text content that matches the target image data is determined, including: Normalize the spatial overlap between the target image data and the text content; The following formula is used to calculate the comprehensive matching degree between the target image data and the text content: ， in, Represents the target image data With text content The comprehensive matching degree between Represents the standardized target image data With text content The spatial overlap between Represents the target image data With text content The semantic similarity between Indicates the Target image data, Indicates the Text content, and are weights, all of which are positive, and ; Based on the comprehensive matching degree between the target image data and the text content, the target text content that matches the target image data is determined.

5. The AI-based multimodal data processing method according to claim 4, characterized in that: Before normalizing the spatial overlap between the target image data and the text content, the method further includes: Performing statistical feature extraction on the spatial overlap between the target image data and the text content to determine a target arrangement pattern between the target image data and the text content, wherein the target arrangement pattern includes an overlapping arrangement pattern, a mixed arrangement pattern, and a separated arrangement pattern; Based on the target arrangement pattern between the target image data and the text content, the weight distribution of the comprehensive matching degree calculation formula is determined.

6. The AI-based multimodal data processing method according to claim 5, characterized in that: When the target layout mode is a mixed layout mode, the weight and The following relations are satisfied: Counting the maximum spatial overlap between each target image data and the text content, and calculating the ratio of the number of target image data corresponding to the maximum spatial overlap exceeding the set value to the total amount of target image data; The weight is calculated using the following formula and : ， ， in, It is the ratio of the number of target image data whose maximum spatial overlap exceeds the set value to the total amount of target image data.

7. The AI-based multimodal data processing method according to claim 2, characterized in that: Based on the text label and feature vector of the text data, feature data corresponding to the text data is determined, including: For each feature vector of the text data, a text label is added before the feature vector, wherein the text label is converted into a feature quantity of a set length; Correspondingly, based on the item label and the feature vector of the text content, the feature data corresponding to the text content is determined, including: For the feature vector of the text content, an item label is added before the feature vector, wherein the item label is converted into a feature quantity of a set length.

Images