Cosmetic product detail picture element extraction method and system based on multi-modal model

Abstract

The invention relates to the technical field of deep learning, in particular to a makeup product detail picture element extraction method and system based on a multi-modal model, and the method comprises the steps: extracting a text region sequence and a visual region sequence in a detail picture, and obtaining the position coordinates of each text region and each visual region; calculating a visual weight according to the information entropy of each visual area, and carrying out weighted correction on the initial visual features of the visual areas to obtain enhanced visual features; calculating the semantic similarity between the text features and the enhanced visual features; constructing a logic correlation coefficient according to a vertical distance between the text region and the visual region, and correcting the semantic similarity to obtain a comprehensive similarity; and determining a visual area corresponding to the text area according to the comprehensive similarity, and outputting an element extraction result. Through the technical scheme of the invention, accurate extraction and alignment of the claim text, the experimental data, the component graph and other visual demonstration elements in the detail graph of the beauty makeup product are realized.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a method and system for extracting elements from beauty product detail images based on a multimodal model. Background Technology

[0002] With the rapid development of mobile internet and e-commerce, online marketing of beauty products mainly relies on vertical long images containing rich text and images. To automate competitor analysis and intelligently generate product detail images, it is crucial to utilize artificial intelligence technology to transform unstructured detail images into structured text and image knowledge elements. In this specific application scenario, accurately mapping textual claims to supporting experimental data, ingredient formulas, or actual product photos is a key step in constructing a high-quality e-commerce product knowledge graph.

[0003] Existing image feature extraction technologies for beauty products typically employ Optical Character Recognition (OCR) to locate text regions and combine it with general object detection algorithms to identify visual elements in the image. When establishing the image-text relationship, the cosine similarity between the text feature vector and the image feature vector is calculated, and the image with the highest semantic score is used as the corresponding image for that text, thus completing the extraction of image and text features.

[0004] However, the above methods have difficulty distinguishing between substantive empirical images and decorative background images, which often leads to the text being incorrectly associated with the frequently appearing background, directly resulting in the inaccurate alignment of the claim text and image elements. Summary of the Invention

[0005] To address the technical problem of inaccurate alignment between claim text and image elements, this application provides a method and system for extracting elements from cosmetic product detail images based on a multimodal model. This method achieves accurate extraction and alignment of claim text with visual evidence elements such as experimental data and ingredient diagrams in cosmetic product detail images, providing auxiliary information for the creation of detail images.

[0006] In a first aspect, this application provides a method for extracting elements from detail images of beauty products based on a multimodal model. The extraction method includes: slicing the detail image to extract text region sequences and visual region sequences, and obtaining the position coordinates of each text region and visual region in the detail image; extracting text features and initial visual features corresponding to the text regions and visual regions; calculating visual weights based on the information entropy of each visual region, and using the visual weights to perform weighted correction on the initial visual features to obtain enhanced visual features; calculating the semantic similarity between the text features and the enhanced visual features; constructing a logical association coefficient based on the vertical distance between the text regions and visual regions, and using the logical association coefficient to correct the semantic similarity to obtain a comprehensive similarity, wherein the logical association coefficient represents the probability that there is a logical association between the text regions and the visual regions; determining the visual region corresponding to the text region based on the comprehensive similarity, and outputting the element extraction result.

[0007] Information entropy can be used to automatically distinguish between substantial content areas such as charts and images and invalid backgrounds. Logical correlation coefficients can be used to eliminate spatially irrelevant semantic mismatches, which significantly improves the accuracy and robustness of extracting image-text element association pairs in complexly formatted long beauty images.

[0008] Preferably, the extraction of text features and initial visual features corresponding to the text region and visual region includes: inputting the text region sequence into a recurrent neural network to obtain the text features of each text region; inputting each visual region image block in the visual region sequence into a convolutional neural network, and using the output of the fully connected layer as the initial visual features.

[0009] It leverages the ability of recurrent neural networks to handle sequence context dependencies and the ability of convolutional neural networks to extract deep image texture features; ensuring that the extracted text features and initial visual features can accurately represent the semantic connotations of each region, providing a data foundation for subsequent cross-modal similarity calculations.

[0010] Preferably, the step of calculating the visual weight based on the information entropy of each visual region includes: calculating the information entropy of each visual region, wherein the information entropy is the image entropy of the visual region or the information entropy of the grayscale histogram corresponding to the visual region; obtaining the maximum and minimum values ​​of the information entropy of all visual regions in the current details image; and normalizing the information entropy of each visual region based on the maximum and minimum values ​​to obtain the visual weight of each visual region.

[0011] The algorithm quantifies the differences in the amount of information contained in different visual regions, enabling it to automatically identify high-value regions with rich textures, such as line graphs containing data, and low-value regions with simple textures, such as solid-color backgrounds, in a data-driven manner. This provides an objective quantitative basis for subsequent feature weighting.

[0012] Preferably, the step of using the visual weights to perform weighted correction on the initial visual features includes: using the visual weights as weighting coefficients, and using the product of the initial visual features and the weighting coefficients as the enhanced visual features.

[0013] This allows high-information visual region features to be amplified in subsequent calculations, while low-information background noise features are suppressed to near zero, thus avoiding invalid decorative elements interfering with the matching process between text and key visual evidence.

[0014] Preferably, the step of constructing a logical correlation coefficient based on the vertical distance between the text region and the visual region includes: obtaining the vertical center coordinates of the text region and the visual region in a normalized coordinate system; calculating the Euclidean distance between the vertical center coordinates of the text region and the visual region; constructing a Gaussian function, and using the output of the Gaussian function with the Euclidean distance as the logical correlation coefficient.

[0015] A spatial decay mechanism that conforms to human reading habits is introduced, which makes the probability of association between text regions and visual regions decrease nonlinearly with the increase of vertical distance. Geometric position constraints eliminate erroneous matches that are semantically similar but have too large spatial spans.

[0016] Preferably, the standard deviation of the Gaussian function is positively correlated with the visual weight.

[0017] Preferably, the semantic similarity is corrected using the logical correlation coefficient to obtain a comprehensive similarity, including: calculating the cosine similarity between text features and enhanced visual features as the semantic similarity, and using the product of the semantic similarity and the logical correlation coefficient as the comprehensive similarity.

[0018] Preferably, before determining the visual region corresponding to the text region based on the comprehensive similarity, the extraction method further includes: normalizing the comprehensive similarity of any text region across all visual regions using the Softmax function to obtain attention weights; and calculating the alignment signal-to-noise ratio of the text region with respect to all visual regions, wherein the alignment signal-to-noise ratio is the ratio of the maximum value of the attention weights of the text region with respect to all visual regions to the average value of all attention weights.

[0019] Preferably, the output element extraction result includes: in response to the alignment signal-to-noise ratio of the text region being greater than the signal-to-noise ratio threshold, and its corresponding maximum comprehensive similarity being greater than the matching score threshold, determining that the text region has valid visual evidence support, and outputting the association pair between the text region and the visual region with the largest attention weight; otherwise, determining that the text region is a plain text claim.

[0020] By setting dual thresholds for alignment signal-to-noise ratio and maximum overall similarity, valid association pairs or plain text claims are determined. Only when the image-text matching has both high similarity and strong directionality, i.e., a high signal-to-noise ratio, is it considered valid. This reduces the output of false positive results and ensures the accuracy of the final product knowledge base.

[0021] In a second aspect, this application also provides a system for extracting elements from detail images of beauty products based on a multimodal model, including a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement the method for extracting elements from detail images of beauty products based on a multimodal model according to the first aspect of this application.

[0022] The technical solution of this application has the following beneficial technical effects: By slicing the detail images and extracting text and visual region sequences, and using visual weights based on information entropy to correct the initial visual features, combined with logical correlation coefficients based on vertical distance to correct semantic similarity, the texture information content and spatial topological relationships of the images can be incorporated into the matching degree calculation. This not only suppresses the noise interference of low-information background decorations on feature matching, but also simulates the cognitive logic of users reading from the nearest point. It effectively solves the problem of long-distance erroneous association caused by simple semantic similarity in long image scenarios, thereby achieving accurate extraction and alignment of claim text and visual evidence elements such as experimental data and ingredient diagrams in beauty product detail images. Attached Figure Description

[0023] Figure 1 This is a flowchart of a method for extracting details from beauty product images based on a multimodal model, according to an embodiment of this application.

[0024] Figure 2 This is a schematic diagram of the text area and visual area in the detailed images according to an embodiment of this application.

[0025] Figure 3 This is a schematic diagram of the information entropy and visual weight of each visual region according to an embodiment of this application.

[0026] Figure 4 This is a schematic diagram of the logical association probability matrix between text regions and visual regions according to an embodiment of this application.

[0027] Figure 5 This is a structural block diagram of a beauty product detail image element extraction system based on a multimodal model, according to an embodiment of this application. Detailed Implementation

[0028] According to the first aspect of this application, this application provides a method for extracting elements from product detail images based on a multimodal model, applied to e-commerce big data analysis in the beauty industry. Currently, product detail pages on major e-commerce platforms are typically vertically distributed long images, with a width of 750 pixels and a height often exceeding 15,000 pixels. These detail images contain complex text and image layouts, including text areas such as core efficacy claims, as well as corresponding visual areas such as experimental data line graphs, ingredient formulas, or efficacy test reports. The purpose of this application is to accurately extract the text areas and their corresponding visual area pairs, thereby constructing a structured product knowledge base and providing auxiliary information for creating product detail images.

[0029] Figure 1 This is a flowchart of a method for extracting details from beauty product images based on a multimodal model, according to an embodiment of this application. Figure 1 As shown, the method for extracting elements from beauty product detail images based on a multimodal model includes steps S101 to S106, which are described in detail below.

[0030] S101, slice the detail image, extract the text region sequence and visual region sequence, and obtain the position coordinates of each text region and visual region in the detail image.

[0031] In one embodiment, since the pixel size of the detail image exceeds the input limit of the neural network model, directly inputting the detail image into the neural network model will cause memory overflow or loss of details. Therefore, it is necessary to slice the detail image.

[0032] Specifically, an overlapping sliding window mechanism is used to segment the detail images. For example, the window height is set to 2048 pixels, and the step size is 1536 pixels, meaning the overlap height of two adjacent sliding windows is 512 pixels. This overlap is set to ensure that text or icons located at the edge of the slice are not truncated, thus guaranteeing the integrity of information across slices.

[0033] After slicing, two operations are performed in parallel on each slice image: First, optical character recognition (OCR) technology is used to extract all text regions and arrange them into a text region sequence according to reading order; second, object detection technology is used to extract all image elements, forming a visual region sequence. Simultaneously, the absolute pixel coordinates (i.e., position coordinates) of each text region and each visual region in the detail image are recorded. The object detection technology can employ the YOLO object detection model to identify each text region and visual region in the detail image; please refer to [link to relevant documentation]. Figure 2 This is a schematic diagram of the text and visual areas in the detail images according to embodiments of this application. Figure 2As can be seen, the details image includes five text areas: T_001 to T_005, and five visual areas: V_001 to V_005.

[0034] In this way, by overlapping slices and coordinate mapping, the problem of large computational load for ultra-long images is solved, while the spatial position of local elements in the global layout is fully preserved, laying a data foundation for subsequent logical relationship analysis.

[0035] S102, extract the text features and initial visual features corresponding to the text region and the visual region.

[0036] In one embodiment, after obtaining the text region sequence and the visual region sequence, each text region in the text region sequence is converted into a text feature, and each visual region in the visual region sequence is converted into an initial visual feature.

[0037] Specifically, extracting text features and initial visual features corresponding to text regions and visual regions includes: inputting the text region sequence into a recurrent neural network to obtain the text features of each text region; inputting each visual region image patch in the visual region sequence into a convolutional neural network, and using the output of the fully connected layer as the initial visual features.

[0038] Among them, recurrent neural networks can effectively capture the contextual dependencies in text region sequences. For example, this recurrent neural network employs a bidirectional long short-term memory network structure, i.e., Bi-LSTM, to simultaneously acquire forward and backward semantic information. Convolutional neural networks can extract deep features such as texture and shape from images. For example, this convolutional neural network uses a ResNet50 structure. Through mapping by fully connected layers, visual region image patches of different sizes are uniformly mapped into fixed-dimensional vectors, i.e., initial visual features.

[0039] In this way, by using deep learning models to transform text regions and visual regions into high-dimensional feature vectors, computers can measure the similarity between text and images in the feature space, providing a data foundation for subsequently determining the correspondence between text regions and visual regions.

[0040] S103, calculate the visual weights based on the information entropy of each visual region, and use the visual weights to perform weighted correction on the initial visual features to obtain enhanced visual features.

[0041] In one embodiment, beauty product detail images may contain decorative backgrounds such as solid color blocks or water ripples. These areas often do not contain substantial product information but can generate noise interference during calculations. To suppress invalid backgrounds and highlight visual areas containing rich information such as tables and molecular formulas, information entropy is introduced for feature weighting.

[0042] Specifically, calculating the visual weight based on the information entropy of each visual region includes: calculating the information entropy of each visual region, wherein the information entropy is the image entropy of the visual region or the information entropy of the grayscale histogram corresponding to the visual region; obtaining the maximum and minimum values ​​of the information entropy of all visual regions in the current details image; and normalizing the information entropy of each visual region based on the maximum and minimum values ​​to obtain the visual weight of each visual region.

[0043] Visual weight is used to quantify the importance of visual areas. For chart areas with rich textures and containing specific data, their information entropy is high, and their corresponding visual weight should also be large. For background decoration areas with simple textures, their information entropy is low, and their corresponding visual weight is small.

[0044] Wherein, the visual weight of the j-th visual region Satisfying the relation: ; In the formula, This represents the information entropy of the j-th visual region; This represents the maximum value of the information entropy of all visual regions in the current details image; This represents the minimum information entropy of all visual regions in the current details image. Please see [link / reference]. Figure 3 This is a schematic diagram of the information entropy and visual weight of each visual region according to an embodiment of this application.

[0045] The visual weights are used as weighting coefficients, and the product of the initial visual features and the weighting coefficients is used as the enhanced visual features. The enhanced visual features of the j-th visual region... Satisfying the relation: ;in, Let the initial visual features of the j-th visual region be... Let be the visual weight of the j-th visual region. If the j-th visual region contains rich information, Approaching 1, the enhanced visual features at this point Approaching the initial visual features That is, the feature information of the region is fully preserved; conversely, if the j-th visual region is low-information background noise, Approaching 0, the enhanced visual features at this point The values ​​of each parameter approach 0, effectively suppressing the feature information of this visual region.

[0046] In other embodiments, in order to further amplify the signal intensity of the effective region based on information entropy while preserving the initial visual features, the enhanced visual features of the j-th visual region are... It can also satisfy the following relation: ,in, Let the initial visual features of the j-th visual region be... Let be the visual weight of the j-th visual region, where Corresponding to the weighting coefficients.

[0047] Thus, by introducing an information entropy-based attention mechanism, we can automatically focus on high-information-content substantive content while ignoring low-information-content background noise.

[0048] S104, calculate the semantic similarity between text features and enhanced visual features.

[0049] In one embodiment, the cosine similarity between text features and enhanced visual features is used as the semantic similarity. The semantic similarity matrix can be obtained by calculating the cosine similarity between each text feature vector and each enhanced visual feature vector.

[0050] Thus, semantic similarity reflects whether textual features and enhanced visual features are close in semantic content. For example, the text about moisturizing and the image of water droplets are highly semantically related.

[0051] S105, construct a logical association coefficient based on the vertical distance between the text region and the visual region, and use the logical association coefficient to correct the semantic similarity to obtain a comprehensive similarity. The logical association coefficient represents the probability that there is a logical association between the text region and the visual region.

[0052] In one embodiment, simple semantic similarity can easily create illusions and lead to incorrect matching. For example, it may incorrectly associate a whitening title at the top of the page with a generic background image at the bottom of the page. In order to reduce incorrect matching caused by semantic similarity, spatial location is introduced to further correct semantic similarity.

[0053] Specifically, a logical correlation coefficient is constructed based on the spatial distribution relationship between the text region and the visual region. The logical correlation coefficient is used to characterize the probability that there is a logical relationship between the text region and the visual region. Based on the cognitive habit of reading from the nearest point of view, text usually has a logical relationship with its adjacent image. Therefore, a Gaussian function is introduced to construct the mapping relationship between spatial location and logical correlation coefficient.

[0054] Logical correlation coefficient between the i-th text region and the j-th visual region Satisfying the relation: ; In the formula, This represents the normalized vertical center coordinates of the i-th text region in the details image; This represents the normalized vertical center coordinates of the j-th visual region in the details image; Indicates the vertical distance between the two; is the standard deviation of the j-th visual region, used to control the effective correlation of the visual field range.

[0055] It should be noted that, in order to simulate the differences in visual appeal to users for images of varying importance, the standard deviation in the Gaussian function is... The visual weight is positively correlated with the visual weight; that is, the larger the visual weight of the j-th visual region, the smaller the standard deviation. The larger the standard deviation, the larger the effective visual field range associated with the j-th visual region. Satisfying the relation: In the formula, The basic field of view is exemplarily set to 0.05; To adjust the gain, an example value of 1.5 is used.

[0056] Understandably, the closer the vertical distance between the text area and the visual area, the closer the logical association coefficient is to 1, indicating a very high probability of logical association; as the vertical distance increases, the logical association coefficient gradually decreases to 0; please refer to [link to relevant documentation]. Figure 4 This is a schematic diagram of the logical association probability matrix between the text region and the visual region according to an embodiment of this application.

[0057] The calculation process of the normalized vertical center coordinates includes: obtaining the local pixel coordinates of the text region or visual region in the slice image; mapping the local pixel coordinates to the absolute center ordinate in the detail image coordinate system according to the sliding step size during slice processing; and normalizing the absolute center ordinate using the total height of the detail image, that is, using the ratio of the absolute center ordinate to the total height as the normalized vertical center coordinate.

[0058] In this embodiment, after obtaining the logical correlation coefficient between any text region and any visual region, the semantic similarity is corrected to obtain a comprehensive similarity. The comprehensive similarity between the i-th text region and the j-th visual region is... Satisfying the relation: ; In the formula, This represents the semantic similarity between the i-th text region and the j-th visual region. Let be the logical correlation coefficient between the i-th text region and the j-th visual region.

[0059] Thus, by introducing a logical correlation coefficient based on vertical distance, false matching over long distances is suppressed, and priority is given to image-text combinations that are spatially close, thereby effectively simulating human reading logic and eliminating the problem of false matching across regions.

[0060] S106: Determine the visual region corresponding to the text region based on the comprehensive similarity and output the element extraction results.

[0061] In one embodiment, for each text region, its overall similarity with all visual regions is iterated. If the maximum overall similarity is greater than the matching score threshold, it is considered a valid extraction, and the text region and the visual region corresponding to the maximum overall similarity are considered as an association pair. Conversely, if the maximum overall similarity is not greater than the matching score threshold, the text region is determined to be a plain text claim, and no corresponding visual region exists. Finally, the plain text claim and the association pair are used as the feature extraction results. For example, the text region "28-day fading lines" and the visual region "experimental data line graph" constitute an association pair.

[0062] The matching score threshold is set to 0.5.

[0063] In another embodiment, to ensure accurate feature extraction results and eliminate ambiguous noise data, before determining the visual region corresponding to the text region based on comprehensive similarity, the extraction method further includes: normalizing the comprehensive similarity of any text region across all visual regions using the Softmax function to obtain attention weights; and calculating the alignment signal-to-noise ratio of the text region with respect to all visual regions, wherein the alignment signal-to-noise ratio is the ratio of the maximum value of the attention weights of the text region with respect to all visual regions to the average value of all attention weights.

[0064] Since the sum of the attention weights for all visual regions is 1, the average of all attention weights is always equal to 1. , For the total number of all visual regions, therefore, the first Alignment signal-to-noise ratio of each text region Satisfying the relation: ; in, For the first Attention weights between the j-th text region and the j-th visual region For the first The maximum attention paid by a text region to all visual regions.

[0065] Understandably, the alignment signal-to-noise ratio can be viewed as the peak of the probability distribution of the overall similarity (i.e., the maximum value of the attention weights) and the uniform distribution (i.e., the average value of all attention weights). This comparison visually reflects the significance of the match. When the probability distribution of the overall similarity approaches a uniform distribution, the maximum value of the attention weight equals... At this point, it is impossible to accurately locate the first... The visual region associated with the text region, the first Alignment signal-to-noise ratio of each text region It equals 1.

[0066] At this point, the output element extraction results include: in response to the text region having an alignment signal-to-noise ratio greater than a signal-to-noise ratio threshold and its corresponding maximum comprehensive similarity greater than a matching score threshold, the text region is determined to have valid visual evidence support, and the association pair between the text region and the visual region with the highest attention weight is output; otherwise, the text region is determined to be a plain text claim. The signal-to-noise ratio threshold is 2; the matching score threshold is 0.5.

[0067] In this way, by comprehensively utilizing multimodal semantic information and spatial layout information, the accurate extraction and alignment of elements in beauty product detail images are achieved, providing reliable data support for building e-commerce product knowledge graphs and creating detail images for new products.

[0068] According to a second aspect of this application, this application also provides a system for extracting elements from beauty product detail images based on a multimodal model. Figure 5 This is a structural block diagram of a multimodal model-based image detail extraction system for beauty products, according to an embodiment of this application. Figure 5 As shown, the system 50 includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement the method for extracting details of beauty product images based on a multimodal model according to the first aspect of this application. The system also includes other components well-known to those skilled in the art, such as a communication bus and a communication interface. Their configurations and functions are known in the art and will not be described further here.

[0069] It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the scope of protection of this application.

Claims

1. A method for extracting elements from product detail images based on a multimodal model, characterized in that, The extraction method includes: slicing the detail image, extracting the text region sequence and the visual region sequence, and obtaining the position coordinates of each text region and visual region in the detail image; Extract text features and initial visual features corresponding to the text region and the visual region; Visual weights are calculated based on the information entropy of each visual region, and the initial visual features are weighted and corrected using the visual weights to obtain enhanced visual features. Calculate the semantic similarity between text features and enhanced visual features; A logical association coefficient is constructed based on the vertical distance between the text region and the visual region. The semantic similarity is then corrected using the logical association coefficient to obtain a comprehensive similarity. The logical association coefficient represents the probability that there is a logical association between the text region and the visual region. Based on the comprehensive similarity, the visual region corresponding to the text region is determined, and the feature extraction results are output.

2. The method for extracting elements from beauty product detail images based on a multimodal model according to claim 1, characterized in that, The extracted text features and initial visual features corresponding to the text region and visual region include: The text region sequence is input into a recurrent neural network to obtain the text features of each text region; Each visual region image patch in the visual region sequence is input into a convolutional neural network, and the output of the fully connected layer is used as the initial visual feature.

3. The method for extracting details from beauty product images based on a multimodal model according to claim 1, characterized in that, The calculation of visual weights based on the information entropy of each visual region includes: Calculate the information entropy of each visual region, where the information entropy is the image entropy of the visual region or the information entropy of the gray-level histogram corresponding to the visual region. Get the maximum and minimum values ​​of information entropy for all visual regions in the current details image; The information entropy of each visual region is normalized based on the maximum and minimum values ​​to obtain the visual weight of each visual region.

4. The method for extracting details from beauty product images based on a multimodal model according to claim 1, characterized in that, The step of using the visual weights to perform weighted correction on the initial visual features includes: using the visual weights as weighting coefficients, and using the product of the initial visual features and the weighting coefficients as the enhanced visual features.

5. The method for extracting details from beauty product images based on a multimodal model according to claim 1, characterized in that, The logical correlation coefficients constructed based on the vertical distance between the text region and the visual region include: Obtain the vertical center coordinates of the text region and the visual region in the normalized coordinate system; Calculate the Euclidean distance between the vertical center coordinates of the text region and the visual region; construct a Gaussian function, and use the Euclidean distance as the output of the Gaussian function as the logical correlation coefficient.

6. The method for extracting details from beauty product images based on a multimodal model according to claim 5, characterized in that, The standard deviation of the Gaussian function is positively correlated with the visual weight.

7. The method for extracting details from beauty product images based on a multimodal model according to claim 1, characterized in that, The semantic similarity is corrected using the logical correlation coefficient to obtain a comprehensive similarity, including: calculating the cosine similarity between text features and enhanced visual features as the semantic similarity, and using the product of the semantic similarity and the logical correlation coefficient as the comprehensive similarity.

8. The method for extracting details from beauty product images based on a multimodal model according to claim 1, characterized in that, Before determining the visual region corresponding to the text region based on the comprehensive similarity, the extraction method further includes: normalizing the comprehensive similarity of any text region across all visual regions using the Softmax function to obtain the attention weight; Calculate the alignment signal-to-noise ratio of the text region with respect to all visual regions. The alignment signal-to-noise ratio is the ratio of the maximum value of the attention weight of the text region with respect to all visual regions to the average value of all attention weights.

9. The method for extracting details from beauty product images based on a multimodal model according to claim 8, characterized in that, The output element extraction results include: in response to the alignment signal-to-noise ratio of the text region being greater than the signal-to-noise ratio threshold and its corresponding maximum comprehensive similarity being greater than the matching score threshold, the text region is determined to have valid visual evidence support, and the association pair between the text region and the visual region with the largest attention weight is output; otherwise, the text region is determined to be a plain text claim.

10. A system for extracting elements from detail images of beauty products based on a multimodal model, characterized in that, It includes a processor and a memory, the memory storing computer program instructions, which, when executed by the processor, implement the method for extracting details of beauty product images based on a multimodal model according to any one of claims 1 to 9.

Images