Picture and title entity recognition unification method based on multi-model collaboration

Abstract

The invention discloses a picture and title entity recognition unification method based on multi-model collaboration, and relates to the technical field of artificial intelligence, and the method comprises the following specific steps: data sampling annotation of an in-station post, unification of a label library, dual-model training verification, in-station mark verification of an online model, entity extraction when a user searches and browses, and retrieval of an entity library to obtain the post. Recommendation is carried out after calculation sorting and risk control filtering, and user behaviors are recorded; through a multi-model collaborative labeling and label unification mechanism, entity recognition accuracy and label normalization are remarkably improved, effective labels are screened by utilizing stratified sampling and multi-model fusion confidence, the problem of single-model errors is solved, meanwhile, through double-model collaborative training and multi-dimensional verification optimization, the model accuracy is ensured, and the accuracy and the reliability of the system are improved. And matching degree calculation and risk control filtering are combined in a recommendation link, so that the user interaction experience is improved, and efficient technical support is provided for platform content distribution.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a unified method for image and title entity recognition based on multi-model collaboration. Background Technology

[0002] With the rapid development of e-commerce and content-driven social platforms, the amount of post data within these platforms has exploded. Accurately identifying entity information within posts has become a core technological support for content categorization, personalized recommendations, and efficient retrieval. Currently, the application of artificial intelligence technology in entity recognition is deepening. Large-scale models, with their strong semantic understanding and cross-modal processing capabilities, are widely used for text and image entity annotation. Simultaneously, the construction and maintenance of entity databases, and multi-model collaborative optimization technologies, have become key directions for improving entity recognition accuracy and adapting to platform business needs.

[0003] Traditional entity recognition technologies have several limitations, making it difficult to meet the platform's requirements for entity recognition accuracy and label standardization. Firstly, most technologies rely on a single model for labeling, making them susceptible to insufficient generalization ability, resulting in high labeling errors. Furthermore, the lack of an effective confidence screening mechanism fails to guarantee the quality of initial labels. Secondly, entity label management lacks unified standards; labels from different sources often result in duplication or confusion due to differences in expression, and the absence of a scientific label association and entity database update mechanism affects the effectiveness of subsequent model training data. Thirdly, text and image entity recognition models are often trained independently, without considering the consistency of their recognition results. Moreover, the lack of multi-dimensional verification methods after deployment leads to insufficient reliability of entity labeling, consequently affecting the accuracy of downstream services such as recommendation and retrieval. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a unified method for image and title entity recognition based on multi-model collaboration. This method obtains on-site post data through stratified sampling, uses multi-model annotation and integrates confidence scores to filter effective tags, solving the problem of single-model annotation error; it adopts a unified entity tag association formula to standardize the tag system and update the entity database; it improves model performance through a dual-model training loss optimization formula; after deployment, it implements multi-dimensional verification to ensure the reliability of entity tagging; and in the recommendation process, it combines user history behavior and risk control filtering to achieve accurate and personalized recommendations, fully adapting to the needs of e-commerce and content social platforms.

[0005] To solve the above-mentioned technical problems, this invention provides the following technical solution: a unified method for image and title entity recognition based on multi-model collaboration, the specific steps of which are as follows: Data sampling and labeling: Stratified sampling of post data on the site by category, including complete text and associated images; select at least two large models, input 16 entity categories to define prompts and labeling examples; label the text and images of the models respectively to obtain initial labels, and use the multi-model labeling confidence fusion formula to filter valid labels; Tag library unification: Valid initial tags are compared with existing entity library tags. First, precise matching of entity phrases is performed. For those that do not match, semantic vector similarity is calculated. A unified entity tag association formula is used to determine the tag association relationship. Tags with semantic differences from the same entity are deduplicated and associated. New entity tags generate IDs and update the library. Dual-model training and validation: The unified label data is divided into training and validation sets. The text data is processed by word segmentation, stop word removal, and word vector embedding, while the image data is processed by resizing, normalization, and enhancement. The text and image entity recognition models are trained separately, and the total loss is optimized using the dual-model training loss optimization formula. After training, the validation set is labeled with the new and old models, and the entity recognition and category matching accuracy is calculated. On-site tag verification: Deploy the compliance model online and expose the API interface; call the interface to identify untagged posts, generate a tag set, merge and deduplicate, and update the post entity association; perform cross-validation, category and brand validation, and category completion on posts, and use a multi-dimensional validation comprehensive score formula to quantify tag quality; Entity association recommendation: Entities are extracted when users search, and historical entities are read to form a set when browsing; the entity ID is used to retrieve the entity database to obtain the post, and the matching degree is calculated and sorted using the search recommendation entity matching degree formula to obtain an initial list; after risk control filtering, the final list is returned and user behavior is recorded.

[0006] Furthermore, the formula for calculating the multi-model annotation confidence fusion in the data sampling and annotation step is as follows: ,in, The final fusion confidence level for entity e; The number of large models participating in the annotation; The weights of the m-th model satisfy... =1; This is the confidence level weighting coefficient with a value of 0.6; Let be the confidence score of the m-th model for the annotation of entity e; The semantic similarity between entity e labeled for the m-th model and the existing entity library E; when filtering the initial labels, retain Tags with a value of ≥0.7; When using stratified sampling, the sample size for each category is calculated as the total number of posts in that category on the site multiplied by the total sampling ratio, and then multiplied by the ratio of the total number of posts in that category to the total number of posts in all categories on the site, with a minimum of 50 samples per category.

[0007] Furthermore, the formula for calculating the unified association of entity tags in the tag library unification step is as follows: ,in, This refers to the degree of uniform association between entity e and similar tags in the entity database; The number of tags in the entity library that are of the same category as entity e; For the first entity in the database A label that is the same as the category of entity e; For entity e and label semantic similarity; (·) is an indicator function, and when equal hour The value is 1 if it is not 1, and 0 otherwise. The category of entity e; For entity library tags The category, with The category system is consistent; when When the similarity is ≥0.85, entity 'e' will be replaced with the tag in the entity library that has the highest similarity to 'e'. New entity IDs are generated according to the rule of first-level category ID plus entity category ID plus a 3-digit serial number, with the 3-digit serial number starting from 001 and increasing sequentially according to the order of addition; after the entity database is updated, 5% of the newly added entities are manually sampled for inspection.

[0008] Furthermore, the formula for optimizing the dual-model training loss in the dual-model training and validation step is as follows: ,in, The total loss during joint training of the two models; The text model loss weights are set to 0.5. The loss for the text entity recognition model; The loss of the image entity recognition model; The consistency loss weight is set to 0.3; This represents the consistency loss between text and image entity recognition results.

[0009] Furthermore, in the dual-model training and verification step, the text entity recognition model is an improvement on the Roberta-base model, with a classification head containing 16 types of entities added to the model output layer. The classification head adopts a combination structure of fully connected layers and Softmax activation function. The image entity recognition model is an improvement on the ResNet-50 model, with the original classification layer removed and entity detection branch and category classification branch added. The entity detection branch adopts the region proposal network structure of Faster R-CNN.

[0010] Furthermore, the formula for calculating the comprehensive score of multi-dimensional verification in the on-site tag verification step is as follows: ,in, This is the final validation score for feedf; , , These are weighting coefficients, with values ​​of 0.4, 0.3, and 0.3 respectively. The score for cross-validation of text and image entities; The score is used to verify category and brand recognition. Complete the score for the category.

[0011] Furthermore, during the text and image entity cross-validation step in the site tag verification process, images that do not contain category tags are removed from subsequent processing, and entity tag characters are replaced with numeric IDs according to the entity ID mapping table; when completing first- and second-level category tags, third-level categories use unrestricted tags, and standard third-level category names are determined through synonym mapping; when summarizing categories, if the category identification of one image among multiple images contained in a single feed is accurate, it is considered that the corresponding level category of that feed is accurate; when summarizing brands, if the brand identification of one image among multiple images contained in a single feed is accurate, it is considered that the brand identification of that feed is accurate.

[0012] Furthermore, the formula for calculating the matching degree of the search recommended entity in the entity association recommendation step is as follows: ,in, The entity matching degree between user u and feedf; Let u be the set of entities for user u; For the set of entities in feedf; for and The intersection of entities; The weight of entity e; for The number of entities included; for The number of entities included; for and The maximum value in the range; θ is the personalized recommendation weight and takes a value of 0.2; This refers to the historical interaction rate of user u with similar feeds; during risk control filtering, posts with inconsistent entity tags and content, violations of platform rules, and user-blocked categories are excluded.

[0013] Compared with existing technologies, this unified method for image and title entity recognition based on multi-model collaboration has the following advantages: I. This invention significantly improves the accuracy of entity recognition and the standardization of the labeling system through a multi-model collaborative annotation and label unification mechanism. First, stratified sampling of on-site post data by category is performed. At least two large models are selected to input 16 entity category definition prompts and annotation examples for annotation. Then, a multi-model annotation confidence fusion formula is used to filter effective labels, reducing the annotation error of a single model. Subsequently, a unified entity label association formula is used to de-duplicate and associate semantically different labels that point to the same entity, and the entity database is updated. This solves the problems of traditional label confusion and duplication, providing high-quality unified label data for model training and laying a solid foundation for entity recognition.

[0014] II. This invention balances technical performance and business application value through dual-model collaborative training and multi-dimensional application optimization. During the training phase, unified labeled data is divided into training and validation sets. Text and image data are preprocessed separately and then input into the corresponding models. The total loss is optimized using a dual-model training loss optimization formula, and the model accuracy is ensured through validation. After deployment, the compliant model is used to identify unlabeled posts on the platform. Tag quality is quantified using a multi-dimensional validation comprehensive score formula. In the recommendation phase, the matching degree is calculated using a search recommendation entity matching degree formula and combined with risk control filtering. This not only improves the user interaction experience but also provides efficient technical support for platform content distribution.

[0015] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0017] Figure 1 A flowchart of a unified method for image and title entity recognition based on multi-model collaboration; Figure 2 This is a framework diagram of a unified method for image and title entity recognition based on multi-model collaboration. Detailed Implementation

[0018] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0019] Example 1: Product Entity Recognition and Recommendation Scenario on E-commerce Platforms The specific implementation steps are as follows: Figure 1 As shown: Data Sampling and Labeling: For post data within the apparel category of an e-commerce platform, stratified sampling was conducted by subcategories such as dresses, coats, and pants. Each category sample was ensured to include product title and description text, as well as product detail images, with a minimum of 50 samples. Two large models were selected, and their input consisted of definition prompts and annotation examples covering 16 entity categories, including category, brand, material, and size. The annotation examples covered all 16 entity categories, with at least two examples per category. The models then labeled the text and images in the sampled data to obtain initial entity labels. A multi-model labeling confidence fusion formula was used to filter out valid initial labels. The multi-model labeling confidence fusion calculation formula is as follows: ,in, The final fusion confidence level for entity e; The number of large models participating in the annotation; The weights of the m-th model satisfy... =1; This is the confidence level weighting coefficient with a value of 0.6; Let be the confidence score of the m-th model for the annotation of entity e; The semantic similarity between entity e labeled in the m-th model and the existing entity library E.

[0020] Tag Library Unification: Valid initial tags are compared with the platform's existing apparel entity library. First, precise matching of entity phrases is used; for example, "pure cotton dress" is directly matched with "pure cotton dress" in the library. For tags that do not match, semantic vector similarity is calculated. A unified entity tag association formula is used to determine tag relationships. Tags with semantic differences but pointing to the same entity, such as "all-cotton dress" and "pure cotton dress," are deduplicated. New entity tags, such as "ice silk dress," generate new entity IDs. These new entity IDs are generated according to the rule of adding the primary category ID, the entity category ID, and a 3-digit sequence number, starting from 001 and incrementing. After the entity library is updated, 5% of newly added entities are manually checked. The unified entity tag association calculation formula is as follows: ,in, This refers to the degree of uniform association between entity e and similar tags in the entity database; The number of tags in the entity library that are of the same category as entity e; For the first entity in the database A label that is the same as the category of entity e; For entity e and label semantic similarity; (·) is an indicator function, and when equal hour The value is 1 if it is not 1, and 0 otherwise. The category of entity e; For entity library tags The category, with The category system is consistent.

[0021] Dual-model training and validation: The unified label data was divided into training and validation sets. For text data, jieba segmentation combined with a custom e-commerce dictionary was used for word segmentation, followed by stop word removal and word vector embedding preprocessing. For image data, resizing and normalization were performed, along with random horizontal flipping and brightness adjustment enhancements. The preprocessed data were then input into the text entity recognition model and the image entity recognition model for training. The text entity recognition model was an improvement on the Roberta-base model with an added classification head containing 16 entity classes in the output layer. The image entity recognition model was an improvement on the ResNet-50 model, removing the original classification layer and adding entity detection and category classification branches. During training, the dual-model training loss optimization formula was used to optimize the total loss. The dual-model training loss optimization formula is as follows: ,in, The total loss during joint training of the two models; The text model loss weights are set to 0.5. The loss for the text entity recognition model; The loss of the image entity recognition model; The consistency loss weight is set to 0.3; To account for the consistency loss between text and image entity recognition results, after training, the validation set data is labeled with the model before and after training, and the entity recognition accuracy and category matching accuracy are calculated to ensure that the model meets the standards.

[0022] Internal tagging verification: The qualified text and image entity recognition models are deployed online using Docker containerization and their API interfaces are exposed. The text entity recognition model container is configured with a 2-core CPU and 8GB of memory, while the image entity recognition model container is configured with a 4-core CPU, 16GB of memory, and one NVIDIA Tesla T4 GPU. Kubernetes is used to achieve automatic container scaling: one container replica is added when the API request volume exceeds 100 QPS, and one container replica is removed when the request volume is below 50 QPS. The API interface uses HTTPS protocol to transmit data, and API requests must include the API key and timestamp, with the timestamp valid for 5 minutes. The API is called to recognize untagged clothing posts on the platform, generating a set of text and image entity tags for each post. After merging and deduplication, the entity associations of the posts are updated. After marking, the posts undergo text and image entity cross-validation, category and brand multi-model validation, and first- and second-level category tag completion. During text and image entity cross-validation, images without category tags are marked and removed, and entity tag characters are replaced with numeric IDs according to the entity ID mapping table. For first- and second-level category tag completion, third-level categories use unrestricted tags, and standard names are determined through synonym mapping. A multi-dimensional validation comprehensive score formula is used to quantify tag quality, and posts with substandard tag quality are removed. The multi-dimensional validation comprehensive score calculation formula is as follows: ,in, This is the final validation score for feedf; , , These are weighting coefficients, with values ​​of 0.4, 0.3, and 0.3 respectively. The score for cross-validation of text and image entities; The score is used to verify category and brand recognition. Complete the score for the category.

[0023] Entity-based recommendation: When a user searches for "summer dress," a text entity recognition model is used to extract entities such as "summer dress." When a user has viewed posts related to "cotton dress," entity tags such as "cotton dress" associated with historical behavior are retrieved from the user's feature database to form a user entity set. The entity ID is then input into the entity database to retrieve the corresponding clothing post. The search recommendation entity matching degree formula is used to calculate the user-post matching degree and sort the posts to generate an initial recommendation list. After filtering out posts with entity tags that do not match the content, violate platform rules, or belong to user-blocked categories using risk control logic, the final recommendation list is returned to the user, and user behavior is recorded. The search recommendation entity matching degree calculation formula is: ,in, The entity matching degree between user u and feedf; Let u be the set of entities for user u; For the set of entities in feedf; for and The intersection of entities; The weight of entity e; for The number of entities included; for The number of entities included; for and The maximum value in the range; θ is the personalized recommendation weight and takes a value of 0.2; This represents the historical interaction rate of user u with similar feeds. After recording user behavior, the user's preference for various entities is calculated. The preference is calculated by dividing the number of interactions a user has with a certain type of entity by the total number of exposures for that type of entity. The top 5 entities with the highest preference are identified as the user's core entities, and posts containing these core entities are prioritized for matching in subsequent recommendation processes.

[0024] In summary, in the context of product entity recognition and recommendation on e-commerce platforms, this invention ensures data representativeness through stratified sampling, selects effective labels using a multi-model collaborative annotation and multi-model annotation confidence fusion formula, and then standardizes the label system and updates the entity database using a unified entity label association formula. During the training phase, jieba word segmentation and targeted data augmentation are used to optimize data preprocessing, and a dual-model training loss optimization formula is employed to improve model performance. During deployment, Docker containerization and Kubernetes scaling ensure stability. The recommendation process combines user preference calculation and risk control filtering to achieve accurate product entity recognition and efficient recommendation, fully adapting to the business needs of e-commerce platforms.

[0025] Example 2: Entity Recognition and Content Distribution Scenario on a Home Furnishing Content Social Platform The specific implementation steps are as follows: Figure 2 As shown: Data Sampling and Labeling: Home-related sharing posts published by users on home-related social media platforms were sampled stratified by space category (living room, bedroom, kitchen, etc.). The sampled data included user-shared text such as renovation tips, furniture recommendations, and home scene images, with at least 50 samples per category. Two large models were selected, with input definitions and annotation examples covering 16 entity categories, including furniture type, material, style, and space. Each entity category contained at least two examples. The models labeled the text and image entities in the sampled data, obtaining initial labels. Valid initial labels were then filtered using a multi-model labeling confidence fusion formula.

[0026] Tag Library Unification: Valid initial tags are compared with the platform's home furnishing entity library. First, precise matching of entity phrases is performed; for example, matching "solid wood sofa" with "solid wood sofa" in the library. For tags that don't match, semantic similarity is calculated. A unified entity tag association formula is used to deduplicate tags such as "solid wood fabric sofa" and "fabric solid wood sofa"—tags with semantic differences but pointing to the same entity. For new entity tags like "smart lift coffee table," a new ID is generated according to the rule of adding the first-level category ID, the entity category ID, and a 3-digit serial number. The 3-digit serial number increments from 001. After the entity library is updated, 5% of newly added entities are manually sampled for inspection.

[0027] Dual-model training and validation: The unified-label data was divided into training and validation sets. For text data, jieba segmentation combined with a custom dictionary for the home furnishing domain was used, followed by stop word removal and word vector embedding. For image data, after resizing and normalization, data augmentation operations such as random horizontal flipping and brightness adjustment were performed. The processed data were then input into a text entity recognition model improved based on Roberta-base and an image entity recognition model improved based on ResNet-50 for training. The text entity recognition model included 16 entity classification heads, while the image entity recognition model included entity detection and category classification branches. During training, a dual-model training loss optimization formula was used to optimize the total loss. After training, the validation set was labeled using both the old and new models, and the entity recognition and category matching accuracy were calculated to ensure the model met the platform's recognition requirements.

[0028] Internal tagging verification: Compliant models are deployed online using Docker containers and their API interfaces are exposed. The text entity recognition model container is configured with a 2-core CPU and 8GB of memory, while the image entity recognition model container is configured with a 4-core CPU, 16GB of memory, and one NVIDIA Tesla T4 GPU. Kubernetes is used to automatically scale the containers; when the API request volume exceeds 100 QPS, one container replica is added, and when the request volume is below 50 QPS, one container replica is removed. The API interface uses HTTPS to transmit data, and API requests must include the API key and a timestamp, with the timestamp valid for 5 minutes. The API is used to identify untagged home furnishing posts on the platform, generate a set of text and image entity tags, merge and deduplicate them, and update the entity associations of the posts. After marking, the post is subjected to cross-validation of text and image entities, multi-model validation of categories and brands, and completion of first- and second-level category target tags. During the cross-validation of text and image entities, images without category target tags are removed and the tag characters are converted into numeric IDs. When completing the first- and second-level category target tags, the standard name of the third-level category is determined by synonym mapping. The tag quality is quantified by a multi-dimensional validation comprehensive score formula to ensure the reliability of post entity marking.

[0029] Entity-based recommendation: When a user searches for "modern minimalist living room decoration," a text entity recognition model is used to extract entities such as "modern minimalist living room." When a user browses posts related to "Nordic style bedroom furniture," related entities such as "Nordic style bedroom" are retrieved from the user's feature database to form a user entity set. The entity ID is input into the entity database to retrieve the corresponding home furnishing post. The matching degree is calculated using a search recommendation entity matching degree formula, and the list is sorted to generate an initial list. After risk control filtering to exclude illegal and low-quality content, the list is distributed to the user's homepage, while simultaneously recording user behavior. Based on the recorded user behavior, the preference degree for various entities is calculated. The preference degree is calculated by dividing the number of times a user interacts with a certain type of entity by the total number of times that type of entity is exposed. The top 5 entities with the highest preference degrees are identified as the user's core entities, and subsequent home furnishing posts containing these core entities are given priority in recommendations.

[0030] In summary, in the context of entity recognition and content distribution on home furnishing content social platforms, this invention starts with stratified sampling and multi-model labeling injection, constructing a standardized label library using a multi-model labeling confidence fusion formula and a unified entity label association formula. During training, data quality is improved through a domain-defined dictionary and data augmentation, while model accuracy is ensured by utilizing a dual-model training loss optimization formula. During deployment, containerization and HTTPS protocols ensure reliable system operation. The distribution process incorporates user core entity screening and risk control logic, achieving accurate entity recognition and personalized distribution of home furnishing content, effectively supporting content operation and user experience improvement on home furnishing content social platforms.

[0031] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A unified method for image and title entity recognition based on multi-model collaboration, characterized in that, The specific steps of this method are as follows: Data sampling and labeling: Stratified sampling of post data on the site by category, including complete text and associated images; select at least two large models, input 16 entity categories to define prompts and labeling examples; label the text and images of the models respectively to obtain initial labels, and use the multi-model labeling confidence fusion formula to filter valid labels; Tag library unification: Valid initial tags are compared with existing entity library tags. First, precise matching of entity phrases is performed. For those that do not match, semantic vector similarity is calculated. A unified entity tag association formula is used to determine the tag association relationship. Tags with semantic differences from the same entity are deduplicated and associated. New entity tags generate IDs and update the library. Dual-model training and validation: The unified label data is divided into training and validation sets. The text data is processed by word segmentation, stop word removal and word vector embedding, and the image data is processed by resizing, normalization and enhancement. Train text and image entity recognition models separately, and optimize the total loss using the dual-model training loss optimization formula; After training, the new and old models are used to label the validation set, and the accuracy of entity recognition and category matching is calculated. Internal tagging and verification: Deploy the compliant model online and expose the API interface; The API call identifies untagged POST requests, generates a tag set, merges and deduplicates the tags, and updates the POST entity associations. Cross-validation, category and brand validation, and category completion are performed on the posts. A multi-dimensional validation comprehensive score formula is used to quantify the quality of the tags. Entity association recommendation: Entities are extracted when users search, and historical entities are read to form a set when browsing; The entity ID is used to retrieve the post from the entity database. The matching degree is then calculated and sorted using the search recommendation entity matching degree formula to obtain an initial list. After being filtered by risk control, the final list is returned and user behavior is recorded.

2. The unified method for image and title entity recognition based on multi-model collaboration as described in claim 1, characterized in that, The formula for calculating the confidence fusion of multiple model annotations in the data sampling and annotation step is as follows: ,in, The final fusion confidence level for entity e; The number of large models participating in the annotation; The weights of the m-th model satisfy... =1; This is the confidence level weighting coefficient with a value of 0.6; Let be the confidence score of the m-th model for the annotation of entity e; The semantic similarity between entity e labeled in the m-th model and the existing entity library E.

3. The unified method for image and title entity recognition based on multi-model collaboration as described in claim 1, characterized in that, The formula for calculating the unified association of entity tags in the unified tag library step is as follows: ,in, This refers to the degree of uniform association between entity e and similar tags in the entity database; The number of tags in the entity library that are of the same category as entity e; For the first entity in the database A label that is the same as the category of entity e; For entity e and label semantic similarity; (·) is an indicator function, and when equal hour The value is 1 if it is not 1, and 0 otherwise. The category of entity e; For entity library tags The category, with The category system is consistent.

4. The unified method for image and title entity recognition based on multi-model collaboration as described in claim 1, characterized in that, The formula for optimizing the dual-model training loss in the dual-model training and validation step is as follows: ,in, The total loss during joint training of the two models; The text model loss weights are set to 0.

5. The loss for the text entity recognition model; The loss of the image entity recognition model; The consistency loss weight is set to 0.3; This represents the consistency loss between text and image entity recognition results.

5. The unified method for image and title entity recognition based on multi-model collaboration according to claim 1, characterized in that, In the dual-model training and verification step, the text entity recognition model is an improvement on the Roberta-base model, with a classification head containing 16 types of entities added to the model output layer. The classification head adopts a combination structure of fully connected layers and Softmax activation function. The image entity recognition model is an improvement on the ResNet-50 model, with the original classification layer removed and entity detection branch and category classification branch added. The entity detection branch adopts the region proposal network structure of Faster R-CNN.

6. The unified method for image and title entity recognition based on multi-model collaboration according to claim 1, characterized in that, The formula for calculating the comprehensive score of multi-dimensional verification in the site marker verification step is as follows: ,in, This is the final validation score for feedf; , , These are weighting coefficients, with values ​​of 0.4, 0.3, and 0.3 respectively. The score for cross-validation of text and image entities; The score is used to verify category and brand recognition. Complete the score for the category.

7. The unified method for image and title entity recognition based on multi-model collaboration according to claim 1, characterized in that, In the site tag verification step, during the cross-verification of text and image entities, images that do not contain category tags are removed from the subsequent processing flow, and the entity tag characters are replaced with numeric IDs according to the entity ID mapping table; when completing first-level and second-level category tags, third-level categories use unrestricted tags, and the standard third-level category name is determined through synonym mapping; when summarizing categories, if the category identification of one image among multiple images contained in a single feed is accurate, it is considered that the corresponding level category of the feed is accurate; when summarizing brands, if the brand identification of one image among multiple images contained in a single feed is accurate, it is considered that the brand identification of the feed is accurate.

8. The unified method for image and title entity recognition based on multi-model collaboration according to claim 1, characterized in that, The formula for calculating the matching degree of the search recommended entity in the entity association recommendation step is as follows: ,in, The entity matching degree between user u and feedf; Let u be the set of entities for user u; For the set of entities in feedf; for and The intersection of entities; The weight of entity e; for The number of entities included; for The number of entities included; for and The maximum value in the range; θ is the personalized recommendation weight and takes a value of 0.2; This represents the historical interaction rate of user u with the same type of feedf.

Images