Semantic Tag-Driven Car Image Tagging Method

By constructing a semantic tag graph for automobiles, combining visual information and textual metadata, generating semantic tag probes and propagating the graph, the problems of low efficiency and poor accuracy in labeling automobile materials in existing technologies are solved, and automated and accurate tag recognition and management are achieved.

CN123115886APending Publication Date: 2026-11-17TAIDOU TECHNOLOGY (SHANGHAI) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611026445.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-10
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing methods for labeling automotive materials rely on manual annotation, which is inefficient and costly. Furthermore, keyword-based or single-image classification models struggle to simultaneously account for semantic constraints among vehicle instances, local regions, keyframes, shot clips, text information, and business tags, leading to inaccurate tag recognition and management difficulties.

Method used

A semantic tag graph for automobiles is constructed. By using visual information and text metadata to build tag nodes and semantic relationships between tags, semantic tag probes are generated. By combining graph propagation and region segmentation, the semantic confidence and region support values ​​of the tags are calculated to achieve automated tagging.

Benefits of technology

It improves the accuracy, applicability, and verifiability of automotive material labeling, reduces missing labels, incorrect labels, and conflicting labels, and supports refined management of various materials such as single images, videos, and image sets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115886A_ABST
    Figure CN123115886A_ABST
Patent Text Reader

Abstract

This invention relates to the field of automotive material labeling technology, specifically to a semantic tag-driven method for automotive material labeling. The method includes: acquiring automotive materials to be labeled and preprocessing them, extracting visual information and textual metadata; constructing an automotive semantic tag graph containing tag nodes and semantic relationships between tags, and configuring tag attribute records; generating semantic tag probes based on the graph, determining initial semantic evidence scores by combining visual information, textual metadata, and probes, and obtaining the graph semantic confidence score through graph propagation; dividing the image content into regions, calculating the regional support representation coefficients of each candidate region's tags, aggregating the regional support representation coefficients of all candidate regions to obtain the regional semantic support value of the tags; then determining the tag output state by combining the material quality confidence representation value, and binding the tag's scope of action to obtain the automotive material labeling result. This invention can reduce tag conflicts and omissions, and improve the convenience of verifying the labeling results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of automotive material marking technology, specifically to a semantic tag-driven automotive material marking method. Background Technology

[0002] With the development of automotive marketing, vehicle model management, material review, and content distribution, the number of automotive images, videos, serialized pictorials, marketing images, close-ups of vehicle details, interior materials, and materials including exterior effects is rapidly increasing. To facilitate material retrieval, content review, vehicle model matching, marketing campaigns, and asset management, automotive materials typically require tagging, such as by brand, model series, model, year, color, body parts, interior parts, shooting scene, marketing style, and content type.

[0003] Current methods for tagging automotive materials typically rely on manual annotation, simple keyword matching, single image classification models, or rule-based labeling. Manual annotation is inefficient and costly, and different annotators may have varying understandings of tags related to specific parts, shooting angles, and marketing styles, making it difficult to meet the needs of rapid data entry and refined management of large-scale automotive materials. Keyword- or metadata-based tagging methods mainly rely on filenames, notes, task fields, or business system fields, making it difficult to fully identify the true visual content in images or videos. Meanwhile, single image classification models typically focus on overall image-level visual recognition, making it difficult to simultaneously consider semantic constraints between vehicle instances, local areas, keyframes, shot clips, text information, and business tags.

[0004] Therefore, there is an urgent need for a semantic tag-driven method for marking automotive materials to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide a semantic tag-driven method for marking automotive materials: to perform accurate, stable and interpretable automatic marking of automotive materials, thereby improving the accuracy, applicability and verifiability of automotive material marking.

[0006] The objective of this invention can be achieved through the following technical solutions: A semantic tag-driven method for marking automotive materials, characterized in that the method includes: Obtain the car images to be labeled and preprocess them. The car images to be labeled include visual information and text metadata. Based on visual information and text metadata, a semantic tag graph for automobiles is constructed, which includes tag nodes and semantic relationships between tags. Tag attribute records containing tag dimensions, default scope, observable objects, and keywords are configured for tag nodes. The semantic tag probe is generated based on the semantic tag graph of automobiles. The initial semantic evidence score is determined according to the visual information of the material, the text metadata and the semantic tag probe. Based on the initial semantic evidence score, the tags are propagated in the graph according to the semantic relationship between the tags to obtain the graph semantic confidence of the tags. The image content in the visual information of the material is divided into regions to obtain several candidate regions. The region support representation coefficient of each candidate region is calculated, and the region support representation coefficients of all candidate regions are aggregated to obtain the region semantic support value of the label. The label's material quality confidence value is calculated, and the label's output state is determined based on the label's graph semantic confidence, regional semantic support value, and material quality confidence value. The label's scope of action is bound based on the label attribute record, and the labeling result of the car material to be labeled is obtained.

[0007] Furthermore, preprocessing includes: visual normalization of the car images to be tagged, quality inspection, text region recognition, keyframe extraction, and reading of associated metadata.

[0008] Furthermore, constructing a vehicle semantic tag graph containing tag nodes and semantic relationships between tags based on visual information and text metadata includes: The semantic relationships between tags are determined based on historical annotation samples, tag attribute records, business rules, and tag semantic associations. These semantic relationships include co-occurrence, mutual exclusion, and hierarchical relationships between tag nodes. Furthermore, the compatibility of tag scope is introduced when determining the semantic relationships between tags to ensure that the tag relationships correspond to the overall material, vehicle instances, local areas, keyframes, or shot clips.

[0009] Furthermore, calculating the compatibility of tag scopes when determining semantic relationships between tags includes: For any two labels, when both labels correspond to local regions in the image, the compatibility of the scope of action is calculated based on the area of ​​the label's scope of action. When both labels are full-image level labels, material level labels, or labels within the same shot, the compatibility of the label's scope of action is 1. When both labels belong to different vehicle instances, and the label dimensions of the two labels need to be judged within the same vehicle instance, the compatibility of the label's scope of action is 0.

[0010] Furthermore, determining the semantic relationships between tags includes: Calculate the co-occurrence strength between any two tags, and determine the co-occurrence relationship between the tags based on the co-occurrence strength; The mutual exclusion relationship between tags is determined based on the basic mutual exclusion strength and the effective mutual exclusion strength of any two tags; Calculate the hierarchy score between any two tags, and determine the hierarchy between the tags based on the hierarchy score.

[0011] Furthermore, the initial semantic evidence score is determined based on the visual information of the material, text metadata, and semantic tag probes, including: Visual features of the material are extracted from the visual information of the material, and the matching relationship between the visual features of the material and the semantic tag probe is determined; the matching relationship between the text metadata and the tag keywords is determined; the hit relationship between the vehicle database, business thesaurus or project thesaurus associated with the material and the candidate tags is determined based on the semantic tag probe; and the initial semantic evidence score is determined based on the matching relationship between the visual features of the material and the semantic tag probe, the matching relationship between the text metadata and the tag keywords and the hit relationship.

[0012] Furthermore, the calculation of the regional support characterization coefficients for each candidate region includes: Calculate the regional weight and component matching degree of each candidate region, and calculate the regional support characterization coefficient of each candidate region based on the regional weight and component matching degree of each candidate region.

[0013] Furthermore, calculating the regional weights of each candidate region includes: Calculate the area ratio of the candidate region based on the pixel area of ​​the candidate region; Calculate the centrality of the candidate region based on the coordinates of its center point. The ambiguity of the candidate region is calculated based on the Laplace variance. The regional weights of candidate regions are calculated based on their area proportion, centrality, and fuzziness.

[0014] Furthermore, the output state of the tag is determined based on the map semantic confidence, region semantic support value, and material quality confidence representation value, including: The joint semantic confidence of a tag is calculated based on the tag's graph semantic confidence, regional semantic support value, and material quality confidence representation value. When the joint semantic confidence is greater than a preset joint semantic confidence threshold, the tag is determined as an output tag. If the joint semantic confidence is less than or equal to the preset joint semantic confidence threshold, the tag is determined as a candidate pending tag.

[0015] Compared to existing solutions, the beneficial effects achieved by this invention are: This invention constructs a semantic tag graph for automobiles, organizing candidate tags such as brand, model series, vehicle type, model year, color, exterior components, interior components, shooting scene, marketing style, and content into a graph structure with co-occurrence, mutual exclusion, and hierarchical relationships. This allows the semantic associations between tags to participate in the tag judgment process. It reduces problems such as missed tags, mislabeling, coexistence of conflicting tags, and missing higher-level tags caused by isolated tag identification, thus improving the overall consistency of tagging results.

[0016] This invention targets various automotive materials, including single images, videos, image galleries, marketing images, close-ups of vehicles, and complex scenes with multiple vehicles. It comprehensively utilizes visual content, text metadata, business rules, semantic relationships between tags, and local area information to accurately, stably, and interpretably automatically tag automotive materials.

[0017] This invention distinguishes different scopes of application during the labeling process, such as the overall material, vehicle instances, local areas, text areas, keyframes, shot clips, and the entire video, to avoid accidental deletion or conflicting judgments of labels due to simple material-level judgments; and provides the scope of application of the labels and the source of evidence when outputting the labeling results, thereby improving the accuracy, applicability, and verifiability of labeling automotive materials. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0019] Figure 1 This is a flowchart of the first semantic tag-driven automotive material tagging method according to an embodiment of the present invention; Figure 2 This is a flowchart of the second semantic tag-driven automotive material tagging method according to an embodiment of the present invention; Figure 3 This is a flowchart of the third semantic tag-driven automotive material tagging method according to an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0022] This invention provides a semantic tag-driven method for marking automotive materials. Figure 1 This is a flowchart of a semantic tag-driven method for marking automotive materials according to an embodiment of the present invention. Figure 1 As shown, the method includes the following procedures: Step 1: Acquisition and preprocessing of the car images to be labeled; Collect the images of the cars to be labeled. These images include visual information and text metadata. Visual information materials can include single images, videos, serialized magazine photo albums, marketing images with text descriptions, close-ups of vehicle details, interior images, and images with exterior effects. Let's denote the car material to be tagged as _____. ,when When using image materials, record them as .when When using video footage, record it as The video footage was then processed to extract image frames, resulting in several image assets. Metadata can include filename, upload time, shooting task number, vehicle model information number, source of material, manual notes, and the project to which the material belongs. (Setting up car materials to be tagged...) The text metadata is denoted as ,in, Indicates the car material to be labeled The associated text information includes one or more of the following: file name, remarks, optical character recognition results, shooting task information, and database field information.

[0023] Preprocessing is performed on the car images to be marked. In the preprocessing stage, the car images to be marked can be normalized in size, converted in color space, detected in blur, detected in exposure, detected in text region, extracted in keyframe and read in metadata.

[0024] Step 2: Construction of the vehicle semantic tag graph; A semantic tag graph for automobiles is constructed based on visual information and textual metadata. The semantic tag graph for automobiles is denoted as: In the formula: Represents a semantic tag graph for automobiles; Represents a set of car tag nodes; Represents the set of tag co-occurrence relationships; Represents a set of mutually exclusive labels; This represents the set of tag hierarchy relationships. It also includes a set of candidate tag nodes. It includes multiple tag nodes. Tag co-occurrence, tag mutual exclusion, and tag hierarchy are considered as semantic relationships between tags, with any tag node being... ,in This represents a candidate car tag. Candidate car tags can include brand tags, model series tags, model year tags, body tags, interior tags, vehicle type tags, color tags, exterior option tags, exterior component tags, interior accessory tags, shooting scene tags, marketing style tags, and content tags.

[0025] In some embodiments, Figure 2 This is a flowchart of the second semantic tag-driven automotive material tagging method according to an embodiment of the present invention, as follows: Figure 2 As shown, the process of determining the semantic relationship between tags includes the following steps: Pre-determine each label Establish tag attribute records. Tag attribute records should include at least the tag's dimension, tag value, tag path, default scope, set of observable components, and set of keywords. The tag's dimension can be brand, vehicle series, model, year, color, shooting angle, external components, interior components, scene, or risky content; the default scope can be the entire image, vehicle instance, local area, keyframe, shot clip, or the entire video.

[0026] Obtain a set of historical annotation samples. Each historical annotation sample includes vehicle material, confirmed tags, tag scope, and sample confidence weight. Confirmed tags can come from manual review results, vehicle model databases, approved material libraries, or structured annotation records in business systems.

[0027] Determining the compatibility of the scope of action: For any two tags and tags First, determine the compatibility of their respective scopes of action. .in, and Each represents any tag in the set of semantic tags for automobiles. Indicates label and tags Whether it is suitable for both parties to participate in the relationship calculation within the same judgment scope.

[0028] When the label and tags When both correspond to local regions in the image, the compatibility of the effective range is calculated according to the following formula: in, Indicates label The area of ​​effect, Indicates label The area of ​​effect, Indicates label The area of ​​the effective region Indicates label The area of ​​the effective region This indicates the overlapping area of ​​two active regions. This indicates taking the smaller value. This indicates an extremely small positive number to prevent the denominator from being zero. The scope of a tag refers to the actual spatial and temporal range within the current automotive material where the tag is valid, as well as the vehicle instance to which it belongs. For image materials, the scope can be the entire image, the entire vehicle area, a specific component area, the background area, or the text area; for video materials, the scope also includes the corresponding keyframes or shot segments.

[0029] When the label and tags When all tags are full-image level tags, material level tags, or tags within the same shot clip, then... When the label and tags When they belong to different vehicle instances, and their respective label dimensions need to be determined within the same vehicle instance, let For example, in a car show image containing two cars, "white body" and "black body" might correspond to different vehicle instances. In this case, they are not simply treated as mutually exclusive tags in the same material. Instead, their scope compatibility and the vehicle instance are used to determine which tags apply to different vehicle instances.

[0030] Determining the co-occurrence relationship of tags: Tag co-occurrence relationships are used to indicate that two tags are valid simultaneously in the same material, the same vehicle instance, the same area, or the same shot.

[0031] For any two tags and tags Calculate the strength of the co-occurrence relationship between the two. : In the formula, This indicates the statistical co-occurrence level obtained based on historical labeled samples; Indicates label and tags Semantic probe similarity; This indicates the co-occurrence support value given by the business rules; Indicates label and tags Average range of action compatibility in historical samples; , and These represent the weight coefficients for statistical co-occurrence, semantic similarity, business rules, and scope compatibility, respectively. Optionally, , and The values ​​can be 0.3, 0.4, and 0.3 respectively. All values ​​in the above formula have been normalized before being substituted into the formula for calculation. Among them, the statistical co-occurrence degree... Labels can be used based on historically labeled samples and tags Simultaneous occurrence frequency determination; semantic probe similarity Through tags and tags The cosine similarity between semantic tag probe encoded vectors is determined.

[0032] when Greater than or equal to the preset co-occurrence threshold And the tag and tags If the relationship is not determined to be strong mutual exclusion, the label will be... and tags Write the tag co-occurrence relationship set.

[0033] Determining the mutual exclusion relationship of tags: Label mutual exclusion is used to indicate that two labels should not be simultaneously output as the definitive labels within the same decision scope. For any two labels... and tags First determine the basic mutual exclusion strength. When the label and tags When attributes belong to the same single-valued attribute dimension but have different values, or when they belong to a preset physical conflict tag pair, the base mutual exclusion strength is higher. The base mutual exclusion strength is preset in the tag map. During actual material tagging, the effective mutual exclusion strength in the current car material to be tagged is further calculated. In the above formula, This indicates the current automotive materials awaiting labeling. Chinese label and tags Effective mutual exclusion strength; Indicates label and tags Current automotive materials awaiting labeling Compatibility of the scope of action in the context; Indicates label and tags The indicator value indicating whether they belong to the same vehicle instance or the same judgment object. When the label and tags When they belong to the same vehicle instance or the same judgment object, When two items belong to different vehicle instances, different local regions, or different shot segments, and their respective label dimensions allow for coexistence across instances or shot segments, .

[0034] Based on effective mutual exclusion strength, mutual exclusion judgment is activated only when the label applies to the same vehicle instance, the same region, or the same judgment object. This avoids accidental label deletion caused by simple mutual exclusion at the material level in multi-vehicle images, multi-region materials, or multi-camera videos, improving the accuracy, stability, and interpretability of automotive material labeling results. It achieves case-by-case processing of "mutual exclusion within the same vehicle, non-mutual exclusion between different vehicles." For example, when an image contains both a white car and a black car, it is allowed to output "white body" and "black body" simultaneously, but they will be bound to different vehicle instances respectively; if the same car is identified as both white and black, mutual exclusion arbitration is triggered.

[0035] Determining the hierarchical relationship of tags: Tag hierarchy is used to represent parent-child, ancestor, or directory-level inclusion relationships between tags.

[0036] For candidate parent tags and candidate child tags Calculate the hierarchical relationship score : In this formula, Indicates label As a label The hierarchy score of the parent tag; Indicates the extent to which observable components are included; Indicates the degree of historical connotation, that is, when the label When it appears, the tag The probability that both are true; Indicates label With tags The basic mutual exclusion strength; , , These represent the weighting coefficients for each item. In some embodiments, , , The values ​​can be 0.3, 0.4, and 0.3 respectively.

[0037] Among them, the degree of inclusion of observable components The following formula can be used for calculation: Indicates label The set of observable components, Indicates label The set of observable components, This represents the number of intersections between two sets of observable components. Indicates label The number of observable components, This indicates an extremely small positive number to prevent the denominator from being zero. For example, the observable components of "headlight close-up" include the headlight, light strip, and lens, while the observable components of "front fascia close-up" include the headlight, grille, logo, and front bumper. Therefore, "headlight close-up" can be classified as a sub-tag of "front fascia close-up." Similarly, "steering wheel close-up" can be classified as a sub-tag of "interior material." And "center console close-up" can be classified as a sub-tag of "interior center console."

[0038] When a candidate parent tag Pair of child tags Hierarchical relationship score Greater than the preset level threshold When its score is higher than other candidate parent tags by a preset difference, the tag will be... With tags Write the tag hierarchy relationship set. If multiple candidate parent tags have similar scores, then add the tag... Mark as a level-based label pending review, and update the label path after manual review.

[0039] When the same tag pair simultaneously meets the criteria for co-occurrence, mutual exclusion, or hierarchical relationship, it is processed according to a preset priority: when the tag With tags When a strong mutual exclusion relationship is determined, it is written first into the tag mutual exclusion relationship set, and ordinary co-occurrence relationships between the two are suppressed; when the tag With tags When a parent-child or ancestor relationship is identified, it is preferentially written into the tag hierarchy set. During subsequent graph propagation, the confidence level is passed between the two through their hierarchical relationship, rather than being treated simply as ordinary co-occurrence tags; when the tag... With tags It does not belong to a strong mutually exclusive relationship, nor does it constitute a hierarchical relationship, but when the co-occurrence relationship strength reaches the co-occurrence threshold, it is written into the tag co-occurrence relationship set.

[0040] Set of tag co-occurrence relationships This is used to indicate the reasonableness of two labels appearing simultaneously in the same material or area. For example, "Interior - Steering Wheel Label" can co-occur, and "Exterior" and "Front 45 Degree Angle" can co-occur.

[0041] Set of mutually exclusive labels This indicates that two labels should not be output as distinct labels simultaneously within the same local area. For example, within the same local area, "front close-up" and "rear close-up" can be mutually exclusive, and "white body" and "black body" can be mutually exclusive within the same whole vehicle area.

[0042] Tag hierarchy relationship set Used to indicate the parent-child hierarchy between tags. For example, "car material" can be the parent tag of "exterior material", and "exterior material" can be the parent tag of "front close-up".

[0043] Step 3: Generate semantic tag probes; Generate a semantic tag probe for each candidate tag based on the vehicle semantic tag map. For any candidate tag... Its semantic tag probe is denoted as: in: Indicates label The corresponding semantic tag probe; Indicates label Visual semantic description; Indicates label A set of text keywords, which can be identified using the bag-of-words model; Indicates label The corresponding observable target or observable component; Indicates label The corresponding set of business judgment rules; for example, for the tag "front face close-up", its visual semantic description It can include the "local area directly in front of the vehicle"; its observable set This can include "grille", "car logo", and "front bumper"; its business determination rules. Rules can include "the front area ratio reaches a preset threshold" and "rear components do not appear at the same time with high confidence".

[0044] Step 4: Calculate the initial semantic evidence; For any candidate label Calculate the car material to be labeled Initial semantic evidence score for the label : in, This indicates automotive materials to be tagged. For tags The initial semantic evidence score; , , and These are the weight coefficients for different evidence items, which can be obtained through training on historical labeled samples or set manually. This represents the function for calculating cosine similarity. This indicates that the automotive materials to be tagged will be... The visual feature vector obtained after inputting into the visual encoding model, i.e., the visual features of the material, is obtained by concatenating or fusing multiple sub-features, including at least: overall visual semantic features of the material, vehicle instance features, local region features, observable component features, visual attribute features, scene background features, composition and shooting angle features, video keyframe temporal features, and visual quality features. The overall visual semantic features of the material are used to characterize the overall content semantics of the car material to be labeled at the whole image or whole frame level. The vehicle instance features are used to characterize the number, position, scale, bounding box, instance mask, and main appearance of vehicle targets in the image. The local region features are used to characterize the visual content of candidate regions obtained by target detection, instance segmentation, or semantic segmentation, including the whole vehicle region, front face region, rear region, headlight region, wheel hub region, window region, interior region, steering wheel region, and center console region. The data includes: instrument panel area, body texture area, background area, etc.; observable component features are used to characterize the matching of observable objects or observable components in the candidate area with the corresponding labels; visual attribute features are used to characterize vehicle color, body shape, texture, material, lighting, exterior optional parts, and local shape structure; scene background features are used to characterize the shooting environment, road, showroom, auto show, outdoor, studio shooting, urban road, natural background, or marketing scene in the material; composition and shooting angle features are used to characterize the vehicle's centrality, area proportion, viewing direction, foreground and background, close-up, and layout in the image; when the car material to be labeled is video material, video keyframe temporal features are used to characterize the visual changes between different keyframes or shot segments and the continuously appearing vehicle semantics; visual quality features are used to characterize the material's clarity, blur, exposure, occlusion, salience, and regional visibility. Indicates the tag probe The probe encoding model, which encodes semantic tag probes into tag probe feature vectors, is used to calculate similarity between the semantic tag probes and the visual feature vectors of the source material and the visual feature vectors of the region. The probe encoding model can include sub-models for text semantic encoding, visual description encoding, observable object encoding, business rule encoding, tag graph encoding, scope encoding, and feature fusion projection. The text semantic encoding sub-model encodes tag names, tag paths, tag dimensions, and tag keyword sets to obtain tag text semantic features; the visual description encoding sub-model encodes the visual semantic descriptions of tags to obtain corresponding visual semantic description features; the observable object encoding sub-model encodes observable targets or observable components corresponding to tags to obtain component semantic features and component visibility constraint features; the business rule encoding sub-model encodes the set of business judgment rules corresponding to tags to obtain rule constraint features; the tag graph encoding sub-model encodes the co-occurrence, mutual exclusion, and hierarchical relationships of tags in the automotive semantic tag graph to obtain graph structure features; the scope encoding sub-model encodes the default scope of tags to obtain scope features, where the default scope includes the entire material, a single image, a vehicle instance, a local region, a text region, a keyframe, a shot clip, or the entire video; the feature fusion projection sub-model concatenates, weights, or fuses the above sub-features, and maps them to a vector space that is the same as or comparable to the visual feature vector of the material to obtain the tag probe feature vector; This indicates that the source text metadata will be displayed. The text feature vector obtained after inputting the text encoding model; Indicates that the label Text keyword set Keyword feature vectors obtained after inputting the text encoding model; Indicates label The hard match score in the material metadata is calculated as follows: When At least one keyword appears in middle, When it is 1, No keywords appeared in the text. middle, =0; Indicates label The hit score in the vehicle model database, business terminology database, or project terminology database can be calculated as follows: when the vehicle model database field associated with the material matches the tag... Consistent, The value is 1, when the vehicle model database field associated with the material matches the tag. Partially consistent, The value is 0.5 when the vehicle model database field associated with the material matches the tag. Inconsistent =0; In some embodiments, the visual encoding model may be a convolutional network, a Vision Transformer, or a multimodal visual encoder; the text encoding model may be a BERT-type text encoder; the input is a car image frame, OCR text, file name, car model database field, and label description text; the output is a normalized visual vector, a text vector, and a label probe vector.

[0045] Step 5: Perform graph propagation on the initial semantic evidence based on the vehicle semantic tag graph to obtain the graph semantic confidence of the candidate tags.

[0046] No. After rounds of propagation, the label The semantic confidence of the graph is denoted as .

[0047] Graph propagation can be performed according to the following formula: In the above formula: Indicates the first Post-propagation label The semantic confidence of the graph; This indicates that the scores for all candidate labels are normalized; The propagation coefficient representing the co-occurrence relationship of labels; The attenuation coefficient representing the mutual exclusion relationship between tags; Support coefficients represent the hierarchical relationship between tags; Indicates label For tags The strength of the co-occurrence relationship; Label For tags The mutual exclusion suppression strength can be equivalent to the effective mutual exclusion strength of the two. ; Indicates label For tags The strength of hierarchical support can be equivalent to the score of the hierarchical relationship between the two. Indicates label No. Confidence level after round propagation.

[0048] During the initial propagation, we can let: Indicates label The initial normalized confidence; Represents a set of tags Any tag in; Indicates label The initial semantic evidence score.

[0049] Calculate the car materials to be labeled Tag entropy : Indicates the label after the map propagation ends. The semantic confidence of the graph; when Greater than the preset entropy threshold When this occurs, it means that multiple tags have a certain probability of being judged, and in this case, the relevant tags can be recorded as candidate tags to be determined. Step 6: Divide the image into regions and calculate the semantic support value of each region; Figure 3 This is a flowchart of the third semantic tag-driven automotive material tagging method according to an embodiment of the present invention, as follows: Figure 3 As shown, the process of dividing an image into regions and calculating the semantic support values ​​of those regions is as follows: When the car material to be labeled is an image or an image frame extracted from a video, the image is divided into regions. Let the input image be... For images Perform object detection, instance segmentation, or semantic segmentation to obtain a set of candidate regions: ;in, Represents the set of candidate regions; Indicates the first One candidate region; Indicates the sequence number of the candidate region; This indicates the total number of candidate areas. Candidate areas can include the whole vehicle area, instrument panel area, rear area, headlight area, wheel area, window area, interior area, steering wheel area, individual component area, body texture area, background area, and people area.

[0050] for Calculate the region weight : in: This indicates that the calculation results will be constrained to... The cutoff function between; , , , and These represent the weighting coefficients for area proportion, centrality, saliency, ambiguity, and occlusion, respectively. Indicates candidate region The area percentage; Indicates candidate region Centrality; Indicates candidate region Visual salience; Indicates candidate region ambiguity; Indicates candidate region The occlusion percentage, i.e., the candidate region Area of ​​the obscured part and candidate region The ratio of areas.

[0051] Candidate region area ratio It can be calculated using the formula: ; Indicates candidate region The pixel area; Indicates the input image The total pixel area.

[0052] Candidate region Centrality The following formula can be used for calculation: in: Indicates the input image The width; Indicates the input image Height; Indicates candidate region The coordinates of the center point; Candidate region ambiguity It can be calculated using Laplace variance: In this formula: This represents the variance calculation function; For candidate regions The response result obtained after processing with the Laplace operator.

[0053] Based on regional weight Calculate candidate regions For tags Regional support characterization coefficient : in, , , These are the weight coefficients for region similarity, component matching degree, and region weight, respectively. express function, Indicates candidate region The region's visual feature vector; Indicates label The corresponding label semantic feature vector; Indicates candidate region With tags The degree of matching of corresponding components; where, component matching degree The following formula can be used for calculation: Indicates label observable component set One of the components; Indicates components Corresponding features.

[0054] Aggregate all regions to tags The region support representation coefficients are used to obtain the region semantic support value. : After obtaining the graph semantic confidence and regional semantic support values ​​of the candidate labels, the two are fused to obtain the joint semantic confidence of the candidate labels.

[0055] Step 7: Calculate the joint semantic confidence. Acquire tags after the graph propagation ends Graph semantic confidence ,Label Regional semantic support value Reliable quality characterization values ​​of automotive materials to be labeled After normalizing the semantic confidence score, region semantic support value, and quality confidence factor of the graph, the labels are calculated by substituting them into the formula. Joint semantic confidence The formula is as follows: ;in, , and These are weighting coefficients, preferably set to 0.25, 0.45, and 0.3. The quality reliability characterization value of the automotive material to be labeled is also mentioned. The blurriness of the car image to be labeled is determined by the following process: the blurriness of the image is obtained by calculating the Laplacian variance of the corresponding image, which can be referenced from the candidate region above. ambiguity The calculation method involves recording the ambiguity of the vehicle image to be labeled as the reliable quality characterization value of the labeled vehicle image. When joint semantic confidence If the confidence level is greater than the preset joint semantic confidence threshold, then the label is determined. For the output label, if the semantic confidence is combined... The label is determined when it is less than or equal to the preset joint semantic confidence threshold. Candidate pending tags; After obtaining the joint semantic confidence, the candidate labels are... Perform scope binding. For any candidate tag Determine the tag based on the default scope recorded in the tag attribute record. The candidate scope types include one or more of the following: overall footage, single image, vehicle instance, local area, text area, keyframe, shot clip, and overall video.

[0056] Step 8: Output the marking results for the car images.

[0057] For any tag The output can be represented as: in: Indicates label The tag name; Indicates label Scope of application; Indicates label Source of evidence; Indicates label The output status includes output label or candidate pending label.

[0058] After the above processing, a tagging result containing the tag name, scope of application, and source of evidence can be generated. This method unifies the modeling of global semantics, local regions, vehicle instances, video time segments, text metadata, and tag graph relationships in automotive materials, enabling automatic tagging of single images, videos, image sets, marketing images, close-ups of vehicles, and complex materials involving multiple vehicles. This method not only outputs tagging results but also the scope of tag application and explanations of the evidence.

Claims

1. A semantic tag-driven method for marking automotive materials, characterized in that, The methods include: Obtain the car images to be labeled and preprocess them. The car images to be labeled include visual information and text metadata. Based on visual information and text metadata, a semantic tag graph for automobiles is constructed, which includes tag nodes and semantic relationships between tags. Tag attribute records containing tag dimensions, default scope, observable objects, and keywords are configured for tag nodes. The semantic tag probe is generated based on the semantic tag graph of automobiles. The initial semantic evidence score is determined according to the visual information of the material, the text metadata and the semantic tag probe. Based on the initial semantic evidence score, the tags are propagated in the graph according to the semantic relationship between the tags to obtain the graph semantic confidence of the tags. The image content in the visual information of the material is divided into regions to obtain several candidate regions. The region support representation coefficient of each candidate region is calculated, and the region support representation coefficients of all candidate regions are aggregated to obtain the region semantic support value of the label. The label's material quality confidence value is calculated, and the label's output state is determined based on the label's graph semantic confidence, regional semantic support value, and material quality confidence value. The label's scope of action is bound based on the label attribute record, and the labeling result of the car material to be labeled is obtained.

2. The semantic tag-driven automotive material marking method according to claim 1, characterized in that, Preprocessing includes: visual normalization of the car images to be tagged, quality inspection, text region recognition, keyframe extraction, and reading of associated metadata.

3. The semantic tag-driven automotive material marking method according to claim 1, characterized in that, A semantic tag graph for automobiles, constructed based on visual information and text metadata, including tag nodes and semantic relationships between tags, is as follows: The semantic relationships between tags are determined based on historical annotation samples, tag attribute records, business rules, and tag semantic associations. These semantic relationships include co-occurrence, mutual exclusion, and hierarchical relationships between tag nodes. Furthermore, the compatibility of tag scope is introduced when determining the semantic relationships between tags to ensure that the tag relationships correspond to the overall material, vehicle instances, local areas, keyframes, or shot clips.

4. The semantic tag-driven automotive material marking method according to claim 3, characterized in that, When determining the semantic relationships between tags, the compatibility of calculating the scope of tags includes: For any two labels, when both labels correspond to local regions in the image, the compatibility of the scope of action is calculated based on the area of ​​the label's scope of action. When both labels are full-image level labels, material level labels, or labels within the same shot, the compatibility of the label's scope of action is 1. When both labels belong to different vehicle instances, and the label dimensions of the two labels need to be judged within the same vehicle instance, the compatibility of the label's scope of action is 0.

5. The semantic tag-driven automotive material marking method according to claim 3, characterized in that, Determining the semantic relationships between tags includes: Calculate the co-occurrence strength between any two tags, and determine the co-occurrence relationship between the tags based on the co-occurrence strength; The mutual exclusion relationship between tags is determined based on the basic mutual exclusion strength and the effective mutual exclusion strength of any two tags; Calculate the hierarchy score between any two tags, and determine the hierarchy between the tags based on the hierarchy score.

6. The semantic tag-driven automotive material marking method according to claim 1, characterized in that, The initial semantic evidence score is determined based on visual information of the material, text metadata, and semantic tag probes, including: Visual features of the material are extracted from the visual information of the material, and the matching relationship between the visual features of the material and the semantic tag probe is determined; the matching relationship between the text metadata and the tag keywords is determined; the hit relationship between the vehicle database, business thesaurus or project thesaurus associated with the material and the candidate tags is determined based on the semantic tag probe; and the initial semantic evidence score is determined based on the matching relationship between the visual features of the material and the semantic tag probe, the matching relationship between the text metadata and the tag keywords and the hit relationship.

7. The semantic tag-driven automotive material marking method according to claim 1, characterized in that, The calculation of the regional support characterization coefficients for each candidate region includes: Calculate the regional weight and component matching degree of each candidate region, and calculate the regional support characterization coefficient of each candidate region based on the regional weight and component matching degree of each candidate region.

8. The semantic tag-driven automotive material marking method according to claim 7, characterized in that, Calculating the regional weights for each candidate region includes: Calculate the area ratio of the candidate region based on the pixel area of ​​the candidate region; Calculate the centrality of the candidate region based on the coordinates of its center point. The ambiguity of the candidate region is calculated based on the Laplace variance. The regional weights of candidate regions are calculated based on their area proportion, centrality, and fuzziness.

9. The semantic tag-driven method for marking automotive materials according to claim 1, characterized in that, The output state of a tag is determined based on the tag's graph semantic confidence, region semantic support value, and material quality confidence representation value. The joint semantic confidence of a tag is calculated based on the tag's graph semantic confidence, regional semantic support value, and material quality confidence representation value. When the joint semantic confidence is greater than a preset joint semantic confidence threshold, the tag is determined as an output tag. If the joint semantic confidence is less than or equal to the preset joint semantic confidence threshold, the tag is determined as a candidate pending tag.