A label information management and abnormality identification method and system

Abstract

The application discloses a kind of label information management and abnormality identification method and system, belong to the field of computer vision, natural language processing and information management technical field, comprising: fusion computer vision and natural language processing technology, first through multi-task learning network extraction image feature, simultaneously complete label positioning, definition judgment and inclination angle correction;Again, visual language big model is used to identify label type and text content;Finally, in combination with search enhancement generation professional knowledge base comparison and visual language big model deep reasoning, identify translation error, illegal advertising, etc. Abnormality, output result and trigger work order and alarm.The application degree of automation is high, environmental adaptability is strong, scalable, suitable for smart city, traffic management and other fields, significantly improve the efficiency and intelligent level of label information management.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, natural language processing and information management technology, and in particular relates to a method and system for signage information management and anomaly recognition. Background Technology

[0002] Signage (such as road signs, directional signs, and notice boards) is an indispensable information carrier in social life and industrial production. With the acceleration of globalization and the increasing frequency of international exchanges, multilingual signage (such as directional signs, road signs, warning signs, and explanatory signs) has become an essential infrastructure in public places such as public transportation hubs (e.g., airports and train stations), tourist attractions, city streets, and international conference centers. Accurate and efficient identification and understanding of this signage information is crucial for pedestrian flow, safe crowd control, information access, and the city's image. Relying on manual inspections for signage management is inefficient, costly, prone to omissions, and highly subjective. Currently, automated processing technologies for signage information, especially those involving detection, recognition, translation, and anomaly detection, have become an important application area in computer vision and natural language processing.

[0003] Existing technical solutions typically employ a phased, sequential processing approach, which has numerous limitations:

[0004] (1) In the sign detection and text recognition stage:

[0005] Existing technologies largely rely on traditional object detection algorithms (such as models based on HOG features and SVM classifiers) or early deep learning models (such as YOLO and Faster R-CNN) to locate the signage area. Subsequently, optical character recognition (OCR) technology (such as Tesseract OCR or its deep learning-based improved version PaddleOCR) is used to extract text from the cropped signage area.

[0006] Existing defects:

[0007] Poor environmental adaptability: Existing models show a significant decrease in detection and recognition accuracy in complex scenarios such as drastic changes in lighting (e.g., reflections, shadows), severe weather (e.g., rain, snow, fog), partial occlusion, tilted or distorted shooting angles.

[0008] Insufficient support for mixed language text: Signage in public places often features a mix of Chinese and English text, as well as numbers and symbols. General-purpose OCR engines are prone to character segmentation errors and recognition confusion when faced with such mixed text, resulting in a chaotic extracted text sequence.

[0009] Ignoring non-text elements: Most existing solutions only focus on text content, while ignoring key visual information such as icons, logos, and color codes on the sign. This information often complements the meaning of the text, and its absence can lead to misunderstandings later on.

[0010] (2) In the text translation and semantic correction stage:

[0011] After obtaining the original text recognized by OCR, existing technologies typically call general machine translation services for direct translation.

[0012] Existing defects:

[0013] Lack of domain adaptability: General-purpose translation engines are not optimized for the language style specific to signage (such as the frequent use of imperative sentences, noun phrases, abbreviations, and proper nouns). For example, translating "No Entry" literally as "no entrance" instead of the correct "prohibited from entering" demonstrates poor contextual adaptability.

[0014] (3) In the anomaly identification and analysis stage:

[0015] Anomaly identification and analysis primarily involves identifying anomalies in the signage itself (such as damage, soiling, obstruction, or loss) and anomalies in its content (such as spam or advertisements). Currently, object detection and classification are mainly used for identification and analysis.

[0016] Existing defects:

[0017] Limited generalization ability: Small models are single-task-specific networks and lack the ability to recognize unknown types or types that have not appeared in the dataset.

[0018] Lack of contextual understanding: Small models rely heavily on local pixel information for recognition, lacking an understanding of scene semantics and spatiotemporal context.

[0019] Lack of zero-sample / few-sample recognition capability: Small models must be trained with a large number of samples to complete specific tasks, while large VLM models only need appropriate prompt words to identify and analyze them.

[0020] In summary, existing technologies suffer from significant drawbacks, including fragmented processing workflows, amplified error propagation between modules, poor environmental robustness, lack of semantic understanding and correction capabilities, and inability to automatically identify various anomalies. Therefore, there is an urgent need for a highly integrated and intelligent solution capable of fusing image and semantic recognition and analysis, and possessing robust anomaly detection and self-correction capabilities to improve the automation and reliability of signage information management. Summary of the Invention

[0021] To address the aforementioned technical problems, the present invention also provides a signage information management and anomaly identification system, comprising:

[0022] Image acquisition and input module, multi-task feature extraction and preprocessing module, sign classification and content recognition module, sign anomaly analysis module, output and response module;

[0023] The image acquisition and input module is used to acquire the sign image to be processed through the image acquisition device and transmit it to the multi-task feature extraction and preprocessing module;

[0024] The multi-task feature extraction and preprocessing module is used to extract features from the sign image, output the sign bounding box position, sharpness judgment result and tilt angle, and filter the sign image according to the sharpness judgment result;

[0025] The sign classification and content recognition module is used to crop and rotate the filtered image according to the sign bounding box and the tilt angle to obtain a corrected sign image; and to use a large visual language model to recognize the corrected sign image to obtain the sign type and text content.

[0026] The signage anomaly analysis module is used to build a professional knowledge base, compare and retrieve signage types and text content, combine visual language big data model reasoning analysis, and output signage anomaly analysis results.

[0027] Optionally, the multi-task feature extraction and preprocessing module includes a backbone feature extraction network and parallel access to a sign detection network, a sharpness classification network, and an angle classification network;

[0028] The backbone feature extraction network is used to extract multi-scale shared features of the sign image;

[0029] The sign detection network is based on the multi-scale shared features and adopts an anchor box target detection structure to output the sign bounding box and confidence score.

[0030] The sharpness classification network is used to output sharpness judgment results based on multi-scale shared features and a lightweight multilayer perceptron.

[0031] The angle classification network performs global average pooling on the multi-scale shared features and then performs angle classification, outputting the tilt angle; the tilt angle is taken as the midpoint value of the corresponding angle interval.

[0032] Optionally, the corrected sign image is input into a visual language model, and the model is guided by preset prompts to complete the sign type classification and text recognition extraction, and output the sign type and text content in a structured format, wherein the preset prompts are dynamic prompts.

[0033] Optionally, the professional knowledge base construction in the signage anomaly analysis module includes a data acquisition unit, a structured organization unit, a vectorization processing unit, and a vector database;

[0034] The data acquisition unit is used to acquire multi-source signage data;

[0035] The structured organization unit uses a hierarchical structured organization to store the multi-source signage data. The hierarchical structured organization includes six levels: signage category, standard text, multilingual comparison, image template, legal basis, and common error patterns.

[0036] The vectorization processing unit obtains text semantic vectors and image embedding vectors based on the multi-source signage data, performs joint encoding to generate a unified semantic space vector, stores the unified semantic space vector, and establishes an inverted index.

[0037] Optionally, the signage anomaly analysis module performs a professional knowledge base retrieval, including the following steps:

[0038] The sign type and text content are input into a multilingual model to generate a text semantic vector. An image vector is generated based on the corrected sign image to obtain a query vector. Based on the query vector, an approximate nearest neighbor search is performed in the vector database to obtain a preset number of candidate knowledge entries, completing the coarse search. A cross encoder is used to finely score the query vector and the preset number of candidate knowledge entries, and additional features are incorporated for re-sorting. A preset number of relevant knowledge fragments are output to complete the fine sorting.

[0039] Optionally, based on a preset number of relevant knowledge fragments, six levels of structured information are extracted to obtain variable content for generating appropriate prompt words; based on the variable content and the predefined anomaly analysis prompt word template, the variable content is filled into the corresponding placeholders in the template to obtain dynamic prompt words that can guide accurate anomaly analysis.

[0040] Optionally, the signage anomaly analysis module also includes automatic updates to the professional knowledge base. This involves obtaining knowledge base update requests through an automatic monitoring and triggering update module; collecting new knowledge based on these requests; verifying the collected new knowledge using a built-in expert rule engine or a manual review interface; vectorizing the verified new knowledge using a model consistent with the original knowledge base to obtain its embedding vector; inserting the new knowledge's embedding vector into the existing vector database and updating the index; and simultaneously implementing version control and rollback through knowledge base snapshots to obtain the updated professional knowledge base.

[0041] Optionally, after inserting the embedding vector of new knowledge into the existing vector database and updating the index, if the new knowledge in the updated professional knowledge base conflicts with the old knowledge, the new and old knowledge are resolved by prioritizing timestamps, weighting by authority, and providing a manual confirmation interface for major changes, thereby updating the professional knowledge base.

[0042] Optionally, the output and response module receives the sign boundary frame, sign type, text content, and anomaly analysis results, stores and visualizes the data, and triggers work order generation and system alerts when an anomaly is detected.

[0043] This invention provides a method for signage information management and anomaly identification, comprising:

[0044] The image of the sign to be processed is acquired using an image acquisition device;

[0045] Feature extraction is performed on the sign image, and the position of the sign bounding box, the sharpness judgment result and the tilt angle are output. The sign image is then filtered based on the sharpness judgment result.

[0046] Based on the sign's bounding box and the tilt angle, the filtered image is cropped and rotated to obtain a corrected sign image; a large visual language model is used to recognize the corrected sign image to obtain the sign type and text content.

[0047] A professional knowledge base is constructed, and the sign types and text content are compared and retrieved in the professional knowledge base. Combined with visual language big data model reasoning analysis, the sign anomaly analysis results are output.

[0048] The system receives the sign boundary frame, sign type, text content, and anomaly analysis results, stores and visualizes the data, and triggers work order generation and system alerts when anomalies are detected.

[0049] Compared with the prior art, the present invention has the following advantages and technical effects:

[0050] (1) High degree of automation and integration: It realizes full-process automation from image input to anomaly analysis, and seamlessly integrates multiple discrete technologies into a unified framework.

[0051] (2) Strong robustness: Through clarity judgment and angle correction preprocessing, the performance of subsequent recognition modules in complex real-world environments is improved.

[0052] (3) Deeper understanding: It goes beyond traditional OCR and uses VLM and RAG+LLM to achieve semantic understanding and knowledge-level reasoning and judgment of sign content, and can discover more complex anomalies.

[0053] (4) High scalability: The knowledge base can be continuously updated and expanded, enabling the system to adapt to new scenarios and identify new anomaly types. By modifying the prompt words, new analysis dimensions (such as style compliance checks, content sentiment analysis, etc.) can be easily added. Attached Figure Description

[0054] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0055] Figure 1 This is a schematic diagram of the technical process of an embodiment of the present invention. Detailed Implementation

[0056] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0057] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0058] Example 1;

[0059] like Figure 1 As shown, this embodiment provides a signage information management and anomaly recognition system, including:

[0060] The system includes an image acquisition and input module, a multi-task feature extraction and preprocessing module, a sign classification and content recognition module, a sign anomaly analysis module, and an output and response module.

[0061] The specific implementation of each module is as follows:

[0062] Module 1: Image Acquisition and Input Module;

[0063] The system acquires images of the signage to be processed using image acquisition devices (such as cameras, smartphones, drones, vehicle-mounted cameras, etc.) and inputs them into the system.

[0064] Module 2: Multi-task feature extraction and preprocessing module;

[0065] A shared backbone feature extraction network (such as ResNet, ConvNeXt, or other convolutional neural networks, or ViT, or other Transformer structures) is used to extract multi-scale shared features from the input image. The choice of specific network structure should comprehensively consider computational efficiency, feature representation capability, and compatibility with downstream tasks. In this embodiment, which focuses more on global modeling capabilities, a Transformer structure like ViT can be selected. To further adapt to the specific needs of this embodiment, the backbone feature extraction network can be specifically optimized, including but not limited to: introducing a feature pyramid structure to enhance multi-scale feature representation, employing an attention mechanism to strengthen the response of key regions, using knowledge distillation or model pruning to improve inference efficiency, and enhancing the domain adaptability of feature extraction by fine-tuning on a road scene dataset.

[0066] Based on the multi-scale features extracted from the shared backbone network, three dedicated task heads are connected in parallel, each responsible for the prediction of different sub-tasks, achieving end-to-end joint learning. The three dedicated task heads are as follows:

[0067] (1) Sign detection network: used to regress the bounding box positions of all signs in the image.

[0068] This approach employs target detection structures, including but not limited to anchor-based structures such as RetinaNet or YOLO, combined with an FPN (Feature Pyramid Network) structure to fuse multi-scale shared features, thereby improving the detection capability for signs of different sizes. The task head includes a classification branch (determining whether it is a sign) and a regression branch (predicting bounding box coordinates), outputting the sign's location and confidence score.

[0069] During training, a weighted sum of classification loss (such as Focal Loss, BCE Loss) and localization loss (such as IoU Loss or GIoULoss) is used as the optimization objective.

[0070] (2) Sharpness classification network: a classifier used to determine the sharpness of each candidate region image.

[0071] The classifier is designed as a lightweight multilayer perceptron (MLP), with input consisting of high-dimensional feature vectors obtained by applying multi-scale features to RoI Align or Global Average Pooling (GAP). The MLP contains 1–2 fully connected layers, using ReLU activation functions in between, and introduces Dropout to prevent overfitting. The classification task is defined as binary classification (sharp / blurred). To enhance discriminative ability, sharpness-annotated data or pseudo-labels generated using image quality assessment metrics (such as Laplacian variance) are introduced during training for weakly supervised learning.

[0072] This network is used to determine the sharpness of each candidate region image. In actual operation, after a large number of tests, the threshold is set to 0.7. If it is lower than this threshold, it is judged as blurry and no further processing is performed.

[0073] (3) Angle classification network: A classification network used to predict the tilt angle of an image.

[0074] Based on the multi-scale features extracted by the shared backbone network, an angle classification network is added to predict the global tilt angle of the input image, providing key parameters for subsequent image rotation correction. The task head is designed as a 90-class multi-classifier, uniformly dividing the 360° azimuth angle into 90 categories, each corresponding to a 4° angle interval, thereby achieving fine-grained discretization of image orientation.

[0075] The angle classification network consists of lightweight fully connected modules, and the specific processing steps include:

[0076] First, global average pooling (GAP) is performed on the multi-scale feature maps output by the backbone network to obtain a one-dimensional feature vector. Then, 1-2 fully connected layers (e.g., 512-dimensional hidden layers) are connected, with ReLU activation function and Batch Normalization and Dropout (e.g., 0.5) to improve generalization ability. Finally, the output layer is a 90-dimensional linear projection layer, followed by a Softmax function to output the probability distribution of each angle category.

[0077] The category division and classification threshold determination include:

[0078] Classification principle: Divide 360° into 90 equal categories, each covering a 4° interval, to ensure that the angular resolution is sufficient to support subsequent correction accuracy (e.g., within ±2°). This granularity strikes a balance between computational complexity and positioning accuracy.

[0079] Label generation method: The angle labels of the training samples are calculated based on the rotation angle θ∈[0°,360°) of the actual annotations to obtain the corresponding category index.

[0080] Boundary handling mechanism: For samples close to the 0° / 360° boundary (such as 358°), Circular Cross-Entropy Loss or label smoothing + circular neighborhood supervision (such as setting 1-2 adjacent classes as soft labels) can be used to alleviate the problem of discontinuous classification boundaries caused by the periodicity of angle.

[0081] The loss function is optimized using the standard cross-entropy loss function.

[0082] Random Rotation Augmentation is introduced during training to cover the entire angle range and improve the model's robustness to arbitrary tilt angles. At the same time, a resampling strategy or class-balanced loss is used to alleviate the class imbalance problem.

[0083] During inference, the model outputs the maximum probability index k of the angle category. * The corresponding predicted angle center value is:

[0084] θpred=4×k * +2;

[0085] That is, the midpoint of the 4° interval is taken as the final estimated angle.

[0086] The predicted angles are directly used in the affine transformation or perspective correction module of the image to achieve automatic rotation correction of the image.

[0087] Module 3: Signage Classification and Content Recognition Module;

[0088] This module is for signage content recognition based on the VLM multimodal large model.

[0089] Using the bounding box and rotation angle parameters obtained from module 2, each sign area is cropped and rotated to obtain a regular sign image.

[0090] The normalized sign image is input into a large visual language model (including but not limited to Qwen2.5-VL).

[0091] By designing specific prompts, VLM can be guided to complete two tasks simultaneously:

[0092] 1) Sign type classification: Output the category of the sign (such as "prohibition sign", "warning sign", "direction sign", etc.).

[0093] 2) Text recognition and extraction: Output all text content on the sign and output it in a structured text format (such as JSON).

[0094] Module 4: Signage Anomaly Analysis Module;

[0095] This module uses RAG and VLM to further analyze signage, such as identifying translation errors and illegal small advertisements.

[0096] A professional knowledge base was built based on RAG, which stores correct and standardized information related to various signs (such as standard traffic rules, precautions for various signs, multilingual translation tables, etc.).

[0097] Module 4.1, Building the RAG Professional Knowledge Base;

[0098] (1) Definition and functional positioning of professional knowledge base;

[0099] The aforementioned professional knowledge base is a structured, multimodal, and searchable knowledge system used to support semantic understanding, standardization verification, and retrieval of common anomaly information for signage content. Its core functions include:

[0100] Provides authoritative references for standard signage text, images, and semantics;

[0101] Store traffic regulations, urban management regulations, and other policy provisions to determine the compliance of signs;

[0102] Supports multilingual translation comparison to identify translation errors;

[0103] It supports collaboration with Visual Language Models (VLM) and Retrieval Augmented Generative Models (RAG) to enable context-aware intelligent reasoning.

[0104] (2) Data sources and collection methods for the knowledge base;

[0105] The data sources for the professional knowledge base include, but are not limited to:

[0106] Official regulatory documents: such as the "Road Traffic Safety Law of the People's Republic of China" and the "Regulations on Urban Appearance and Environmental Sanitation Management," and other publicly available government documents;

[0107] Standard Signage Atlas: Standard signage design specifications (including images and text) issued by traffic management departments and urban planning departments;

[0108] Multilingual translation comparison table: A bilingual / multilingual comparison table of standard terminology provided by language experts or official institutions (such as Chinese-English, Chinese-French, Chinese-Arabic, etc.);

[0109] Historical error correction data: Cases of erroneous signs accumulated from past manual reviews or system identification (such as mistranslations, misspellings, and unauthorized postings);

[0110] Expert knowledge input: Rules and experience provided by domain experts (such as traffic engineers, linguists, and city managers).

[0111] Data collection methods include:

[0112] Automated acquisition of publicly available information from government websites and standards databases;

[0113] The combination of manual data entry by operators and expert annotation ensures the accuracy of key clauses;

[0114] Access API interfaces of authoritative third-party knowledge platforms (such as the National Standards Full-Text Disclosure System);

[0115] Relevant data accumulated during system operation, such as abnormal local signs.

[0116] (3) The structured organization of the knowledge base;

[0117] To support efficient retrieval and semantic matching, the professional knowledge base is organized in a hierarchical structure, with the main levels shown in Table 1:

[0118] Table 1

[0119]

[0120] This knowledge base is based on vector databases (such as FAISS, Pinecone, and Milvus), embedding text, images, and semantics into vectorized storage, and supporting cross-modal retrieval.

[0121] (4) Vectorization and embedding representation;

[0122] To support efficient retrieval in the RAG model, each piece of knowledge in the professional knowledge base generates an embedding vector in the following way:

[0123] Text content: Semantic vectors are generated using multilingual Qwen3-Embedding and other models;

[0124] Image content: Image embeddings are extracted using VLM models such as CLIP and BLIP;

[0125] Multimodal fusion: Jointly encode image-text pairs to generate vector representations in a unified semantic space.

[0126] All vectors are stored in a vector database and an inverted index is built to support fast retrieval based on semantic similarity.

[0127] Module 4.2, Update Mechanism of Professional Knowledge Base;

[0128] (1) The necessity of a real-time update mechanism;

[0129] Because urban regulations, language usage habits, signage standards, and signage anomaly information may change over time, static knowledge bases cannot meet long-term application needs. Therefore, this invention proposes a closed-loop dynamic update mechanism to ensure the timeliness and accuracy of the knowledge base.

[0130] (2) Technical implementation of the update mechanism;

[0131] (a) Automatic monitoring and triggering of updates;

[0132] External data source monitoring: The system regularly crawls government websites and standard update announcements to detect regulatory revisions or the release of new signs;

[0133] User feedback channel: Receive feedback from city management units, the public, or review personnel to correct errors;

[0134] Model confidence monitoring: When the confidence of the RAG / VLM model in recognizing a certain type of signage continues to fall below the threshold, a knowledge base update request is triggered.

[0135] System supplement: Operators enter information through the annotation platform.

[0136] (b) Incremental knowledge injection process;

[0137] The update process is as follows:

[0138] New knowledge acquisition: Obtaining new regulations, new signage samples, or user feedback;

[0139] Expert review module: New knowledge is verified by a built-in expert rule engine or a manual review interface;

[0140] Vectorization: Generate embedding vectors using a model consistent with the original knowledge base;

[0141] Incremental update of vector database: Insert new vectors into the existing vector database and update the index;

[0142] Version control and rollback mechanism: Supports knowledge base snapshot saving to prevent system failure caused by erroneous updates.

[0143] (c) Knowledge conflict detection and resolution;

[0144] When new knowledge conflicts with old knowledge (such as when old regulations are repealed), the system handles it in the following ways:

[0145] Timestamp priority: based on the latest release time;

[0146] Authority weighting: Official government releases > Third-party sources;

[0147] Manual confirmation mechanism: Provides a manual confirmation interface for major changes.

[0148] (3) Incremental learning and model collaborative updating;

[0149] The updating of the professional knowledge base not only affects the search results, but should also be fed back to the RAG and VLM models. The specific process is as follows:

[0150] Regularly fine-tune the RAG search engine using newly acquired knowledge;

[0151] By continuously adjusting the prompts on the VLM using new samples, its ability to identify signage anomalies can be improved.

[0152] Construct a closed loop for the collaborative evolution of "knowledge base-model" to continuously improve the overall intelligence level of the system.

[0153] Module 4.3, RAG Professional Knowledge Base Retrieval;

[0154] In this embodiment, the retrieval module of RAG acts as a "knowledge hub," responsible for quickly retrieving the most relevant sign types, standard clauses, correct translations, legal basis, common anomalies, and other content from the professional knowledge base after receiving the sign image, sign type, and OCR information from module 3, so as to provide the generation model with comparison and reasoning.

[0155] (1) Query vectorization;

[0156] To achieve efficient retrieval, the input content needs to be transformed into a vector with a unified semantic space:

[0157] (a) Signage image vectorization: Extract image vectors using CLIP or BLIP models;

[0158] (b) Text vectorization of signage type and OCR information: semantic vectors are generated using the Qwen3-Embedding model.

[0159] (2) Retrieval mechanism;

[0160] The knowledge base is stored in vector databases (such as FAISS, Pinecone, Milvus) and uses Approximate Nearest Neighbor (ANN) retrieval to quickly find the content with the closest semantics.

[0161] (3) Multi-stage retrieval and re-ranking;

[0162] To improve accuracy, a two-stage strategy of coarse detection + fine sorting is adopted:

[0163] Phase 1: Coarse similarity check of vectors in the professional knowledge base;

[0164] Retrieve the Top-100 candidate knowledge entries from the professional knowledge base vector library; sort them based on cosine similarity or inner product distance.

[0165] Second stage: Semantic re-ranking;

[0166] A cross-encoder is used to finely score the query and candidate knowledge; additional features are incorporated, such as language consistency, font standardization, and layout similarity; the final Top-5 relevant knowledge fragments are output.

[0167] This module performs the following functions:

[0168] Use retrieval to achieve image search by image, obtain the sign type, and compare it with the sign type output by Module 3. If they are different, transfer it to manual special review.

[0169] Use retrieval to obtain the text translation of the sign.

[0170] Use retrieval to obtain the specifications, precautions, and common abnormal problem points related to the current sign.

[0171] Module 4.4, VLM recognition and analysis;

[0172] (1) Generation of VLM prompts;

[0173] Generate prompts by combining the sign type, text translation, specification precautions, and common abnormal problem points retrieved from the RAG professional knowledge base in Module 4.3 with predefined anomaly analysis instructions.

[0174] Organize the knowledge returned by RAG into structured data for subsequent templated generation:

[0175] Example:

[0176] “{

[0177] "sign_type": "Guide sign",

[0178] "required_elements": ["Destination name", "Direction arrow", "Distance (including unit)", "Exit number"],

[0179] "language_rules": {

[0180] "zh": "Use standard place names and prohibit using abbreviations (e.g., using 'Jing' instead of 'Beijing')",

[0181] "en": "The English translation should comply with the GA / T 1548-2019 standard, e.g., 'Airport' → '机场'"

[0182] },

[0183] "layout_rules": "The arrow should be on the right side of the text and point in the same direction; when the Chinese and English are in separate lines, the Chinese should be on top and the English should be at the bottom",

[0184] "common_errors":

[0185] "The unit of distance is not marked (e.g., '500' should be '500 meters')",

[0186] "English capitalization error (e.g., 'aIrPoRt')",

[0187] "The direction arrow does not match the text description."

[0188] "Check whether the sign is faded, dirty, tilted, etc."

[0189] "Check whether there is any bad information such as small advertisements on the sign."

[0191] }”。

[0192] Examples of predefined exception analysis instructions are as follows:

[0193] (a)For translation error recognition;

[0194] You are an expert in traffic sign language compliance. Please analyze the following bilingual road sign image:

[0195] This sign is a highway exit preview sign, which should include the destination, direction arrow, distance, and exit number;

[0196] According to the national standard (GA / T 1548-2019), the English word "Airport" should be translated as "机场", and cannot be written as "AirPort" or "Ariport";

[0197] Chinese place names should use the full name (e.g., "北京" cannot be abbreviated as "京");

[0198] The English case should be standardized (e.g., "Turn Right" cannot be written as "tUrN rIgHt");

[0199] Please check whether there are any spelling mistakes, inappropriate word usage, or non-standard expressions in the English translation on the sign.

[0200] If any problems are found, please indicate the specific location, type of error, and give suggestions for correct translation.

[0201] (b)For direction and layout compliance judgment;

[0202] Please analyze whether the visual layout of this road sign complies with the national standard:

[0203] Is the arrow on the right side of the text and aligned? For example, in "前方右转 →", the arrow should be on the right side of the text;

[0204] Is the arrow on the right side of the text and aligned? For example, in "前方右转 →", the arrow should be on the right side of the text;

[0205] Does it contain all the necessary information: destination name, distance (including the unit "meter"), exit number (such as "出口2")?

[0206] ​Is the font compliant: Use bold for Chinese and Arial Bold for English? Is the color white text on a green background?

[0207] Are there any layout issues such as text overlap, misalignment, or inconsistent font sizes?

[0208] Please compare with the standard templates in the professional knowledge base and point out any details that do not meet the specifications.

[0209] (c) For checking Chinese-English consistency;

[0210] This is a Chinese-English bilingual road sign. Please check its language consistency:

[0211] Does the English content accurately convey the Chinese meaning? For example, "Turn Right Ahead, 500m" should be translated for "前方500米右转", not "Right Turn 500m Front";

[0212] Is the order of Chinese and English information consistent? For example, if the Chinese is "机场 → 300米", the English should be "Aiport →300m";

[0213] Does the font size ratio meet the requirements? Chinese characters should be slightly larger than English (recommended 1.2 times);

[0214] Is there a situation where there is information only in Chinese and the English is missing?

[0215] If inconsistencies are found, please specify and provide the correct version that meets the standards.

[0216] Automated Prompt generation template:

[0217] This invention designs a dynamic prompt word generation engine to automatically fill in the template according to the RAG retrieval results:

[0218] You are an expert in traffic sign compliance analysis. Please analyze the following {sign_type} image:

[0219] According to the professional knowledge base, such signs should meet the following requirements:

[0220] - Required elements: {required_elements};

[0221] - Language specifications: {language_rules};

[0222] - Layout requirements: {layout_rules};

[0223] - Common errors include: {common_errors}.

[0224] Please check carefully:

[0225] 1. Does the text use standard names and standardized translations?

[0226] 2. Whether the Chinese and English content are semantically consistent and in the correct order;

[0227] 3. Does the arrow direction match the text description?

[0228] 4. Do the font, color, and spacing conform to national standards?

[0229] 5. Does it contain illegal content or advertising information?

[0230] 6. Check if the sign is faded, dirty, or tilted.

[0231] Please output a structured report, including: problem location, error type, relevant clause, and suggested corrections.

[0232] By filling the template with RAG search results, the system can automatically generate targeted and context-rich Chinese prompts, significantly improving the accuracy of VLM analysis.

[0233] (2) VLM analysis;

[0234] The signage images from Module 3 are analyzed in conjunction with the dynamic prompts mentioned above.

[0235] Based on its powerful reasoning capabilities, VLM outputs analysis results such as: "Content is correct and there are no anomalies", "There is a translation error, it should be 'XXX'", "Suspected small advertisement content 'XXX' was found", "Signage is faded and dirty", etc.

[0236] Module 5: Output and Response Module;

[0237] The system ultimately outputs the detection box, category, text content, and anomaly analysis results for each sign. These results are stored in the database and displayed on the system interface. If translation errors or other anomalies are detected, the system triggers work order systems, alarm systems, etc.

[0238] (1) Data storage: Persistently store the analysis results in a structured database (such as PostgreSQL or MySQL).

[0239] (2) Visualization: The analysis results can be presented in the user interface in the following ways:

[0240] Map visualization: Mark the locations of anomaly signs on the GIS map, with different colors indicating the error type (red = severe, yellow = warning, green = normal);

[0241] List view: Displays all test results in a table format, and supports filtering by category, region, time, and error type;

[0242] Details pop-up window: After clicking on a sign, the original image, detection box, OCR text, anomaly details, standard basis, and correction suggestions are displayed;

[0243] Comparison Mode: Displays "Actual Signage" and "Standard Template" side-by-side to visually demonstrate the differences.

[0244] (3) Triggering subsequent business systems (work orders, alarms, etc.):

[0245] The system supports automatically converting analysis results into actionable business processes, achieving a closed loop of "detection → response". If translation errors or other anomalies are identified, a work order is generated, including: a problem description, image and detection bounding box, correction suggestions, etc. After setting a processing time limit, the work order is dispatched and a warning is issued in the system.

[0246] User feedback and result correction mechanism (closed-loop update);

[0247] The system supports a closed-loop process of manual review and feedback to ensure that results can be corrected and knowledge can be evolved.

[0248] User correction process:

[0249] 1) Reviewers mark a result as "false positive" or "fixed" on the interface;

[0250] 2) The system records user actions;

[0251] 3) If it is a "false alarm", add the sample to the "negative sample library" for subsequent model optimization;

[0252] 4) If it is a "new error type", you can manually add an error description, and the system will learn it automatically.

[0253] Reverse updates of the knowledge base and model;

[0254] Knowledge base update:

[0255] New standards or revision suggestions confirmed by users can be submitted to the RAG professional knowledge base; the system will automatically trigger knowledge vectorization and index updates.

[0256] Model retraining: After accumulating a certain amount of manually corrected samples, start incremental training of the VLM / RAG model.

[0257] This embodiment also provides a method for signage information management and anomaly identification based on the above system, including:

[0258] The system acquires sign images to be processed using image acquisition equipment; extracts features from the sign images, outputting the sign bounding box position, sharpness judgment results, and tilt angle; filters the sign images based on the sharpness judgment results; crops and rotates the filtered images according to the sign bounding box and the tilt angle to obtain corrected sign images; uses a visual language model to recognize the corrected sign images, obtaining the sign type and text content; constructs a professional knowledge base, compares and retrieves the sign type and text content in the professional knowledge base, and combines the visual language model for reasoning analysis to output sign anomaly analysis results; receives the sign bounding box, sign type, text content, and anomaly analysis results, stores and visualizes them, and triggers work order generation and system alerts when a sign anomaly is detected.

[0259] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A signage information management and anomaly identification system, characterized in that, include: Image acquisition and input module, multi-task feature extraction and preprocessing module, sign classification and content recognition module, sign anomaly analysis module, output and response module; The image acquisition and input module is used to acquire the sign image to be processed through the image acquisition device and transmit it to the multi-task feature extraction and preprocessing module; The multi-task feature extraction and preprocessing module is used to extract features from the sign image, output the sign bounding box position, sharpness judgment result and tilt angle, and filter the sign image according to the sharpness judgment result; The sign classification and content recognition module is used to crop and rotate the filtered image according to the sign bounding box and the tilt angle to obtain a corrected sign image; and to use a large visual language model to recognize the corrected sign image to obtain the sign type and text content. The signage anomaly analysis module is used to build a professional knowledge base, compare and retrieve signage types and text content, combine visual language big data model reasoning analysis, and output signage anomaly analysis results; The corrected sign image is input into the visual language model, and the visual language model is guided by preset prompt words to complete the sign type classification and text recognition extraction, and output the sign type and text content in a structured format, wherein the preset prompt words are dynamic prompt words; The professional knowledge base construction in the signage anomaly analysis module includes a data acquisition unit, a structured organization unit, a vectorization processing unit, and a vector database. The data acquisition unit is used to acquire multi-source signage data; The structured organization unit uses a hierarchical structured organization to store the multi-source signage data. The hierarchical structured organization includes six levels: signage category, standard text, multilingual comparison, image template, legal basis, and common error patterns. The vectorization processing unit obtains text semantic vectors and image embedding vectors based on the multi-source signage data, performs joint encoding to generate a unified semantic space vector, stores the unified semantic space vector, and establishes an inverted index. The signage anomaly analysis module performs a professional knowledge base search, which includes the following steps: The sign type and text content are input into a multilingual model to generate a text semantic vector. An image vector is generated based on the corrected sign image to obtain a query vector. Based on the query vector, an approximate nearest neighbor search is performed in the vector database to obtain a preset number of candidate knowledge items, completing the coarse search. A cross encoder is used to finely score the query vector and the preset number of candidate knowledge items, and additional features are incorporated for re-sorting. A preset number of relevant knowledge fragments are output to complete the fine sorting. Based on a preset number of relevant knowledge fragments, extract structured information of six levels to obtain variable content for generating appropriate prompt words; based on the variable content and the predefined anomaly analysis prompt word template, fill the variable content into the corresponding placeholders in the template to obtain dynamic prompt words that can guide accurate anomaly analysis.

2. The system according to claim 1, characterized in that, The multi-task feature extraction and preprocessing module includes a backbone feature extraction network and parallel access to a sign detection network, a sharpness classification network, and an angle classification network; The backbone feature extraction network is used to extract multi-scale shared features of the sign image; The sign detection network is based on the multi-scale shared features and adopts an anchor box target detection structure to output the sign bounding box and confidence score. The sharpness classification network is used to output sharpness judgment results based on multi-scale shared features and a lightweight multilayer perceptron. The angle classification network performs global average pooling on the multi-scale shared features and then performs angle classification, outputting the tilt angle; the tilt angle is taken as the midpoint value of the corresponding angle interval.

3. The system according to claim 1, characterized in that, The signage anomaly analysis module also includes automatic updates to the professional knowledge base. It obtains knowledge base update requests through an automatic monitoring and triggering update module; collects new knowledge based on these requests; verifies the collected new knowledge using a built-in expert rule engine or a manual review interface; vectorizes the verified new knowledge using a model consistent with the original knowledge base to obtain the new knowledge's embedding vector; inserts the new knowledge's embedding vector into the existing vector database and updates the index; and simultaneously achieves version control and rollback through knowledge base snapshot saving to obtain the updated professional knowledge base.

4. The system according to claim 3, characterized in that, After inserting the embedding vector of new knowledge into the existing vector database and updating the index, if the new knowledge in the updated professional knowledge base conflicts with the old knowledge, the new and old knowledge will be resolved by prioritizing timestamps, weighting by authority, and providing a manual confirmation interface for major changes, thereby updating the professional knowledge base.

5. The system according to claim 1, characterized in that, The output and response module receives the sign boundary frame, sign type, text content, and anomaly analysis results, stores and visualizes the data, and triggers work order generation and system alerts when an anomaly is detected.

6. A method for signage information management and anomaly identification based on the system described in any one of claims 1-5, characterized in that, include: The image of the sign to be processed is acquired using an image acquisition device; Feature extraction is performed on the sign image, and the position of the sign bounding box, the sharpness judgment result and the tilt angle are output. The sign image is then filtered based on the sharpness judgment result. Based on the sign's bounding box and the tilt angle, the filtered image is cropped and rotated to obtain a corrected sign image; The visual language model is used to identify the corrected sign image to obtain the sign type and text content; A professional knowledge base is constructed, and the sign types and text content are compared and retrieved in the professional knowledge base. Combined with visual language big data model reasoning analysis, the sign anomaly analysis results are output. The system receives the sign boundary frame, sign type, text content, and anomaly analysis results, stores and visualizes the data, and triggers work order generation and system alerts when anomalies are detected.

Images