An intelligent matching method and system for on-site images and monitoring cameras and a medium

Abstract

The embodiment of the application discloses a kind of field image and the intelligent matching method and system of monitoring camera, comprising: obtaining field image and come associated image, pre-process it, obtain structured document, carry out validity check, multi-dimension similarity fusion calculation, obtain comprehensive matching score;According to the total score threshold of pre-set, filter out Top10 matching result.The present application can overcome the problem that the method of early dependence SIFT, SURF and other bottom visual features is poor in robustness to light, shooting angle change and difficult to associate camera semantic information (such as scene type, landmark), make up the deficiency that the parsing dimension in the matching method based on text description is single, and multi-dimension feature fusion adopts fixed weight, while solving the problem that part of multi-dimension feature scheme does not filter invalid feature, and text extraction information is not sufficient in cleaning, finally improve the accuracy, robustness and generalization ability in complex urban scene of image and monitoring camera matching.

Description

Technical Field

[0001] This invention relates to the field of urban security and monitoring system deployment technology, specifically to a method, system, and storage medium for intelligent matching of on-site images and surveillance cameras. Background Technology

[0002] Early image matching methods relied heavily on low-level visual features, comparing key points, textures, colors, and other information extracted from images. However, these methods are sensitive to changes in lighting and shooting angles, have poor robustness in complex urban environments, and are difficult to directly correlate with the semantic information of surveillance cameras.

[0003] With the development of natural language processing technology, text-based matching methods have gradually emerged. These methods convert image content into text descriptions and then use text similarity algorithms to achieve matching. However, this technology often has two limitations: first, it has a single dimension in text parsing of images, focusing mainly on the overall description of the scene and ignoring fine-grained information such as landmarks and infrastructure, resulting in insufficient matching accuracy; second, it uses a fixed weight strategy when fusing multi-dimensional features, making it impossible to dynamically adjust the importance of each dimension according to the actual scene.

[0004] Furthermore, while some technical solutions attempt to combine multi-dimensional features, they fail to consider the effectiveness screening of features during similarity calculation. When a feature dimension is empty or has high noise, it will still participate in the calculation, further affecting matching accuracy. Simultaneously, existing technologies lack sufficient cleaning and processing of extracted text information, making them susceptible to interference from redundant information, leading to a decrease in the relevance of the text description to the actual scenario.

[0005] Furthermore, existing technologies still have some obvious limitations, such as the following:

[0006] (a) Although text cleaning can filter redundant information, it is insufficient in processing derivative words that are strongly related to monitoring, which may lead to the accidental deletion of valid information.

[0007] (ii) The camera vector pre-calculation adopts the full processing mode. When adding or deleting a device, all vectors need to be recalculated, which results in low dynamic maintenance efficiency. Especially when the number of monitored devices is large, it will significantly increase resource consumption and latency.

[0008] (iii) The first-level scenario adopts a strict matching strategy, which lacks tolerance for similar scenarios such as "main roads" and "secondary roads", thus limiting the adaptability to complex scenarios.

[0009] (iv) The weight parameters optimized by the genetic algorithm are overly dependent on the quality and coverage of the training samples, which can easily lead to overfitting in new scenarios. Furthermore, the weight allocation does not take into account the dynamic increase or decrease of feature dimensions under different scenarios, resulting in insufficient flexibility. Summary of the Invention

[0010] The purpose of this invention is to provide an intelligent matching method, system, and storage medium for on-site images and surveillance cameras, aiming to address the shortcomings of existing technologies in the field of intelligent matching of images and surveillance cameras. Specifically, it overcomes the problems of poor robustness to changes in lighting and shooting angle and difficulty in associating semantic information (such as scene type and landmarks) of early methods that relied on low-level visual features such as SIFT and SURF. It also makes up for the shortcomings of text-based matching methods, such as the single parsing dimension and the use of fixed weights for multi-dimensional feature fusion. At the same time, it solves the problems of some multi-dimensional feature schemes failing to filter invalid features and insufficient text information cleaning. Ultimately, it improves the accuracy, robustness, and generalization ability of image and surveillance camera matching in complex urban scenarios.

[0011] To achieve the above objectives, in a first aspect, embodiments of the present invention provide an intelligent matching method between on-site images and surveillance cameras, comprising:

[0012] The system acquires on-site images from grid workers and related images from surveillance cameras deployed on city streets.

[0013] The on-site images and associated images are preprocessed to obtain a structured document; the preprocessing includes image watermark removal, text extraction and deduplication, and structured parsing.

[0014] The structured document is subjected to a first-level scene matching check and a second-level scene intersection check to obtain the check results;

[0015] Based on the inspection results, a multi-dimensional similarity fusion calculation is performed to obtain a comprehensive matching score for multiple related images in the surveillance camera image library;

[0016] Based on a preset total score threshold, the Top 10 matching results are selected from multiple comprehensive matching scores to complete the intelligent matching of on-site images and surveillance cameras.

[0017] As a specific implementation of this application, the on-site images and associated images are preprocessed, specifically as follows:

[0018] An image-to-text conversion model is used to convert the on-site images and associated images into structured text descriptions containing multi-dimensional information. The text descriptions include scene classification, landmark network, structural features, infrastructure, ground features, environmental factors, and text extraction. The scene classification includes primary scenes and secondary scenes.

[0019] A multi-rule cleaning mechanism is used to adaptively clean the text description information; the multi-rule cleaning mechanism includes filtering timestamps and fixing irrelevant identifiers.

[0020] Retain derivative terms that are strongly related to monitoring, and standardize the format of null text annotations to obtain structured documents.

[0021] As a specific implementation of this application, the structured document is subjected to a first-level scene matching check and a second-level scene intersection check to obtain the check results, specifically as follows:

[0022] A pre-trained semantic model is used to vectorize the content of each dimension in the text description information to obtain multi-dimensional semantic features;

[0023] The validity of the multi-dimensional semantic features is determined, invalid features are excluded, and the valid features are obtained as the inspection result; the valid features include structural features, landmark networks, infrastructure, ground features, and text extraction.

[0024] As a specific implementation of this application, a multi-dimensional similarity fusion calculation is performed based on the inspection results, specifically as follows:

[0025] A genetic algorithm is introduced, which optimizes the initial weights of multiple effective features based on training samples and through a fitness function to obtain optimized weights.

[0026] Based on the optimized weights, the similarity between the scene image and the associated image is calculated for each of the multiple effective features, resulting in multiple similarity scores;

[0027] The combined matching score is obtained by weighted summation of multiple similarity scores.

[0028] Secondly, embodiments of this application provide an intelligent matching system for on-site images and surveillance cameras, comprising:

[0029] The acquisition unit is used to acquire on-site images from grid workers and related images from surveillance cameras deployed on city streets.

[0030] The image processing unit is used to preprocess the on-site images and related images to obtain a structured document; the preprocessing includes image watermark removal, text extraction and deduplication, and structured parsing.

[0031] The validity check unit is used to perform a first-level scene matching check and a second-level scene intersection check on the structured document to obtain the check results;

[0032] Similarity calculation is used to perform multi-dimensional similarity fusion calculation based on the inspection results to obtain a comprehensive matching score of multiple related images in the surveillance camera image library;

[0033] The intelligent matching unit is used to filter the Top 10 matching results from multiple comprehensive matching scores according to a preset total score threshold, so as to complete the intelligent matching of the on-site image and the surveillance camera.

[0034] As one specific implementation of this application, the image processing unit is specifically used for:

[0035] An image-to-text conversion model is used to convert the on-site images and associated images into structured text descriptions containing multi-dimensional information. The text descriptions include scene classification, landmark network, structural features, infrastructure, ground features, environmental factors, and text extraction. The scene classification includes primary scenes and secondary scenes.

[0036] A multi-rule cleaning mechanism is used to adaptively clean the text description information; the multi-rule cleaning mechanism includes filtering timestamps and fixing irrelevant identifiers.

[0037] Retain derivative terms that are strongly related to monitoring, and standardize the format of null text annotations to obtain structured documents.

[0038] As a specific implementation of this application, the validity checking unit is specifically used for:

[0039] A pre-trained semantic model is used to vectorize the content of each dimension in the text description information to obtain multi-dimensional semantic features;

[0040] The validity of the multi-dimensional semantic features is determined, invalid features are excluded, and the valid features are obtained as the inspection result; the valid features include structural features, landmark networks, infrastructure, ground features, and text extraction.

[0041] As one specific implementation of this application, the similarity calculation unit is specifically used for:

[0042] A genetic algorithm is introduced, which optimizes the initial weights of multiple effective features based on training samples and through a fitness function to obtain optimized weights.

[0043] Based on the optimized weights, the similarity between the scene image and the associated image is calculated for each of the multiple effective features, resulting in multiple similarity scores;

[0044] The combined matching score is obtained by weighted summation of multiple similarity scores.

[0045] Thirdly, embodiments of this application also provide another intelligent matching system for on-site images and surveillance cameras, including a processor, an input device, an output device, and a memory, wherein the processor, input device, output device, and memory are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method described in the first aspect.

[0046] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the method described in the first aspect.

[0047] By implementing the embodiments of the present invention, the problems of poor robustness to changes in lighting and shooting angle and difficulty in associating semantic information of the camera (such as scene type and landmark) can be overcome by early methods that rely on low-level visual features such as SIFT and SURF. It also makes up for the shortcomings of text-based matching methods that have a single parsing dimension and use fixed weights for multi-dimensional feature fusion. At the same time, it solves the problems of some multi-dimensional feature schemes failing to filter invalid features and insufficient cleaning of text extraction information. Ultimately, it improves the accuracy, robustness and generalization ability of image matching with surveillance cameras in complex urban scenes.

[0048] More specifically, the advantages of the present invention are as follows:

[0049] Firstly, through image-to-text conversion and semantic encoding, it breaks free from the dependence of underlying visual features on lighting and shooting angle, and can directly associate with camera semantic information (such as scene and landmark), making it more robust in complex urban scenes.

[0050] Secondly, based on multi-dimensional fine-grained semantic parsing, combined with text cleaning and feature selection, the matching accuracy is greatly improved, solving the problem of the single dimension of existing text matching.

[0051] Third, by using genetic algorithms to dynamically optimize the weights of each dimension, it can be adapted to different scenarios and overcome the bottleneck of weak generalization ability of fixed weights.

[0052] Fourth, by calculating similarity across dimensions, determining the total score threshold, and outputting the Top 10 results, the reliability of the matching is ensured. At the same time, it is adapted to the dynamic maintenance of the monitoring database, making it more practical in scenarios such as urban security and grid worker dispatch. Attached Figure Description

[0053] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below.

[0054] Figure 1This is a flowchart of the intelligent matching method between on-site images and surveillance cameras provided in an embodiment of the present invention;

[0055] Figure 2 yes Figure 1 The system architecture diagram corresponding to the method shown is as follows;

[0056] Figure 3 yes Figure 1 The data processing flowchart corresponding to the method shown is as follows;

[0057] Figure 4 This is a flowchart of the multi-dimensional matching algorithm provided in an embodiment of the present invention;

[0058] Figure 5 This is a structural diagram of the intelligent matching system between on-site images and surveillance cameras provided in an embodiment of the present invention;

[0059] Figure 6 This is a screenshot of the user interface.

[0060] Figure 7 This is a screenshot of the monitoring information interface;

[0061] Figures 8a to 8b This is a screenshot of the interface for adding monitoring data;

[0062] Figures 9a to 9c This is a screenshot of the interface for deleting monitoring data;

[0063] Figures 10a to 10c This is a screenshot of the interface for uploading images and performing a matching process;

[0064] Figure 11 This is a screenshot of the interface for viewing matching results. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0067] Please refer to Figure 2 and Figure 3 The inventive concept of this invention is:

[0068] To address the shortcomings of existing technologies, a new image-camera matching scheme that integrates multi-dimensional semantic features and adaptive weight optimization is proposed. The corresponding system architecture and data flow are as follows: Figure 2 and Figure 3 As shown in the diagram, this scheme first transforms the on-site images from grid workers and the associated images from surveillance cameras into structured text descriptions encompassing multiple dimensions, including scene classification (primary and secondary scenes), landmark networks, structural features, infrastructure, ground features, environmental factors, and text extraction, using an image analysis model. Second, it encodes the text for each dimension to generate feature vectors and uses a validity check mechanism to filter out valuable feature dimensions, avoiding interference from invalid information. Third, it introduces a genetic algorithm to dynamically optimize the weights of each feature dimension, automatically adjusting the weight ratios of each dimension (such as landmarks, structures, and text extraction) based on the matching requirements of different scenes in the training samples, improving adaptability in complex scenarios. Finally, it weights and fuses the similarity of the multi-dimensional feature vectors based on the optimized weights to obtain a comprehensive matching score, achieving accurate matching between on-site images from grid workers and surveillance cameras. This technical approach significantly improves the accuracy and robustness of matching through comprehensive fusion of multi-dimensional features, dynamic adjustment of adaptive weights, and effective cleaning of invalid information, overcoming the shortcomings of existing technologies in terms of single feature dimensions, fixed weights, and weak anti-interference capabilities.

[0069] Please refer to Figure 1 and Figure 4 The present invention provides an intelligent matching method for on-site images and surveillance cameras, which may include the following steps:

[0070] S1, acquire on-site images from grid workers and related images from surveillance cameras.

[0071] Surveillance cameras are deployed on city streets.

[0072] S2 preprocesses the on-site images and related images to obtain structured documents.

[0073] Preprocessing includes, but is not limited to, image watermark removal, text extraction and deduplication, and structured parsing.

[0074] More specifically, in this embodiment, the on-site images and associated images are preprocessed, specifically as follows:

[0075] (1) The image-to-text conversion model qwen-vl-max is used to convert the scene image and associated image into structured text description information containing multi-dimensional information.

[0076] The text description information includes seven dimensions: scene classification, landmark network, structural features, infrastructure, ground features, environmental factors, and text extraction. The scene classification includes primary scenes and secondary scenes, with secondary scenes supporting multi-label isolation. This embodiment breaks through the limitation of existing text descriptions that only focus on the overall scene, constructing a structured semantic system containing seven core dimensions. In particular, through multi-dimensional structured text parsing design such as secondary scene multi-labels and ground-environment feature linkage, it achieves comprehensive coverage of fine-grained information, making up for the shortcomings of existing text parsing with a single dimension, and laying a data foundation for accurate matching.

[0077] (2) The text description information is adaptively cleaned using a multi-rule cleaning mechanism; the multi-rule cleaning mechanism includes filtering timestamps and fixing irrelevant identifiers.

[0078] (3) Retain derivative terms that are strongly related to monitoring and label null text in a unified format to obtain structured documents.

[0079] In other words, this embodiment designs a multi-rule cleaning mechanism to address redundant interference in the extracted text information: filtering timestamps, fixing irrelevant identifiers, retaining derivative words that are strongly related to monitoring, and standardizing the format of null text annotations, thereby improving the relevance of text descriptions to actual scenarios and solving the problem of insufficient text cleaning in existing methods.

[0080] S3 performs a first-level scene matching check and a second-level scene intersection check on the structured document to obtain the check results.

[0081] In specific implementation, step S3 includes:

[0082] Semantic feature vector encoding: The pre-trained semantic model bge-large-zh-v1.5 is used to encode the content of each dimension of the structured text. In particular, special instructions are added for the text query by grid workers to distinguish the encoding logic of the query end and the monitoring end, enhance the targeting of semantic matching, and overcome the defect that the early low-level visual features are difficult to associate with semantic information.

[0083] Feature validity screening mechanism: Before similarity calculation, the validity of features in each dimension is determined. For other dimensions, the existence of valid content in the corresponding list is directly checked. If a feature in a certain dimension is empty or has high noise, it is excluded from the similarity calculation to avoid interference from invalid features in the matching results. Valid features include, but are not limited to, structural features, landmark networks, infrastructure, ground features, and text extraction.

[0084] As can be seen, this embodiment provides an adaptive text cleaning and feature selection collaborative mechanism: through a three-level processing logic of redundant information filtering, effective derivative vocabulary retention, and invalid feature exclusion, it solves the dual problems of text interference and invalid feature participation in calculation, thereby improving the quality of semantic data.

[0085] Furthermore, this embodiment also provides a matching logic that combines semantic encoding with dynamic weights: combining exclusive instruction semantic encoding with genetic algorithm-optimized weights not only strengthens the semantic correlation between the query end and the monitoring end, but also realizes the scenario-based adaptation of the importance of each dimension, breaking through the bottleneck of weak generalization ability of fixed weights and insufficient semantic correlation of underlying visual features, and ultimately improving the matching accuracy and robustness in complex urban scenarios.

[0086] S4. Based on the inspection results, perform multi-dimensional similarity fusion calculation to obtain the comprehensive matching score of multiple related images in the surveillance camera image library.

[0087] In practice, step S4 includes:

[0088] A genetic algorithm is introduced, based on training samples, to dynamically optimize the initial weights of multiple effective features through a fitness function (combined with matching metrics such as hit@1, hit@3, hit@5, and hit@10) to obtain optimized weights. This enables adaptive adjustment of weights under different scenarios, replacing the existing fixed weight strategy and enhancing the generalization ability in complex scenarios.

[0089] Based on the optimized weights, the similarity between the on-site image and the associated image is calculated for multiple effective features to obtain multiple similarity scores; for example, cosine similarity is calculated for structural features, landmark networks, infrastructure and ground features, while edit distance similarity is calculated for text extraction.

[0090] The combined matching score is obtained by weighted summation of multiple similarity scores.

[0091] S5. Based on the preset total score threshold, the Top 10 matching results are selected from multiple comprehensive matching scores to complete the intelligent matching of the on-site image and the surveillance camera.

[0092] For example, if the total score threshold is set to 0.35, the Top 10 with a comprehensive matching score exceeding 0.35 will be filtered out.

[0093] The intelligent matching method for on-site images and surveillance cameras provided in this embodiment of the invention can overcome the problems of poor robustness to changes in lighting and shooting angle and difficulty in associating semantic information (such as scene type and landmarks) of early methods that relied on low-level visual features such as SIFT and SURF. It also makes up for the shortcomings of text-based matching methods, such as single parsing dimension and fixed weights in multi-dimensional feature fusion. At the same time, it solves the problems of some multi-dimensional feature schemes failing to filter invalid features and insufficient text information cleaning. Ultimately, it improves the accuracy, robustness and generalization ability of image and surveillance camera matching in complex urban scenes.

[0094] More specifically, the advantages of the present invention are as follows:

[0095] Firstly, through image-to-text conversion and semantic encoding, it breaks free from the dependence of underlying visual features on lighting and shooting angle, and can directly associate with camera semantic information (such as scene and landmark), making it more robust in complex urban scenes.

[0096] Secondly, based on multi-dimensional fine-grained semantic parsing, combined with text cleaning and feature selection, the matching accuracy is greatly improved, solving the problem of the single dimension of existing text matching.

[0097] Third, by using genetic algorithms to dynamically optimize the weights of each dimension, it can be adapted to different scenarios and overcome the bottleneck of weak generalization ability of fixed weights.

[0098] Fourth, by calculating similarity across dimensions, determining the total score threshold, and outputting the Top 10 results, the reliability of the matching is ensured. At the same time, it is adapted to the dynamic maintenance of the monitoring database, making it more practical in scenarios such as urban security and grid worker dispatch.

[0099] Based on the same inventive concept, embodiments of the present invention provide an intelligent matching system for on-site images and surveillance cameras, comprising:

[0100] The acquisition unit is used to acquire on-site images from grid workers and related images from surveillance cameras deployed on city streets.

[0101] The image processing unit is used to preprocess the on-site images and related images to obtain a structured document; the preprocessing includes image watermark removal, text extraction and deduplication, and structured parsing.

[0102] The validity check unit is used to perform a first-level scene matching check and a second-level scene intersection check on the structured document to obtain the check results;

[0103] Similarity calculation is used to perform multi-dimensional similarity fusion calculation based on the inspection results to obtain a comprehensive matching score of multiple related images in the surveillance camera image library;

[0104] The intelligent matching unit is used to filter the Top 10 matching results from multiple comprehensive matching scores according to a preset total score threshold, so as to complete the intelligent matching of the on-site image and the surveillance camera.

[0105] In specific implementation, the image processing unit is specifically used for:

[0106] An image-to-text conversion model is used to convert the on-site images and associated images into structured text descriptions containing multi-dimensional information. The text descriptions include scene classification, landmark network, structural features, infrastructure, ground features, environmental factors, and text extraction. The scene classification includes primary scenes and secondary scenes.

[0107] A multi-rule cleaning mechanism is used to adaptively clean the text description information; the multi-rule cleaning mechanism includes filtering timestamps and fixing irrelevant identifiers.

[0108] Retain derivative terms that are strongly related to monitoring, and standardize the format of null text annotations to obtain structured documents.

[0109] In specific implementation, the validity check unit is used for:

[0110] A pre-trained semantic model is used to vectorize the content of each dimension in the text description information to obtain multi-dimensional semantic features;

[0111] The validity of the multi-dimensional semantic features is determined, invalid features are excluded, and the valid features are obtained as the inspection result; the valid features include structural features, landmark networks, infrastructure, ground features, and text extraction.

[0112] In specific implementation, the similarity calculation unit is used for:

[0113] A genetic algorithm is introduced, which optimizes the initial weights of multiple effective features based on training samples and through a fitness function to obtain optimized weights.

[0114] Based on the optimized weights, the similarity between the on-site image and the associated image is calculated for each of the multiple effective features, resulting in multiple similarity scores;

[0115] The combined matching score is obtained by weighted summation of multiple similarity scores.

[0116] It should be noted that the specific workflow of this embodiment is described in the foregoing method embodiment section, and will not be repeated here.

[0117] Furthermore, another embodiment of the present invention also provides an intelligent matching system for on-site images and surveillance cameras. For example... Figure 5As shown, the system may include one or more processors 101, one or more input devices 102, one or more output devices 103, and a memory 104. The processors 101, input devices 102, output devices 103, and memory 104 are interconnected via a bus 105. The memory 104 stores a computer program, which includes program instructions. The processor 101 is configured to invoke the program instructions to execute the method described in the above-described method embodiment.

[0118] It should be understood that, in this embodiment of the invention, the processor 101 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0119] Input device 102 may include a keyboard, etc., and output device 103 may include a display (LCD, etc.), a speaker, etc.

[0120] The memory 104 may include read-only memory and random access memory, and provides instructions and data to the processor 101. A portion of the memory 104 may also include non-volatile random access memory. For example, the memory 104 may also store device type information.

[0121] In specific implementations, the processor 101, input device 102, and output device 103 described in the embodiments of the present invention can execute the implementation methods described in the embodiments of the intelligent matching method between on-site images and monitoring cameras provided in the embodiments of the present invention, which will not be repeated here.

[0122] To better understand the technical solutions provided in the embodiments of the present invention, the specific process of system operation is shown below.

[0123] 1. Start the display system

[0124] Open the Python terminal and enter the command: streamlit run app.py. After successful startup, the following link will be displayed. Copy the link and open it in your browser to access the display interface.

[0125] 2. Display Interface Functions and Operations

[0126] Display interface as follows Figure 6 As shown, it is divided into two main modules: "Model Analysis Results" and "Monitoring Information". The specific operation is as follows:

[0127] (1) View surveillance camera information (e.g.) Figure 7 (As shown)

[0128] The interface displays the "serial number" and "information" of all monitored items by default.

[0129] Search Monitoring: Enter keywords (such as "Hongqihegou" or "CAM002") in the "Search Monitoring Information" input box, and the interface will filter and match the monitoring data in real time.

[0130] View details: Click the monitoring number to enlarge the image, and hover the mouse over it to view the complete monitoring information (the code supports scrolling the list, so there is no need to click "View complete information" separately).

[0131] (2) Add new monitoring data (such as Figure 8a and Figure 8b (As shown)

[0132] When adding new monitoring points, there's no need to manually run camera_i2w.py; you can do it directly through the interface.

[0133] In the "Add Monitoring Data" area of ​​the "Manage Monitoring Data" module, click "Browse files" or drag and drop images (multiple file uploads are supported, in JPG / JPEG / PNG formats).

[0134] After uploading, the App will automatically perform the following operations:

[0135] Save the image to the CAM folder.

[0136] Call generate_camera_description() to generate monitoring semantic information and write it to camera.txt.

[0137] The clean_watermark.py script is run automatically to remove the watermark and the regenerate_vector_cache() function is called to regenerate the camera_vectors_bge.pkl cache.

[0138] Note: If the uploaded monitoring sequence number already exists (e.g., CAM001 is uploaded repeatedly), the App will prompt "Camera XX already exists, skip processing" to avoid duplicate data.

[0139] (3) Delete monitoring data (e.g.) Figure 9a and Figure 9b (As shown)

[0140] When you need to remove invalid monitoring data, follow these steps: Click "Delete Monitoring Data" to expand the deletion panel.

[0141] To filter monitoring data for deletion: Enter keywords in the "Search Monitoring Data (for Deletion)" input box to filter target monitoring data. Check the checkbox before a single monitoring data point, or check "Select All" to select all data points at once.

[0142] Clicking "Confirm Delete" will cause the app to automatically perform the following actions: delete the corresponding surveillance image from the CAM folder; remove the semantic information for that surveillance image from camera.txt; and call regenerate_vector_cache() to regenerate the cache, ensuring accurate matching results.

[0143] Note: Deletion is irreversible. It is recommended to confirm that the monitoring sequence number is correct before proceeding.

[0144] (4) Upload images and perform matching

[0145] Upload image: such as Figure 10a and Figure 10b As shown, in the "Upload Grid Worker's On-Site Photos" module of the "Model Analysis Results" area, click "Browse files" to select images (drag and drop is supported, single file ≤200MB). After uploading, the interface will preview the photos.

[0146] Perform matching: such as Figure 10c As shown, clicking the "Perform Matching" button will display loading prompts such as "Analyzing the image..." and "Matching analysis in progress...", eliminating the need to manually run other scripts.

[0147] Note: If the match fails, the interface will display an error message (such as "match.py ​​execution error" or "image encoding failure"). You can check the image format or the match.py ​​code according to the message.

[0148] (5) View the matching results (e.g.) Figure 11 (As shown)

[0149] After matching is complete, the results are divided into two parts: "Visual Cards" and "Data Tables".

[0150] Top 20 Visual Cards:

[0151] The interface displays the top 20 surveillance videos with a similarity score greater than 0 in a 5-column layout. Each card contains a “survey image”, a “survey number”, and a “similarity score” (retaining 4 decimal places).

[0152] If no results with a similarity greater than 0 are found, the interface will display the message "No matching results with a similarity greater than 0".

[0153] Data table (right-hand "Monitoring Information" area):

[0154] The table automatically adds a "Similarity" column and sorts the data in descending order of similarity.

[0155] Highlighting rules: The background color of the first 10 results is #ffcccc (light red), and the background color of results 11-20 is #ffe5e5 (very light red), which facilitates quick identification of high-match monitoring.

[0156] View full results:

[0157] Image analysis results: Semantic information including scene classification, landmark network, and structural features of the grid worker images (from test.txt).

[0158] Similarity results: Click "similarity_results.json" to view the Top 20 score details.

[0159] Total score results: Click "total_scores.json" to view the complete scores for all monitored data.

[0160] Accordingly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, implement the above-described intelligent matching method between on-site images and surveillance cameras.

[0161] The computer-readable storage medium can be an internal storage unit of the system described in any of the foregoing embodiments, such as the system's hard disk or memory. The computer-readable storage medium can also be an external storage device of the system, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the system. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0162] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0163] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.

[0164] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0165] Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units can be implemented in hardware or as software functional units. When using each module, user information is collected and stored only with the user's full authorization and in compliance with relevant laws and regulations, protecting the security and privacy of user data, and strictly prohibiting unauthorized access; data processing will be conducted within the scope stipulated by law and will not exceed the purpose and scope authorized by the user; at the same time, users have the rights to access, correct, delete, restrict processing, and refuse their personal data; and must strictly comply with applicable laws and regulations and conduct compliance reviews.

[0166] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for intelligent matching of on-site images with surveillance cameras, characterized in that, include: Acquire on-site images from grid workers and related images from surveillance cameras; The surveillance cameras are deployed on city streets; The on-site images and associated images are preprocessed to obtain a structured document; The preprocessing includes image watermark removal, text extraction and deduplication, and structured parsing; The structured document is subjected to a first-level scene matching check and a second-level scene intersection check to obtain the check results; Based on the inspection results, a multi-dimensional similarity fusion calculation is performed to obtain a comprehensive matching score for multiple related images in the surveillance camera image library; Based on a preset total score threshold, the Top 10 matching results are selected from multiple comprehensive matching scores to complete the intelligent matching of on-site images and surveillance cameras.

2. The intelligent matching method between on-site images and surveillance cameras as described in claim 1, characterized in that, The on-site images and associated images are preprocessed, specifically as follows: An image-to-text conversion model is used to convert the on-site images and associated images into structured text descriptions containing multi-dimensional information. The text descriptions include scene classification, landmark network, structural features, infrastructure, ground features, environmental factors, and text extraction. The scene classification includes primary scenes and secondary scenes. A multi-rule cleaning mechanism is used to adaptively clean the text description information. The multi-rule cleaning mechanism includes filtering timestamps and fixing irrelevant identifiers; Retain derivative terms that are strongly related to monitoring, and standardize the format of null text annotations to obtain structured documents.

3. The intelligent matching method between on-site images and surveillance cameras as described in claim 2, characterized in that, The structured document is subjected to a first-level scene matching check and a second-level scene intersection check to obtain the check results, specifically: A pre-trained semantic model is used to vectorize the content of each dimension in the text description information to obtain multi-dimensional semantic features; The validity of the multi-dimensional semantic features is determined, invalid features are excluded, and the valid features are obtained as the inspection result; the valid features include structural features, landmark networks, infrastructure, ground features, and text extraction.

4. The intelligent matching method between on-site images and surveillance cameras as described in claim 3, characterized in that, Based on the inspection results, a multi-dimensional similarity fusion calculation is performed, specifically as follows: A genetic algorithm is introduced, which optimizes the initial weights of multiple effective features based on training samples and through a fitness function to obtain optimized weights. Based on the optimized weights, the similarity between the on-site image and the associated image is calculated for each of the multiple effective features, resulting in multiple similarity scores; The combined matching score is obtained by weighted summation of multiple similarity scores.

5. An intelligent matching system for on-site images and surveillance cameras, characterized in that, include: The acquisition unit is used to acquire on-site images from grid workers and related images from surveillance cameras; The surveillance cameras are deployed on city streets; The image processing unit is used to preprocess the on-site images and related images to obtain a structured document; the preprocessing includes image watermark removal, text extraction and deduplication, and structured parsing. The validity check unit is used to perform a first-level scene matching check and a second-level scene intersection check on the structured document to obtain the check results; Similarity calculation is used to perform multi-dimensional similarity fusion calculation based on the inspection results to obtain a comprehensive matching score of multiple related images in the surveillance camera image library; The intelligent matching unit is used to filter the Top 10 matching results from multiple comprehensive matching scores according to a preset total score threshold, so as to complete the intelligent matching of the on-site image and the surveillance camera.

6. The intelligent matching system for on-site images and surveillance cameras as described in claim 5, characterized in that, The image processing unit is specifically used for: An image-to-text conversion model is used to convert the on-site images and associated images into structured text descriptions containing multi-dimensional information. The text descriptions include scene classification, landmark network, structural features, infrastructure, ground features, environmental factors, and text extraction. The scene classification includes primary scenes and secondary scenes. A multi-rule cleaning mechanism is used to adaptively clean the text description information. The multi-rule cleaning mechanism includes filtering timestamps and fixing irrelevant identifiers; Retain derivative terms that are strongly related to monitoring, and standardize the format of null text annotations to obtain structured documents.

7. The intelligent matching system for on-site images and surveillance cameras as described in claim 6, characterized in that, The validity check unit is specifically used for: A pre-trained semantic model is used to vectorize the content of each dimension in the text description information to obtain multi-dimensional semantic features; The validity of the multi-dimensional semantic features is determined, invalid features are excluded, and the valid features are obtained as the inspection result; the valid features include structural features, landmark networks, infrastructure, ground features, and text extraction.

8. The intelligent matching system for on-site images and surveillance cameras as described in claim 7, characterized in that, The similarity calculation unit is specifically used for: A genetic algorithm is introduced, which optimizes the initial weights of multiple effective features based on training samples and through a fitness function to obtain optimized weights. Based on the optimized weights, the similarity between the on-site image and the associated image is calculated for each of the multiple effective features, resulting in multiple similarity scores; The combined matching score is obtained by weighted summation of multiple similarity scores.

9. An intelligent matching system for on-site images and surveillance cameras, characterized in that, The system includes a processor, an input device, an output device, and a memory, which are interconnected. The memory is used to store a computer program, which includes program instructions. The processor is configured to invoke the program instructions to perform the method as described in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-4.

Images