Method for automatically identifying landmark from video

Abstract

The invention discloses a method for automatically identifying landmarks from videos, and relates to the technical field of adaptive attention identification mechanisms and identification framework dynamic adjustment technologies, and the method comprises the following steps: receiving video streams from different video sources, and identifying the sources of the video streams; analyzing metadata and video content features of the video stream, performing feature extraction, and determining a video source; different preprocessing algorithms are selected and applied according to video source types; according to a preprocessing result of the video preprocessing algorithm, identifying landmarks in the video by using an adaptive identification framework; after landmark recognition is completed, counting the number of recognition errors, and calculating a recognition error rate; classifying error data of videos of different sources, and then adjusting algorithm parameters in an identification framework of a corresponding video source by using an identification error dynamic adjustment method; and finally, outputting the dynamically adjusted identification result for further use and analysis by a user or a system.

Description

Technical Field

[0001] The present invention relates to the technical field of adaptive attention recognition mechanism and dynamic adjustment of recognition framework, and specifically to a method for automatically recognizing landmarks from a video. Background Art

[0002] With the continuous development of video surveillance and intelligent analysis technology, automatic landmark recognition from videos has become an important research field. Landmark recognition not only plays an important role in urban security monitoring, autonomous driving navigation, and tourism recommendation systems, but also provides effective support for urban planning and environmental monitoring. However, due to factors such as the large amount of video data, complex and changeable scenes, and the diversity of landmark features, how to achieve efficient and accurate recognition and analysis of video sources to lay the foundation for subsequent landmark recognition is still a challenging problem. Therefore, the present invention aims to overcome the limitations of the prior art and proposes an adaptive attention mechanism and a dynamic adjustment system of the recognition framework to realize a method of identifying landmarks from videos, adapt to the diversity and dynamic changes of landmark features, improve the recognition accuracy, and provide technical support for smart cities, traffic monitoring and other fields. Summary of the invention

[0003] The object of the present invention is to provide a method for automatically identifying landmarks from a video to solve the problems raised in the prior art.

[0004] To achieve the above object, the present invention provides the following technical solutions: A method for automatically identifying landmarks from a video, comprising the following steps: Step 1: Collect and receive real-time video stream data from different video sources, identify the source of the video stream, obtain device permissions and establish a connection with the device to obtain the video stream; Step 2: Analyze the metadata and video content features of the video stream, use the convolutional neural network deep learning model to extract features from the video frames that reflect the video characteristics in the video stream, and use the machine learning model to classify the extracted features to determine the video source type; Step 3: Select and apply different preprocessing algorithms according to the video source type, and perform video preprocessing operations after determining the preprocessing algorithm; Step 4: Based on the preprocessing results of the video preprocessing algorithm, a convolutional neural network deep learning model is constructed, and deep features are extracted using feature mapping and feature fusion methods. Then, the landmarks in the video are identified using an adaptive recognition framework. Step 5: After the landmark recognition is completed, the recognition results output by the recognition framework are evaluated, the number of recognition errors is counted, and the recognition error rate is calculated; Step 6: first classify the error data of videos from different sources according to the calculated error rate, and then use the recognition error dynamic adjustment method to adjust the algorithm parameters in the recognition framework of the corresponding video source; Step 7: After the dynamic adjustment is completed, the recognition result after dynamic adjustment is output for further use and analysis by the user or system.

[0005] The receiving of video streams from different video sources further comprises the following steps: Step 1.1: Determine the video source: Identify the source of the video stream, including but not limited to webcams, live broadcast platforms, and video surveillance systems, and analyze the data source and characteristics of the video stream; Step 1.2: Obtain device permissions: Use the front-end controller and integrated API provided by the video source to ensure that you have access to the device. Step 1.3: Establish connection: To ensure real-time video stream data is obtained from the video source, use the corresponding library and software tools to connect and obtain the video stream from the video source.

[0006] The analysis of the metadata and video content features of the video stream comprises the following steps: Step 2.1: After the connection is established, the metadata information of the video file is extracted based on the acquired video stream, the file name of the video file is analyzed, and a preliminary visual analysis of the video frame is performed; in addition, based on the analysis results, a convolutional neural network deep learning model is used to extract features of the frames in the video that reflect the video characteristics for further video source analysis; Step 2.2: After extracting the features, use the machine learning model to classify the extracted features, classify the analysis results, distinguish the category of the video source, and use the ensemble learning method to determine the source type of the video.

[0007] The steps of selecting and applying different preprocessing algorithms according to the video source type include the following: Step 3.1: Determine the corresponding processing algorithm based on the identified video source type; for 3D videos from natural scenes, apply array camera calibration and camera geometric view correction algorithms; for videos from surveillance systems, apply image denoising and motion compensation algorithms; for videos from social media, apply compression artifact repair and color correction algorithms; for videos from professional recordings, apply video stabilization and scene segmentation algorithms; Step 3.2: Perform the video preprocessing operation after determining the preprocessing algorithm, and output the processed video stream for subsequent processing.

[0008] The method of using an adaptive recognition framework to identify landmarks in a video comprises the following steps: Step 4.1: Deep feature extraction: Based on the preprocessing results of the video preprocessing algorithm, first build a convolutional neural network deep learning model, then train the model by defining the loss function and optimization algorithm. After the training is completed, use feature mapping and feature fusion methods to extract features. For multimodal data or when multiple features need to be fused, use an adaptive attention mechanism to fuse features from different sources, automatically learn and focus on the key areas and dynamic features of landmarks in the video; Step 4.2: Adaptive attention mechanism: After deep feature extraction using the convolutional neural network deep learning model, the adaptive attention mechanism is applied to dynamically adjust the degree of attention to different frames and different regions within the frame, automatically focusing on the key areas and dynamic features of landmarks in the video, and improving the accuracy and real-time performance of recognition; this mechanism includes two steps: attention weight calculation and attention weighted feature fusion, using the attention network and the feature vector fused with attention weights for calculation; the calculation formula of attention weight is: W = att [F(t),F(t-1),...,F(tn)] Where W represents the attention weight calculated at time t; att is an attention network that considers the features of the current frame and the past frame n, and the attention weight output by the attention network is 1; The calculation formula of attention weighted fusion is: , Among them, F att is the feature vector fused with attention weights; F is the feature vector of each frame; n represents the total number of frames in the historical data, t represents the current frame, which refers to the index of the frame currently being considered; ti represents a time step in the past relative to the current time step t, where i is an index from 1 to n, indicating the number of frames backtracked from the current time step. If t is the current frame, t-1 is the previous frame, t-2 is the previous two frames, and so on, until tn, which is the oldest frame in the time window; Features are extracted through a deep learning model, and an adaptive attention mechanism is used to dynamically focus on key frames and key areas in the video to quickly locate and identify landmarks in the video stream.

[0009] The evaluation of the recognition result and calculation of the error rate comprises the following steps: Step 5.1: Collect the recognition results and the real landmark labels to determine whether the recognition results are correct; Step 5.2: Count the number of correctly identified landmarks and calculate the ratio of the number of correctly identified landmarks to the total number of landmarks to obtain the recognition accuracy; at the same time, calculate the error rate based on the number of incorrectly identified landmarks and the total number of landmarks. Finally, count the surface recognition error rates of different video sources. The calculation method is the number of incorrectly identified landmarks divided by the total number of identified landmarks.

[0010] The dynamic adjustment of the algorithm parameters in the recognition framework according to the error rate includes the following steps: Step 6.1: Error classification: First, determine the error rate of landmark recognition from different video sources, then set an error rate threshold θ to distinguish the preset range of error rates, and classify and summarize the error rates above and below the threshold; Step 6.2: Dynamic adjustment: When the recognition error is higher than the threshold θ, the recognition algorithm parameters of the corresponding video source are adjusted, including but not limited to increasing the sample training amount, optimizing the algorithm structure and adjusting the model parameters, and the adjusted algorithm parameters are calculated according to the following formula: New parameter = current parameter + α (target error rate - current error rate) Among them, α is the adjustment coefficient, which is used to control the step size of parameter adjustment, and the target error rate is set to a predetermined value lower than the threshold θ; Then, the adjusted algorithm parameters are applied to re-identify the landmarks, and the above steps are repeated until the recognition error rate drops below the threshold θ or reaches the predetermined number of iterations; When the recognition error rate is lower than the threshold θ, the current recognition algorithm parameters are maintained unchanged or fine-tuned according to the following formula: New parameter = current parameter + β (current error rate - target error rate) Among them, β is the fine-tuning coefficient, which is smaller than α and is used to make fine adjustments when the error rate is already low; The purpose of adjusting the model is to reduce the error rate and make the recognition result more accurate. In the formula, the target error rate is the preset lower error rate, and the current error rate is the actual calculated error rate. By calculating the difference between the target error rate and the current error rate and multiplying it by the adjustment coefficient α or β, an adjustment amount can be obtained to update the model parameters. The adjustment coefficients α and β ensure that the amplitude of the parameter adjustment is proportional to the magnitude of the error rate deviation, thereby ensuring the stability and effectiveness of the adjustment process. By iteratively applying this formula, the error rate can be gradually reduced until it reaches or falls below the threshold θ, proving that the formula can effectively guide the dynamic adjustment of the recognition framework.

[0011] The outputting of the recognition result comprises the following steps: Step 7.1: After completing the dynamic adjustment of the model parameters, first confirm whether the recognition model parameters after dynamic adjustment are kept within the preset threshold range. The stability of the parameter model after dynamic adjustment can be measured by calculating the standard deviation and coefficient of variation of the parameter changes to ensure that the parameter fluctuation does not exceed the preset threshold and the error rate is reduced to the preset target error rate range. Then, use the recognition model after stability verification and error rate detection to re-recognize the landmarks in the video and record the landmark recognition results output by the recognition model. Step 7.2: Organize the recognition results into a file of a unified format, output the organized recognition results to the user interface, and store them in the database for user query and analysis.

[0012] Compared with the prior art, the present invention has the following beneficial effects: 1. Improve recognition accuracy: Through deep learning models, especially in combination with adaptive attention mechanisms, landmarks in videos can be located and identified more accurately; the attention mechanism enables the model to focus on key information in the video, thereby improving the accuracy of landmark recognition.

[0013] 2. Adaptive optimization: The system can dynamically adjust the error rate threshold based on historical recognition data and real-time recognition results to adapt to recognition requirements in different scenarios and lighting conditions, thereby improving the accuracy of landmark recognition.

[0014] 3. Reduce misidentification and missed identification: Dynamically adjusting the error rate can significantly reduce misidentification and missed identification of landmarks, ensuring that users can quickly and accurately obtain the required landmark information.

[0015] 4. Real-time feedback and adjustment: When the system detects a recognition error, the algorithm parameters can be adjusted immediately to reduce errors in subsequent recognition and improve overall recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a processing data flow structure of a method for automatically identifying landmarks from a video according to the present invention; Figure 2 A schematic diagram of an event flow structure of a method for automatically identifying landmarks from a video according to the present invention; DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Example: Figure 1-Figure 2 As shown, the present invention provides a technical solution. A method for automatically identifying landmarks from a video, comprising the following steps: Step 1: Collect and receive real-time video stream data from different video sources, identify the source of the video stream, obtain device permissions and establish a connection with the device to obtain the video stream; Step 2: Analyze the metadata and video content features of the video stream, use the convolutional neural network deep learning model to extract features from the video frames that reflect the video characteristics in the video stream, and use the machine learning model to classify the extracted features to determine the video source type; Step 3: Select and apply different preprocessing algorithms according to the video source type, and perform video preprocessing operations after determining the preprocessing algorithm; Step 4: Based on the preprocessing results of the video preprocessing algorithm, a convolutional neural network deep learning model is constructed, and deep features are extracted using feature mapping and feature fusion methods. Then, the landmarks in the video are identified using an adaptive recognition framework. Step 5: After the landmark recognition is completed, the recognition results output by the recognition framework are evaluated, the number of recognition errors is counted, and the recognition error rate is calculated; Step 6: first classify the error data of videos from different sources according to the calculated error rate, and then use the recognition error dynamic adjustment method to adjust the algorithm parameters in the recognition framework of the corresponding video source; Step 7: After the dynamic adjustment is completed, the recognition result after dynamic adjustment is output for further use and analysis by the user or system.

[0019] The receiving of video streams from different video sources further comprises the following steps: Step 1.1: Determine the video source: Identify the source of the video stream, including but not limited to webcams, live broadcast platforms, and video surveillance systems, and analyze the data source and characteristics of the video stream; Step 1.2: Obtain device permissions: Use the front-end controller and integrated API provided by the video source to ensure that you have access to the device. Step 1.3: Establish connection: To ensure real-time video stream data is obtained from the video source, use the corresponding library and software tools to connect and obtain the video stream from the video source.

[0020] Suppose you are developing a system to automatically identify landmarks from videos. The system needs to receive video streams from three different video sources: Source A: A network camera located at the company's gate, which supports the RTSP protocol and provides real-time video streaming.

[0021] Source B: An online live streaming platform that provides live streams of different channels that can be accessed via HTTP requests.

[0022] Source C: The company’s internal security video surveillance system, which consists of multiple cameras and is accessed through a dedicated API.

[0023] For source A, the camera's IP address is identified as 192.168.0.101, port 554, and supports the RTSP protocol.

[0024] For source B, several live streaming URLs are determined, and API authentication is required to obtain access permissions.

[0025] For source C, the unique identifier of each camera, such as Camera1, Camera2, etc., is known through internal documents, and the video stream needs to be accessed through the API of the internal network.

[0026] Through the received video stream, we then start requesting device permissions: For source A, the administrator account and password were entered through the camera's web interface to obtain permission to access the camera's video stream.

[0027] For source B, the developer API provided by the live broadcast platform was used to authenticate through the application ID and key, and obtain permission to access the specific live stream.

[0028] For Source C, access to the security video surveillance system was obtained through the company’s IT department and the necessary API keys were obtained.

[0029] After obtaining device permissions, connect the system to the video stream: For source A, an RTSP connection is established using the FFmpeg library to get the video stream in real time.

[0030] For source B, an HTTP client is written that periodically sends requests to the live streaming platform’s API to obtain the latest live video stream.

[0031] For source C, a secure connection was established through the API of the internal network using the SDK provided by the company, which enabled real-time access to the video streams of all cameras.

[0032] Through these steps, the video surveillance system successfully receives video streams from different video sources and is ready for further video analysis and processing.

[0033] The analysis of the metadata and video content features of the video stream comprises the following steps: Step 2.1: After the connection is established, the metadata information of the video file is extracted based on the acquired video stream, the file name of the video file is analyzed, and a preliminary visual analysis of the video frame is performed; in addition, based on the analysis results, a convolutional neural network deep learning model is used to extract features of the frames in the video that reflect the video characteristics for further video source analysis; Step 2.2: After extracting the features, use the machine learning model to classify the extracted features, classify the analysis results, distinguish the category of the video source, and use the ensemble learning method to determine the source type of the video.

[0034] Assuming that the system has completed receiving the video stream, the following analysis of the video stream is performed to determine the video source type: Once a connection is established with the video source, it starts receiving the video stream and does the following: Extract metadata: Use a video processing library (such as OpenCV or FFmpeg) to extract metadata information of the video stream, such as encoding format, resolution, frame rate, etc.

[0035] Analyze file names: If the video stream is read from a file, analyze the file name for possible source clues, such as date, time, device ID, etc.

[0036] Visual analysis: Perform preliminary visual analysis on the video frames, such as using color histograms, edge detection, etc. to obtain frame features.

[0037] Feature Extraction: Next, feature extraction is performed on the video frames using a pre-trained convolutional neural network (CNN) model that has been trained on a large amount of video data with source labels.

[0038] After the features are extracted, a machine learning model is used to classify these features to determine the source type of the video; Classification models: Use an ensemble learning method such as random forest or gradient boosted trees, which perform well in multi-class classification tasks; Training and prediction: First, the classification model is trained on labeled training data, and then prediction is performed on the extracted video features; Assuming that features have been extracted and models have been trained, through this process, video streams are classified according to source types, such as natural scenes, surveillance systems, social media, and professional recordings. In the end, a list of video source types is obtained, which can help the development team better understand the video data and provide a basis for further video processing and analysis.

[0039] The steps of selecting and applying different preprocessing algorithms according to the video source type include the following: Step 3.1: Determine the corresponding processing algorithm based on the identified video source type; for 3D videos from natural scenes, apply array camera calibration and camera geometric view correction algorithms; for videos from surveillance systems, apply image denoising and motion compensation algorithms; for videos from social media, apply compression artifact repair and color correction algorithms; for videos from professional recordings, apply video stabilization and scene segmentation algorithms; Step 3.2: Perform the video preprocessing operation after determining the preprocessing algorithm, and output the processed video stream for subsequent processing.

[0040] Based on the video source types classified in the above steps, it is assumed that a specific processing algorithm can be determined for each video type. The following is a hypothetical output of the algorithm determined after classification: For three-dimensional videos from natural scenes, it is determined to use an array camera calibration algorithm to calibrate the intrinsic and extrinsic parameters of the camera, and a camera geometric view correction algorithm to correct the geometric distortion caused by the arrangement and viewing angle differences of the camera array.

[0041] For videos originating from surveillance systems, it is determined to apply image denoising algorithms to reduce random noise in the video, and motion compensation algorithms to smooth video jitter caused by camera or object movement.

[0042] For videos sourced from social media, we plan to use compression artifact repair algorithms to improve image distortion caused by compression, as well as color correction algorithms to adjust the color of the video to make it look more natural.

[0043] For videos originating from professional recordings, video stabilization algorithms are applied to reduce camera shake, and scene segmentation algorithms are applied to identify and separate different scenes in the video.

[0044] After determining the corresponding processing algorithm, the following hypothetical preprocessing operations are performed on each video stream: Preprocessing of natural scene 3D video: First, use the array camera calibration toolbox (such as MATLAB's CameraCalibrator) to calibrate the camera parameters. Then, apply the camera geometric view correction algorithm to correct the geometric distortion of the video frame through mapping and resampling techniques.

[0045] Surveillance system video preprocessing: For surveillance system videos, use denoising algorithms (such as bilateral filters) in image processing libraries (such as OpenCV) to reduce noise. Next, apply motion compensation algorithms to compensate for motion in the video by analyzing motion vectors between consecutive frames.

[0046] Social media video preprocessing: For social media videos, a specially designed compression artifact repair algorithm is used to reduce blockiness and artifacts. Then, a color correction algorithm is used to adjust the hue, saturation, and brightness of the video frames to improve the overall visual effect of the video.

[0047] Professionally recorded video preprocessing: For professionally recorded videos, video stabilization techniques are used to reduce unintended camera movement. In addition, we use scene segmentation algorithms to identify scene cuts and segment the video stream accordingly for subsequent content analysis and editing.

[0048] After the preprocessing operation is completed, we output the processed video stream for subsequent processing. These processed video streams may be used for video analysis, content distribution, storage, or further video editing.

[0049] The method of using an adaptive recognition framework to identify landmarks in a video comprises the following steps: Step 4.1: Deep feature extraction: Based on the preprocessing results of the video preprocessing algorithm, first build a convolutional neural network deep learning model, then train the model by defining the loss function and optimization algorithm. After the training is completed, use feature mapping and feature fusion methods to extract features. For multimodal data or when multiple features need to be fused, use an adaptive attention mechanism to fuse features from different sources, automatically learn and focus on the key areas and dynamic features of landmarks in the video; Step 4.2: Adaptive attention mechanism: After deep feature extraction using the convolutional neural network deep learning model, the adaptive attention mechanism is applied to dynamically adjust the degree of attention to different frames and different regions within the frame, automatically focusing on the key areas and dynamic features of landmarks in the video, and improving the accuracy and real-time performance of recognition; this mechanism includes two steps: attention weight calculation and attention weighted feature fusion, using the attention network and the feature vector fused with attention weights for calculation; the calculation formula of attention weight is: W = att [F(t),F(t-1),...,F(tn)] Where W represents the attention weight calculated at time t; att is an attention network that considers the features of the current frame and the past frame n, and the attention weight output by the attention network is 1; The calculation formula of attention weighted fusion is:

[0050] Among them, F att is the feature vector fused with attention weights; F is the feature vector of each frame; n represents the total number of frames in the historical data, t represents the current frame, which refers to the index of the frame currently being considered; ti represents a time step in the past relative to the current time step t, where i is an index from 1 to n, indicating the number of frames backtracked from the current time step. If t is the current frame, t-1 is the previous frame, t-2 is the previous two frames, and so on, until tn, which is the oldest frame in the time window; Features are extracted through a deep learning model, and an adaptive attention mechanism is used to dynamically focus on key frames and key areas in the video to quickly locate and identify landmarks in the video stream.

[0051] Assume that the above feature extraction and preprocessing operations have been completed, there is a short video sequence containing 5 frames, each of which extracts a 100-dimensional feature vector through a deep learning model (CNN). For simplicity, assume that the feature vector has been normalized in some way, and the sum of the attention weights output by the attention network (AttNet) is 1.

[0052] Here are the feature vectors for each frame: F_{1} = [0.1,0.2,0.3,..., 0.1] F_{2} = [0.2,0.3,0.4,..., 0.2] F_{3} = [0.3,0.4,0.5,..., 0.3] (assuming this frame contains a landmark) F_{4} = [0.2,0.3,0.4,..., 0.2] F_{5} = [0.1,0.2,0.3,..., 0.1] Calculation process 1. Calculate attention weight: Assume that the attention network AttNet is a simple linear model and calculates the attention weights based on the differences between feature vectors. Here we simply assume that F_{3} has a higher uniqueness compared to other frames, so it should get a higher attention weight.

[0053] W{1} = 0.1 W{2} = 0.2 W{3} = 0.5 (because this frame contains the landmark) W{4} = 0.2 W{5} = 0 (assuming this frame is similar to the first frame and has the lowest weight) Attention-weighted feature fusion: Based on the calculated attention weights, we can calculate the weighted fused feature vector F_{att}.

[0054] F{att} = W{1} · F{1} + W{2} · F{2} + W{3} · F{3} + W{4} · F{4} +W{5} · F{5} Since the feature vector dimension is 100, we can calculate only the first element as an example: F_{att}(1) = (0.1· 0.1) + (0.2· 0.2) + (0.5 · 0.3) + (0.2· 0.2) +(0 · 0.1) F_{att}(1) = 0.01 + 0.04 + 0.15 + 0.04 + 0 F_{att}(1) = 0.24 The calculation process is the same for other dimensions Landmark recognition: Assume that classifier C is a simple threshold model that identifies an object as a landmark if the first element of F_{att} is greater than 0.2.

[0055] In this example, F_{att}(1) = 0.24, which is greater than the threshold 0.2, so classifier C will recognize that the third frame contains a landmark.

[0056] Through the above hypothetical data and calculation process, we can see how the adaptive attention mechanism increases the importance of key frames (frames containing landmarks) by assigning different weights to different frames. In this simplified example, the attention weight makes the frame containing the landmark F{3} contribute the most to the final fused feature vector F{att}, so that the classifier can correctly identify the landmark.

[0057] The evaluation of the recognition result and calculation of the error rate comprises the following steps: Step 5.1: Collect the recognition results and the real landmark labels to determine whether the recognition results are correct; Step 5.2: Count the number of correctly identified landmarks and calculate the ratio of the number of correctly identified landmarks to the total number of landmarks to obtain the recognition accuracy; at the same time, calculate the error rate based on the number of incorrectly identified landmarks and the total number of landmarks. Finally, count the surface recognition error rates of different video sources. The calculation method is the number of incorrectly identified landmarks divided by the total number of identified landmarks.

[0058] The dynamic adjustment of the algorithm parameters in the recognition framework according to the error rate includes the following steps: Step 6.1: Error classification: First, determine the error rate of landmark recognition from different video sources, then set an error rate threshold θ to distinguish the preset range of error rates, and classify and summarize the error rates above and below the threshold; Step 6.2: Dynamic adjustment: When the recognition error is higher than the threshold θ, the recognition algorithm parameters of the corresponding video source are adjusted, including but not limited to increasing the sample training amount, optimizing the algorithm structure and adjusting the model parameters, and the adjusted algorithm parameters are calculated according to the following formula: New parameter = current parameter + α (target error rate - current error rate) Among them, α is the adjustment coefficient, which is used to control the step size of parameter adjustment, and the target error rate is set to a predetermined value lower than the threshold θ; Then, the adjusted algorithm parameters are applied to re-identify the landmarks, and the above steps are repeated until the recognition error rate drops below the threshold θ or reaches the predetermined number of iterations; When the recognition error rate is lower than the threshold θ, the current recognition algorithm parameters are maintained unchanged or fine-tuned according to the following formula: New parameter = current parameter + β (current error rate - target error rate) Among them, β is the fine-tuning coefficient, which is smaller than α and is used to make fine adjustments when the error rate is already low; The purpose of adjusting the model is to reduce the error rate and make the recognition result more accurate. In the formula, the target error rate is the preset lower error rate, and the current error rate is the actual calculated error rate. By calculating the difference between the target error rate and the current error rate and multiplying it by the adjustment coefficient α or β, an adjustment amount can be obtained to update the model parameters. The adjustment coefficients α and β ensure that the amplitude of the parameter adjustment is proportional to the magnitude of the error rate deviation, thereby ensuring the stability and effectiveness of the adjustment process. By iteratively applying this formula, the error rate can be gradually reduced until it reaches or falls below the threshold θ, proving that the formula can effectively guide the dynamic adjustment of the recognition framework.

[0059] Assume that landmark recognition has been completed, and a large amount of data recognition and storage has been performed for each category, and there are statistical results of the recognition error rate. After obtaining the error rate data, the dynamic adjustment method is used to dynamically adjust the error rate of the system. The following is the assumed output of dynamic adjustment: Assume that the initial error rate of landmark recognition of video source A is 8%, the threshold θ is set to 5%, and the target error rate is set to 3%. The adjustment coefficient α is set to 0.1, and the fine-tuning coefficient β is set to 0.05; Initial parameters: Current error rate = 8% Target error rate = 3% Adjustment factor α = 0.1 Fine-tuning factor β = 0.05 First adjustment: The current error rate is higher than the threshold θ and needs to be adjusted New parameter = current parameter + α × (target error rate - current error rate) New parameter = current parameter + 0.1 × (3% - 8%) New parameter = current parameter - 0.1 × 5% New parameter = current parameter - 0.5% After the adjustment, the error rate is reduced to 6%. Second adjustment: The current error rate is still higher than the threshold θ, so continue adjusting.

[0060] New parameter = current parameter + α × (target error rate - current error rate) New parameter = current parameter + 0.1 × (3% - 6%) New parameter = current parameter - 0.1 × 3% New parameter = current parameter - 0.3% After the assumption adjustment, the error rate is reduced to 4.7% The third adjustment: The current error rate is lower than the threshold θ, but higher than the target error rate, and needs to be fine-tuned.

[0061] New parameter = current parameter + β × (current error rate - target error rate) New parameter = current parameter + 0.05 × (4.7% - 3%) New parameter = current parameter + 0.05 × 1.7% New parameter = current parameter + 0.085% Assume that after adjustment, the error rate is reduced to 3.5%, which is lower than the target error rate.

[0062] Through the above adjustment process, we can see that the error rate has gradually decreased from 8% to 3.5%, close to the target error rate of 3%. This example proves that by iteratively applying the formula, the recognition framework can be dynamically adjusted to reduce the error rate, thereby improving the accuracy of landmark recognition. In actual applications, the parameter adjustment process will be iterated multiple times according to the specific situation until the error rate drops to the preset range.

[0063] The outputting of the recognition result comprises the following steps: Step 7.1: After completing the dynamic adjustment of the model parameters, first confirm whether the recognition model parameters after dynamic adjustment are kept within the preset threshold range. The stability of the parameter model after dynamic adjustment can be measured by calculating the standard deviation and coefficient of variation of the parameter changes to ensure that the parameter fluctuation does not exceed the preset threshold and the error rate is reduced to the preset target error rate range. Then, use the recognition model after stability verification and error rate detection to re-recognize the landmarks in the video and record the landmark recognition results output by the recognition model. Step 7.2: Organize the recognition results into a file of a unified format, output the organized recognition results to the user interface, and store them in the database for user query and analysis.

[0064] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for automatically identifying landmarks from a video, characterized in that: The following steps are involved: Step 1: Collect and receive real-time video stream data from different video sources, identify the source of the video stream, obtain device permissions and establish a connection with the device to obtain the video stream; Step 2: Analyze the metadata and video content features of the video stream, use the convolutional neural network deep learning model to extract features from the video frames that reflect the video characteristics in the video stream, and use the machine learning model to classify the extracted features to determine the video source type; Step 3: Select and apply different preprocessing algorithms according to the video source type, and perform video preprocessing operations after determining the preprocessing algorithm; Step 4: Based on the preprocessing results of the video preprocessing algorithm, a convolutional neural network deep learning model is constructed, and deep features are extracted using feature mapping and feature fusion methods. Then, the landmarks in the video are identified using an adaptive recognition framework. Step 5: After the landmark recognition is completed, the recognition results output by the recognition framework are evaluated, the number of recognition errors is counted, and the recognition error rate is calculated; Step 6: first classify the error data of videos from different sources according to the calculated error rate, and then use the recognition error dynamic adjustment method to adjust the algorithm parameters in the recognition framework of the corresponding video source; Step 7: After the dynamic adjustment is completed, the recognition result after dynamic adjustment is output for further use and analysis by the user or system.

2. The method for automatically identifying landmarks from a video according to claim 1, characterized in that: The receiving of video streams from different video sources further comprises the following steps: Step 1.1: Determine the video source: Identify the source of the video stream, including but not limited to webcams, live broadcast platforms, and video surveillance systems, and analyze the data source and characteristics of the video stream; Step 1.2: Obtain device permissions: Use the front-end controller and integrated API provided by the video source to ensure that you have access to the device. Step 1.3: Establish connection: To ensure real-time video stream data is obtained from the video source, use the corresponding library and software tools to connect and obtain the video stream from the video source.

3. The method for automatically identifying landmarks from a video according to claim 1, characterized in that: The analysis of the metadata and video content features of the video stream comprises the following steps: Step 2.1: After the connection is established, the metadata information of the video file is extracted based on the acquired video stream, the file name of the video file is analyzed, and a preliminary visual analysis of the video frame is performed; in addition, based on the analysis results, a convolutional neural network deep learning model is used to extract features of the frames in the video that reflect the video characteristics for further video source analysis; Step 2.2: After extracting the features, use the machine learning model to classify the extracted features, classify the analysis results, distinguish the category of the video source, and use the ensemble learning method to determine the source type of the video.

4. The method for automatically identifying landmarks from a video according to claim 1, characterized in that: The steps of selecting and applying different preprocessing algorithms according to the video source type include the following: Step 3.1: Determine the corresponding processing algorithm according to the identified video source type; For 3D videos from natural scenes, we apply array camera calibration and camera geometric view correction algorithms; for videos from surveillance systems, we apply image denoising and motion compensation algorithms; for videos from social media, we apply compression artifact repair and color correction algorithms; for videos from professional recordings, we apply video stabilization and scene segmentation algorithms; Step 3.2: Perform the video preprocessing operation after determining the preprocessing algorithm, and output the processed video stream for subsequent processing.

5. The method for automatically identifying landmarks from a video according to claim 1, characterized in that: The method of using an adaptive recognition framework to identify landmarks in a video comprises the following steps: Step 4.1: Deep feature extraction: Based on the preprocessing results of the video preprocessing algorithm, first build a convolutional neural network deep learning model, then train the model by defining the loss function and optimization algorithm. After the training is completed, use feature mapping and feature fusion methods to extract features. For multimodal data or when multiple features need to be fused, use an adaptive attention mechanism to fuse features from different sources, automatically learn and focus on the key areas and dynamic features of landmarks in the video; Step 4.2: Adaptive attention mechanism: After deep feature extraction using the convolutional neural network deep learning model, the adaptive attention mechanism is applied to dynamically adjust the degree of attention to different frames and different regions within the frame, automatically focusing on the key areas and dynamic features of landmarks in the video, and improving the accuracy and real-time performance of recognition; this mechanism includes two steps: attention weight calculation and attention weighted feature fusion, using the attention network and the feature vector fused with attention weights for calculation; the calculation formula of attention weight is: W = att [F(t),F(t-1),...,F(tn)] Where W represents the attention weight calculated at time t; att is an attention network that considers the features of the current frame and the past frame n, and the attention weight output by the attention network is 1; The calculation formula of attention weighted fusion is: ， Among them, F att is the feature vector fused with attention weights; F is the feature vector of each frame; n represents the total number of frames in the historical data, t represents the current frame, which refers to the index of the frame currently being considered; ti represents a time step in the past relative to the current time step t, where i is an index from 1 to n, indicating the number of frames backtracked from the current time step. If t is the current frame, t-1 is the previous frame, t-2 is the previous two frames, and so on, until tn, which is the oldest frame in the time window; Features are extracted through a deep learning model, and an adaptive attention mechanism is used to dynamically focus on key frames and key areas in the video to quickly locate and identify landmarks in the video stream.

6. The method for automatically identifying landmarks from a video according to claim 1, characterized in that: The evaluation of the recognition result and calculation of the error rate comprises the following steps: Step 5.1: Collect the recognition results and the real landmark labels to determine whether the recognition results are correct; Step 5.2: Count the number of correctly identified landmarks and calculate the ratio of the number of correctly identified landmarks to the total number of landmarks to obtain the recognition accuracy; at the same time, calculate the error rate based on the number of incorrectly identified landmarks and the total number of landmarks. Finally, count the surface recognition error rates of different video sources. The calculation method is the number of incorrectly identified landmarks divided by the total number of identified landmarks.

7. The method for automatically identifying landmarks from a video according to claim 1, characterized in that: The dynamic adjustment of the parameters in the recognition framework algorithm according to the error rate includes the following steps: Step 6.1: Error classification: First, determine the error rate of landmark recognition from different video sources, then set an error rate threshold θ to distinguish the preset range of error rates, and classify and summarize the error rates above and below the threshold; Step 6.2: Dynamic adjustment: When the recognition error is higher than the threshold θ, the recognition algorithm parameters of the corresponding video source are adjusted, including but not limited to increasing the sample training amount, optimizing the algorithm structure and adjusting the model parameters, and the adjusted algorithm parameters are calculated according to the following formula: New parameter = current parameter + α (target error rate - current error rate) Among them, α is the adjustment coefficient, which is used to control the step size of parameter adjustment, and the target error rate is set to a predetermined value lower than the threshold θ; Then, the adjusted algorithm parameters are applied to re-identify the landmarks, and the above steps are repeated until the recognition error rate drops below the threshold θ or reaches the predetermined number of iterations; When the recognition error rate is lower than the threshold θ, the current recognition algorithm parameters are maintained unchanged or fine-tuned according to the following formula: New parameter = current parameter + β (current error rate - target error rate) Among them, β is the fine-tuning coefficient, which is smaller than α and is used to make fine adjustments when the error rate is already low; In the formula, the target error rate is the preset error rate, and the current error rate is the actually calculated error rate. By calculating the difference between the target error rate and the current error rate and multiplying it by the adjustment coefficient α or β, an adjustment amount can be obtained to update the model parameters.

8. The method for automatically identifying landmarks from a video according to claim 1, characterized in that: The outputting of the recognition result comprises the following steps: Step 7.1: After completing the dynamic adjustment of the model parameters, first confirm whether the recognition model parameters after dynamic adjustment are kept within the preset threshold range. The stability of the parameter model after dynamic adjustment can be measured by calculating the standard deviation and coefficient of variation of the parameter changes to ensure that the parameter fluctuation does not exceed the preset threshold and the error rate is reduced to the preset target error rate range. Then, use the recognition model after stability verification and error rate detection to re-recognize the landmarks in the video and record the landmark recognition results output by the recognition model; Step 7.2: Organize the recognition results into a file of a unified format, output the organized recognition results to the user interface, and store them in the database for user query and analysis.

Images