A method and system for fast sparse map reconstruction based on video images

Abstract

The present invention discloses a method and system for fast sparse map reconstruction based on video images, which belongs to the field of video image processing technology, including motion analysis, key frame extraction, spatiotemporal graph convolution network feature extraction, image segmentation and multi-view fusion, to achieve fast reconstruction of sparse maps in dynamic environments. Technicians can correct maps in real time through an interactive interface, automatically adjust weights based on feedback, and provide personalized correction suggestions. The present invention optimizes the speed and accuracy of map reconstruction, enhances adaptability to dynamic environments, and improves the practicality and personalization of maps through user interaction, which has significant technical progress and practical value.

Description

Technical Field

[0001] The present invention relates to the technical field of video image processing, and in particular to a method and system for fast sparse map reconstruction based on video images. Background Art

[0002] In the field of video image processing, with the continuous development of science and technology, sparse map reconstruction technology has become a research hotspot in fields such as computer vision and robot navigation. Traditional sparse map reconstruction methods mainly rely on high-precision sensors and data preprocessing technologies. Although these methods can achieve map reconstruction to a certain extent, they generally have problems such as slow processing speed and poor adaptability to dynamic environments. In recent years, sparse map reconstruction technology based on video images has gradually attracted attention, and researchers have achieved remarkable results in this field, but there are still certain limitations.

[0003] The shortcomings of the existing related technologies are mainly reflected in the following aspects: First, when processing high-frame rate video data, the existing methods are often difficult to effectively identify and extract key frames, resulting in loss or redundancy of reconstructed map information; second, the existing technologies are not accurate in segmenting dynamic objects and backgrounds, which is prone to misjudgment and affects the accuracy of map reconstruction; third, in the process of multi-view fusion and sparse map optimization, the existing methods often have high computational complexity and are difficult to achieve fast reconstruction. The fast sparse map reconstruction method based on video images proposed by our invention optimizes and improves the above problems. Through the steps of motion analysis, key frame extraction, spatiotemporal graph convolutional network feature extraction, image segmentation and multi-view fusion, the sparse map is reconstructed quickly and accurately in a dynamic environment, which has good application prospects and practical value. On this basis, combined with the shortest path algorithm and user real-time correction, the efficiency and accuracy of sparse map reconstruction are further improved, providing new ideas for the development of technology in related fields. Summary of the invention

[0004] In view of the above existing problems, the present invention aims to solve multiple technical problems in sparse map reconstruction, including improving reconstruction speed, enhancing adaptability to dynamic environments, improving map accuracy, and realizing real-time user interaction and personalized correction. By optimizing steps such as motion analysis, key frame extraction, and spatiotemporal graph convolutional network feature extraction, this method significantly speeds up the reconstruction speed, and uses deep learning models to perform image segmentation and multi-view fusion, effectively coping with dynamic environments and improving map accuracy. At the same time, the shortest path algorithm is used to optimize the spatial relationship of the point cloud, and combined with the user correction interface, users are allowed to provide real-time feedback and personalized adjustments, thereby improving the user experience and the practicality of the map.

[0005] To solve the above technical problems, a fast sparse map reconstruction method based on video images is proposed, including:

[0006] Collect raw video data, perform motion analysis on the raw video data stream through the specified motion model, calculate the difference in motion changes and identify dynamic objects and backgrounds, apply motion compensation technology to preferentially select frames containing key features as key frames, blur the slow-moving background, and output a key frame sequence; use the spatiotemporal graph convolutional network to extract features in the key frame sequence, extract features to construct a feature adjacency matrix, establish a spatiotemporal relationship model, and output a spatiotemporal consistent feature map; apply the spatiotemporal consistent feature map to the deep learning model to perform image segmentation on the feature map of the key frame, judge the feature map and the set threshold to segment the dynamic foreground and the relatively static background, and output the segmented video frame and the separated dynamic foreground and the relatively static background; evaluate the quality of multiple perspectives, sort the multiple perspectives, select the best perspective for fusion, and output the fused sparse map structure; use the shortest path algorithm to optimize the spatial relationship between sparse map point clouds. During the optimization process, if the set conditions are met, the optimized fast sparse map point cloud will be output, and if the conditions are not met, the re-optimization mechanism will be triggered; technicians update the optimized map information in real time by clicking and dragging, allowing correction and supplementary data, and output the user-corrected and enhanced fast sparse map.

[0007] As a preferred solution of the method for fast sparse map reconstruction based on video images described in the present invention, wherein: the original video data includes video format, frame resolution, illumination change and moving object speed acquired at a high frame rate;

[0008] The motion analysis includes inputting the original video data stream into a set motion model and analyzing the motion change difference between two consecutive frames in any area:

[0009] ;

[0010] in, Indicates at location and time The difference in movement, Indicates at location and time The image intensity at Indicates time The next frame after Indicates time The previous frame before, Indicates at location and time The image intensity at Indicates at location and time Image intensity at ;

[0011] when ≤ parameter 1, it is marked as a relatively static background; when > parameter 1, it is marked as a candidate area for dynamic objects. If the mean motion difference in the current area is greater than 10, it is marked as a dynamic object. When the mean motion difference in the current area does not exceed 10, manual judgment by technicians is performed.

[0012] As a preferred solution of the method for fast sparse map reconstruction based on video images described in the present invention, the blurring process includes applying motion compensation technology and preferentially selecting frames containing key features as key frames: , where n represents the total number of frames analyzed and i is the variable index. Represents the time interval between consecutive frames, is the threshold set to form a keyframe; Indicates the time stamp of each frame;

[0013] After extracting the keyframes, blur the background that moves slowly in the video:

[0014] ;

[0015] in, Indicates at location The intensity of the blurred image is Indicates the position in the current image frame The strength value of is a constant that normalizes the blur effect, represents the weight function of different blur levels, Representation and location The degree of motion blur, For the location and weighted time frames The image intensity at represents the number of different time frames used for weighting, and j is the variable index;

[0016] After the blurring is complete, output the resulting keyframe sequence , At the point in time The blurred image of Indicates the number of key frames extracted.

[0017] As a preferred solution of the method for fast sparse map reconstruction based on video images described in the present invention, the spatiotemporal relationship model includes: , where each keyframe is encoded into a corresponding feature map , , in the spatiotemporal graph convolutional network, features are extracted to construct the feature adjacency matrix:

[0018] ;

[0019] in, is the feature map and The measure of similarity between is the feature adjacency matrix representing the Time Keyframe and keyframes The connection strength between To determine whether to connect, the normalized threshold is To further restrict the connectivity of the feature adjacency matrix, Respectively represent the indexes of adjacent key frames in the key frame sequence;

[0020] Through the convolution operation, effective information is extracted from the spatiotemporal characteristics of the feature graph to build a spatiotemporal relationship model:

[0021] ;

[0022] Output the spatiotemporal consistency feature map, where Indicates at time The extracted spatiotemporal features, Keyframes The neighbor set of Indicates at time The spatial and temporal characteristics of the input, represents the weight matrix of the spatiotemporal convolution kernel, represents the comprehensive spatiotemporal feature map obtained through multi-layer processing; L is the number of layers of graph convolution, is the variable index, is a non-linear activation function.

[0023] As a preferred solution of the method for fast sparse map reconstruction based on video images described in the present invention, the image segmentation includes: When the threshold is less than 1, segmentation is performed to extract the dynamic foreground from the background;

[0024] when When the threshold is less than 2, no segmentation is performed and the video frame is regarded as an intermediate state without any processing;

[0025] when When the threshold is ≥ 2, segmentation is performed on the relatively static background extracted from the background;

[0026] Output segmented video frames with separated dynamic foreground and static background.

[0027] As a preferred solution of the method for fast sparse map reconstruction based on video images described in the present invention, the selection of the best perspective for fusion includes evaluating the quality of multiple perspectives:

[0028] ;

[0029] in, Representation perspective Quality score, Perspective The corresponding feature map set, Represents different feature maps from multiple perspectives The normalized value of Representing different feature maps from multiple perspectives Stability measure of ;

[0030] According to the evaluation results, multiple views are sorted and B views with the best or second best quality are selected for fusion:

[0031] ;

[0032] in, The final fused feature map, The number of perspectives involved in the fusion, is the variable index, is the dynamically adjusted weight;

[0033] Extract sparse point cloud based on the final fused feature map and output the fused sparse map structure:

[0034] ;

[0035] in, is the output sparse point cloud set; Indicates The position coordinates of sparse points, For sparse points The coordinates in three-dimensional space, Index of the variable.

[0036] As a preferred solution of the method for fast sparse map reconstruction based on video images described in the present invention, the fast sparse map point cloud includes optimizing the spatial relationship between sparse map point clouds using the shortest path algorithm:

[0037] ;

[0038] in, From point cloud To point cloud The optimized path cost is Representing point clouds To point cloud The direct distance is an exponential decay function, is a scaling factor that controls the impact of environmental costs on the optimization path. Indicates that in the current optimization process, the point cloud arrive environmental costs, Indicates passing point cloud arrive The path cost of

[0039] During the optimization process, satisfy Less than 7, when When it exceeds 7, increase the scaling factor ;

[0040] Must be less than 10, when When the conditions are not met, the re-optimization mechanism will be triggered;

[0041] Output the optimized fast sparse map point cloud. The technician interface will display the current fast sparse map view. Click on the point cloud in the map to add or correct data, mark the area that needs correction on the map, and provide correction information on the selected point cloud. The technician will be prompted to enter new coordinates or feature descriptions and choose to adjust the weights. After confirming the correction, the data will be included in the preliminary fast sparse map to synthesize a new map.

[0042] Introduce the point cloud updates provided by the user into the map, and adjust the fusion ratio according to the weight of user feedback. After the correction is completed, you will be prompted to view the updated map and asked if you are satisfied with the changes. If you are satisfied, the correction results will be permanently saved; if you are not satisfied, you will be allowed to continue to adjust.

[0043] The correction weight is automatically updated for each correction, and user feedback is recorded to provide personalized correction suggestions based on historical data and user preferences.

[0044] Another object of the present invention is to provide a fast sparse map reconstruction system based on video images. The present invention aims to solve a series of technical problems, including the problem of accurately reconstructing sparse maps in dynamic scenes, effectively identifying and separating dynamic objects and static backgrounds through motion analysis and segmentation technology; the problem of improving map reconstruction efficiency, by selecting key frames and applying motion compensation technology to reduce the amount of processed data; the challenge of multi-view data fusion, by evaluating the quality of the view and selecting the best view for fusion, to ensure the integrity and accuracy of the map; optimizing the spatial relationship between sparse map point clouds, using the shortest path algorithm to ensure the rationality of spatial distribution; and the problem of user interaction and map update, by providing a user interaction correction module and personalized correction suggestions, to enhance the flexibility and user-friendliness of the system. In general, it is committed to achieving high-efficiency and high-precision sparse map reconstruction in a dynamic environment, and optimizing map quality through user participation to adapt to complex and changing application requirements.

[0045] As a preferred solution of the fast sparse map reconstruction system based on video images described in the present invention, it is characterized by comprising a video data preprocessing module, a feature extraction and segmentation module, a view evaluation and fusion module, a sparse map optimization module and a user interaction correction module;

[0046] The video data preprocessing module receives the original video data stream, processes the video data with a motion analysis algorithm, identifies and marks dynamic objects and backgrounds, uses motion compensation technology to select key frames representing scene changes, performs blur processing on frames with less intense background motion, and outputs a processed key frame sequence;

[0047] The feature extraction and segmentation module receives a key frame sequence, extracts features using a spatiotemporal graph convolutional network, constructs a feature adjacency matrix, establishes a spatiotemporal relationship model, applies a deep learning model to segment the feature graph, distinguishes between dynamic foreground and static background, and outputs the segmentation result;

[0048] The perspective evaluation and fusion module evaluates the quality of feature maps from different perspectives, selects the optimal or suboptimal perspective for fusion based on the evaluation results, fuses the feature maps, extracts sparse point clouds, and constructs a sparse map structure;

[0049] The sparse map optimization module uses the shortest path algorithm to optimize the spatial relationship between the sparse point clouds, adjusts the path cost and the spatial relationship, and outputs the optimized sparse map point cloud;

[0050] In the user interactive correction module, technicians interact with the map through an interface to perform corrections and supplements, record user operations, update map information, provide personalized suggestions, adjust correction weights based on user preferences and historical data, confirm correction results, save if satisfied, and continue to adjust if dissatisfied.

[0051] A computer device includes a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of a method for fast sparse map reconstruction based on video images are implemented.

[0052] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of a method for fast sparse map reconstruction based on video images are implemented.

[0053] Beneficial effects of the present invention: The present invention effectively identifies dynamic objects and backgrounds by performing motion analysis on raw video data, and optimizes the selection of key frames using motion compensation technology, thereby reducing information loss and redundancy and improving the accuracy of key frame extraction. At the same time, blurring the slow-moving background reduces computational complexity, improves processing speed, and provides accurate basic data for sparse map reconstruction. Features in key frame sequences are extracted through a spatiotemporal graph convolutional network, and a spatiotemporal relationship model is established. This process not only integrates temporal and spatial information, but also improves the comprehensiveness and accuracy of feature extraction, providing rich feature information for image segmentation and map reconstruction.

[0054] Furthermore, the feature map is segmented through a deep learning model, which accurately separates the dynamic foreground and the relatively static background, reduces misjudgment in the segmentation process, and improves the accuracy of map reconstruction. The integrity and stability of the sparse map are enhanced by evaluating the quality of multiple perspectives and selecting the best perspective for fusion. The spatial relationship of the sparse map point cloud is optimized using the shortest path algorithm to ensure the robustness of map reconstruction. Finally, technicians are allowed to correct and supplement data in real time, and personalized correction suggestions are used to improve the practicality of the map and the user's interactive experience, making the fast sparse map more accurate and adaptable to different scene requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0056] Figure 1 An overall flow chart of a method for fast sparse map reconstruction based on video images provided for one embodiment of the present invention.

[0057] Figure 2 A system solution module diagram of a fast sparse map reconstruction system based on video images provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0059] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0060] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it an embodiment that is mutually exclusive with other embodiments, either individually or selectively.

[0061] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0062] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0063] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0064] Example 1, reference Figure 1, which is the first embodiment of the present invention, provides a method for fast sparse map reconstruction based on video images, comprising:

[0065] S1: Collect raw video data, perform motion analysis on the raw video data stream using the specified motion model, calculate motion change differences and identify dynamic objects and backgrounds, apply motion compensation technology to preferentially select frames containing key features as key frames, blur the slow-moving background, and output a key frame sequence.

[0066] Specifically, the raw video data includes video format acquired at a high frame rate, frame resolution, lighting changes, and speed of moving objects.

[0067] It should be noted that the original video data stream is input into the set motion model to analyze the motion change difference between two consecutive frames in any area:

[0068] ;

[0069] in, Indicates at location and the difference in motion at time t, Indicates at location and the image intensity at time t, represents the next frame after time t, represents the previous frame before time t, Indicates at location and time The image intensity at Indicates at location and time Image intensity at ;

[0070] when ≤15, it is marked as a relatively static background; >15, it is marked as a candidate area for dynamic objects. If the mean motion difference in the current area is greater than 10, it is marked as a dynamic object. When the mean motion difference in the current area does not exceed 10, manual judgment by technicians is performed.

[0071] Among them, by performing statistical analysis on the overall distribution of motion differences, the brightness changes in dynamic objects and static background areas are observed; the brightness changes of feature points in multiple sample frames are measured to obtain the average value and standard deviation of the value; multiple rounds of experiments are carried out, and the threshold is continuously adjusted to find the value that can best distinguish dynamic objects from static backgrounds; based on experimental data, it is found that the changes in static areas are usually concentrated in a small value range, that is, less than 15, while the changes in dynamic areas are significantly increased, and this threshold is selected.

[0072] The threshold value is set to 15, which can effectively exclude small motion noises, so that only motion exceeding this value will be regarded as a dynamic object, which helps to reduce misidentification caused by environmental fluctuations. In many practical environments, the movement speed and change amplitude of moving objects (such as pedestrians and vehicles) have a certain range. A dynamic threshold adjustment mechanism based on real-time data feedback is implemented to automatically update the threshold setting according to changes in the current environmental conditions to ensure higher accuracy and flexibility.

[0073] Furthermore, motion compensation technology is applied to preferentially select frames containing key features as key frames: , where n represents the total number of frames analyzed and i is the variable index. Represents the time interval between consecutive frames, is the set threshold, Indicates the time stamp of each frame;

[0074] Among them, by performing statistical analysis on motion changes of a set of annotated video data, the relationship between effective key frames and motion changes is observed. Through experiments, it is found that the sum of motion changes greater than or equal to 50 often corresponds to frames with obvious scene changes, which can be set as the threshold; Dynamic objects in highly complex scenes often contain more motion changes; in static scenes, a lower threshold can effectively extract key frames. It has been debugged to effectively capture obvious motion changes while avoiding unnecessary keyframes, thus balancing the efficiency and accuracy of extraction.

[0075] After extracting the keyframes, blur the background that moves slowly in the video:

[0076] ;

[0077] in, Indicates at location The intensity of the blurred image is Indicates the position in the current image frame The strength value of is a constant that normalizes the blur effect, represents the weight function of different blur levels, Representation and location The degree of motion blur, For the location and weighted time frames The image intensity at represents the number of different time frames used for weighting, and j is the variable index;

[0078] After the blurring is complete, output the resulting keyframe sequence , At the point in time The blurred image of Indicates the number of key frames extracted.

[0079] S2: Use the spatiotemporal graph convolutional network to extract features from the key frame sequence, extract features to construct a feature adjacency matrix, establish a spatiotemporal relationship model, and output a spatiotemporal consistent feature map.

[0080] Furthermore, according to the input key frame sequence , where each keyframe is encoded into a corresponding feature map , , in the spatiotemporal graph convolutional network, features are extracted to construct the feature adjacency matrix:

[0081] ;

[0082] in, is the feature map and The measure of similarity between is the feature adjacency matrix representing the Time Keyframe and keyframes The connection strength between To determine whether to connect, the normalized threshold is To further restrict the connectivity of the feature adjacency matrix, Respectively represent the indexes of adjacent key frames in the key frame sequence;

[0083] Through the convolution operation, effective information is extracted from the spatiotemporal characteristics of the feature graph to build a spatiotemporal relationship model:

[0084] ;

[0085] Output the spatiotemporal consistency feature map, where Indicates at time The extracted spatiotemporal features, Keyframes The neighbor set of Indicates at time The spatial and temporal characteristics of the input, represents the weight matrix of the spatiotemporal convolution kernel, Represents the comprehensive spatiotemporal feature map obtained through multi-layer processing, which contains the information integration of time and space; L is the number of layers of graph convolution, l is the variable index, is a non-linear activation function.

[0086] S3: Apply the spatiotemporal consistency feature map to the deep learning model to perform image segmentation on the feature map of the key frame, judge the feature map and the set threshold to segment the dynamic foreground and the relatively static background, and output the segmented video frame and the separated dynamic foreground and the relatively static background.

[0087] It should be noted that when When <0.3, segmentation is performed to extract the dynamic foreground from the background;

[0088] when When <0.7, no segmentation is performed and the video frame is regarded as an intermediate state without any processing;

[0089] when When ≥0.7, segmentation is performed on the relatively static background extracted from the background;

[0090] Output segmented video frames with separated dynamic foreground and static background.

[0091] Among them, the threshold 1 is set to 0.3 to judge the dynamic foreground and effectively identify dynamic objects (such as moving objects) from non-dynamic relatively low-brightness backgrounds; the threshold 2 is set to 0.7 to exclude high-brightness areas in order to capture strong change features or dynamic foregrounds, while avoiding mistakenly classifying some high-intensity backgrounds (such as reflective or highlight areas) as dynamic foregrounds. Dynamic objects generally have more obvious brightness changes than static backgrounds, so setting a higher threshold can better distinguish them.

[0092] By setting two thresholds, it is possible to effectively reduce erroneous segmentation caused by noise, segment out irrelevant static areas or backgrounds with excessive brightness, and enhance the robustness of the entire segmentation process. In video streams, the background is usually relatively static, while the brightness of dynamic objects changes more significantly. By managing two thresholds, these dynamic elements can be accurately captured and the recognition rate of dynamic foregrounds can be improved. The setting of threshold 1 and threshold 2 enables the algorithm to be flexibly adjusted in different scenarios. For example, in complex environments (such as scenes with significant changes in lighting), these thresholds need to be adjusted accordingly to ensure the segmentation effect. It can also adapt to different types of dynamic objects, whether more sensitive detection is required.

[0093] S4: Evaluate the quality of multiple views, sort the multiple views, select the best view for fusion, and output the fused sparse map structure.

[0094] Going a step further, evaluate the quality of multiple views:

[0095] ;

[0096] in, Representation perspective Quality score, Perspective The corresponding feature map set, Represents different feature maps from multiple perspectives The normalized value of Representing different feature maps from multiple perspectives Stability measure of ;

[0097] According to the evaluation results, multiple views are sorted and B views with the best or second best quality are selected for fusion:

[0098] ;

[0099] in, The final fused feature map, The number of perspectives involved in the fusion, is the variable index, is the dynamically adjusted weight;

[0100] Extract sparse point cloud based on the final fused feature map and output the fused sparse map structure:

[0101] ;

[0102] in, is the output sparse point cloud set, which contains all the sparse points in the map; Indicates The position coordinates of sparse points, For sparse points The coordinates in three-dimensional space, Index of the variable.

[0103] S5: Use the shortest path algorithm to optimize the spatial relationship between sparse map point clouds. During the optimization process, if the set conditions are met, the optimized fast sparse map point cloud will be output. If the conditions are not met, the re-optimization mechanism will be triggered.

[0104] Furthermore, the shortest path algorithm is used to optimize the spatial relationship between sparse map point clouds:

[0105] ;

[0106] in, From point cloud To point cloud The optimized path cost is Representing point clouds To point cloud The direct distance is an exponential decay function, is a scaling factor that controls the impact of environmental costs on the optimization path. Indicates that in the current optimization process, the point cloud arrive environmental costs, Indicates passing point cloud arrive The path cost of

[0107] During the optimization process, satisfy Less than 7, when When it exceeds 7, increase the scaling factor ;

[0108] Must be less than 10, when When the conditions are not met, the re-optimization mechanism will be triggered.

[0109] S6: Technicians update the optimized map information in real time by clicking and dragging, allowing correction and supplementation of data, and outputting a fast sparse map enhanced by user correction.

[0110] It should be noted that when outputting the optimized fast sparse map point cloud, the technician interface will display the view of the current fast sparse map. By clicking on the point cloud in the map to add or correct data, the area that needs correction is marked on the map. The technician provides correction information on the selected point cloud, and will be prompted to enter new coordinates or feature descriptions, and choose to adjust the weights. After confirming the correction, the data will be included in the preliminary fast sparse map to synthesize a new map;

[0111] Introduce the point cloud updates provided by the user into the map, and adjust the fusion ratio according to the weight of the user's feedback. After the correction is completed, you will be prompted to view the updated map and asked if you are satisfied with the changes. If you are satisfied, the correction results will be permanently saved; if you are not satisfied, you will be allowed to continue to adjust.

[0112] The correction weight is automatically updated for each correction, and user feedback is recorded to provide personalized correction suggestions based on historical data and user preferences.

[0113] Embodiment 2, the second embodiment of the present invention, is different from the first two embodiments in that:

[0114] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0115] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0116] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0117] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0118] Example 3, reference Figure 2 , which is the third embodiment of the present invention, and which provides a fast sparse map reconstruction system based on video images, including a video data preprocessing module, a feature extraction and segmentation module, a view evaluation and fusion module, a sparse map optimization module, and a user interaction correction module;

[0119] The video data preprocessing module receives the original video data stream, processes the video data with motion analysis algorithms, identifies and marks dynamic objects and backgrounds, uses motion compensation technology to select key frames representing scene changes, blurs frames with less intense background motion, and outputs the processed key frame sequence;

[0120] The feature extraction and segmentation module receives the key frame sequence, uses the spatiotemporal graph convolutional network to extract features, constructs the feature adjacency matrix, establishes the spatiotemporal relationship model, applies the deep learning model to segment the feature map, distinguishes the dynamic foreground from the static background, and outputs the segmentation result;

[0121] The view evaluation and fusion module evaluates the quality of feature maps from different viewpoints, selects the optimal or suboptimal viewpoint for fusion based on the evaluation results, fuses the feature maps, extracts sparse point clouds, and constructs a sparse map structure;

[0122] The sparse map optimization module uses the shortest path algorithm to optimize the spatial relationship between sparse point clouds, adjust the path cost and spatial relationship, and output the optimized sparse map point cloud;

[0123] User Interaction Correction Module Technicians interact with the map through the interface, make corrections and additions, record user operations, update map information, provide personalized suggestions, adjust correction weights based on user preferences and historical data, confirm correction results, save if satisfied, and continue to adjust if dissatisfied.

[0124] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A fast sparse map reconstruction method based on video images, characterized in that: include, Collect raw video data, perform motion analysis on the raw video data stream using a specified motion model, calculate motion change differences and identify dynamic objects and backgrounds, apply motion compensation technology to preferentially select frames containing key features as key frames, blur the slow-moving background, and output a key frame sequence; The spatiotemporal graph convolutional network is used to extract features from the key frame sequence, extract features to construct a feature adjacency matrix, establish a spatiotemporal relationship model, and output a spatiotemporal consistent feature map; Apply the spatiotemporal consistency feature map to the deep learning model to perform image segmentation on the feature map of the key frame, judge the feature map and the set threshold to segment the dynamic foreground and the relatively static background, and output the segmented video frame and the separated dynamic foreground and the relatively static background; Evaluate the quality of multiple views, sort the multiple views, select the best view for fusion, and output the fused sparse map structure; Use the shortest path algorithm to optimize the spatial relationship between sparse map point clouds. During the optimization process, if the set conditions are met, the optimized fast sparse map point cloud will be output. If the conditions are not met, the re-optimization mechanism will be triggered. By clicking and dragging the information of the current fast sparse map, the map information is updated in real time to obtain the real-time updated and optimized map information, allowing correction and supplementary data, and outputting the user-corrected and enhanced fast sparse map.

2. A method for fast sparse map reconstruction based on video images as claimed in claim 1, characterized in that: The raw video data includes video format, frame resolution, illumination changes, and speed of moving objects acquired at a high frame rate; The motion analysis includes inputting the original video data stream into a set motion model and analyzing the motion change difference between two consecutive frames in any area: ， in, Indicates at location and time The difference in movement, Indicates at location and time The image intensity at Indicates time The next frame after Indicates time The previous frame before, Indicates at location and time The image intensity at Indicates at location and time Image intensity at ; when ≤15, it is marked as a relatively static background; When the average motion difference in the current area is greater than 10, it is marked as a dynamic object. When the average motion difference in the current area does not exceed 10, manual judgment is performed.

3. The method for fast sparse map reconstruction based on video images according to claim 2, characterized in that: The blur processing includes applying motion compensation technology, selecting frames containing key features as key frames, and after extracting the key frames, blurring the background moving slowly in the video: ， in, Indicates at location The intensity of the blurred image is Indicates the position in the current image frame The strength value of is a constant that normalizes the blur effect, represents the weight function of different blur levels, Representation and location The corresponding degree of motion blur, For the location and weighted time frames The image intensity at represents the number of different time frames used for weighting, and j is the variable index; After the blurring is complete, output the resulting keyframe sequence , At the point in time The blurred image intensity value, Indicates the number of key frames extracted.

4. The method for fast sparse map reconstruction based on video images according to claim 3, characterized in that: The spatiotemporal relationship model includes: , where each keyframe is encoded into a corresponding feature map , , in the spatiotemporal graph convolutional network, features are extracted to construct the feature adjacency matrix: ， in, is the feature map and The measure of similarity between is the feature adjacency matrix representing the Time Keyframe and keyframes The connection strength between To determine whether to connect, the normalized threshold is To further restrict the connectivity of the feature adjacency matrix, Respectively represent the indexes of adjacent key frames in the key frame sequence; Through the convolution operation, effective information is extracted from the spatiotemporal characteristics of the feature graph to build a spatiotemporal relationship model: ， ， Output the spatiotemporal consistency feature map, where Indicates at time The extracted spatiotemporal features, Represents a key frame The neighbor set of Indicates at time The spatial and temporal characteristics of the input, represents the weight matrix of the spatiotemporal convolution kernel, Represents the comprehensive spatiotemporal feature map obtained through multi-layer processing, which contains the information integration of time and space; L is the number of layers of graph convolution, l is the variable index, is a non-linear activation function.

5. The method for fast sparse map reconstruction based on video images according to claim 4, characterized in that: The image segmentation includes: When <0.3, segmentation is performed to extract the dynamic foreground from the background; when When <0.7, no segmentation is performed and the video frame is regarded as an intermediate state without any processing; when When ≥0.7, segmentation is performed on the relatively static background extracted from the background; Output segmented video frames with separated dynamic foreground and static background.

6. A method for fast sparse map reconstruction based on video images as claimed in claim 5, characterized in that: The selection of the best view for fusion includes evaluating the quality of multiple views: ， in, Representation perspective Quality score, Perspective The corresponding comprehensive spatiotemporal feature map under the condition of Represents different feature maps from multiple perspectives The normalized value of Representing different feature maps from multiple perspectives Stability measure of ; According to the evaluation results, multiple views are sorted and B views with the best or second best quality are selected for fusion: ， in, The final fused feature map, The number of perspectives involved in fusion, b is the variable index, is the dynamically adjusted weight; Extract sparse point cloud based on the final fused feature map and output the fused sparse map structure: ， in, is the output sparse point cloud set, which contains all the sparse points in the map; Indicates The position coordinates of sparse points, For sparse points The coordinate in three-dimensional space, o is the variable index.

7. The method for fast sparse map reconstruction based on video images according to claim 6, characterized in that: The fast sparse map point cloud includes optimizing the spatial relationship between sparse map point clouds using the shortest path algorithm: ， in, From point cloud To point cloud The optimized path cost is Representing point clouds To point cloud The direct distance is an exponential decay function, is a scaling factor that controls the impact of environmental costs on the optimization path. Indicates that in the current optimization process, the point cloud arrive environmental costs, Indicates passing point cloud arrive The path cost of During the optimization process, satisfy Less than 7, when When it exceeds 7, increase the scaling factor ; Must be less than 10, when When the conditions are not met, the re-optimization mechanism will be triggered; Output the optimized fast sparse map point cloud. The interface will display the current fast sparse map view. Click on the point cloud in the map to add or correct data, mark the area that needs correction on the map, provide correction information on the selected point cloud, and prompt to enter new coordinates or feature descriptions, and choose to adjust the weight. After confirming the correction, the data will be included in the preliminary fast sparse map to synthesize a new map. Introduce the point cloud updates provided by the user into the map, and adjust the fusion ratio according to the weight of the user's feedback. After the correction is completed, you will be prompted to view the updated map. The system will ask whether you are satisfied with the changes. If you are satisfied, the correction results will be permanently saved; if you are not satisfied, you will be allowed to continue to adjust. The correction weight is automatically updated for each correction, and user feedback is recorded to provide personalized correction suggestions based on historical data and user preferences.

8. A system using a method for fast sparse map reconstruction based on video images as claimed in any one of claims 1 to 7, characterized in that: It includes video data preprocessing module, feature extraction and segmentation module, viewpoint evaluation and fusion module, sparse map optimization module and user interaction correction module; The video data preprocessing module, the system receives the original video data stream, the motion analysis algorithm processes the video data, identifies and marks dynamic objects and backgrounds, and the motion compensation technology is used to select key frames representing scene changes, blur the frames with less intense background motion, and output the processed key frame sequence; The feature extraction and segmentation module receives a key frame sequence, extracts features using a spatiotemporal graph convolutional network, constructs a feature adjacency matrix, establishes a spatiotemporal relationship model, applies a deep learning model to segment the feature graph, distinguishes between dynamic foreground and static background, and outputs the segmentation result; The perspective evaluation and fusion module evaluates the quality of feature maps from different perspectives, selects the optimal or suboptimal perspective for fusion based on the evaluation results, fuses the feature maps, extracts sparse point clouds, and constructs a sparse map structure; The sparse map optimization module uses the shortest path algorithm to optimize the spatial relationship between the sparse point clouds, adjusts the path cost and the spatial relationship, and outputs the optimized sparse map point cloud; The user interactive correction module interacts with the map through an interface to perform corrections and supplements. The system records user operations, updates map information, provides personalized suggestions, adjusts correction weights based on user preferences and historical data, confirms correction results, saves them if satisfied, and continues to adjust if not satisfied.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the processor implements the steps of a method for fast sparse map reconstruction based on video images as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method for fast sparse map reconstruction based on video images described in any one of claims 1 to 7 are implemented.

Images