A method and system for surveying and mapping geospatial big data processing

Abstract

This application proposes a method and system for processing geospatial big data in the field of geospatial big data technology. The method includes acquiring a data stream to be processed and identifying and labeling it to distinguish between ordinary and emergency data; continuously monitoring the real-time operating status of all specialized processing units, including image recognition and text semantic analysis units; dynamically adjusting resource allocation among the specialized processing units based on their real-time operating status; and establishing a high-priority network transmission channel to transmit emergency data to the specialized processing units when emergency data occurs. The specialized processing units then perform image and semantic processing on the emergency data to obtain processing results, including incident type, location coordinates, event labels, and semantic vectors. This application can improve the system's response speed and processing efficiency to emergencies.

Description

Technical Field

[0001] This application relates to the field of surveying and mapping geospatial big data technology, and more specifically, to a surveying and mapping geospatial big data processing method and system. Background Technology

[0002] In processing massive amounts of geospatial data, existing technologies have significant limitations, struggling to adapt to scenarios with large data volumes, diverse sources, and varied formats. They are particularly inadequate for emergency response to sudden events and real-time analysis and decision-making. These systems can only handle single, well-structured data relatively well. When faced with complex and ever-changing unstructured information, they exhibit problems such as low indexing efficiency and uneven distribution of computing resources, severely restricting overall performance and response speed. In fields such as urban traffic management, to improve emergency response capabilities and prediction accuracy, engineers have introduced unstructured internet data such as social media text and images. This data contains timely descriptions of traffic events, but integration has encountered severe challenges. Existing multi-dimensional indexing mechanisms are designed specifically for handling numerical information such as coordinate points, line segments, and timestamps. They cannot understand and index fuzzy geographical descriptions, nor can they establish effective retrieval relationships for event semantic keywords. This results in complex queries combining spatial, temporal, and semantic elements being completed only through inefficient line-by-line matching. To solve the indexing problem, engineers attempted to build an independent full-text search engine, but this led to a mismatch at the computational level. The original parallel computing framework was optimized for homogeneous numerical computation tasks. However, unstructured data processing requires the introduction of heterogeneous tasks such as natural language processing and image content recognition. These tasks have higher computational complexity and significantly different resource consumption patterns. The original framework could not accurately predict their execution time, resulting in an imbalance in task allocation: some nodes became heavily loaded and bottlenecked due to the large number of complex text analysis tasks, while others were idle after completing simple numerical calculations. This led to a serious waste of computing resources, ultimately slowing down the system's processing speed and increasing the latency of data fusion and analysis. As a result, the goal of improving real-time response capabilities by introducing new data sources was difficult to achieve. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a method and system for processing geospatial big data in mapping, aiming to improve the system's response speed and processing efficiency to emergencies.

[0004] In a first aspect, embodiments of this application provide a method for processing geospatial big data, including:

[0005] Acquire the data stream to be processed and identify and label the data stream to distinguish between ordinary data and urgent data;

[0006] Continuously monitor the real-time working status of all professional processing units, including image recognition professional units and text semantic analysis professional units;

[0007] The resource allocation of the specialized processing unit is dynamically adjusted based on the real-time working status.

[0008] When the emergency data appears, a high-priority network transmission channel is established to transmit the emergency data to the professional processing unit. The professional processing unit performs image and semantic processing on the emergency data to obtain the processing results, which include accident type, location coordinates, event tags, and semantic vectors.

[0009] According to some embodiments of this application, the step of identifying and labeling the data stream to be processed to distinguish between ordinary data and urgent data includes:

[0010] When the data stream to be processed is text data, a matching analysis is performed based on a preset keyword matcher and a preset sentiment analyzer to obtain the number of urgent keywords and sentiment tendency scores.

[0011] When the number of emergency keywords exceeds a preset number or the sentiment score is lower than a preset score, the text data is marked as emergency data, wherein the preset score is obtained based on negative sentiment keywords.

[0012] According to some embodiments of this application, the step of identifying and labeling the data stream to be processed to distinguish between ordinary data and urgent data includes:

[0013] When the data stream to be processed is image data, visual feature extraction and scene classification are performed based on a preset shallow feature extractor to obtain detection confidence and classification results;

[0014] When the detection confidence level is greater than the preset confidence level and the classification result is consistent with the preset result, or when the classification result is consistent with the preset result, the image data is marked as emergency data, wherein the preset result represents the event result related to the emergency event.

[0015] According to some embodiments of this application, the continuous monitoring of the real-time operating status of all specialized processing units includes:

[0016] The CPU utilization, GPU utilization, memory usage, and list of tasks being processed for all professional processing units are obtained according to a preset cycle.

[0017] According to some embodiments of this application, the step of dynamically adjusting the resource allocation of the specialized processing unit based on the real-time working status includes:

[0018] The current load status of the specialized processing unit is obtained based on the CPU utilization, GPU utilization, memory usage, and the list of tasks being processed.

[0019] Resource allocation is dynamically adjusted based on the current load status of the specialized processing unit.

[0020] According to some embodiments of this application, the step of performing image and semantic processing on the emergency data to obtain processing results includes:

[0021] The emergency data is processed to obtain the accident type and location coordinates;

[0022] The emergency data is semantically processed to obtain event tags and semantic vectors.

[0023] According to some embodiments of this application, semantic processing is performed on the emergency data to obtain event tags and semantic vectors, including:

[0024] The keywords in the emergency data are converted into semantic vectors based on a pre-trained word vector model.

[0025] Event tags are extracted from the emergency data based on pattern matching according to preset rules.

[0026] According to some embodiments of this application, after obtaining the processing result, the process further includes:

[0027] The processing result is added to the data stream to be processed to obtain the target information and store it in the existing multidimensional spatiotemporal index;

[0028] When a user enters query conditions and performs a query based on the multidimensional spatiotemporal index, the query results are obtained and pushed to the user.

[0029] According to some embodiments of this application, when a user inputs query conditions and performs a query based on the multidimensional spatiotemporal index, obtaining the query results and pushing the query results to the user includes:

[0030] When a user enters query conditions and queries based on the multidimensional spatiotemporal index, the time index and geospatial index are used for preliminary filtering according to the query conditions to obtain preliminary filtering results;

[0031] Based on the preliminary screening results, the processing results are matched and queried to obtain the query results, which are then pushed to the user.

[0032] Secondly, this application also discloses a geospatial big data processing system for surveying and mapping, comprising:

[0033] The acquisition module is used to acquire the data stream to be processed and to identify and mark the data stream to be processed in order to distinguish between ordinary data and urgent data;

[0034] The monitoring module is used to continuously monitor the real-time working status of all professional processing units, including an image recognition unit and a text semantic analysis unit.

[0035] An adjustment module is used to dynamically adjust the resource allocation of the specialized processing unit based on the real-time working status.

[0036] The processing module is used to establish a high-priority network transmission channel when the emergency data occurs, and transmit the emergency data to the professional processing unit through the high-priority network transmission channel. The professional processing unit performs image and semantic processing on the emergency data to obtain the processing result, which includes the accident type, location coordinates, event label and semantic vector.

[0037] The technical solution according to the embodiments of this application has at least the following beneficial effects: The surveying and mapping geospatial big data processing method disclosed in this application achieves differentiated processing of different types of data by acquiring the data stream to be processed and identifying and labeling it to distinguish between ordinary data and emergency data. Simultaneously, it continuously monitors the real-time working status of all professional processing units and dynamically adjusts the resource allocation of these units based on the real-time working status, effectively solving the problem of uneven distribution of computing resources in the prior art. More importantly, when emergency data occurs, this application can establish a high-priority network transmission channel to quickly transmit the emergency data to the professional processing units for image and semantic processing, obtaining processing results including accident type, location coordinates, event labels, and semantic vectors. This method overcomes the shortcomings of existing technologies in processing massive and diverse data, especially when rapid response to emergencies is required, such as low indexing efficiency, uneven distribution of computing resources, and difficulty in effectively integrating unstructured information. Through high-priority processing of emergency data and dynamic resource adjustment, the system's response speed and processing efficiency to emergencies are significantly improved.

[0038] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0039] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0040] Figure 1A flowchart illustrating a geospatial big data processing method for surveying and mapping provided in one embodiment of this application;

[0041] Figure 2 This is a schematic diagram of a geospatial big data processing system provided in one embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical methods, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] It should be noted that the meaning of "multiple" (or "more than") in the description of the embodiments of this application refers to two or more, and "greater than," "less than," "exceeding," etc. are understood to exclude the number itself, while "above," "below," "within," etc. are understood to include the number itself. If "first," "second," etc. are used in the description, they are only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0044] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: the existence of a alone, the existence of b alone, the existence of c alone, the simultaneous existence of a and b, the simultaneous existence of a and c, the simultaneous existence of b and c, or the simultaneous existence of a, b, and c, where a, b, and c can be single or multiple.

[0045] In the description of this application, unless otherwise expressly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0046] Based on the above, this application proposes a surveying and mapping geospatial big data processing method and system, aiming to improve the system's response speed and processing efficiency to emergencies.

[0047] The geospatial big data processing method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms; the software can be an application that implements the geospatial big data processing method, but is not limited to the above forms.

[0048] This application can be applied to numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via communication networks. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices. It should be noted that in various specific embodiments of this invention, when processing is required based on data related to the characteristics of an object (e.g., user attributes or sets of attribute information), permission or consent from the corresponding object is obtained first, and the collection, use, and processing of this data comply with relevant laws and standards. Furthermore, when the embodiments of the present invention need to obtain the attribute information of an object, they will obtain the separate permission or separate consent of the corresponding object through pop-up windows or redirection to a confirmation page. After obtaining the separate permission or separate consent of the corresponding object, they will then obtain the relevant data of the object necessary for the embodiments of the present invention to operate normally.

[0049] See Figure 1 , Figure 1This is a flowchart illustrating a surveying and mapping geospatial big data processing method according to an embodiment of this application. The surveying and mapping geospatial big data processing method provided in this embodiment includes, but is not limited to, steps S110 to S140, which will be described in detail below.

[0050] Step S110: Obtain the data stream to be processed and identify and label the data stream to be processed to distinguish between ordinary data and urgent data;

[0051] Step S120: Continuously monitor the real-time working status of all professional processing units, including image recognition professional unit and text semantic analysis professional unit.

[0052] Step S130: Dynamically adjust the resource allocation of the professional processing unit according to the real-time working status;

[0053] Step S140: When emergency data appears, a high-priority network transmission channel is established to transmit the emergency data to the professional processing unit. The professional processing unit performs image and semantic processing on the emergency data to obtain the processing results, which include accident type, location coordinates, event label and semantic vector.

[0054] It's important to clarify that "data stream" refers to a continuously generated data sequence that requires real-time processing. Its sources can include various channels such as sensor networks, social media, and satellite remote sensing, and its data types encompass multiple formats including text, images, and video. "Specialized processing units" refer to computing modules or services specifically designed to perform particular data processing tasks. For example, an image recognition unit is responsible for analyzing image data, extracting visual features, and identifying targets; a text semantic analysis unit is responsible for performing natural language processing on text data, understanding the meaning of the text, and extracting key information. These specialized processing units can be virtual instances deployed on cloud computing platforms or specific nodes in a physical server cluster. Their operational status (such as CPU utilization, memory usage, and task queue length) is continuously monitored for dynamic resource scheduling.

[0055] In one embodiment, the system acquires and identifies and labels the data stream to distinguish between ordinary and emergency data. In practical applications, data stream acquisition can be achieved in various ways. For example, a data acquisition agent can be configured to capture data in real time from various data sources (such as social media APIs, sensor data interfaces, and geographic information system databases). This data needs to undergo preliminary identification and labeling before entering the processing system. One implementation is that the system can preset a set of rules to make preliminary judgments based on the data source, data type, or specific keywords in the data content. For example, data streams originating from emergency management departments can be directly labeled as emergency data; text data containing specific sensitive words (such as "fire," "accident," and "rescue") can be preliminarily identified as emergency data. Another implementation is that the system can use a machine learning model to classify the data stream and train the model to distinguish between ordinary and emergency data. For example, a text classification model can be used to assess the urgency of text data, or an image classification model can be used to detect abnormal events in image data.

[0056] In one embodiment, the real-time operational status of all specialized processing units is continuously monitored. Monitoring the real-time operational status of these specialized processing units is crucial to ensuring efficient system operation and timely response to emergencies. One implementation involves deploying a monitoring agent to periodically (e.g., every few seconds or minutes) collect performance metrics for each specialized processing unit, including CPU utilization, GPU utilization, memory usage, and a list of currently processed tasks. This data is then aggregated and analyzed on a central monitoring platform. Another implementation involves specialized processing units proactively reporting their operational status and load to the monitoring system; for example, they may proactively send alerts when their task queue length exceeds a certain threshold.

[0057] In one embodiment, resource allocation for specialized processing units is dynamically adjusted based on real-time operational status. Based on monitoring results of the real-time operational status of these units, the system needs to be able to flexibly adjust resource allocation to optimize overall processing efficiency. One implementation is that when the monitoring system detects that a specialized processing unit is overloaded (e.g., CPU utilization consistently exceeds 90% or the task queue is too long), the resource scheduling module can automatically allocate more computing resources to it, such as increasing the number of CPU cores, expanding memory capacity, or starting new instances to share the load. Conversely, if a specialized processing unit is idle for an extended period, the system can reclaim some resources for use by other busier units. Another implementation is that the system can pre-adjust resource allocation according to preset strategies at different time periods or when specific events occur. For example, during peak periods when a large number of emergency events are expected, more resources can be reserved in advance for image recognition and text semantic analysis specialized units.

[0058] In one embodiment, when emergency data occurs, a high-priority network transmission channel is established to transmit the emergency data to a specialized processing unit. The specialized processing unit performs image and semantic processing on the emergency data to obtain processing results, including accident type, location coordinates, event tags, and semantic vectors. Once the system identifies and marks emergency data, it needs to ensure that this data reaches the corresponding specialized processing unit as quickly as possible. One implementation method is to utilize a Quality of Service (QoS) mechanism to set higher priority for emergency data packets, ensuring their priority transmission even during network congestion. Simultaneously, a dedicated, higher-bandwidth network channel can be established for emergency data to avoid delays caused by sharing the same channel with regular data streams. Once the emergency data arrives at the specialized processing unit, the image recognition unit performs image processing, for example, using object detection and scene recognition algorithms to identify the accident type (e.g., fire, traffic accident) and location coordinates in the image. The text semantic analysis unit then performs semantic processing on the text data, for example, using named entity recognition and sentiment analysis to extract event tags (e.g., "traffic congestion," "personal injury") and semantic vectors for subsequent retrieval and analysis.

[0059] The core innovation of this application lies in its intelligent identification and priority processing mechanism for data streams, as well as dynamic resource scheduling based on real-time operating status. By acquiring and identifying the data streams to be processed, the system can effectively distinguish between ordinary and urgent data. This step is the foundation for subsequent efficient processing, enabling the system to adopt different processing strategies for different types of data. For example, when urgent data appears, the system immediately establishes a high-priority network transmission channel to ensure that this critical information can be quickly transmitted to specialized processing units. Furthermore, this application significantly optimizes the utilization efficiency of computing resources by continuously monitoring the real-time operating status of all specialized processing units and dynamically adjusting resource allocation based on these statuses. In existing technologies, the allocation of computing resources is often static or based on simple load balancing strategies, which are difficult to adapt to complex and ever-changing heterogeneous computing tasks. The dynamic resource adjustment mechanism of this application can flexibly increase or decrease resources according to the actual load of the image recognition and text semantic analysis specialized units, thereby avoiding the waste of resources where some nodes are overloaded while others are idle, and improving the overall system throughput and response speed. Through the aforementioned technical solution, this application can effectively perform image and semantic processing on emergency data, obtaining processing results including accident type, location coordinates, event labels, and semantic vectors. These results provide accurate and real-time support for subsequent geospatial analysis, emergency decision-making, and information dissemination. This application not only improves the processing efficiency of unstructured data but also constructs an intelligent processing framework capable of rapidly responding to emergencies, optimizing resource utilization, and providing multi-dimensional analysis results, thus achieving significant technological progress in the field of surveying and mapping geospatial big data processing.

[0060] Specifically, the process of identifying and labeling the data streams to be processed to distinguish between ordinary data and urgent data may include the following steps:

[0061] When the data stream to be processed is text data, matching analysis is performed based on a preset keyword matcher and a preset sentiment analyzer to obtain the number of urgent keywords and sentiment tendency scores.

[0062] When the number of urgent keywords exceeds the preset number or the sentiment score is lower than the preset score, the text data is marked as urgent data. The preset score is obtained based on negative sentiment keywords.

[0063] The preset keyword matcher can be understood as a pre-configured set of rules or a dictionary containing a series of keywords related to emergency events, such as "fire," "accident," "explosion," and "distress." When text data is received, the keyword matcher scans the text content to identify and count the number of emergency keywords. The preset sentiment analyzer is an algorithmic model used to assess the sentiment tendency of text. It analyzes the sentiment polarity, intensity, and context of words in the text to calculate the overall sentiment tendency score of the text. This sentiment tendency score is typically used to quantify the positive, neutral, or negative emotions expressed in the text. Furthermore, the preset quantity refers to the minimum threshold number of emergency keywords pre-set when judging emergency data. When the number of emergency keywords identified in the text data exceeds this preset quantity, it indicates that the text data may be highly relevant to an emergency event. The preset score is a threshold trained or set based on negative sentiment keywords to determine whether the sentiment tendency of the text data is negative enough to be considered an emergency. For example, when the sentiment tendency score is lower than the preset score, it indicates that the text data contains significant negative emotions, such as panic, despair, and anger, which are usually closely related to the occurrence of an emergency event.

[0064] This application's solution effectively identifies potential emergency information from massive data streams by performing keyword matching and sentiment analysis on text data. When text data is received, a pre-set keyword matcher is used for initial screening to quickly locate core words related to the emergency. Simultaneously, a pre-set sentiment analyzer performs in-depth analysis of the overall sentiment of the text to capture negative emotional expressions that strongly suggest an emergency, even if they don't directly contain explicit keywords. By comprehensively considering the number of emergency keywords and sentiment scores, this solution can more comprehensively and accurately determine the urgency of the text data, thus avoiding misjudgments or omissions that may result from relying on a single indicator. Through the above technical solution, this application can achieve rapid and accurate identification and labeling of emergency information in text data streams. This method combines direct keyword matching with in-depth sentiment analysis, significantly improving the sensitivity and accuracy of emergency data identification, especially when facing emergency events with indirect or emotional expressions, effectively capturing potential emergency signals. Therefore, it ensures that emergency text data is promptly identified and enters subsequent high-priority processing flows, providing a solid foundation for the response of the geospatial big data processing system to emergencies.

[0065] Specifically, in the above-mentioned geospatial big data processing method, the data stream to be processed is identified and labeled to distinguish between ordinary data and urgent data, which can further process the image data.

[0066] When the data stream to be processed is image data, visual feature extraction and scene classification are performed based on a preset shallow feature extractor to obtain detection confidence and classification results;

[0067] When the detection confidence level is greater than the preset confidence level and the classification result is consistent with the preset result, or when the classification result is consistent with the preset result, the image data is marked as emergency data. Here, the preset result represents the event result related to the emergency event.

[0068] Specifically, a shallow feature extractor can be understood as an algorithm or model used to extract low-level visual features from raw image data, such as edges, corners, textures, and color histograms. Its purpose is to provide basic, distinguishable visual information for subsequent scene classification. In practical applications, shallow feature extractors can employ classic feature extraction algorithms such as SIFT, SURF, and ORB, or utilize early layers of convolutional neural networks (CNNs) for feature extraction. Visual feature extraction refers to the process of automatically identifying and quantifying visual elements from an image using a shallow feature extractor. Scene classification refers to categorizing an image into a predefined scene category based on the extracted visual features, such as "fire scene," "traffic accident," or "natural disaster." Detection confidence is a numerical value that measures the probability or certainty of the existence of a specific emergency event in an image, while the classification result refers to the specific scene category to which the image is classified. Preset confidence is a pre-set threshold used to judge the reliability of the detection result. When the detection confidence exceeds this threshold, it indicates a high probability that an emergency event exists in the image. Preset results represent event outcomes related to emergency events. For example, specific event types such as "fire," "flood," and "collapse" can be preset as classification results for emergency events.

[0069] This application's solution extracts key visual cues from complex image information by using a pre-defined shallow feature extractor when the data stream to be processed is image data. Subsequently, these visual cues are matched with predefined emergency event scenarios through scene classification to obtain preliminary detection confidence and classification results. Furthermore, by comparing the detection confidence with a pre-set confidence level and determining whether the classification results are consistent with the preset results, potential emergency events in the image data can be effectively identified. This recognition mechanism based on visual features and scene classification enables the system to intelligently analyze unstructured image data, thereby achieving rapid detection and response to emergencies. Through the above technical solution, this application can effectively process data streams in image form and accurately identify emergency data. This allows the system to not only process emergency information in text form but also perform in-depth analysis of visual information, greatly expanding the detection range and accuracy of emergency events. Especially in the field of geospatial big data processing, it can promptly detect emergencies such as fires, floods, and traffic accidents from satellite images, drone aerial photographs, or surveillance video streams, providing timely and reliable input for subsequent high-priority processing and resource scheduling, thereby significantly improving the emergency response capability and efficiency of the entire big data processing system.

[0070] Specifically, the aforementioned continuous monitoring of the real-time operational status of all specialized processing units includes:

[0071] The system acquires CPU utilization, GPU utilization, memory usage, and a list of tasks being processed for all specialized processing units according to a preset period. The preset period refers to the time interval at which the system collects status data. This period can be flexibly configured according to actual application needs; for example, it can be set to collect data once per second, per minute, or per hour to balance real-time monitoring and system overhead. Specifically, CPU utilization refers to the percentage of a specialized processing unit's CPU that is occupied within a specific time period, reflecting the workload of its computing resources. GPU utilization refers to the percentage of a specialized processing unit's GPU that is occupied within a specific time period, which is particularly critical for image recognition units involving image processing tasks. Memory usage refers to the total amount of physical or virtual memory currently occupied by the specialized processing unit, revealing its consumption of storage resources. The list of tasks being processed contains detailed information on all tasks currently being executed by the specialized processing unit, such as task identifier, task type, priority, start time, and estimated completion time, which helps to understand the specific work content and load composition of the processing unit. All of the above indicators can be obtained through the API interface provided by the operating system, the monitoring tools of the virtualization platform, or the log system within the specialized processing unit.

[0072] This application's solution, by periodically acquiring CPU utilization, GPU utilization, memory usage, and a list of tasks being processed for each specialized processing unit according to a preset cycle, can comprehensively and accurately grasp the real-time operating status of each specialized processing unit. CPU and GPU utilization directly reflect the workload of computing resources, memory usage reveals resource consumption, and the list of tasks being processed provides a detailed view of the workload type and quantity. These data collectively form the basis for an accurate assessment of the current load and availability of the specialized processing units, thus providing a reliable basis for subsequent dynamic resource adjustments. Through this technical solution, more detailed and comprehensive real-time operating status data of the specialized processing units can be obtained. This detailed monitoring enables the system to more accurately assess the actual load and potential bottlenecks of each processing unit, thereby providing more precise decision support for subsequent dynamic resource adjustments. This helps optimize resource utilization efficiency, ensures stable system operation when processing ordinary data, and enables rapid and effective resource allocation when urgent data occurs, ensuring the timely completion of high-priority tasks.

[0073] In this regard, this application further proposes that the steps for dynamically adjusting the resource allocation of the professional processing unit based on the real-time working status include:

[0074] The current load status of the specialized processing unit is obtained based on the CPU utilization, GPU utilization, memory usage, and the list of tasks being processed.

[0075] Resource allocation is dynamically adjusted based on the current load status of the specialized processing units.

[0076] Specifically, CPU utilization refers to the percentage of CPU time occupied by a specialized processing unit within a specific time period, reflecting its computing power load. Graphics processor utilization refers to the percentage of GPU time occupied by a specialized processing unit within a specific time period; this metric is particularly crucial for image recognition and text semantic analysis units. Memory usage refers to the amount of memory currently occupied by a specialized processing unit, reflecting its data caching and processing capacity requirements. The list of tasks being processed refers to the collection of all tasks currently being executed by the specialized processing unit, which can be further refined into information such as the number, type, and priority of tasks. Obtaining the current load status of a specialized processing unit can be understood as a comprehensive evaluation and quantification of the above real-time working status indicators. For example, through weighted averaging, fuzzy logic inference, or machine learning models, data from multiple dimensions such as CPU utilization, GPU utilization, memory usage, and the list of tasks being processed can be integrated into a single metric that characterizes the overall busyness or resource pressure of the specialized processing unit. This load status can be a discrete level (e.g., "low load," "medium load," "high load," "overload") or a continuous value. Dynamically adjusting resource allocation based on the current load status of a specialized processing unit refers to intelligently adjusting the computing resources (such as the number of CPU cores, GPU computing power), storage resources (such as memory size), or network bandwidth allocated to that unit based on the obtained load status. For example, when the load status is "high load" or "overload," the system can increase the resources allocated to that specialized processing unit; when the load status is "low load," the resource allocation can be appropriately reduced to free up resources for other specialized processing units with greater demand.

[0077] This application's solution first comprehensively analyzes multi-dimensional real-time operational status data (including CPU utilization, GPU utilization, memory usage, and the list of tasks being processed) to obtain a unified and accurate reflection of the current load status of specialized processing units. It is precisely this in-depth integration and interpretation of the raw data that allows the system to more comprehensively and accurately understand the operational status of each specialized processing unit. Based on this, resource allocation adjustments are made according to the quantified or tiered load status, avoiding potential biases caused by judging based on a single or isolated indicator, and ensuring the scientific and rational nature of resource allocation decisions. Through the above technical solution, refined management of resource allocation for specialized processing units can be achieved. By comprehensively evaluating multiple real-time indicators, the system can more accurately identify the true load situation of specialized processing units, thereby making more timely and reasonable resource adjustment decisions. This not only helps avoid resource waste and improve resource utilization, but more importantly, it ensures that specialized processing units receive sufficient resource support when high-priority tasks such as urgent data processing occur, thus significantly improving the overall response speed, processing efficiency, and stability of the surveying and mapping geospatial big data processing system.

[0078] In one embodiment, the current load status of a specialized processing unit can be represented by a comprehensive score, which is calculated by weighting CPU utilization, GPU utilization, memory usage, and the number of tasks being processed. A scoring function can be defined as: Load Score = 0.3 * CPU Utilization + 0.4 * GPU Utilization + 0.2 * Memory Usage + 0.1 * Number of Tasks. When the load score is below a certain threshold (e.g., below 40), the specialized processing unit is determined to be "low load"; when the score is between 40 and 70, it is determined to be "medium load"; and when the score is above 70, it is determined to be "high load". When the system detects that the load score of an image recognition specialized unit is consistently above 70 (high load state), the system can immediately trigger a resource adjustment strategy, such as adding 2 CPU cores and 4GB of memory, or migrating the ordinary data tasks it is processing to other specialized processing units with lower loads to ensure that it can prioritize processing urgent data. Conversely, if the load score of a certain text semantic analysis unit is consistently below 40 (low load state), the system may consider reclaiming some of its idle resources or allocating them to other specialized processing units that require more resources, thereby optimizing the overall resource allocation.

[0079] In some embodiments of this application, the steps of performing image and semantic processing on emergency data to obtain processing results may specifically include:

[0080] Image processing is performed on emergency data to obtain the accident type and location coordinates;

[0081] It also performs semantic processing on emergency data to obtain event tags and semantic vectors.

[0082] Specifically, image processing of emergency data aims to extract event-related visual features from the image information contained within the data. For example, deep learning models can be used for object detection, scene classification, and event recognition in images to identify the type of accident present, such as fires, traffic accidents, or building collapses, and to accurately determine the location coordinates of the accident. These location coordinates can be inferred and determined through geotag information in the image, image content analysis, or by combining other geospatial data. Furthermore, semantic processing of emergency data aims to extract textual information from the data or identify key semantic information from the textual descriptions derived from image processing results. For example, natural language processing techniques can be used to perform entity recognition, sentiment analysis, and topic modeling on the textual content, generating event tags describing the core content of the event, such as "highway rear-end collision" or "factory fire," and converting these tags into semantic vectors for subsequent advanced analysis. Semantic vectors can capture the deeper meaning of words and phrases, facilitating similarity matching, cluster analysis, and trend prediction.

[0083] This application's solution clearly divides the emergency data processing into two independent sub-processes: image processing and semantic processing. This allows for the application of the most suitable analysis techniques for different types of data features. Image processing focuses on the interpretation of visual information to obtain intuitive accident types and precise location information; while semantic processing focuses on the understanding of text and concepts to generate abstract event labels and computable semantic vectors. This specialized division of labor and collaboration model enables the processing results to more comprehensively and accurately reflect all aspects of the emergency, avoiding information omissions or processing biases that may result from a single processing method. Through the above technical solution, the processing of emergency data is refined and specialized, ensuring the accurate extraction of accident types and location coordinates from images, while efficiently obtaining event labels and semantic vectors from text or semantic levels. This structured processing approach improves the accuracy and completeness of the processing results, providing high-quality input for subsequent geospatial big data analysis and decision-making, thereby more effectively supporting emergency response and resource allocation.

[0084] Specifically, in some implementations of the above-mentioned geospatial big data processing method, the step of performing semantic processing on emergency data to obtain event labels and semantic vectors can be further refined as follows: Performing semantic processing on emergency data to obtain event labels and semantic vectors includes:

[0085] For emergency data, a pre-trained word vector model is used to convert keywords in the emergency data into semantic vectors;

[0086] Extract event tags from emergency data using pattern matching based on preset rules.

[0087] Pre-trained word vector models refer to models trained on large-scale corpora that can map words to a continuous vector space, such as Word2Vec, GloVe, or BERT. These models can capture the semantic relationships between words, making semantically similar words closer together in the vector space. Converting keywords in emergency data into semantic vectors aims to transform unstructured text information into a computable and comparable numerical representation, providing a foundation for subsequent semantic analysis and event recognition. Specifically, emergency data can first be segmented to identify key entities, verbs, or adjectives as keywords, and then pre-trained word vector models can be used to obtain the vector representations corresponding to these keywords. Pre-defined rule pattern matching refers to a method of identifying and extracting specific information by defining a series of grammatical, syntactic, or semantic patterns. These rules can be constructed based on expert knowledge, domain dictionaries, or statistical analysis to identify phrases, sentence structures, or named entities in the text that are related to specific event types. For example, rules can be set to identify event types such as "fire," "traffic accident," and "natural disaster," and extract related information such as location, time, and personnel. By extracting event tags through pattern matching, the aim is to identify specific event categories or themes from the semantic vectors of emergency data, thereby enabling rapid classification and understanding of emergency events.

[0088] This application's solution converts keywords in emergency data into semantic vectors, achieving quantification and semantic representation of textual information, enabling computers to understand and process complex linguistic information. Based on this, through pattern matching with pre-defined rules, event tags can be accurately identified from this semantic data, effectively transforming unstructured emergency text data into structured information usable for subsequent analysis and decision-making. This processing method ensures rapid and accurate identification and classification of emergency events, providing high-quality input for subsequent geospatial big data processing.

[0089] In this regard, this application further proposes that after obtaining the processing result, it also includes:

[0090] The processing results are added to the data stream to be processed, the target information is obtained and stored in the existing multidimensional spatiotemporal index;

[0091] When a user enters query criteria and performs a query based on a multidimensional spatiotemporal index, the query results are obtained and pushed to the user.

[0092] Specifically, the processing results refer to the accident type, location coordinates, event labels, and semantic vectors obtained after image and semantic processing of emergency data by a specialized processing unit. These processing results are then correlated and integrated with relevant contextual information (such as timestamps and data sources) in the original data stream to obtain more complete and richer target information. The target information is designed to comprehensively characterize the key elements of the emergency event. Furthermore, the target information is stored in an existing multidimensional spatiotemporal index. A multidimensional spatiotemporal index is a data structure specifically designed for efficient management and querying of data with time, space, and other attributes. By storing the target information in such an index, efficient organization and rapid retrieval of emergency event data can be achieved; for example, rapid location can be performed based on time range, geographical region, event type, or specific keywords. In addition, to enable users to fully utilize the stored emergency event information, this application also includes obtaining query results and pushing them to the user when the user inputs query conditions and performs a query based on the multidimensional spatiotemporal index. Query conditions may include, but are not limited to, specific time periods, geographical regions, accident types, event labels, or semantic vectors. Leveraging the advantages of multidimensional spatiotemporal indexing, the system can quickly match and retrieve query results that meet user needs. These results are then pushed to the user through user interface, report generation, or real-time notifications, ensuring that the user can promptly obtain the information they require.

[0093] This application's solution integrates the processing results obtained in real-time with the original data stream into target information and stores it in a multi-dimensional spatiotemporal index, thereby solving the problems of persistent storage and efficient retrieval mechanisms for processing results in basic solutions. The introduction of the multi-dimensional spatiotemporal index enables the system to perform structured management of massive amounts of surveying and mapping geospatial big data and supports multi-dimensional, high-efficiency query operations. When a user enters query conditions, the multi-dimensional spatiotemporal index can quickly locate and extract relevant target information without linearly scanning all data, greatly improving the efficiency and response speed of data retrieval. Furthermore, by pushing the query results to the user, this solution ensures that the processed information can be delivered to decision-makers or relevant personnel in a timely and effective manner, thus combining real-time processing capabilities with long-term information management and utilization capabilities.

[0094] In this regard, this application further proposes the following: when a user inputs query conditions and performs a query based on a multidimensional spatiotemporal index, the query results are obtained and pushed to the user, including:

[0095] When a user enters query conditions and queries based on a multidimensional spatiotemporal index, the time index and geospatial index are used for preliminary filtering based on the query conditions to obtain preliminary filtering results.

[0096] Based on the initial screening results, the processing results are matched and queried to obtain the query results, which are then pushed to the user.

[0097] Specifically, a time index refers to quickly locating and extracting all event data occurring within a specific time range based on the time range information contained in the user's query conditions within a multi-dimensional spatiotemporal index. For example, a user might query events "within the past 24 hours" or "between a specific date and a specific date." A geospatial index refers to quickly locating and extracting all event data occurring within a specific geographic space based on the geographic location information (e.g., the radius around a region or point) contained in the user's query conditions within a multi-dimensional spatiotemporal index. For example, a user might query events "within a city" or "within 5 kilometers of a coordinate point." By combining time and geospatial indexes, a subset of data that initially matches the user's query conditions can be efficiently filtered from massive amounts of data, forming preliminary filtering results. These preliminary filtering results are a data set constrained by time and spatial dimensions, and their volume is much smaller than the original total processed results. Matching queries based on the preliminary filtering results refer to further precise matching within the initially filtered subset of data based on other non-spatiotemporal attributes (e.g., accident type, event tags, semantic vectors, etc.) that may be included in the user's query conditions. For example, after initially filtering all events within a certain area and time period, a user might further query for events related to "fire" or those containing specific "keywords." This step-by-step querying method can significantly improve the accuracy and efficiency of the search.

[0098] This application's solution effectively addresses the efficiency issues that may arise from directly querying large-scale multidimensional spatiotemporal indexes by decomposing the query process into two stages: preliminary screening and matching query. First, leveraging the efficient retrieval capabilities of the multidimensional spatiotemporal index in both time and geospatial dimensions, the user-input query conditions are initially analyzed, quickly locating a smaller subset of data within the massive dataset—the preliminary screening result. This step utilizes the advantages of spatiotemporal indexes, avoiding the scanning of irrelevant data. Second, after obtaining the preliminary screening result, a more refined matching query is performed on this smaller subset. Since the matching query is performed within a significantly reduced data range, computational complexity is significantly reduced, and query response speed is improved. This hierarchical filtering mechanism enables the system to handle complex query requests more efficiently and quickly provide users with accurate query results. Through the above technical solution, this application significantly improves the efficiency and accuracy of query operations in surveying and mapping geospatial big data processing systems. By introducing time and geospatial indexes for preliminary screening, the query scope can be significantly narrowed, reducing the amount of data for subsequent matching queries, thereby reducing system resource consumption and shortening query response time. Furthermore, based on the initial screening results, precise matching queries are performed to ensure that the final query results are highly relevant to user needs, avoiding unnecessary redundant information. Therefore, the solution in this application not only optimizes the performance of big data queries but also improves the convenience and accuracy for users to obtain the information they need.

[0099] See Figure 2 , Figure 2 This is a schematic diagram of a geospatial big data processing system for surveying and mapping provided in one embodiment of this application. The geospatial big data processing system 200 includes:

[0100] The acquisition module 210 is used to acquire the data stream to be processed and to identify and label the data stream to be processed in order to distinguish between ordinary data and urgent data;

[0101] The monitoring module 220 is used to continuously monitor the real-time working status of all professional processing units, including image recognition professional units and text semantic analysis professional units.

[0102] Adjustment module 230 is used to dynamically adjust the resource allocation of the professional processing unit according to the real-time working status;

[0103] The processing module 240 is used to establish a high-priority network transmission channel when emergency data occurs, and transmit the emergency data to the professional processing unit through the high-priority network transmission channel. The professional processing unit performs image and semantic processing on the emergency data to obtain the processing results, which include accident type, location coordinates, event label and semantic vector.

[0104] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0105] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0106] The foregoing has provided a detailed description of the preferred embodiments of this application. However, this application is not limited to the above-described embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined in this application.

Claims

1. A method for processing geospatial big data in surveying and mapping, characterized in that, include: Acquire the data stream to be processed and identify and label the data stream to distinguish between ordinary data and urgent data; Continuously monitor the real-time working status of all professional processing units, including image recognition professional units and text semantic analysis professional units; The resource allocation of the specialized processing unit is dynamically adjusted based on the real-time working status. When the emergency data appears, a high-priority network transmission channel is established to transmit the emergency data to the professional processing unit. The professional processing unit performs image and semantic processing on the emergency data to obtain the processing results, which include accident type, location coordinates, event tags, and semantic vectors.

2. The method according to claim 1, characterized in that, The process of identifying and labeling the data stream to be processed to distinguish between ordinary data and urgent data includes: When the data stream to be processed is text data, a matching analysis is performed based on a preset keyword matcher and a preset sentiment analyzer to obtain the number of urgent keywords and sentiment tendency scores. When the number of emergency keywords exceeds a preset number or the sentiment score is lower than a preset score, the text data is marked as emergency data, wherein the preset score is obtained based on negative sentiment keywords.

3. The method according to claim 1, characterized in that, The process of identifying and labeling the data stream to be processed to distinguish between ordinary data and urgent data includes: When the data stream to be processed is image data, visual feature extraction and scene classification are performed based on a preset shallow feature extractor to obtain detection confidence and classification results; When the detection confidence level is greater than the preset confidence level and the classification result is consistent with the preset result, or when the classification result is consistent with the preset result, the image data is marked as emergency data, wherein the preset result represents the event result related to the emergency event.

4. The method according to claim 1, characterized in that, The continuous monitoring of the real-time operating status of all specialized processing units includes: The CPU utilization, GPU utilization, memory usage, and list of tasks being processed for all professional processing units are obtained according to a preset cycle.

5. The method according to claim 4, characterized in that, The step of dynamically adjusting the resource allocation of the specialized processing unit based on the real-time working status includes: The current load status of the specialized processing unit is obtained based on the CPU utilization, GPU utilization, memory usage, and the list of tasks being processed. Resource allocation is dynamically adjusted based on the current load status of the specialized processing unit.

6. The method according to claim 5, characterized in that, The image and semantic processing of the emergency data to obtain the processing result includes: The emergency data is processed to obtain the accident type and location coordinates; The emergency data is semantically processed to obtain event tags and semantic vectors.

7. The method according to claim 6, characterized in that, The emergency data is semantically processed to obtain event tags and semantic vectors, including: The keywords in the emergency data are converted into semantic vectors based on a pre-trained word vector model. Event tags are extracted from the emergency data based on pattern matching according to preset rules.

8. The method according to claim 1, characterized in that, After obtaining the processing result, the process also includes: The processing result is added to the data stream to be processed to obtain the target information and store it in the existing multidimensional spatiotemporal index; When a user enters query conditions and performs a query based on the multidimensional spatiotemporal index, the query results are obtained and pushed to the user.

9. The method according to claim 8, characterized in that, When a user inputs query conditions and performs a query based on the multidimensional spatiotemporal index, obtaining the query results and pushing the query results to the user includes: When a user enters query conditions and queries based on the multidimensional spatiotemporal index, the time index and geospatial index are used for preliminary filtering according to the query conditions to obtain preliminary filtering results; Based on the preliminary screening results, the processing results are matched and queried to obtain the query results, which are then pushed to the user.

10. A geospatial big data processing system for surveying and mapping, characterized in that, include: The acquisition module is used to acquire the data stream to be processed and to identify and mark the data stream to be processed in order to distinguish between ordinary data and urgent data; The monitoring module is used to continuously monitor the real-time working status of all professional processing units, including an image recognition unit and a text semantic analysis unit. An adjustment module is used to dynamically adjust the resource allocation of the specialized processing unit based on the real-time working status. The processing module is used to establish a high-priority network transmission channel when the emergency data occurs, and transmit the emergency data to the professional processing unit through the high-priority network transmission channel. The professional processing unit performs image and semantic processing on the emergency data to obtain the processing result, which includes the accident type, location coordinates, event label and semantic vector.

Images