Image processing method and device for spatial governance scene, medium and product

Abstract

The application discloses a kind of image processing methods, devices, media and products for spatial governance scene.It relates to image processing or artificial intelligence field, the method comprises: obtaining video image, wherein;Video image is input into target multimodal big model, and the scene information of video image is output;Scene information and video image are input into dispatch hub, and the target event involved in video image is processed by dispatching multiple processing tools by dispatch hub, wherein, multiple processing tools include at least one of the following: visual scene tool, remote sensing positioning tool and governance evaluation tool, visual scene tool is used for target detection to video image, remote sensing positioning tool is used for positioning target event, and governance evaluation tool is used for evaluating the governance effect of target event after target event is governed.By the present application, the problem of low processing efficiency of spatial governance in related art under complex spatial governance scene is solved.

Description

Technical Field

[0001] This application relates to the fields of image processing or artificial intelligence, and more specifically, to an image processing method, apparatus, medium, and product for spatial governance scenarios. Background Technology

[0002] Currently, urban spatial governance (e.g., environmental pollution control, and illegal activity control) presents complex scenarios. For example, environmental pollution is highly concealed, making it difficult to accurately pinpoint the source of early warning targets. The instantaneous nature of illegal activities leads to difficulties in investigation and evidence collection, resulting in low governance efficiency. Furthermore, urban spatial governance relies heavily on significant human and material resources, sometimes resulting in high governance outcomes.

[0003] Among related technologies, there are methods for governing urban space using multi-source heterogeneous data composed of satellite imagery, real-time sensor readings, relevant regulations and texts, and geographic coordinate data. However, due to the complexity of urban space governance scenarios, this method is relatively inefficient in governing urban space.

[0004] There is currently no effective solution to the problem of low efficiency in handling space governance issues in complex scenarios. Summary of the Invention

[0005] The main objective of this application is to provide an image processing method, apparatus, medium, and product for spatial governance scenarios, in order to solve the problem of low processing efficiency of spatial governance problems in complex spatial governance scenarios in related technologies.

[0006] To achieve the above objectives, according to one aspect of this application, an image processing method for spatial governance scenarios is provided. The method includes: acquiring video images, wherein the video images include images acquired by an image acquisition device mounted on a metal frame; inputting the video images into a target multimodal large model and outputting scene information of the video images; inputting the scene information and the video images into a scheduling center, and scheduling multiple processing tools through the scheduling center to process target events involved in the video images, wherein the multiple processing tools include at least one of the following: a visual sceneization tool, a remote sensing positioning tool, and a governance evaluation tool; the visual sceneization tool is used to perform target detection on the video images; the remote sensing positioning tool is used to locate the target events; and the governance evaluation tool is used to evaluate the governance effect of the target events after they have been governed.

[0007] Furthermore, a large language model is deployed in the scheduling center. Multiple processing tools are scheduled through the scheduling center to process target events involved in the video image, including: determining target prompts based on the scene information, wherein the target prompts are used to guide the large language model to plan tasks according to the scene information; inputting the target prompts into the large language model to obtain multiple sub-tasks, and semantically encoding each sub-task to obtain a semantic vector for each sub-task; matching the semantic vector of each sub-task with a vector tool mapping relationship to obtain a matching result, wherein the vector tool mapping relationship includes: a mapping relationship between the semantic vector associated with the metadata of each processing tool and the metadata of that processing tool; selecting multiple processing tools based on the matching result, and using the multiple processing tools to process the target events involved in the video image.

[0008] Furthermore, processing the target events involved in the video image using multiple processing tools includes: generating input data conforming to the input format of each processing tool based on the parameter information of each processing tool; inputting the associated input data of each processing tool into the processing tool, receiving the processing result returned by the processing tool, generating a processing report based on the processing result returned by each processing tool, and transmitting the processing report to the target storage space; in the process of processing the target events involved in the video image using multiple processing tools, the method further includes: performing anomaly detection on the processing process of each processing tool, and handling the anomaly through a preset processing strategy if an anomaly is detected, wherein the anomaly detection includes at least one of the following: parameter verification, verification of the processing result, runtime error detection, and call timeout detection.

[0009] Furthermore, the large language model is trained in the following manner: using a first language model to simulate the task initiator generating task requirements, and using a second language model to simulate the calling process and processing results of the processing tool, to obtain synthetic data; verifying the synthetic data to obtain a verification result; if the verification result indicates that the synthetic data has passed verification, training the initial model based on the synthetic data using a supervised fine-tuning strategy to obtain the large language model, wherein the initial model includes: the large language model before supervised fine-tuning; and adjusting the large language model using a reinforcement learning strategy.

[0010] Furthermore, before processing the target events involved in the video image using multiple processing tools, the method further includes: allocating computing resources to each processing tool based on the scene information and each sub-task using a preset resource allocation strategy to obtain a resource allocation result, wherein the preset resource allocation strategy includes at least one of the following: allocating computing resources based on the urgency of the sub-task, allocating computing resources to periodic sub-tasks using a staggered resource allocation strategy, and calling redundant computing resources to balance the computing pressure.

[0011] Further, the target events include at least one of the following: smoke and fire, garbage accumulation, and illegal construction; the visual scene-based tools include at least one of the following: a smoke detection tool, a garbage detection tool, and a construction detection tool; the video image type includes: visible light image and infrared image; when the visual scene-based tools are scheduled by the scheduling center to process the target events involved in the video image, the method includes at least one of the following: based on the visible light image and the infrared image, identifying smoke and fire features in the video image using the smoke detection tool; based on the visible light image and the infrared image, identifying garbage accumulation in the video image using the garbage detection tool, wherein the garbage accumulation includes one of the following: the garbage accumulation area exceeds a preset area threshold; based on the visible light image and the infrared image, identifying illegal construction in the video image using the construction detection tool.

[0012] Furthermore, the target multimodal large model includes: a multimodal large model with fewer than a preset threshold number of parameters, and supporting input of the visible light image and the infrared image. The target multimodal large model includes: a data layer, a training layer, an inference layer, and an application layer. The data layer is used to receive the video image and preprocess the video image. The preprocessing method includes at least: data cleaning and data fusion. The training layer is used for model training and model parameter updating. The inference layer is used for inference and task processing. The application layer is used to determine the scene information.

[0013] To achieve the above objectives, according to another aspect of this application, an image processing apparatus for spatial governance scenarios is provided. The apparatus includes: an acquisition unit for acquiring video images, wherein the video images include images acquired by an image acquisition device mounted on a steel frame; a first processing unit for inputting the video images into a target multimodal large model and outputting scene information of the video images; and a second processing unit for inputting the scene information and the video images into a scheduling center, through which the scheduling center schedules multiple processing tools to process target events involved in the video images, wherein the multiple processing tools include at least one of the following: a visual sceneization tool, a remote sensing positioning tool, and a governance evaluation tool, wherein the visual sceneization tool is used to detect targets in the video images, the remote sensing positioning tool is used to locate the target events, and the governance evaluation tool is used to evaluate the governance effect of the target events after they have been governed.

[0014] Furthermore, a large language model is deployed in the scheduling center, and the second processing unit includes: a determination subunit, used to determine target prompt words based on the scene information, wherein the target prompt words are used to guide the large language model to perform task planning according to the scene information; a first processing subunit, used to input the target prompt words into the large language model to obtain multiple subtasks, and to perform semantic encoding on each subtask to obtain a semantic vector for each subtask; a matching subunit, used to match the semantic vector of each subtask with the vector tool mapping relationship to obtain a matching result, wherein the vector tool mapping relationship includes: the mapping relationship between the semantic vector associated with the metadata of each processing tool and the metadata of the processing tool; and a second processing subunit, used to select multiple processing tools based on the matching result, and to use the multiple processing tools to process the target events involved in the video image.

[0015] Further, the second processing subunit includes: a first processing module, configured to generate input data conforming to the input format of each processing tool based on the parameter information of each processing tool; input the associated input data of each processing tool into the processing tool; receive the processing result returned by the processing tool; generate a processing report based on the processing result returned by each processing tool; and transmit the processing report to a target storage space; the second processing subunit further includes: a second processing module, configured to perform anomaly detection on the processing process of each processing tool; and, if an anomaly is detected, handle the anomaly using a preset processing strategy, wherein the anomaly detection includes at least one of the following: parameter verification, verification of the processing result, runtime error detection, and call timeout detection.

[0016] Furthermore, the large language model is trained through the following sub-units: a third processing sub-unit, used to simulate the task initiator generating task requirements using a first language model, and to simulate the calling process and processing results of the processing tool using a second language model to obtain synthetic data; a verification sub-unit, used to verify the synthetic data to obtain a verification result; a training sub-unit, used to train the initial model based on the synthetic data using a supervised fine-tuning strategy when the verification result indicates that the synthetic data has passed verification, to obtain the large language model, wherein the initial model includes: the large language model before supervised fine-tuning; and a reinforcement sub-unit, used to adjust the large language model using a reinforcement learning strategy.

[0017] Furthermore, the second processing unit further includes: an allocation subunit, configured to allocate computing resources to each processing tool based on the scene information and each subtask using a preset resource allocation strategy before processing the target events involved in the video image using the multiple processing tools, and obtain a resource allocation result, wherein the preset resource allocation strategy includes at least one of the following: allocating computing resources based on the urgency of the subtask, allocating computing resources for periodic subtasks using a staggered resource allocation strategy, and calling redundant computing resources to balance the computing pressure.

[0018] Furthermore, the target events include at least one of the following: smoke and fire, garbage accumulation, and illegal construction. The visual scene-based tools include at least one of the following: a smoke detection tool, a garbage detection tool, and a construction detection tool. The video image type includes: visible light image and infrared image. When the visual scene-based tools are scheduled by the scheduling center to process the target events involved in the video image, at least one of the following is included: based on the visible light image and the infrared image, the smoke detection tool identifies the smoke and fire features in the video image; based on the visible light image and the infrared image, the garbage detection tool identifies the garbage accumulation in the video image, wherein the garbage accumulation includes one of the following: the garbage accumulation area exceeds a preset area threshold; based on the visible light image and the infrared image, the construction detection tool identifies illegal construction in the video image.

[0019] Furthermore, the target multimodal large model includes: a multimodal large model with fewer than a preset threshold number of parameters, and supporting input of the visible light image and the infrared image. The target multimodal large model includes: a data layer, a training layer, an inference layer, and an application layer. The data layer is used to receive the video image and preprocess the video image. The preprocessing method includes at least: data cleaning and data fusion. The training layer is used for model training and model parameter updating. The inference layer is used for inference and task processing. The application layer is used to determine the scene information.

[0020] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the image processing method for a space governance scenario.

[0021] According to another aspect of this application, an electronic device is provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the image processing method for a space governance scenario during runtime.

[0022] According to another aspect of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the image processing method for spatial governance scenarios.

[0023] In this application, the following method is adopted: acquiring video images, wherein the video images include images acquired by an image acquisition device mounted on a steel structure; inputting the video images into a target multimodal large model, outputting scene information of the video images; inputting the scene information and video images into a scheduling center, and scheduling multiple processing tools to process target events involved in the video images through the scheduling center, wherein the multiple processing tools include at least one of the following: a visual sceneization tool, a remote sensing positioning tool, and a governance evaluation tool. The visual sceneization tool is used to detect targets in the video images, the remote sensing positioning tool is used to locate target events, and the governance evaluation tool is used to evaluate the governance effect of target events after they have been governed. This solves the technical problem of low processing efficiency of spatial governance problems in complex spatial governance scenarios in related technologies. In this application, scene recognition is performed through a target multimodal large model, and multiple processing tools are scheduled through a scheduling center to govern target events, avoiding the situation of high target positioning difficulty and low spatial governance efficiency in complex spatial governance scenarios in related technologies, thereby achieving the technical effect of improving spatial governance efficiency. Attached Figure Description

[0024] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 A hardware structure block diagram of a computer terminal for implementing an image processing method for space governance scenarios is shown.

[0026] Figure 2 This is a flowchart of an image processing method for space governance scenarios provided according to an embodiment of this application;

[0027] Figure 3 This is a flowchart of image processing for space governance scenarios provided according to embodiments of this application;

[0028] Figure 4 This is a structural diagram of image processing for spatial governance scenarios provided according to embodiments of this application;

[0029] Figure 5 This is a structural diagram of the target multimodal large model provided according to the embodiments of this application;

[0030] Figure 6 This is a schematic diagram of the development of a lightweight multimodal large model according to an embodiment of this application;

[0031] Figure 7 This is a schematic diagram of an image processing device for space governance scenarios provided according to an embodiment of this application;

[0032] Figure 8 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws and standards, have taken necessary security measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse. For example, this system has interfaces with relevant users or organizations, providing users with corresponding operation entry points for users to choose to agree to or refuse automated decision results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0036] Example 1

[0037] According to an embodiment of this application, a method embodiment for image processing in a space governance scenario is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0038] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing an image processing method for spatial governance scenarios is shown. Figure 1 As shown, computer terminal 10 (or mobile device) may include one or more ( Figure 1The processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions may also be included. In addition, it may include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0039] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image processing method for space governance scenarios in this embodiment of the application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned image processing method for space governance scenarios. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0041] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0042] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0043] Under the aforementioned operating environment, this application provides the following: Figure 2 The image processing method shown is for spatial governance scenarios. Figure 2 This is a flowchart of an image processing method for a space governance scenario according to Embodiment 1 of this application.

[0044] Step S201: Acquire video images, wherein the video images include images acquired by an image acquisition device mounted on the iron frame.

[0045] The aforementioned video images include images captured by an image acquisition device mounted on a metal frame. This image acquisition device can be a camera. In one optional example, the image acquisition device can be a dual-channel camera that integrates infrared thermography and visible light image recognition technologies. The visible light channel of the dual-channel camera can clearly capture detailed information such as the color and shape of objects, making it suitable for identifying types of waste, construction equipment, etc. The infrared channel can reflect the thermal characteristics of objects through the temperature field distribution, accurately detecting high-temperature points and areas with abnormal temperatures.

[0046] Step S202: Input the video image into the target multimodal large model and output the scene information of the video image.

[0047] The aforementioned target multimodal large model can be a lightweight multimodal large model, which can support mid-to-high point visible light images and infrared images as input; it can achieve target detection with few or zero samples through Chinese commands, and not only can the detection categories be dynamically expanded, but it also has open set recognition capability; at the same time, the model has low memory usage and strong compatibility, and can be flexibly deployed on a variety of common graphics cards, effectively reducing the difficulty of application deployment and operating costs.

[0048] In one alternative example, the scenario information mentioned above may include, but is not limited to: emergency scenarios involving emergency events such as fireworks, environmental protection scenarios related to environmental issues such as garbage, and construction scenarios related to construction.

[0049] In another alternative example, the scene information mentioned above may include scenes such as city roads, parks, and fields.

[0050] In this embodiment, video images can be input into a target multimodal large model, and the target multimodal large model can perform scene recognition on the video images to obtain scene information of the video images. Figure 3 This is a flowchart of image processing for space governance scenarios provided according to embodiments of this application, such as... Figure 3 As shown, video images captured by video surveillance can be input into a pre-trained lightweight multimodal large model for scene understanding. If environmental pollution-related elements such as garbage or smoke are present, the scene information of the video image will be output.

[0051] Step S203: Input scene information and video images into the scheduling center. The scheduling center schedules multiple processing tools to process the target events involved in the video images. The multiple processing tools include at least one of the following: a visual sceneization tool, a remote sensing positioning tool, and a governance evaluation tool. The visual sceneization tool is used to detect targets in the video images, the remote sensing positioning tool is used to locate the target events, and the governance evaluation tool is used to evaluate the governance effect of the target events after they have been governed.

[0052] The aforementioned processing tools may include, but are not limited to: visual scene-based tools, remote sensing positioning tools, and governance assessment tools. Visual scene-based tools may include, but are not limited to: smoke detection tools, garbage detection tools, and construction detection tools. The processing tools may also include: alarm task processing tools.

[0053] The aforementioned remote sensing spatial positioning tool can overcome the difficulty of accurately locating target sources using video images. It employs fully automated spatial positioning technology based on high-point cameras to provide efficient and accurate spatial location information for smart city environmental protection applications, laying a solid spatial data foundation for subsequent event analysis, data statistics, and intelligent decision-making. After receiving alarm information from the visual scene-based tool, the dispatch center can then dispatch the remote sensing spatial positioning module to obtain the 3D geographic coordinates of the alarm event. The remote sensing spatial positioning tool can efficiently and accurately acquire the intrinsic and extrinsic parameters of the high-point camera through a rigorous offline and online process. Based on this, orthophotos (DOM) and digital elevation models (DEM) can be introduced as core geographic reference data, aligning the camera attitude from the local coordinate system to the global geographic coordinate system through a georegistration process. Furthermore, the remote sensing spatial positioning tool can integrate depth estimation technology into the core positioning process. Based on the obtained accurate camera parameters, the depth estimation model will be used to provide precise depth information for each pixel in the image. By combining pixel-level depth data with the camera's precise attitude, geometric calculations can directly convert each pixel into its corresponding 3D geographic coordinates.

[0054] like Figure 3 As shown, in one optional example, after obtaining the scene information from the video image, the scene information and the corresponding video image can be transmitted to an intelligent scheduling and decision-making center based on LLM (Large Language Model) (corresponding to the scheduling center). This center then routes and schedules the corresponding visual scene-based tools to finely detect the type of the target and its bounding box in the video image. After obtaining the bounding box, the scheduling center can call a remote sensing spatial positioning tool to obtain the 3D geographic coordinates of the alarm event (corresponding to the target event). The alarm information corresponding to the alarm event (e.g., 3D geographic coordinates, scene information, video image, etc.) can be integrated by the scheduling center and uploaded to a cloud database for storage. Simultaneously, the scheduling center can identify alarm tasks requiring immediate processing and send them to the relevant environmental enforcement agencies via SMS or email. After completing the corresponding environmental governance operations, the implementers of the environmental enforcement agencies can feed back the processing results to the scheduling center. The scheduling center can then re-schedule the video images of the area and detect relevant pollutants, evaluating the results of the environmental governance and forming an intelligent closed loop from monitoring to governance to evaluation.

[0055] Through the above steps, this embodiment utilizes a target multimodal large model for scene recognition and a scheduling center to dispatch multiple processing tools to manage target events. This avoids the difficulties in target localization and low efficiency in complex spatial governance scenarios found in related technologies, thereby improving the technical effect of spatial governance efficiency. Furthermore, it solves the technical problem of low processing efficiency in complex spatial governance scenarios in related technologies.

[0056] Figure 4 This is a structural diagram of image processing for spatial governance scenarios provided in the embodiments of this application, such as... Figure 4 As shown, the overall technical approach includes the following components: a lightweight multimodal large model, a visual scene-based model, remote sensing spatial positioning, and an NLP (Natural Language Processing) agent. The lightweight multimodal large model corresponds to the multimodal large model scene understanding in the scheme design diagram; the visual scene-based model and remote sensing spatial positioning correspond to the modules with the same names in the diagram; and the NLP agent includes various linking capabilities centered around an LLM-based intelligent scheduling and decision-making hub.

[0057] Optionally, in the image processing method for spatial governance scenarios provided in this application embodiment, a large language model is deployed in the scheduling center. Multiple processing tools are scheduled through the scheduling center to process target events involved in the video image. This includes: determining target prompts based on scene information, wherein the target prompts are used to guide the large language model to plan tasks according to the scene information; inputting the target prompts into the large language model to obtain multiple sub-tasks, and semantically encoding each sub-task to obtain a semantic vector for each sub-task; matching the semantic vector of each sub-task with the vector tool mapping relationship to obtain a matching result, wherein the vector tool mapping relationship includes: the mapping relationship between the semantic vector associated with the metadata of each processing tool and the metadata of that processing tool; selecting multiple processing tools based on the matching result, and using the multiple processing tools to process the target events involved in the video image.

[0058] In this embodiment, a large language model (LLM) is deployed in the scheduling center. The large language model (LLM) has a strong text semantic understanding capability. As an intelligent scheduling center, it processes various text language tasks, and can intelligently schedule various scenario-based models and make decisions on various alarm tasks. This allows the image processing method for regional spatial governance provided in this embodiment to efficiently and automatically improve the efficiency of environmental protection problem monitoring and governance.

[0059] In this embodiment, the scheduling center not only performs question-and-answer dialogue functions but also possesses scheduling and decision-making capabilities, undertaking the overall coordination function of the entire process from environmental monitoring data analysis to the issuance of governance tasks and the evaluation of governance effects. The scheduling center can use "tool learning" as its core support, and through collaborative operations in task planning, tool selection, tool invocation, and response generation, it can encapsulate and connect scattered tools and modules such as mid-to-high-level visual recognition small models, multimodal scene understanding, remote sensing positioning, database query interfaces, and email / SMS sending interfaces using MCP (Model Context Protocol), forming a closed-loop control of "monitoring-analysis-decision-execution-evaluation".

[0060] In this embodiment, after inputting scene information and corresponding video images into the scheduling center, task planning can be performed first. For example, vague environmental protection requirements (such as "identifying straw burning in a certain area") can be broken down into executable sub-tasks (such as "video image scene recognition → fine detection of burning points → remote sensing coordinate positioning"). This three-tiered task planning process of "scene recognition - dedicated guidance - precise planning" avoids the problem of "environmental protection scene adaptation deviation" that often arises from the use of generic prompts in related technologies. For example, the task of "nighttime smoke detection" might be mistakenly broken down into "ordinary smoke recognition algorithm," or key steps such as "remote sensing positioning and matching with law enforcement areas" might be omitted.

[0061] In one optional example, a corresponding prompt word template can be determined based on scenario information to guide the LLM to accurately decompose multiple sub-tasks. In this embodiment, environmental protection scenarios can be divided into three core scenarios: "sudden pollution alarm," "routine inspection analysis," and "treatment effect evaluation." For each of these three core scenarios, a dedicated prompt word template can be designed. A structured approach—"scenario features - sub-task decomposition rules - tool association guidance"—guides the large language model to follow the dedicated prompt word instructions and perform targeted task planning for each scenario. Alternatively, the prompt word retrieval process can be transformed into a tool learning process. Specifically, one scenario corresponds to one tool method, and all tools for each scenario are aggregated into an MCP service. The large language model uses a tool learning framework and is constrained to perform tool retrieval within a scenario-specific toolset, ultimately returning the prompt word template corresponding to the target scenario, enabling the configuration of specific task planning prompt words for specific scenarios.

[0062] After task planning is performed at the scheduling center and multiple sub-tasks are obtained, tool selection can be performed. The optimal tool corresponding to the sub-task can be matched from various MCP services (including mid-to-high-point visual recognition small models, multimodal scene understanding, remote sensing positioning, database query interfaces, email and SMS sending interfaces, and scene-based prompt word templates, etc.). In this embodiment, a three-layer tool selection process of "pre-retrieval - lightweight fine-tuning - dynamic adaptation" can be used to avoid the problems of "low efficiency of full tool traversal", "high cost of large model calling" and "poor adaptation to scene changes" faced by tool selection in related technologies.

[0063] For pre-retrieval, a pre-semantic retrieval method can be used to filter out the processing corresponding to multiple sub-tasks. For example, in response to the problem of "the tool pool is too large, resulting in long selection time", a semantic vector-driven tool selection mechanism can be constructed: (1) Tool vector library construction: Semantically encode the metadata (functional description, applicable scenarios, input and output parameters, call time) of all tools, generate high-dimensional vectors and store them, and establish a "vector-tool metadata" mapping relationship (corresponding to the vector tool mapping relationship). This mapping relationship can be stored in the vector database in the form of a table; (2) Task-tool semantic matching: Semantically encode the sub-tasks (such as "nighttime fireworks detection") after decomposition, and select the TopN (e.g., Top5, i.e. the top 5 with the highest similarity) high matching tools (such as "general fireworks detection algorithm" and "nighttime infrared fireworks detection algorithm") from the vector library through cosine similarity calculation; (3) Candidate tool ranking: Combine the relevance of the tools, historical call success rate, response time, etc. to perform secondary ranking and output the optimal tool recommendation list.

[0064] For lightweight fine-tuning, this embodiment balances performance and cost by fine-tuning a small-parameter model. In this embodiment, a model with fewer parameters (i.e., a small model) can be used to select tools from a tool recommendation list. For example, while large-parameter models have higher tool selection accuracy, their invocation cost is 8-10 times that of small models, and their response latency is higher, making it difficult to meet the "low latency, large scale" requirements of environmental protection scenarios. Therefore, this embodiment proposes a "large model generating data + small model dedicated fine-tuning" scheme. After fine-tuning, the accuracy of the small model in environmental tool selection tasks can approach the level of the large-parameter model, significantly reducing invocation cost and response latency.

[0065] For dynamic adaptation, since the tool call in the environmental protection scenario is easily affected by variables such as "image blur", "interface timeout" and "temporary adjustment of task requirements", the "one-time call" mode in the related technology has poor fault tolerance. In this embodiment, the tool call strategy can be adjusted through the dynamic adaptation mechanism and real-time information feedback. For example, result verification adaptation: if the processing tool returns an incomplete result (such as remote sensing positioning only returning latitude and longitude, missing altitude), it can automatically trigger "supplementary parameter call" and pass the instruction "must include altitude information" to the interface; (2) abnormal handling adaptation: if a certain visual scene tool fails to detect twice in a row (such as due to blurred image on rainy days), it can automatically switch to "rainy day optimized detection model" and adjust the camera parameters simultaneously (such as increasing exposure).

[0066] Optionally, in the image processing method for spatial governance scenarios provided in this application embodiment, multiple processing tools are used to process target events involved in video images, including: generating input data conforming to the input format of each processing tool based on the parameter information of each processing tool; inputting the associated input data of each processing tool into the processing tool, receiving the processing results returned by the processing tool, generating a processing report based on the processing results returned by each processing tool, and transmitting the processing report to the target storage space; in the process of using multiple processing tools to process target events involved in video images, the method further includes: performing anomaly detection on the processing process of each processing tool, and processing the anomaly through a preset processing strategy when an anomaly is detected, wherein the anomaly detection includes at least one of the following: parameter verification, verification of processing results, runtime error detection, and call timeout detection.

[0067] In this embodiment, the tool execution process can be automatically triggered according to the selected tool, efficiently completing parameter transmission and exception feedback processing, ensuring the stable operation and closed-loop processing of the task chain. Specifically, a three-stage tool invocation mechanism of "rule triggering - parameter verification - exception management" can be used to avoid problems such as parameter type mismatch and lack of exception handling that often occur in tool invocation due to model illusion, which can lead to task interruption or unreliable results.

[0068] An optional tool execution flow may include:

[0069] 1. Intelligent Parameter Mapping and Validation: It can automatically extract parameter information from the output of large language models through predefined processing tool parameter specifications and type constraints, and map it into input data that conforms to the input format required by the processing tools. Simultaneously, it incorporates strong type validation and logical verification mechanisms to effectively identify and filter erroneous or fictitious parameters generated by model illusion, ensuring call reliability and data security.

[0070] 2. Multi-layered anomaly monitoring and handling: Multi-layered anomaly monitoring can be implemented during the execution of the processing tool, covering parameter verification, call timeout, runtime errors, and result integrity checks. Once an anomaly is detected, it can automatically respond according to preset handling strategies, which may include, but are not limited to: retrying, switching to backup tools, triggering manual review, or executing task rollback, thereby ensuring the robustness of space governance in complex environments.

[0071] In one optional example, the system can also integrate the output of processing tools, follow user instructions, and refer to uploaded documents to generate structured alarm information, enforcement task instructions, or governance effectiveness evaluation reports, which are then simultaneously pushed to the cloud database and the implementing unit. Furthermore, the dispatch center can periodically execute evaluation and analysis programs, allowing users to observe global alarm events at different times and locations, and generate visualized governance effectiveness reports for user analysis.

[0072] Optionally, in the image processing method for space governance scenarios provided in this application embodiment, the large language model is trained in the following manner: using a first language model to simulate the task initiator generating task requirements, and using a second language model to simulate the calling process and processing results of the generated processing tool, to obtain synthetic data; verifying the synthetic data to obtain a verification result; if the verification result indicates that the synthetic data has passed verification, training the initial model based on the synthetic data using a supervised fine-tuning strategy to obtain the large language model, wherein the initial model includes: the large language model before supervised fine-tuning; and adjusting the large language model using a reinforcement learning strategy.

[0073] Improving LLM invocation capabilities relies on high-quality data. However, environmental scenarios suffer from issues such as "limited availability of truly labeled data (high cost of manual labeling)" and "insufficient scenario diversity (e.g., few cases of extreme weather pollution)." To address this, this embodiment employs a dual-loop architecture of "synthesis-driven cold start - feedback-enhanced continuous optimization." Specifically:

[0074] (I) Cold Start Phase: Synthesizing High-Quality Training Data from Large Models

[0075] By employing large-scale model synthesis technology, we can quickly construct training corpora covering multiple scenarios for tool calls, thus solving the problem of "scarcity of initial data." The core steps include:

[0076] (1) Synthesis logic design: Based on the existing processing tool function definition, the large model with a high number of parameters (corresponding to the first language model) is called to collaboratively generate data. The first language model can simulate the diverse output requirements of the "task initiator" (such as "straw burning detection" and "nighttime construction site dust identification"). The second language model simulates the "LLM calling processing tool" output tool call process and results, and finally generates a large batch of synthetic data.

[0077] (2) Data quality screening: Synthetic data is verified by rule verification (such as whether the processing tool calling logic conforms to the environmental protection process) and manual sampling (10% data verification) to be used for supervised fine-tuning (SFT) of LLM, so that the model can quickly master the basic logic of tool calling in the environmental protection scenario.

[0078] (II) Application Phase: Optimization of Reinforcement Learning Driven by Real Trajectories

[0079] In this embodiment, the model strategy can be continuously optimized based on user interaction data (feedback from governance personnel, tool call logs) using the GRPO reinforcement learning method to optimize the large language model. The core steps include:

[0080] (1) Real data collection: Under the premise of compliance, collect three main categories of data:

[0081] (2) Tool call log: tool response time, accuracy, and exception type (such as interface timeout, detection failure);

[0082] (3) Execution feedback data: the satisfaction of governance personnel with alarm information (such as "accurate location" and "complete information"), and the completion rate of governance tasks;

[0083] (4) Effectiveness evaluation data: changes in pollutants before and after treatment, frequency of secondary alarms;

[0084] (5) Reward function design: Design a multi-dimensional reward mechanism around the core objectives of environmental protection scenarios.

[0085] Optionally, in the image processing method for spatial governance scenarios provided in this application embodiment, before processing the target events involved in the video image using multiple processing tools, the method further includes: allocating computing resources to each processing tool based on scene information and each sub-task using a preset resource allocation strategy to obtain resource allocation results. The preset resource allocation strategy includes at least one of the following: allocating computing resources based on the urgency of the sub-task, allocating computing resources to periodic sub-tasks using a staggered resource allocation strategy, and calling redundant computing resources to balance the computing pressure.

[0086] In this embodiment, intelligent management and scheduling through algorithms optimize the allocation of intelligent analysis tasks and resources. For example, computing power and storage resources can be automatically allocated for different scenarios (such as fireworks, garbage, and construction) to ensure the efficiency and accuracy of the platform's alarm judgment output, enabling faster regulatory response. This embodiment can incorporate an intelligent scheduling engine that dynamically adjusts resource allocation based on the recognition difficulty and urgency of different scenarios. For example, for high-urgency tasks like fireworks detection, high-performance computing nodes can be prioritized to ensure that video images and scene information analysis and alarm generation are completed within a preset time (e.g., 0.5 seconds). For periodic tasks like garbage detection, a staggered computing mode can be used to complete large-area inspection and analysis during idle computing periods, avoiding resource waste. Simultaneously, the computing pressure of each node can be automatically balanced based on real-time data such as device load and network bandwidth. When a concentrated alarm occurs in a certain area, redundant computing power from surrounding nodes can be quickly dispatched to provide support, ensuring the platform remains stable during peak periods and reducing the average time from alarm generation to regulatory response to less than 5 minutes.

[0087] Optionally, in the image processing method for space governance scenarios provided in this application embodiment, the target events include at least one of the following: smoke and fire, garbage accumulation, and illegal construction; the visual scene-based tools include at least one of the following: smoke detection tool, garbage detection tool, and construction detection tool; the video image types include: visible light image and infrared image; when the visual scene-based tools are scheduled by the scheduling center to process the target events involved in the video image, the method includes at least one of the following: based on the visible light image and infrared image, identifying the smoke and fire features in the video image using the smoke detection tool; based on the visible light image and infrared image, identifying the garbage accumulation in the video image using the garbage detection tool, wherein the garbage accumulation includes one of the following: the garbage accumulation area exceeds a preset area threshold; based on the visible light image and infrared image, identifying the illegal construction in the video image using the construction detection tool.

[0088] In this embodiment, the image acquisition device can be a dual-channel camera, integrating infrared thermometry and visible light image recognition technologies. This leverages the advantages of different technologies to overcome the limitations of traditional single-technology applications, providing more comprehensive data support for monitoring diverse scenarios. The visible light channel can clearly capture details such as the color and shape of objects, suitable for identifying types of waste and construction equipment. The infrared channel reflects the thermal characteristics of objects through temperature field distribution, accurately detecting high-temperature points and areas of abnormal temperature. For example, in construction monitoring, visible light images can identify construction equipment and work areas to determine if fencing has not been properly set up, while infrared images can monitor temperature anomalies in the construction area. When high-temperature points exceeding 300°C are detected, risks such as unauthorized open flames or welding operations without fire prevention measures can be quickly identified. In monitoring straw burning, even if dense smoke obscures the visible light image, the infrared channel can still pinpoint the fire location through the high-temperature characteristics of the burning point, ensuring comprehensive monitoring without blind spots.

[0089] The visual scene-based tool provided in this embodiment relies on deep learning networks and comprehensively utilizes technologies such as image enhancement, multi-scale fusion, and scene-based analysis to create a series of core algorithms. Combined with abundant high-point tower resources, it uses binocular cameras and a "front-end + back-end" high-point AI algorithm to accurately capture various environmental anomalies in a patrol mode, achieving efficient alarms. Among them, high-point tower resources, with their advantages of wide coverage and unobstructed view, break through the spatial limitations of ground monitoring in related technologies, enabling 24-hour uninterrupted monitoring of key areas such as forests, farmland, urban open spaces, and riverbanks. Binocular cameras can improve the ability to capture image depth information through stereo vision technology. Combined with the collaborative mode of front-end edge computing and back-end cloud analysis, it not only ensures the speed of real-time early warning but also meets the needs of accurate identification in complex scenarios, transforming environmental anomalies from "passive discovery" to "active identification."

[0090] In this embodiment, a multi-level monitoring platform can be built, such as a provincial, municipal, and county-level joint intelligent and intensive monitoring platform. Visual scene-based tools can be deployed on the multi-level monitoring platform. The multi-level monitoring platform can use distributed technology to build a three-dimensional command and dispatch system that spans levels, domains, and departments, completely changing the fragmented and delayed response situation in relevant environmental protection supervision. The multi-level monitoring platform can achieve distributed deployment of computing power and data through a distributed architecture, supporting the concurrent processing of tens of thousands of monitoring videos. Even environmental anomalies in remote areas can be synchronized to the provincial and municipal platforms within 3 seconds, realizing "one screen to view the whole area, one network to manage the whole process". At the same time, the multi-level monitoring platform takes openness as its core, seamlessly connecting with multiple management department systems related to spatial governance through standardized interfaces, integrating video resources and alarm functions, and promoting the co-construction and sharing of multiple scenarios such as forest and grassland fire prevention, straw burning ban, garbage management, and illegal construction supervision. For example, departments related to forestry supervision can directly access tower monitoring resources through the platform to conduct forest fire prevention patrols, while departments related to environmental governance can simultaneously obtain early warning information on garbage accumulation, significantly reducing the cost of redundant construction of monitoring systems by various departments and improving the efficiency of environmental supervision collaboration by more than 60%.

[0091] In this embodiment, the environmental scenario-based tools may include: smoke detection tools, garbage detection tools, and construction detection tools. The smoke detection tools may include smoke and fire recognition algorithms, the garbage detection tools may include garbage detection algorithms, and the construction detection tools may include construction detection algorithms, forming an intelligent monitoring network covering multiple scenarios. The smoke and fire recognition algorithm can utilize image enhancement technology to solve image blurring problems under complex weather conditions such as fog, rain, and backlighting. Through a multi-scale fusion algorithm, it accurately extracts the characteristics of smoke and fire of different sizes and combustion stages, shortening the single-frame image recognition time to less than 0.2 seconds, with a detection accuracy of 98%. Compared to manual inspections in related technologies, it can detect potential fire hazards 10-15 minutes earlier. The garbage detection algorithm can be trained using deep learning. A large number of samples of domestic waste, industrial waste, and construction waste can accurately identify the type, accumulation area, and scale of openly dumped waste. When the area of ​​waste accumulation exceeds a set threshold, an early warning report containing the location, area, and estimated removal volume can be automatically generated, providing timely warnings of environmental pollution and preventing soil and water pollution caused by long-term waste accumulation. The construction detection algorithm can intelligently monitor construction behaviors such as unauthorized construction and illegal operations by building a feature library of construction scenarios. It can not only identify whether the construction area exceeds the approved scope, but also determine whether effective dust prevention and control measures have been taken by analyzing the operating status of construction equipment and the coverage of bare soil. This allows environmental supervision to shift from a "human wave tactic" to "intelligent identification," reducing labor costs while increasing the detection rate of illegal construction to over 95%.

[0092] A visual scene visualization tool may include algorithms as shown in Table 1.

[0093] Table 1

[0094]

[0095]

[0096] Optionally, in the image processing method for space governance scenarios provided in the embodiments of this application, the target multimodal large model includes: a multimodal large model with fewer than a preset threshold number of parameters and supporting input of visible light images and infrared images. The target multimodal large model includes: a data layer, a training layer, an inference layer, and an application layer. The data layer is used to receive video images and preprocess the video images. The preprocessing methods include at least: data cleaning and data fusion. The training layer is used for model training and model parameter updating. The inference layer is used for inference and task processing. The application layer is used to determine scene information.

[0097] In this embodiment, the number of model parameters of the target multimodal large model can be less than a preset number threshold, which can be 1 billion (1B). It can support the input of visible light images and infrared images at mid-to-high point positions. It can achieve target detection with few or zero samples through Chinese commands. Not only can the detection categories be dynamically expanded, but it also has open set recognition capability. At the same time, the target multimodal large model has low memory usage and strong compatibility. It can be flexibly deployed on a variety of common graphics cards, effectively reducing the application deployment difficulty and operating costs.

[0098] In addition, the target multimodal large model has four core advantages, including: (1) outstanding semantic understanding ability: the model can achieve deep alignment between images and language text, and performs well in core scenarios such as open set recognition and small sample detection, and can more accurately adapt to complex business needs; (2) efficient and flexible demand response: it can rely on text command interaction mode to quickly respond to new customer needs and demand changes, without the need for a lot of secondary development, which greatly improves the efficiency of business adaptation; (3) significant computing power utilization efficiency: a single AI analysis can complete dozens of target recognition tasks. Through result reuse and post-processing optimization, computing power consumption can be greatly reduced and hardware resource investment costs can be reduced; (4) controllable maintenance and operation costs: only 1-2 of the multimodal large models are needed to meet the core business needs of customers. The configuration work in the project delivery stage is simplified, the later maintenance process is clear, and the full life cycle operation cost is effectively reduced.

[0099] This multimodal large model adopts a "four-layer distributed architecture" design to achieve deep integration of technology and business. The architecture layers are shown in Table 2:

[0100] Table 2

[0101]

[0102] An optional structure for a target multimodal large model is as follows: Figure 5 As shown, it includes: a text encoder, a visual encoder, and a cross-modal feature fusion module. The cross-modal feature fusion module may include: a text-guided query selection module and a cross-modal decoder. The text encoder can extract text features, and the visual encoder can extract image features. The text features and image features can be fused through the cross-modal feature fusion model to finally obtain the detection result.

[0103] This four-layer distributed architecture supports horizontal scaling, allowing for flexible addition of data sources and task modules based on industry needs. Furthermore, its loosely coupled inter-layer design ensures that upgrading any layer does not affect the overall system operation. A schematic diagram of an optional lightweight multimodal large model development approach is shown below. Figure 6 As shown, it includes: large-scale dataset construction, design and training of a lightweight multimodal large model, and multi-industry adaptation.

[0104] In an optional example, in order to implement the image processing method for spatial governance scenarios provided in this embodiment, a standardized hardware and software deployment and service system can be established in this embodiment. The overall architecture of this system uses a three-level collaborative mode of "central cloud - regional edge cloud - site edge node" to achieve on-demand sinking and optimal configuration of computing resources, including: (1) Central cloud: deployed in a large data center, with powerful computing and storage capabilities. It is mainly responsible for training large models, global data analysis, model refinement and distribution, as well as unified management and operation of the entire network. (2) Regional edge cloud: deployed in provincial and municipal branch offices or large site computer rooms, serving as a data aggregation and localized processing center. Its role is to connect the upper and lower levels, and it can aggregate and analyze data from multiple site edge nodes, and also distribute models and instructions issued by the central cloud to each site node. (3) Site edge node: deployed on communication tower sites in various regions, serving as the front line. It directly processes real-time data from front-end devices, performs local real-time inference and decision-making, and only transmits high-value structured data back to the upper network. The three-tier architecture effectively solves the problems of bandwidth, latency, and cost. By offloading tasks with high real-time requirements to edge nodes and keeping tasks requiring global knowledge in the central cloud, on-demand allocation of computing resources is achieved, avoiding the "one-size-fits-all" centralized processing in the cloud and thus achieving optimal resource allocation.

[0105] For hardware deployment, to build an independent, controllable, secure, and reliable technological foundation, and to fully leverage the advantages of the existing technological ecosystem, this embodiment proposes a hybrid computing power adaptation strategy: high-performance GPUs are used during the large-scale model training phase in the cloud, while AI acceleration cards are used comprehensively during the inference deployment phase. Specifically, this includes:

[0106] (1) Cloud training: High-performance GPUs:

[0107] Training large models in the cloud can utilize high-performance GPUs, providing superior throughput and memory bandwidth for large-scale training. This stage can fully leverage its mature software ecosystem and toolchain.

[0108] In this environment, deep learning frameworks are used for model training.

[0109] Model inference: Employing a target computing power card (e.g., a high-performance AI processor):

[0110] After cloud-based training, the model is deployed to edge computing cards through model conversion and compression. Target computing cards are primarily used; although defined as training processors, their powerful performance makes them suitable for high-performance cloud-side inference deployments as well. They support computationally intensive tasks such as convolutional models and can be used to build multi-machine inference clusters to meet diverse customer needs. Target computing cards are specifically designed for AI inference, with typical power consumption as low as 25W, making them ideal for deployment on edge nodes.

[0111] This hybrid adaptation strategy leverages NVIDIA's leading advantage in the training ecosystem while ensuring technological autonomy and controllability at the widely deployed edge.

[0112] The foundational software layer in software deployment, including operating system and database support, is detailed below:

[0113] Cloud-edge operating systems provide edge-side inference capabilities. By decoupling applications from models and models from AI chips, they support heterogeneous computing power such as CPUs and GPUs. The operating system may include intelligent desktop assistant functionality and supports online access to large models and custom model configurations. These operating systems provide a stable and reliable operating environment for deploying upper-layer AI models.

[0114] Data storage and analysis: The database used supports various machine learning algorithms, such as linear regression and decision trees, making it a powerful tool for users to build business analysis models. The adoption of this database provides a self-reliant and controllable tool for data governance and analysis.

[0115] In this embodiment, users can select the location of the tower to be monitored on an electronic map, check the AI ​​algorithm functions to be activated (such as "smoke and fire recognition" and "garbage accumulation alarm"), and complete the service order online. The platform provides standardized API interfaces, which can be easily integrated with customers' existing emergency command, grid management, and other business systems, seamlessly integrating AI perception capabilities into their workflows. This "turnkey" service model greatly reduces the threshold and cost for various departments to use advanced AI technologies.

[0116] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0117] Example 2

[0118] This application also provides an image processing apparatus for spatial governance scenarios. It should be noted that this image processing apparatus for spatial governance scenarios can be used to execute the image processing method for spatial governance scenarios provided in this application. The image processing apparatus for spatial governance scenarios provided in this application will be described below.

[0119] According to an embodiment of this application, an apparatus for implementing the above-described image processing method for spatial governance scenarios is also provided, such as... Figure 7 As shown, the device includes: an acquisition unit 71, a first processing unit 72, and a second processing unit 73.

[0120] Acquisition unit 71 is used to acquire video images, wherein the video images include: images acquired by an image acquisition device mounted on the iron frame;

[0121] The first processing unit 72 is used to input video images into the target multimodal large model and output scene information of the video images;

[0122] The second processing unit 73 is used to input scene information and video images into the scheduling center, and to schedule multiple processing tools through the scheduling center to process target events involved in the video images. The multiple processing tools include at least one of the following: a visual sceneization tool, a remote sensing positioning tool, and a governance evaluation tool. The visual sceneization tool is used to detect targets in the video images, the remote sensing positioning tool is used to locate target events, and the governance evaluation tool is used to evaluate the governance effect of target events after they have been governed.

[0123] In the image processing apparatus for space governance scenarios provided in this application embodiment, video images can be acquired by acquisition unit 71. These video images include images acquired by an image acquisition device mounted on a steel frame. A first processing unit 72 inputs the video images into a target multimodal large model and outputs scene information from the video images. A second processing unit 73 inputs the scene information and video images into a scheduling center. The scheduling center then schedules multiple processing tools to process target events involved in the video images. These multiple processing tools include at least one of the following: a visual sceneization tool, a remote sensing positioning tool, and a governance evaluation tool. The visual sceneization tool is used for target detection in the video images, the remote sensing positioning tool is used for locating target events, and the governance evaluation tool is used to evaluate the governance effect of the target events after they have been governed. This solves the technical problem of low processing efficiency for space governance issues in complex space governance scenarios in related technologies. In this embodiment, scene recognition is performed using a target multimodal large model, and multiple processing tools are scheduled through a scheduling center to manage target events. This avoids the problems of high target positioning difficulty and low spatial governance efficiency in complex spatial governance scenarios in related technologies, thereby achieving the technical effect of improving spatial governance efficiency.

[0124] Optionally, in the image processing device for spatial governance scenarios provided in this application embodiment, a large language model is deployed in the scheduling center, and the second processing unit includes: a determination subunit, used to determine target prompt words based on scene information, wherein the target prompt words are used to guide the large language model to perform task planning according to the scene information; a first processing subunit, used to input the target prompt words into the large language model to obtain multiple sub-tasks, and to perform semantic encoding on each sub-task to obtain a semantic vector for each sub-task; a matching subunit, used to match the semantic vector of each sub-task with the vector tool mapping relationship to obtain a matching result, wherein the vector tool mapping relationship includes: the mapping relationship between the semantic vector associated with the metadata of each processing tool and the metadata of the processing tool; and a second processing subunit, used to select multiple processing tools based on the matching result, and to use the multiple processing tools to process the target events involved in the video image.

[0125] Optionally, in the image processing apparatus for spatial governance scenarios provided in this application embodiment, the second processing subunit includes: a first processing module, configured to generate input data conforming to the input format of each processing tool based on the parameter information of each processing tool; input the associated input data of each processing tool into the processing tool; receive the processing results returned by the processing tool; generate a processing report based on the processing results returned by each processing tool; and transmit the processing report to a target storage space; the second processing subunit further includes: a second processing module, configured to perform anomaly detection on the processing process of each processing tool; and, if an anomaly is detected, process the anomaly using a preset processing strategy, wherein the anomaly detection includes at least one of the following: parameter verification, verification of processing results, runtime error detection, and call timeout detection.

[0126] Optionally, in the image processing apparatus for space governance scenarios provided in this application embodiment, the large language model is trained through the following sub-units: a third processing sub-unit, used to simulate the task initiator generating task requirements using a first language model and to simulate the calling process and processing results of the generation processing tool using a second language model to obtain synthetic data; a verification sub-unit, used to verify the synthetic data and obtain verification results; a training sub-unit, used to train the initial model based on the synthetic data and using a supervised fine-tuning strategy when the verification result indicates that the synthetic data has passed verification, to obtain the large language model, wherein the initial model includes: the large language model before supervised fine-tuning; and a reinforcement sub-unit, used to adjust the large language model using a reinforcement learning strategy.

[0127] Optionally, in the image processing apparatus for spatial governance scenarios provided in this application embodiment, the second processing unit further includes: an allocation subunit, used to allocate computing resources to each processing tool based on scene information and each sub-task using a preset resource allocation strategy before processing the target events involved in the video image using multiple processing tools, and obtain a resource allocation result, wherein the preset resource allocation strategy includes at least one of the following: allocating computing resources based on the urgency of the sub-task, allocating computing resources for periodic sub-tasks using a staggered resource allocation strategy, and calling redundant computing resources to balance the computing pressure.

[0128] Optionally, in the image processing apparatus for space governance scenarios provided in this application embodiment, the target events include at least one of the following: smoke and fire, garbage accumulation, and illegal construction. The visual scene-based tools include at least one of the following: a smoke detection tool, a garbage detection tool, and a construction detection tool. The video image types include: visible light images and infrared images. When the visual scene-based tools are scheduled by the scheduling center to process the target events involved in the video images, at least one of the following is included: identifying smoke and fire features in the video images using the smoke detection tool based on visible light images and infrared images; identifying garbage accumulation in the video images using the garbage detection tool based on visible light images and infrared images, wherein the garbage accumulation includes one of the following: the garbage accumulation area exceeds a preset area threshold; and identifying illegal construction in the video images using the construction detection tool based on visible light images and infrared images.

[0129] Optionally, in the image processing device for space governance scenarios provided in the embodiments of this application, the target multimodal large model includes: a multimodal large model with fewer than a preset threshold number of parameters and supporting input of visible light images and infrared images. The target multimodal large model includes: a data layer, a training layer, an inference layer, and an application layer. The data layer is used to receive video images and preprocess the video images. The preprocessing methods include at least: data cleaning and data fusion. The training layer is used for model training and model parameter updating. The inference layer is used for inference and task processing. The application layer is used to determine scene information.

[0130] It should be noted that the acquisition unit 71, the first processing unit 72, and the second processing unit 73 mentioned above correspond to steps S201 to S203 in Embodiment 1. Each unit and the corresponding step implement the same instance and application scenario, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0131] Example 3

[0132] Embodiments of this application may provide an electronic device. Figure 8 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 8 As shown, the electronic device may include: one or more ( Figure 8 (Only one is shown) processor 802, memory 804, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0133] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0134] The processor can access information and applications stored in memory via a transmission device to perform the following steps: acquiring video images, wherein the video images include images acquired by an image acquisition device mounted on a metal frame; inputting the video images into a target multimodal large model and outputting scene information of the video images; inputting the scene information and video images into a scheduling center, and scheduling multiple processing tools to process target events involved in the video images, wherein the multiple processing tools include at least one of the following: a visual sceneization tool, a remote sensing positioning tool, and a governance evaluation tool, wherein the visual sceneization tool is used to detect targets in the video images, the remote sensing positioning tool is used to locate target events, and the governance evaluation tool is used to evaluate the governance effect of target events after they have been governed.

[0135] The processor can also access information and applications stored in the memory via a transmission device to execute the following steps: A large language model is deployed in the scheduling center, which schedules multiple processing tools to process target events involved in the video image, including: determining target prompt words based on scene information, wherein the target prompt words are used to guide the large language model to plan tasks according to the scene information; inputting the target prompt words into the large language model to obtain multiple sub-tasks, and semantically encoding each sub-task to obtain a semantic vector for each sub-task; matching the semantic vector of each sub-task with the vector tool mapping relationship to obtain a matching result, wherein the vector tool mapping relationship includes: the mapping relationship between the semantic vector associated with the metadata of each processing tool and the metadata of that processing tool; selecting multiple processing tools based on the matching result, and using multiple processing tools to process the target events involved in the video image.

[0136] The processor can also invoke information and applications stored in the memory via a transmission device to perform the following steps: processing target events involved in the video image using multiple processing tools, including: generating input data conforming to the input format of each processing tool based on the parameter information of each processing tool; inputting the associated input data of each processing tool into the processing tool, receiving the processing results returned by the processing tool, generating a processing report based on the processing results returned by each processing tool, and transmitting the processing report to the target storage space; in the process of processing target events involved in the video image using multiple processing tools, the processor also includes: performing anomaly detection on the processing process of each processing tool, and handling the anomaly through a preset processing strategy if an anomaly is detected, wherein the anomaly detection includes at least one of the following: parameter verification, verification of processing results, runtime error detection, and call timeout detection.

[0137] The processor can also access information and applications stored in memory via a transmission device to perform the following steps: The large language model is trained in the following ways, including: using a first language model to simulate the task initiator generating task requirements, and using a second language model to simulate the calling process and processing results of the processing tool to obtain synthetic data; verifying the synthetic data to obtain verification results; if the verification results indicate that the synthetic data has passed verification, training the initial model based on the synthetic data using a supervised fine-tuning strategy to obtain the large language model, wherein the initial model includes: the large language model before supervised fine-tuning; and adjusting the large language model using a reinforcement learning strategy.

[0138] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: before processing the target events involved in the video image using multiple processing tools, it further includes: allocating computing resources to each processing tool based on scene information and each sub-task using a preset resource allocation strategy to obtain the resource allocation result, wherein the preset resource allocation strategy includes at least one of the following: allocating computing resources based on the urgency of the sub-task, allocating computing resources to periodic sub-tasks using a staggered resource allocation strategy, and calling redundant computing resources to balance the computing pressure.

[0139] The processor can also invoke information and applications stored in the memory via a transmission device to perform the following steps: the target event includes at least one of the following: smoke and fire, garbage accumulation, and illegal construction; the visual scene-based tools include at least one of the following: a smoke detection tool, a garbage detection tool, and a construction detection tool; the video image type includes: visible light image and infrared image; when the visual scene-based tools are scheduled by the scheduling center to process the target event involved in the video image, the method includes at least one of the following: based on the visible light image and the infrared image, identifying the smoke and fire features in the video image using the smoke detection tool; based on the visible light image and the infrared image, identifying the garbage accumulation in the video image using the garbage detection tool, wherein the garbage accumulation includes one of the following: the garbage accumulation area exceeds a preset area threshold; based on the visible light image and the infrared image, identifying the illegal construction in the video image using the construction detection tool.

[0140] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: The target multimodal large model includes: a multimodal large model with fewer than a preset threshold of parameters and supporting input visible light images and infrared images. The target multimodal large model includes: a data layer, a training layer, an inference layer and an application layer. The data layer is used to receive video images and preprocess the video images. The preprocessing methods include at least: data cleaning and data fusion. The training layer is used for model training and model parameter updating. The inference layer is used for inference and task processing. The application layer is used to determine scene information.

[0141] By employing the embodiments of this application, scene recognition is performed through a target multimodal large model, and multiple processing tools are scheduled through a scheduling center to manage target events. This avoids the situation in related technologies where target localization is difficult and spatial management efficiency is low in complex spatial management scenarios, thereby achieving the technical effect of improving spatial management efficiency.

[0142] Those skilled in the art will understand that Figure 8 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 8 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 8 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 8 The different configurations shown.

[0143] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0144] Example 4

[0145] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image processing method for spatial governance scenarios provided in Embodiment 1.

[0146] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0147] This application also provides a computer program product that, when executed on a data processing device, is suitable for performing image processing method steps for a space governance scenario.

[0148] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0149] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0151] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0152] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0153] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0154] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image processing method for spatial governance scenarios, characterized in that, include: Acquire video images, wherein the video images include: images acquired by an image acquisition device mounted on a metal frame; The video image is input into the target multimodal large model, and the scene information of the video image is output. The scene information and the video image are input into the scheduling center, which then schedules multiple processing tools to process the target events involved in the video image. The multiple processing tools include at least one of the following: a visual scene detection tool, a remote sensing positioning tool, and a governance evaluation tool. The visual scene detection tool is used to detect targets in the video image, the remote sensing positioning tool is used to locate the target events, and the governance evaluation tool is used to evaluate the governance effect of the target events after they have been governed.

2. The image processing method according to claim 1, characterized in that, The scheduling center is equipped with a large language model, and multiple processing tools are scheduled through the scheduling center to process target events involved in the video images, including: Based on the scene information, target prompt words are determined, wherein the target prompt words are used to guide the large language model to perform task planning according to the scene information; The target prompt word is input into the large language model to obtain multiple sub-tasks, and each sub-task is semantically encoded to obtain a semantic vector for each sub-task; The semantic vector and vector tool mapping relationship of each subtask are matched to obtain the matching result, wherein the vector tool mapping relationship includes: the mapping relationship between the semantic vector associated with the metadata of each processing tool and the metadata of the processing tool; Based on the matching results, multiple processing tools are selected, and the target events involved in the video image are processed using the multiple processing tools.

3. The image processing method according to claim 2, characterized in that, The target events involved in the video image are processed using multiple processing tools, including: generating input data that conforms to the input format of each processing tool based on the parameter information of each processing tool; inputting the associated input data of each processing tool into the processing tool; receiving the processing results returned by the processing tool; generating a processing report based on the processing results returned by each processing tool; and transmitting the processing report to the target storage space. In the process of processing the target events involved in the video image using multiple processing tools, the method further includes: performing anomaly detection on the processing process of each processing tool, and processing the anomaly by means of a preset processing strategy when an anomaly is detected. The anomaly detection includes at least one of the following: parameter verification, verification of the processing result, runtime error detection, and call timeout detection.

4. The image processing method according to claim 2, characterized in that, The large language model is trained in the following ways: The task initiator generates task requirements using a first language model, and the call flow and processing results of the processing tool are generated using a second language model to obtain synthetic data. The synthesized data is verified to obtain the verification result; If the verification result indicates that the synthetic data has passed the verification, the initial model is trained based on the synthetic data using a supervised fine-tuning strategy to obtain the large language model, wherein the initial model includes: the large language model before supervised fine-tuning; The large language model is adjusted using a reinforcement learning strategy.

5. The image processing method according to claim 2, characterized in that, Before processing the target events involved in the video image using the multiple processing tools described above, the method further includes: Based on the scenario information and each subtask, a preset resource allocation strategy is used to allocate computing resources to each processing tool to obtain a resource allocation result. The preset resource allocation strategy includes at least one of the following: allocating computing resources based on the urgency of the subtask, allocating computing resources for periodic subtasks using a staggered resource allocation strategy, and calling redundant computing resources to balance the computing pressure.

6. The image processing method according to claim 1, characterized in that, The target events include at least one of the following: smoke and fire, garbage accumulation, and illegal construction; the visual scene-based tools include at least one of the following: smoke detection tool, garbage detection tool, and construction detection tool; the video image type includes: visible light image and infrared image; when the visual scene-based tools are scheduled by the scheduling center to process the target events involved in the video images, the method includes at least one of the following: Based on the visible light image and the infrared image, the smoke detection tool identifies the smoke features in the video image; Based on the visible light image and infrared image, the garbage detection tool identifies the garbage accumulation situation in the video image, wherein the garbage accumulation situation includes one of the following: the garbage accumulation area exceeds a preset area threshold; Based on the visible light and infrared images, the construction detection tool identifies illegal construction activities in the video images.

7. The image processing method according to claim 6, characterized in that, The target multimodal large model includes: a multimodal large model with fewer than a preset threshold number of parameters, and supporting input of the visible light image and the infrared image. The target multimodal large model includes: a data layer, a training layer, an inference layer, and an application layer. The data layer is used to receive the video image and preprocess the video image. The preprocessing method includes at least: data cleaning and data fusion. The training layer is used for model training and model parameter updating. The inference layer is used for inference and task processing. The application layer is used to determine the scene information.

8. An image processing device for spatial governance scenarios, characterized in that, include: An acquisition unit is used to acquire video images, wherein the video images include images acquired by an image acquisition device mounted on a metal frame; The first processing unit is used to input the video image into the target multimodal large model and output the scene information of the video image; The second processing unit is used to input the scene information and the video image into the scheduling center, and to schedule multiple processing tools through the scheduling center to process the target events involved in the video image. The multiple processing tools include at least one of the following: a visual scene detection tool, a remote sensing positioning tool, and a governance evaluation tool. The visual scene detection tool is used to detect targets in the video image, the remote sensing positioning tool is used to locate the target events, and the governance evaluation tool is used to evaluate the governance effect of the target events after they have been governed.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the image processing method for spatial governance scenarios as described in any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the image processing method for spatial governance scenarios as described in any one of claims 1 to 7.

Images