A multi-camera network optimization selection method based on a large model

Abstract

The application discloses a multi-camera network optimization selection method based on a large model, comprising the following steps: extracting multi-dimensional representation information of the camera network; generating a selection scheme of the camera network through step-by-step reasoning of a multi-modal large model and guidance to a greedy search; evaluating the generated selection scheme through multi-dimensional evaluation indexes, and outputting the selection scheme meeting the index requirements; constructing a comprehensive health degree score to detect the abnormality of the camera network; and executing an incremental optimization strategy on the locally abnormal camera network to solve the camera network selection adjustment scheme with the lowest cost. The application has the beneficial effects that: by introducing a multi-modal large model, the system is endowed with the ability to understand the scene and understand the rules; through multi-modal scene understanding, semantic constraint reasoning, hybrid intelligent optimization and dynamic adaptive adjustment, a high-quality and interpretable camera selection scheme can be automatically generated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent traffic monitoring, in particular to a multi-camera network optimization selection method based on a large model. BACKGROUND

[0002] With the rapid development of intelligent transportation systems and computer vision technology, multi-camera monitoring networks have become a core component of modern traffic management systems. In complex scenarios such as highway service areas, urban parking lots, and industrial parks, dozens or even hundreds of cameras are typically deployed to form a monitoring network in order to achieve global vehicle tracking and behavior analysis.

[0003] However, due to factors such as monitoring needs, site conditions, and cost budgets, actual deployment often faces problems such as waste of computing resources, network bandwidth occupation, and high storage costs caused by excessive number of cameras; in addition, different cameras have significantly different contributions to tracking tasks (such as entrance and exit cameras being more valuable than internal parking area cameras), so how to select the smallest and optimal subset of cameras from a large-scale camera network while meeting the global coverage requirement to achieve optimal resource allocation has become a core technical problem that needs to be solved in the deployment and optimization of intelligent monitoring systems.

[0004] Existing technologies for camera selection and optimization generally use a greedy algorithm based on geometric coverage to optimize camera locations, but such methods have the following technical defects:

[0005] (1) Lack of understanding of scene semantics: Traditional methods mainly select cameras based on geometric coverage relationships (such as set coverage problems), only considering spatial location and field of view, ignoring semantic information and business logic of the scene (such as entrances and exits must be covered, and main roads have higher priority than parking areas), resulting in a scheme that mathematically meets the coverage requirements but deviates from the actual monitoring intent and operational requirements.

[0006] (2) Insufficient formalization of constraint expressions: Traditional methods rely on precise mathematical models to describe constraints, making it difficult to effectively model and quantify fuzzy empirical knowledge (such as main road cameras should be given priority) and complex implicit constraints (such as system tolerance for single-point failures), which limits the practicality and adaptability of optimization schemes.

[0007] (3) Low automation level, relying on high-cost manual decision-making: The selection of camera locations for cross-camera tracking is a typical multi-dimensional (spatial, temporal, performance, cost) coupled decision-making process, and traditional methods lack efficient automated reasoning mechanisms, mainly relying on manual experience for repeated testing and manual adjustment. This process not only consumes time and effort, has a long decision-making cycle, and is prone to missing the global optimal solution due to the cognitive limitations of human decision-makers.

[0008] (4) Static optimization models lack dynamic adaptability: In actual operation, camera networks often experience local performance degradation due to dynamic factors such as equipment failure, occlusion, and changes in lighting. Traditional methods perform one-time optimization based on static assumptions and cannot autonomously adjust the camera selection strategy according to real-time monitoring feedback data, causing the system to fail when anomalies occur, lacking the necessary robustness and adaptability. Summary of the Invention

[0009] One objective of this application is to provide a multi-camera network optimization selection method based on a large model that can solve at least one of the defects in the aforementioned background art.

[0010] To achieve at least one of the above objectives, the technical solution adopted in this application is: a multi-camera network optimization selection method based on a large model, comprising the following steps:

[0011] S100: Based on the number of cameras in the camera network and external parameters, extract representational information of the camera network monitoring scene in four dimensions: spatial topology, temporal synchronization, semantic scene, and visualization.

[0012] S200: Combines semantic scene representation and visual representation as input prompts for a multimodal large model. Through stepwise reasoning of the multimodal large model and guidance of greedy search, it generates a selection scheme for the camera network.

[0013] S300: Evaluate the generated selection schemes through multi-dimensional evaluation indicators, output the selection schemes that meet the indicator requirements, and re-execute step S200 for the selection schemes that do not meet the indicator requirements after optimization and adjustment through human-computer interaction.

[0014] S400: Constructs a comprehensive health score based on various performance indicators of the camera; when the camera network executes the output selection scheme, it performs anomaly detection on the camera network based on the comprehensive health score; for camera networks with local anomalies, it executes an incremental optimization strategy to solve the camera network selection and adjustment scheme with the lowest cost.

[0015] Preferably, the extraction of representational information for spatial topology includes the following processes: calculating the field-of-view coverage area and effective coverage quality function of each camera; constructing a spatial adjacency graph of the camera network based on the field-of-view overlap relationship of each camera in the camera network; discretizing the monitoring scene into multiple grid cells to generate a region-camera coverage matrix; the extraction of representational information for temporal synchronization includes the following processes: extracting the temporal synchronization information of each camera in the camera network, calculating a reasonable time window for the vehicle to appear in the field of view of any adjacent camera during driving; defining a temporal compatibility discrimination function based on camera extrinsic parameters to distinguish between true and false reasonable time windows; and for semantic fields... The extraction of scene representation information includes the following processes: dividing the monitoring scene into multiple semantic regions according to functional requirements and assigning semantic weights to each semantic region; constructing a semantic importance function for each region based on region type, traffic density, and monitoring requirements; constructing a semantic region-camera association matrix based on the weighted coverage quality of each camera for each semantic region; generating a structured scene description vector and converting it into natural language text for input to a multimodal large model; the extraction of visualization representation information includes the following processes: generating a top-down view of the scene and marking the positions of all cameras, the field of view coverage of each camera, the main driving routes of vehicles, and the boundaries of key regions in the image.

[0016] Preferred semantic regions Regional semantic importance function The expression is as follows:

[0017] ;

[0018] Camera C i semantic regions Weighted coverage quality S ki The expression is as follows:

[0019] ;

[0020] ;

[0021] Among them, w k This represents the semantic weight, and β represents the vehicle density influence coefficient. Represents semantic regions Average traffic density, This represents the average traffic density across all scenarios. Indicates a forced overlay flag. Represents the region-camera correlation matrix. Represents semantic regions area, The effective coverage quality function, c r This represents the center coordinates of the grid cell r. Indicates camera C i The projected coordinates of the object onto the ground, v i Indicates camera C i The orientation of the unit vector, d i Indicates camera C i Effective monitoring distance.

[0022] Preferably, in step S200, the scene top view and structured text description are combined as input to the multimodal large model to construct a prompt including the task objective and constraints; the task objective is: to select M cameras from N cameras in the camera network to achieve full-domain vehicle tracking; the constraints include a set of hard constraints H and a set of soft constraints; wherein, the set of hard constraints H includes a key area complete coverage constraint H1, an adjacent camera overlap constraint H2, a spatiotemporal compatibility constraint H3, and a connectivity constraint H4; the set of soft constraints is the multi-objective loss function L(x) corresponding to the task objective, where x represents the camera selection vector.

[0023] Preferably, in step S200, the multimodal large model is guided to perform multi-round iterative reasoning using a thought chain reasoning method, specifically including the following process:

[0024] Define the input for the t-th round of inference optimization of a multimodal large model. With output Mapping relationship:

[0025] ;

[0026] ;

[0027] Define the iteration termination condition: ;

[0028] By defining a hard constraint satisfaction scoring function And soft constraint optimization degree scoring function Construct a comprehensive scoring function Evaluate the results of the iteration;

[0029] ;

[0030] , ;

[0031] in, and These represent the camera selection vectors respectively. and The corresponding multi-objective loss function, S VLM This represents the policy vector generated by a multimodal large model. This represents the constraint violation vector in round t. Indicates an indicator function, , , , Let represent the different hard constraints in round t. Let represent the difference between the current loss and the target loss, and let h represent any constraint in the set of hard constraints H. H represents the number of elements in the set of hard constraints, and ε represents the convergence threshold. This represents the theoretically optimal loss. This indicates the worst-case scenario.

[0032] Preferably, the process of guiding the greedy search of the traditional algorithm using a multimodal large model in step S200 is as follows:

[0033] Based on the current selected scheme, the difference in overall score before and after adding the new camera is used to construct a gain function. ;

[0034] Define the weighted gain function of a multimodal large model ;

[0035] ;

[0036] Set a greedy selection strategy to choose the optimal position from the input x. Perform iterative optimization; the iterative update method is to add the optimal position to the current selected solution. The unit vector is 1; the optimized weighted gain function is calculated after each iteration, and the iteration is terminated when the newly added camera cannot provide significant gain;

[0037] Where η represents the heuristic weight coefficient of the multimodal large model. This indicates the priority score of the newly added camera. This represents the highest priority score among all cameras.

[0038] Preferably, in step S300, the multi-dimensional indicators used to evaluate the selected scheme include coverage indicators, efficiency indicators, quality indicators, and robustness indicators. The coverage indicator is obtained by weighting spatial coverage, temporal coverage, and key area weighted coverage. The efficiency indicator is obtained by weighting compression rate, computing resource utilization, and cost savings rate. The quality indicator is obtained by weighting adjacent camera overlap rate and tracking handover success rate estimation. The robustness indicator is obtained by weighting single-point failure tolerance, redundant coverage, and load balancing. After obtaining all the indicators, different weights are assigned according to the importance of each indicator to obtain the overall evaluation score for evaluating the selected scheme.

[0039] Preferably, in step S400, the overall health score is calculated. The expression is:

[0040] ;

[0041] Construct an anomaly detection discriminant function Perform anomaly detection for the camera network;

[0042] ;

[0043] in, Functions representing device states Indicates the tracking loss rate. C represents the false match rate. i This indicates any camera. Indicates the health threshold. This indicates the threshold for the rate at which health status decreases.

[0044] Preferably, after detecting a local anomaly in the camera network in step S400, the multimodal large model adopts a hierarchical fault diagnosis and solution generation framework for adaptive adjustment, specifically including the following processes: constructing a multi-dimensional anomaly information vector as a structured input for multimodal large model analysis, the multi-dimensional anomaly information vector including camera status, performance indicators, environmental factors, and historical information; the multimodal large model uses a thought chain reasoning method to sequentially classify fault types, analyze physical mechanisms, and assess impacts to identify the root cause of the anomaly; based on the analysis results of the root cause of the anomaly, the multimodal large model generates multiple types of candidate solutions and performs an incremental optimization strategy for each candidate solution to evaluate cost-effectiveness.

[0045] Preferably, the incremental optimization strategy includes the following process:

[0046] The current selected option is calculated using Hamming distance. With candidate solutions Difference measurement ;

[0047] Construct an incremental optimization objective function based on the obtained difference metric. ;

[0048] ;

[0049] Given constraints: The faulty camera must be removed, and all hard constraints must be satisfied;

[0050] Quantifying the coverage gap: ;

[0051] Prioritization of Building Backup Cameras ;

[0052] ;

[0053] Based on selection priority The addition of backup cameras is carried out using a greedy selection strategy, and the iteration termination condition is whether the coverage gap is completely filled.

[0054] in, This represents the switching cost weighting coefficient. This represents a set of faulty cameras. Indicate candidate solutions The multi-objective optimization function This represents a vector that sets only the location of the faulty camera to 0. Represents the region-camera correlation matrix. Let represent a unit vector where the j-th position is 1 and the rest are 0. This represents the j-th column of the region-camera association matrix.

[0055] Compared with the prior art, the beneficial effects of this application are as follows:

[0056] (1) The technical solution of this application introduces a multimodal large model to give the system the ability to understand the scene and understand the rules; through multimodal scene understanding, semantic constraint reasoning, hybrid intelligent optimization and dynamic adaptive adjustment, it can automatically generate a high-quality and interpretable camera selection scheme.

[0057] (2) The technical solution of this application integrates the macro-policy generation capability of a multimodal large model with the precise search capability of traditional optimization algorithms. This allows the multimodal large model to be responsible for high-level policy guidance and complex constraint reasoning, while the traditional greedy algorithm performs efficient and precise local search under policy guidance. This hybrid intelligent framework combines breadth and depth, and compared with any single method, it can obtain higher quality and more reliable optimization solutions in a shorter solution time.

[0058] (3) The technical solution of this application is designed with a dynamic adaptation mechanism that runs through the entire process of monitoring, diagnosis, decision-making and execution. The system monitors the health of the cameras. Once an abnormal performance is detected, it can trigger a multimodal large model to perform root cause reasoning and generate multiple solutions, and start an incremental optimization algorithm. This mechanism ensures that the monitoring system maintains high performance and high availability during long-term operation, and significantly reduces operation and maintenance costs. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the overall working steps of this application.

[0060] Figure 2 This is a schematic diagram illustrating the specific workflow of this application. Detailed Implementation

[0061] The present application will now be further described in conjunction with specific embodiments. It should be noted that, in the description of this specification, the use of terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicates that the specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0062] In the description of this application, it should be noted that the terms "center", "lateral", "longitudinal", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., which indicate the orientation and positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and should not be construed as limiting the specific protection scope of this application.

[0063] It should be noted that the terms "first," "second," etc., in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0064] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "joining," and "fixing," etc., should be interpreted broadly. For example, they can refer to a connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0065] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0066] The terms “comprising” and “having”, and any variations thereof, in the specification and claims of this application are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0067] One preferred embodiment of this application, such as Figure 1 and Figure 2 As shown, a multi-camera network optimization selection method based on a large model includes the following steps:

[0068] S100: Based on the number of cameras in the camera network and external parameters, extract representational information of the camera network monitoring scene in four dimensions: spatial topology, temporal synchronization, semantic scene, and visualization.

[0069] S200: Combines semantic scene representation and visual representation as input prompts for a multimodal large model. Through stepwise reasoning of the multimodal large model and guidance of greedy search, it generates a selection scheme for the camera network.

[0070] S300: Evaluate the generated selection schemes through multi-dimensional evaluation indicators, output the selection schemes that meet the indicator requirements, and re-execute step S200 for the selection schemes that do not meet the indicator requirements after optimization and adjustment through human-computer interaction.

[0071] S400: Constructs a comprehensive health score based on various performance indicators of the camera; when the camera network executes the output selection scheme, it performs anomaly detection on the camera network based on the comprehensive health score; for camera networks with local anomalies, it executes an incremental optimization strategy to solve the camera network selection and adjustment scheme with the lowest cost.

[0072] It is understood that the technical solution of this application includes two core elements: one is to optimize the output of the original camera network input, and the other is to perform dynamic adaptive optimization based on the working state of the camera network during its operation after the optimized output of the camera network is completed.

[0073] Specifically, the input camera network is first characterized using a multimodal scene representation. Then, the visual language model within the multimodal large-scale model is invoked for scene understanding, generating a preliminary camera selection strategy. Next, the large language model within the multimodal large-scale model is used for multi-round constraint reasoning, dynamically transforming abstract business rules into distance optimization objectives and constraints. Based on this, a hybrid intelligent optimization framework is constructed to efficiently solve the problem by combining the macro-level strategy guidance of the large language model with the micro-level precise search capabilities of traditional greedy algorithms. For the generated candidate solutions, multi-dimensional quantitative evaluation is used to determine their feasibility. Unsuccessful candidate solutions can be iteratively updated through a human-computer interaction optimization mechanism. During the evaluation and operation of the candidate solutions, a dynamic adaptive adjustment mechanism monitors the camera network data and continuously optimizes the camera network based on the monitoring results.

[0074] Compared to traditional optimization methods based on geometric coverage relationships, the technical solution of this application introduces a multimodal large model, endowing the system with the ability to understand the scene and comprehend the rules. Through multimodal scene understanding, semantic constraint reasoning, hybrid intelligent optimization, and dynamic adaptive adjustment, it can automatically generate high-quality, interpretable camera selection schemes. A comprehensive and accurate data foundation is provided for optimization decisions through a multimodal scene representation system integrating spatial topology, temporal synchronization, semantic scene, and visualization.

[0075] At the solution level, this approach integrates the macro-policy generation capabilities of a multimodal large model with the precise search capabilities of traditional optimization algorithms. Specifically, the large language model handles high-level policy guidance and complex constraint reasoning, while the traditional greedy algorithm performs efficient and accurate local searches under policy guidance. This hybrid intelligent framework combines breadth and depth, achieving higher quality and more reliable optimization solutions in a shorter time compared to any single method. It demonstrates superior solution efficiency and stability in real-world large-scale camera networks (e.g., hundreds of nodes).

[0076] Compared to the fixed deployment of traditional solutions, the technical solution proposed in this application features a dynamic adaptive mechanism that runs through the entire process of monitoring, diagnosis, decision-making, and execution. The system monitors the health of cameras, and once performance anomalies (such as malfunctions or obstructions) are detected, it triggers a large language model to perform root cause inference and generate multiple solutions, and initiates an incremental optimization algorithm. This algorithm aims to minimize switching costs, adjusting only the affected local network, avoiding the high overhead of global re-optimization, and achieving second-level fault response and solution switching. This mechanism ensures that the monitoring system maintains high performance and high availability throughout long-term operation, significantly reducing maintenance costs.

[0077] In this embodiment, the construction of multimodal scene representation mainly involves acquiring basic information about the camera network, including external physical data such as the number of cameras, their location coordinates, orientation angles, field of view, and installation height. Then, representation information is extracted from four dimensions of the monitored scene: spatial topology, temporal synchronization, semantic scene, and visualization. For ease of understanding, the specific extraction process for these four dimensions will be described in detail below.

[0078] Specifically, the extraction of spatial topology representation information includes the following process: for each camera C i (i=1, 2, 3, ..., N), extract its spatial topological feature vector. Where N represents the total number of cameras included in the camera network; p i =(x i y i , z i θ represents the position of the camera in the three-dimensional world coordinate system; i ∈[0, 2π) represents the horizontal rotation angle of the camera relative to due north, i.e., the horizontal orientation angle; This represents the tilt angle of the camera relative to the horizontal plane, i.e., the pitch angle; d i This indicates the maximum clear detection distance of the camera, i.e., the effective monitoring distance; h i This indicates the installation height of the camera relative to the ground.

[0079] Based on the parameters mentioned above, first calculate C for each camera. i Field of view coverage area A i Field of view coverage area A i It can be defined as the intersection of the camera's cone-shaped perception space and the ground in three-dimensional space; the specific expression is as follows:

[0080] .

[0081] in, Indicates camera C iThe projected coordinates of the point onto the ground, where q represents the coordinates of the point on the ground, and v... i =(cosθ i sinθ i ) indicates camera C i The orientation of the unit vector, α i This indicates the horizontal field of view of camera Ci.

[0082] Based on the calculated field of view coverage area A i Camera C can be defined i Effective coverage quality function Q(q, C) i The function's range is [0, 1], indicating that the closer the camera is to the area projected onto the ground and the more angular the location, the higher the coverage quality. The specific expression is as follows:

[0083] .

[0084] Then, based on the overlapping field of view of each camera in the camera network, a spatial adjacency graph of the camera network is constructed. Where the vertex set V = {C1, C2, ..., C} N}; edge set γ represents any two adjacent cameras C i and C j The boundaries overlap, that is... Elements W in the weight matrix W ij C represents any two adjacent cameras. i and C j The overlap and intersection ratio of the field of view, i.e. .

[0085] Then, the monitoring scene is discretized into R=R x ×R y 1 grid cell; where R x and R y These represent the number of grid cells in the x and y directions of the monitored area, respectively, which can then be used to generate a region-camera coverage matrix. The specific representation is as follows:

[0086] ; where c r This represents the center coordinates of the grid cell r.

[0087] Specifically, the extraction of representational information for time synchronization includes the following process: First, extract the time synchronization information of each camera. Among them, offset i Indicates camera C i Startup time The time offset relative to the startup time of the reference camera, using the camera with the shortest startup time as the reference camera, is the time offset. i The calculation formula is: fps i Indicates the camera's video frame rate; scale i The time scale factor is used to correct clock drift between different cameras; the scale is defined. i =T ref / T i T ref Indicates the reference duration, T i Indicates camera C i The actual duration of the same recorded event.

[0088] Then, construct the spatiotemporal compatibility tensor. Its elements Indicates that the vehicle is from camera C i The field of view disappears into the adjacent camera C j The reasonable time window for the appearance of vision; and These represent the vehicle from camera C. i The field of view disappears into the adjacent camera C j The earliest and latest times of visual field appearance are calculated using the following formulas:

[0089] .

[0090] in, Indicates camera C i Field of view boundary to camera C j The shortest distance (in meters) to the boundary of the field of view, v max v represents the vehicle's maximum speed (meters per second). min This indicates the vehicle's minimum speed (meters per second). Indicates the maximum possible dwell time of the vehicle (in seconds), offset j Indicates camera C j The time offset.

[0091] Then, based on the camera extrinsics, a timing compatibility discrimination function is defined. The expression for determining true or false within a reasonable time window is as follows:

[0092] .

[0093] in, This indicates the vehicle's trajectory segment at camera C. i The last timestamp to appear in the middle, This indicates the vehicle's trajectory segment at camera C.j The timestamp first appeared in the text.

[0094] Specifically, the extraction of semantic scene representation information includes the following process: First, the monitoring scene is divided into k semantic regions according to functional requirements. Different semantic regions include entrances / exits, main roads, and parking areas. Semantic weights are assigned to each type of semantic region, resulting in a semantic weight vector. Semantic regions are constructed based on region type, traffic density, and monitoring requirements. Regional semantic importance function The specific expression is as follows:

[0095] .

[0096] Among them, w k This represents the semantic weight, and β represents the vehicle density influence coefficient. Represents semantic regions Average traffic density, This represents the average traffic density across all scenarios. The mandatory coverage indicator is 1 for key areas (entrances and exits, main roads, etc.) and 0 for others.

[0097] Then, based on the weighted coverage quality S of each camera for various semantic regions... ki Construct a semantic region-camera association matrix S; weighted coverage quality S ki The expression is as follows:

[0098] , .

[0099] in, Represents semantic regions area, This represents the effective coverage quality function.

[0100] Then, a structured scene description vector is generated. And convert it into natural language text T scene Used as input for large language models in multimodal large models; where the geometric description d geo The topology description d represents the area, boundary coordinates, and number of semantic regions of the monitored scene. topo Represents the road network topology, number of entrances and exits, and connectivity in a monitoring scenario; semantic description d sem It represents the distribution of regional types, traffic flow patterns, and monitoring targets.

[0101] Specifically, the extraction of representational information for visualization includes the following process: generating a top-down view of the scene (I). layout In image Ilayout The location of all cameras, the field of view of each camera, the main driving route of the vehicle, and the boundaries of key areas are marked.

[0102] In this embodiment, the multimodal large model includes a visual language large model (VLM) and a large language model (LLM). After extracting multi-dimensional representation information, a preliminary strategy can be generated using the visual language large model, specifically including the following: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] layout With structured text description T scene Combined as input to a multimodal large model, it constructs a hint that includes a list of task objectives and constraints.

[0103] The task objective is to select M cameras from N cameras in a camera network to achieve full-area vehicle tracking. The task conditions include hard and soft constraints. The hard constraints require key area coverage to exceed 100%, while the soft constraints require total coverage to exceed 95% and the overlap between adjacent cameras to exceed 10 meters. After receiving element cues, the visual language model analyzes scene visualizations, understands spatial layout relationships, identifies key areas, evaluates the coverage and overlap of each camera, and performs reasoning based on constraints described in the text. Ultimately, it outputs a preliminary selection scheme including a list of required cameras, a priority ranking of optional cameras, an area coverage strategy, and the rationale behind the decision.

[0104] In this embodiment, after the visual language big model generates a preliminary selection scheme through preliminary analysis, constraint optimization reasoning can be performed through the big language model. For ease of understanding, a detailed description will be given below.

[0105] The first step is to formalize the constraints, that is, to transform the business rules into logical constraint expressions. The specific process is as follows:

[0106] We can define an initial selection scheme x for the camera network, represented as a vector, i.e., the camera selection vector. ; where component x i =1 indicates that camera C was selected at the corresponding location in the camera network. i Otherwise, the value is 0. Define the set of hard constraints as H = {H1, H2, H3, H4}. Here, H1 represents the key region complete coverage constraint, H2 represents the adjacent camera overlap constraint, H3 represents the spatiotemporal compatibility constraint, and H4 represents the connectivity constraint. Define the set of soft constraints as the multi-objective loss function L(x) corresponding to the task objective. For ease of understanding, each constraint will be described in detail below.

[0107] Specifically, for the critical region complete coverage constraint H1, the expression is:

[0108] .

[0109] Where K represents the set of key region types, θ min This represents the minimum coverage quality threshold. This represents the minimum number of cameras required to cover a semantic region of type k. This indicates an indicator function that takes the value 1 if the condition within the parentheses is true, and 0 otherwise.

[0110] Based on the above expression, this constraint indicates that the weighted coverage quality S will be applied to the k semantic regions corresponding to the key region. ki Greater than the minimum coverage quality threshold θ min The number of cameras is accumulated to ensure that the accumulated number of cameras is greater than or equal to the minimum required camera coverage.

[0111] For the overlap constraint H2 of adjacent cameras, the expression is:

[0112] .

[0113] Where, ω min x represents the threshold of the visual field overlap coefficient. i and x j This represents two adjacent components in the camera selection vector.

[0114] Based on the above expression, this constraint can ensure that the overlap of the field of view of two adjacent cameras needs to be greater than a certain threshold, so as to ensure that there is no tracking blind spot when the vehicle switches between adjacent cameras.

[0115] For the spatiotemporal compatibility constraint H3, the expression is:

[0116] .

[0117] in, and These represent the spatial adjacency discrimination function and the time window validity discrimination function, respectively.

[0118] Based on the above expression, this constraint ensures that two cameras can be used for cross-camera trajectory tracking of vehicles only if the coverage and time window requirements are met.

[0119] For connectivity constraint H4, it is necessary to ensure that the coverage area of ​​the selected cameras forms a connected graph, so that the vehicle can be tracked from any entrance to any exit.

[0120] For the multi-objective loss function L(x), the expression is:

[0121] .

[0122] in, The penalty for the number of cameras is calculated using the following formula: λ1 represents the weighting coefficient of the camera quantity penalty term; The coverage loss term is calculated using the following formula: λ2 represents the weighting coefficient of the coverage loss term; This represents the weighted coverage quality loss term, and the specific calculation formula is as follows: λ3 represents the weighting coefficient of the weighted coverage quality loss term; The robustness loss term is represented by the following formula: λ4 represents the weighting coefficient of the robustness loss term.

[0123] Step 2: After formalizing the constraints, the multimodal large model can optimize the initial selected solution through multi-round dialogue. Specifically, the multimodal large model is guided through multi-round iterative reasoning using a thought chain reasoning approach, which includes the following process:

[0124] Define the input for the t-th round of inference optimization of a multimodal large model. With output Mapping relationship:

[0125] .

[0126] in, This represents the input to the camera selection vector in the t-th round; This represents the output of the camera selection vector in the t-th round; S represents the multi-target loss function corresponding to the camera selection vector in round t; VLM Represents the policy vector generated by the large visual language model; Let represent the constraint violation vector in round t, specifically defined as follows:

[0127] .

[0128] in, , , , Let represent the different hard constraints in round t. Indicates current loss With target loss The difference.

[0129] Define the iteration termination condition: .

[0130] in, Represents the camera selection vector The corresponding multi-objective loss function; ε represents the convergence threshold, and the specific value can be selected according to the actual needs of those skilled in the art, for example, ε=0.01; indicating that the relative change of the loss function between two consecutive rounds of optimization is less than 1%.

[0131] Specifically, the iteration termination condition is the camera selection vector. The iteration terminates when all hard constraints are met and the relative rate of change of the loss function from the previous round is less than the convergence threshold.

[0132] Step 3: After completing multiple rounds of iterative reasoning, evaluate the constraint satisfaction of the obtained selection schemes; constraint satisfaction can be assessed using a comprehensive scoring function. To evaluate this, we will use a comprehensive scoring function. The construction process includes the following steps:

[0133] Define a hard constraint satisfaction rating function The specific expression is:

[0134] .

[0135] in, This represents the number of elements in the set of hard constraints H.

[0136] Define a soft-constraint optimization degree scoring function The specific expression is:

[0137] .

[0138] in, This represents the theoretically optimal loss. This indicates the worst-case scenario.

[0139] Based on the obtained hard constraint satisfaction scoring function And soft constraint optimization degree scoring function Construct a comprehensive scoring function The specific expression is as follows:

[0140] .

[0141] In this embodiment, as Figure 2 As shown, after completing the stepwise reasoning of the multimodal large model, the large language model can be used to guide the greedy search in traditional algorithms to achieve hybrid intelligent optimization; that is, the priority ranking generated by the large language model is used as a heuristic function. The specific hybrid intelligent optimization process combining the large language model and greedy search is as follows:

[0142] First, based on the difference in overall scores before and after adding the new camera according to the current selected scheme, a gain function is constructed. That is, the gain function. This indicates adding camera C to the currently selected option x. i The subsequent comprehensive evaluation and improvement are expressed in the following formula:

[0143] .

[0144] Among them, e i This represents a unit vector where the i-th position is 1 and the rest are 0; that is, it is represented by the i-dimensional unit vector e. i In the camera selection vector, it is used as the added camera C. i .

[0145] Then, define the weighted gain function of the large language model in the multimodal large model. The specific expression is:

[0146] .

[0147] Where η represents the heuristic weight coefficient of the multimodal large model, which is used to balance the actual gain with the prior knowledge of the large language model scene understanding; This indicates the newly added camera C. i Priority score; This represents the highest priority score among all cameras.

[0148] Then, a greedy selection strategy is set to choose the optimal position from the input selection scheme x. The specific expression is as follows:

[0149] .

[0150] in, Represents all that satisfy x i The position i where x equals 0; that is, the greedy selection strategy is to select from all positions that satisfy x. i Find the corresponding weighted gain function at position i where = 0. The position with the largest value is taken as the optimal position. .

[0151] Then, for the current selected scheme Perform iterative updates, and select the updated option. The specific expression is:

[0152] .

[0153] That is, by finding the optimal position The unit vector with position 1 and all other positions 0 is the same as the current selection scheme. Add them together.

[0154] Then, the convergence condition is set to terminate the iteration when the newly added camera fails to provide significant gain, which can be expressed by the following expression:

[0155] .

[0156] in, This represents the gain threshold, and the specific value can be selected according to the actual needs of those skilled in the art. For example, it can be 0.001. That is, when the increase of the weighted gain function by the newly added camera is less than 0.001, the iteration is considered to have converged and the iteration can be terminated.

[0157] In this embodiment, after generating the camera network selection scheme using a hybrid intelligent framework based on a multimodal large model and a traditional greedy algorithm, the system calls a large language model to perform interpretability analysis on the generated selection scheme to ensure its reliability. Specifically, the large language model analyzes the technical basis of the camera selection scheme, including key area coverage strategies, continuous tracking strategies, key node coverage strategies, and resource optimization strategies; it generates a region-camera coverage allocation table, explaining the camera combination, overlapping area size, overall coverage rate, and coverage quality assessment for each region; and it evaluates the impact of single-point failures of each camera on the overall system, including coverage reduction, coverage gaps, backup schemes, and recovery strategies. The evaluation results of the above analysis process can be represented by multi-dimensional indicators. Specific indicators used for evaluation include the coverage rate indicator E. cov Efficiency index E eff Quality Indicator E qual and robustness index E rob To facilitate understanding, the composition of each indicator will be described in detail below.

[0158] Regarding the coverage metric E cov .

[0159] Its spatial coverage Time coverage and weighted coverage of key areas The weighted average is then applied; the weights for each parameter can be set according to actual needs. For ease of understanding, the coverage index E will be analyzed below using a set of optimized weighting coefficients. cov The expression (x) is used to describe it.

[0160] .

[0161] .

[0162] .

[0163] .

[0164] Among them, T total This indicates the total observation duration. Represents all that satisfy x i =1 at position i.

[0165] Regarding efficiency index E eff .

[0166] It is through compression ratio Calculate resource utilization rate and cost savings The weighted average is then applied; the weights for each parameter can be set according to actual needs. For ease of understanding, the efficiency index E will be analyzed below using a set of optimized weighting coefficients. eff The expression (x) is used to describe it.

[0167]

[0168] .

[0169] .

[0170] .

[0171] in, Indicates frame rate. Indicates the total number of pixels in a single frame. Indicates hardware cost, Indicates bandwidth cost, This indicates storage costs.

[0172] Regarding quality indicator E qual .

[0173] It uses the weighted overlap rate of adjacent cameras and tracking handover success rate estimation The weighted average is then applied; the weights for each parameter can be set according to actual needs. For ease of understanding, the quality index E will be analyzed below using a set of optimized weighting coefficients. qual The expression (x) is used to describe it.

[0174] .

[0175] .

[0176] .

[0177] in, ω1 and ω2 represent the edge set between the selected cameras; ω1 and ω2 both represent weight coefficients, and the specific values ​​can be selected according to the actual needs of those skilled in the art, for example, ω1=0.7 and ω2=0.3.

[0178] Regarding the robustness index E rob .

[0179] Its single-point failure tolerance Redundancy coverage and load balancing The weighted average is then applied; the weights for each parameter can be set according to actual needs. For ease of understanding, the robustness index E will be analyzed below using a set of optimized weighting coefficients. rob The expression (x) is used to describe it.

[0180] .

[0181] .

[0182] .

[0183] .

[0184] in, Indicates camera C i The number of grid cells covered, This represents the average value. The component x in the camera selection vector represents i =1 represents the number of cameras selected in the chosen scheme.

[0185] After obtaining the indicators for all dimensions, different weights are assigned to each indicator based on their importance, resulting in an overall evaluation score for assessing the selected options. The weight values ​​for each indicator can be selected by those skilled in the art based on their actual needs. In this embodiment, the coverage indicator E... cov Efficiency index E eff Quality Indicator E qual and robustness index E rob The preferred weights are 0.4, 0.3, 0.2, and 0.1. Feasibility is determined based on the final overall evaluation score of the selected solution. If the overall evaluation score of the selected solution exceeds the set threshold, the selected solution is deemed acceptable; otherwise, the selected solution is optimized and updated through human-computer interaction.

[0186] Understandably, for human-computer interaction optimization, the system supports adjusting optimization requirements through natural language interaction. The specific process is as follows: Intent recognition: Parse the natural language input by the user to identify the intention to adjust constraints, modify targets, etc.; Constraint update: Update hard and soft constraints according to user needs; Solution regeneration: Re-execute the optimization algorithm based on the updated constraints; Result verification: Verify the effectiveness and feasibility of the newly generated solution; User confirmation: Show the adjustment results to the user and support further interactive optimization.

[0187] In this embodiment, during the operation after the selected scheme is output, the operation of the camera network can be monitored in real time, and online optimization can be dynamically adapted to address local anomalies in the camera network based on the monitoring results. For ease of understanding, the anomaly detection and dynamic optimization process during camera network operation will be described in detail below.

[0188] First, it is necessary to monitor and provide feedback on the camera network in real time, which includes the following:

[0189] Define the performance of the camera .

[0190] in, This represents the device status function. When the camera is working properly, the device status function can be assigned a value of 1. When the camera performance degrades, the device status function can be assigned a value of 0.5. When the camera is faulty or offline, the device status function can be assigned a value of 0.

[0191] The tracking loss rate can be represented by a sliding window, and the specific calculation formula is as follows:

[0192] .

[0193] Where m represents the size of the sliding window (in seconds). This represents the number of vehicles lost at time t'. This indicates the total number of vehicles.

[0194] The false match rate is represented by the following formula:

[0195] .

[0196] in, This represents the number of vehicles that were verified as mismatches at time t'. This indicates the total number of target vehicles to be matched.

[0197] Based on the obtained device state function Tracking loss rate and mismatch rate A comprehensive health score for characterizing camera performance can be constructed as follows. The specific expression is:

[0198] .

[0199] Based on the obtained comprehensive health score Anomaly detection and discrimination functions can be constructed. Anomaly detection and discrimination function for camera network. The specific expression is:

[0200] .

[0201] in, Indicates the health threshold. This indicates the threshold for the rate at which health status decreases.

[0202] In other words, if the overall health score of a camera is less than a set health threshold, or if the rate of decrease in the overall health score is less than a set health decrease rate threshold, the camera can be considered abnormal. In this case, the anomaly detection function can output 1; otherwise, it outputs 0. Therefore, the condition for triggering local optimization of the current selected scheme for the camera network is that the value of the anomaly detection function is at least 1.

[0203] After detecting local anomalies in the camera network, the multimodal large model adopts a hierarchical fault diagnosis and solution generation framework for adaptive adjustment, which includes the following processes:

[0204] Structuring of anomaly information: The system constructs a multi-dimensional anomaly information vector as a structured input for multimodal large model analysis. The multi-dimensional anomaly information vector includes camera status, performance indicators, environmental factors, and historical information.

[0205] Root cause reasoning for anomalies: The large language model in the multimodal large model uses the thought chain reasoning method to classify fault types, analyze physical mechanisms, and assess impacts in sequence to identify the root cause of anomalies.

[0206] Multi-solution generation: The large language model in the multimodal large model generates multiple types of candidate solutions based on the analysis results of the root cause of the anomaly, and performs an incremental optimization strategy for each candidate solution to evaluate cost-effectiveness.

[0207] The execution of the incremental optimization strategy includes the following process:

[0208] Define the objective of incremental optimization: given the current selection scheme Collection of faulty cameras Find the candidate solution with the lowest cost. .

[0209] First, the current choice is calculated using Hamming distance. With candidate solutions Difference measurement The specific expression is as follows:

[0210] .

[0211] Based on the obtained difference measure Constructing an incremental optimization objective function The specific expression is as follows:

[0212] .

[0213] in, This represents the switching cost weighting coefficient. Indicate candidate solutions The multi-objective optimization function.

[0214] Given constraints: the faulty camera must be removed, and all hard constraints must be satisfied.

[0215] Quantifying the coverage gap: .

[0216] This represents a vector that sets only the location of the faulty camera to 0.

[0217] Prioritization of Building Backup Cameras The specific expression is as follows:

[0218] .

[0219] in, This represents a unit vector where the j-th position is 1 and the rest are 0, used to represent adding the current selection scheme. A backup camera; represents the j-th column of the region-camera correlation matrix, used to characterize the coverage capability vector of the j-th backup camera.

[0220] Based on selection priority The optimal position j of the backup camera is determined using a greedy selection strategy. * The determination of the optimal position j * The expression is:

[0221] ;in, Represents all that satisfy x j The position j where =0.

[0222] Then, the iteration continues until the gap is completely filled. The specific method of iteration can be found in the greedy search algorithm described above. The determination of the termination of the iteration can be expressed by the following expression.

[0223] .

[0224] in, Let L1 norm be represented, then This represents the total gap obtained by summing the absolute values ​​of all gaps; This indicates the acceptable coverage gap threshold. The specific value can be determined by those skilled in the art based on their actual needs. For example, it can be 0.01R. That is, when the total gap percentage is less than or equal to 1% of the total area of ​​the monitored scene, it can be assumed that it has been completely filled. At this point, the iteration can be stopped; otherwise, the iteration and update can continue.

[0225] The basic principles, main features, and advantages of this application have been described above. Those skilled in the art should understand that this application is not limited to the above embodiments. The embodiments and descriptions in the specification are merely the principles of this application. Various changes and modifications can be made to this application without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claims. The scope of protection claimed by this application is defined by the appended claims and their equivalents.

Claims

1. A large model-based multi-camera network optimization selection method, characterized in that, Comprise the following steps: S100: According to the number of cameras of the camera network and the external parameter, the spatial topology, time synchronization, semantic scene and visualization of the camera network monitoring scene are extracted; S200: The semantic scene representation and the visualization representation are combined as the input prompt of the multi-modal large model, and the selection scheme of the camera network is generated through the step-by-step reasoning of the multi-modal large model and the guidance of the greedy search; S300: The generated selection scheme is evaluated by multi-dimensional evaluation index, the selection scheme meeting the index requirement is output, and the selection scheme not meeting the index requirement is re-executed after optimization adjustment through human-computer interaction; S400: Based on the performance indicators of the cameras, a comprehensive health score is constructed; When the camera network executes the output selection scheme, the camera network is detected according to the comprehensive health score; the incremental optimization strategy is executed for the local abnormal camera network to solve the camera network selection adjustment scheme with the lowest cost; In step S400, the comprehensive health score is integrated The expression is: ； Building an anomaly detection discriminant function Making an abnormality determination of a camera network ； wherein, represents a device state function, represents a tracking loss rate, represents a false match rate, C i represents an arbitrary camera, represents a health threshold, represents a health rate of decline threshold. 2.The large model-based multi-camera network optimization selection method of claim 1, wherein, The process of extracting the spatial topology representation information includes the following processes: calculating the field of view coverage area and effective coverage quality function of each camera; constructing the spatial adjacency relationship graph of the camera network according to the field of view overlap relationship of each camera in the camera network; discretizing the monitoring scene into multiple grid units to generate a region-camera coverage matrix; The process of extracting the time synchronization representation information includes the following processes: extracting the time synchronization information of each camera in the camera network, calculating the reasonable time window of the vehicle appearing and disappearing in the field of view of any adjacent camera during driving; defining a time compatibility discrimination function according to the camera external parameters to determine the true or false of the reasonable time window; The process of extracting the semantic scene representation information includes the following processes: dividing the monitoring scene into multiple semantic regions according to the function requirements, and assigning semantic weights to each type of semantic region; constructing a region semantic importance function according to the region type, vehicle flow density and monitoring demand; constructing a semantic region-camera association matrix according to the weighted coverage quality of each camera to each type of semantic region; generating a structured scene description vector and converting it into a natural language text for input of the multi-modal large model; The process of extracting the visualization representation information includes the following processes: generating a scene top view, marking the positions of all cameras, the field of view coverage of each camera, the main vehicle driving route and the key region boundary in the image. 3.The method of claim 2, wherein, semantic region of the region semantic importance function The expression of the region semantic importance function is as follows: ； Camera C i To the semantic region The weighted coverage quality S ki The expression is as follows: ； ； where w k denotes semantic weight, β denotes vehicle density influence coefficient, denotes average vehicle flow density of semantic region , denotes average vehicle flow density of whole scene, denotes mandatory coverage flag, denotes region-camera association matrix, denotes area of semantic region , denotes effective coverage quality function, c r denotes center coordinate of grid cell r, denotes projection coordinate position of camera C i onto ground, v i denotes orientation unit vector of camera C i , d i denotes effective monitoring distance of camera C i . 4.The method of claim 1, wherein, In step S200, the scene top view and the structured text description are combined as the input of the multi-modal large model, and the prompt including the task target and the constraint condition is constructed; The task target is to select M cameras from N cameras of the camera network to realize global vehicle tracking; The constraints include a hard constraint set and a soft constraint set The hard constraint set includes a key region complete coverage constraint H1, an adjacent camera overlap constraint H2, a space-time compatibility constraint H3, and a connectivity constraint H4 The soft constraint set is a multi-objective loss function L(x) corresponding to a task target, and x represents a camera selection vector 5.The method of claim 4, wherein, In step S200, the multi-modal large model is guided by the thought chain reasoning method for multi-round iterative reasoning, which includes the following processes: Defining an input for a t-th round of inference optimization of a multi-modal large model and an output mapping relationship: ； ； Define the iteration termination condition: ; By defining a hard constraint satisfaction score function and a soft constraint optimization score function a combined score function is constructed The evaluation of the iteration results is performed; ； ， ； wherein, and respectively represent the camera selection vector and corresponding multi-objective loss function, S VLM represent the strategy vector generated by the multi-modal large model, represent the constraint violation vector of the t-th round, represent the indicator function, , , , respectively represent different hard constraints of the t-th round, represent the difference between the current loss and the target loss, h represents any constraint in the hard constraint set H, represent the number of elements in the hard constraint set H, and ε represents the convergence threshold, represent the theoretical optimal loss, represent the worst loss. 6.The method of claim 5, wherein, The process of guiding the greedy search of the traditional algorithm by the multi-modal large model in step S200 is: The difference of comprehensive score before and after adding new camera according to current selection scheme, construct gain function ; Defining a weighted gain function for multi-modal large models ; ； A greedy selection strategy is set to select the optimal position from the current selection scheme for iterative optimization; the iterative update mode is to add the optimal position to the current selection scheme A unit vector for 1; After each iteration, the weighted gain function of the optimized camera is calculated, and the iteration is terminated when the newly added camera cannot provide significant gain; wherein η represents a heuristic weight coefficient of the multi-modal large model, represents a priority score of the newly added camera, represents a maximum priority score among all cameras.

7. The large model based multi-camera network optimization selection method of any one of claims 1-6, wherein, In step S300, the multi-dimensional indexes for evaluating the selection scheme include coverage index, efficiency index, quality index, and robustness index; The coverage index is obtained by weighting the spatial coverage, the time coverage, and the key area weighted coverage; The efficiency index is obtained by weighting the compression rate, the computing resource utilization rate, and the cost saving rate; The quality index is obtained by weighting the adjacent camera weighted overlap rate and the tracking switching success rate estimation; The robustness index is obtained by weighting the single-point fault tolerance, the redundant coverage, and the load balancing degree; After obtaining all the indexes of the dimensions, different weights are given according to the importance of each dimension index to obtain the overall evaluation score for evaluating the selection scheme. 8.The method of claim 1, wherein, After detecting the local anomaly of the camera network in step S400, the multi-modal large model adopts a hierarchical fault diagnosis and scheme generation framework for adaptive adjustment, including the following processes: A multi-dimensional abnormal information vector is constructed as a structured input for the multi-modal large model analysis, including camera state, performance index, environmental factors, and historical information; The multi-modal large model uses a thinking chain reasoning method to sequentially perform fault type classification, physical mechanism analysis, and impact assessment to identify the root cause of the anomaly; The multi-modal large model generates multiple types of candidate solutions based on the analysis results of the root cause of the anomaly, and performs incremental optimization strategy for cost-benefit evaluation on each candidate solution. 9.The large model based multi-camera network optimization selection method of claim 8, wherein, The incremental optimization strategy includes the following processes: Calculating the difference measure of the current selection scheme by Hamming distance from the candidate solution ;​ constructing an incremental optimization objective function from the obtained difference measure ; ； Given the constraint condition: the faulty camera must be removed, and all hard constraints must be met; Quantification of the coverage gap Gap: ; Building selection priority for backup camera ;​ According to selection priority The addition of the backup camera is performed by a greedy selection strategy, taking whether the gap is completely filled as the iteration termination condition. wherein, denotes the switching cost weight coefficient, denotes the set of faulty cameras, denotes the candidate solutions of the multi-objective optimization function, denotes the vector of setting the faulty camera positions to 0 only, denotes the zone-camera association matrix, denotes the unit vector with the jth position as 1 and the rest as 0, denotes the jth column of the zone-camera association matrix.

Images