Vision-based operation management system

Abstract

The present application relates to the fields of industrial automation and intelligent logistics, and discloses a vision-based operation management system, including: a visual perception module, a traffic modeling module, a path optimization module, a scheduling optimization module, and a feedback adjustment module; the visual perception module obtains image or video data in the operation area and extracts the dynamic state of the target device; the traffic modeling module builds a dynamic weighted graph model based on the device state data; the path optimization module generates the optimal path of the device by solving the dynamic Hamiltonian path problem; the scheduling optimization module optimizes the passage sequence and speed of the device based on the game theory model; the feedback adjustment module monitors the operation status of the device in real time, detects abnormal conditions, and dynamically adjusts the modeling and optimization results. The present invention can effectively solve the path conflict and traffic congestion problems in the collaborative operation of multiple devices, improve the operation efficiency and resource utilization, has strong real-time and adaptability, and is suitable for a variety of scenarios such as logistics warehousing, industrial production and intelligent transportation.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial automation and intelligent logistics, and in particular to a vision-based operation management system. Background Art

[0002] With the development of industrial automation and intelligence, unmanned handling equipment (such as AGV forklifts, unmanned transport vehicles and logistics robots) has been widely used in logistics warehousing, smart factories and other fields. These devices can efficiently complete operations such as material transportation, loading and unloading and inventory management through automatic navigation, path planning and task scheduling. However, in the complex environment of multi-device collaborative operation, the traditional operation management system still has many shortcomings in dynamic adaptability, real-time optimization and multi-objective collaboration.

[0003] In the existing technology, most operation management systems rely on path planning and task scheduling solutions based on static rules. This method usually assumes that the equipment operation environment is static, that is, the node location, the passage path and the equipment status do not change during the planning cycle. However, in actual operation scenarios, the equipment status and environmental conditions are often dynamically changing. For example, path blocking, new equipment joining or equipment failure will cause the established path and scheduling strategy to be unable to be effectively executed. The existing technology lacks the ability to respond quickly to such dynamic environments, resulting in low equipment operation efficiency and even large-scale congestion or task delays.

[0004] In addition, traditional path planning methods are mostly optimized with a single device as the center, ignoring the resource contention problem in the coordinated operation of multiple devices. For example, when multiple devices share the same path or node, the lack of an effective coordination mechanism can easily lead to path conflicts or unfair priority allocation. Especially in scenarios with high-density device operation, this defect can significantly reduce the system's operating efficiency and device utilization.

[0005] On the other hand, existing technologies also have limitations in multi-objective optimization. Most systems focus only on a single objective (such as the shortest path or the lowest energy consumption), while ignoring multiple requirements in equipment operation, such as the balance of comprehensive indicators such as travel time, task priority, energy consumption, and equipment wear. In addition, due to the lack of a closed-loop feedback mechanism, it is difficult for existing systems to monitor equipment status in real time and adjust optimization strategies during operation, and it is impossible to maintain the stability and adaptability of the system in a changing operating environment. Summary of the invention

[0006] In view of the deficiencies of the prior art, the present invention provides a vision-based operation management system, which solves the problems of path conflicts, insufficient adaptability to dynamic environments, and difficulty in balancing multi-objective optimization in a multi-device collaborative operation environment.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a vision-based operation management system, comprising:

[0008] Visual perception module, used to obtain image or video data in the operating area and identify the real-time dynamic status of the target device, including position, speed, direction and target position;

[0009] A traffic modeling module, used to construct a dynamic weighted graph model based on the equipment status data provided by the visual perception module;

[0010] A path optimization module, used to solve the optimal path of the device from the starting point to the target node based on the dynamic weighted graph model;

[0011] A scheduling optimization module, used to optimize the passage sequence and speed of the equipment according to the optimal path provided by the path optimization module;

[0012] The feedback adjustment module is used to monitor the execution status of the equipment in real time and dynamically adjust the input data and optimization results of the traffic modeling module.

[0013] Preferably, the visual perception module identifies the dynamic state of the target device based on a deep learning target detection algorithm, wherein:

[0014] The target equipment includes AGV forklifts, unmanned guided vehicles and mobile robots;

[0015] The dynamic state includes the spatial coordinates, driving speed, driving direction and target node of the device.

[0016] Preferably, the dynamic weighted graph model includes a plurality of nodes and edge weights, wherein:

[0017] Nodes are used to represent the current and target locations of the device;

[0018] Edges are used to represent possible paths between devices, and the weight of the edge is determined based on the device's travel time and priority;

[0019] The edge weight is calculated by the following formula:

[0020] w ij =α·t ij +β·p i ;

[0021] Among them, w ij represents the edge weight from node i to node j; where t ij represents the travel time of the device from node i to node j, d ij is the distance from node i to node j, v i is the current speed of the device; pi Indicates the priority of the device; α and β are weight coefficients, satisfying α+β=1;

[0022] The traffic modeling module updates the status of nodes and edges in real time, including the addition of new equipment, path congestion, or changes in equipment speed, and recalculates the weights in the dynamic weighted graph model based on the updated equipment status.

[0023] Preferably, the path optimization module is used to dynamically optimize the passage path of the equipment based on a dynamic weighted graph model and solve the dynamic Hamiltonian path problem, wherein:

[0024] The dynamic Hamiltonian path problem aims to minimize the total weight of all nodes in the dynamic weighted graph model. The total weight is calculated by the following formula:

[0025]

[0026] Where C(P) represents the total weight of path P; P = {v1, v2, ..., v k} represents the node sequence in the path; v i represents the i-th node in the path; Represents the slave node v i To node v i+1 The edge weight of

[0027] Dynamic path optimization is solved based on a dynamic programming method, which includes:

[0028] Define the subproblem, let D(S,v i ) means starting from the starting point, passing through the node set S and ending at node v i is the shortest path length to the end point;

[0029] Construct a recursive relation:

[0030]

[0031] Among them, D(S,v i ) means passing through the node set S and ending at node v i is the shortest path length to the end point; v j represents a node in the node set S; S\{v i} means removing node v from set S i The collection after Represents the slave node v j To node v i The edge weight of

[0032] Initial conditions:

[0033] D({v1}, v1) = 0;

[0034] Where {v1} represents the node set including the starting node v1; D({v1}, v1) represents the shortest path length starting from the starting node v1 and passing only through the node v1;

[0035] The optimal solution is:

[0036]

[0037] Where V represents the set of all nodes in the dynamic weighted graph model; D(V, v i ) means starting from the starting point, passing through all nodes V and ending at node v i is the shortest path length to the end point; Represents the slave node v i Return the edge weight of the starting point v1;

[0038] The path optimization module updates the path planning results in real time according to the changes in the dynamic weighted graph model, and re-solves the dynamic Hamiltonian path problem when new nodes are added or the path is blocked to ensure the dynamic optimality of the passage path.

[0039] Preferably, the scheduling optimization module optimizes the passage order and speed of the target equipment based on a non-cooperative game model, wherein:

[0040] Scheduling optimization is based on a non-cooperative game model, with the target device as the participant in the game. Each device has an optional strategy set S i , indicating the passage sequence or speed selection of the equipment;

[0041] Define the utility function u i (s i ,s -i ), indicating that the target device i is selecting strategy s i And other devices choose strategies -i The benefit when , the utility function is:

[0042] u i (s i ,s -i )=-(α·t i +β·e i );

[0043] Among them, t i represents the travel time of device i; e i represents the energy consumption of device i; β and β are adjustable weight parameters, satisfying α+β=1; s i is the selection strategy for device i; s -1 A selection policy set for other devices;

[0044] The scheduling optimization module solves the Nash equilibrium point s of the game * , determine the optimal strategy of the device, the Nash equilibrium satisfies the following conditions:

[0045]

[0046] in, represents the optimal strategy of device i; Represents the optimal strategy set for other devices.

[0047] Preferably, the scheduling optimization module adopts an iterative optimal response algorithm to solve the Nash equilibrium, which specifically includes the following steps:

[0048] Initialize the policy set for all target devices;

[0049] For each device, calculate its optimal strategy when the strategies of other devices are fixed;

[0050] Repeatedly update the strategy of each device until it converges to the Nash equilibrium point s * .

[0051] Preferably, the scheduling optimization module combines a multi-objective optimization method to comprehensively optimize the operating efficiency of the target equipment, wherein:

[0052] The goal of scheduling optimization is to simultaneously minimize the following objective functions:

[0053]

[0054] Among them, F1(x) represents the total travel time of all target devices; F2(x) represents the total energy consumption of all target devices; F3(x) represents the total wear of all target devices; t i 、e i 、m i They represent the travel time, energy consumption and equipment wear of target equipment i respectively; n represents the number of target equipment;

[0055] The multi-objective optimization is converted into a single-objective optimization problem by normalizing the weights, and its optimization function is:

[0056] F(x)=λ1·F1(x)+λ2·F2(x)+λ3·F3(x);

[0057] Wherein, F(x) represents the comprehensive optimization objective function; λ1, λ2, and λ3 are weight parameters of the objective function, satisfying λ1+λ2+λ3=1.

[0058] Preferably, the scheduling optimization module solves the optimization problem by means of a genetic algorithm, specifically comprising the following steps:

[0059] Initialize the population and randomly generate the passage order and speed strategy of the equipment;

[0060] Select the population according to the fitness function F(x);

[0061] Generate new strategy populations through crossover and mutation operations;

[0062] Repeat the iteration until the fitness function converges to obtain the optimal strategy for comprehensive optimization.

[0063] Preferably, the feedback adjustment module detects the following abnormal conditions and triggers dynamic adjustment by visually monitoring the operating status of the device:

[0064] Path occupation or blockage;

[0065] The addition of new equipment;

[0066] Equipment operation failure.

[0067] The present invention also provides a vision-based operation management method, comprising the following steps:

[0068] The image data of the operating area is obtained through visual equipment, and the dynamic state of the target device, including position, speed, direction and target node, is extracted using the target detection algorithm;

[0069] Based on the device status data, a dynamic weighted graph model is constructed, wherein nodes represent the current location and target location of the device, and weights are based on the travel time and priority of the device;

[0070] Based on the dynamic weighted graph model, the dynamic Hamiltonian path problem is solved to generate the optimal path for the target device; based on the game theory model, the passage sequence and speed of the target device are optimized to generate the device passage instructions;

[0071] Monitor equipment operating status in real time, detect abnormal conditions during operation, and dynamically adjust traffic modeling and optimization results.

[0072] The present invention provides a vision-based operation management system, which has the following beneficial effects:

[0073] 1. The present invention realizes the global optimization of target equipment from path planning to traffic scheduling by solving dynamic weighted graph model and dynamic Hamiltonian path problem, combined with game theory model and multi-objective optimization method. The equipment can perform tasks according to the optimal path and traffic sequence in a complex dynamic environment, effectively reducing path conflicts, equipment waiting time and operating energy consumption, and improving overall operating efficiency.

[0074] 2. The feedback adjustment module of the present invention can monitor the operating status of the equipment in real time, and dynamically adjust the traffic modeling and optimization results by detecting abnormal situations (such as path obstruction, equipment failure or new equipment), thereby ensuring the stability and continuity of the system in a changing environment. Compared with the traditional static scheduling method, the present invention can respond to environmental changes more quickly and ensure the normal operation of the equipment.

[0075] 3. The present invention solves the traffic conflict problem of multiple devices in a shared transportation network by introducing a game theory model in the scheduling optimization module. The system can optimize the traffic order according to the device priority and task requirements, avoiding resource contention and path congestion, thereby improving the efficiency and fairness of multi-device collaborative operations.

[0076] 4. The architecture design and method flow of the present invention can adapt to a variety of operating scenarios, including industrial logistics, warehouse management, intelligent transportation, etc. By flexibly adjusting model parameters (such as weight coefficients, priority rules) and supporting multi-objective optimization, the present invention can be extended to operating environments of different scales and complexities, and has good versatility and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is a schematic diagram of the system architecture of the present invention;

[0078] Figure 2 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] Please see attached Figure 1 The vision-based operation management system of the present invention is used to dynamically manage and optimize the operation status of target equipment such as AGV forklifts and mobile robots in logistics, warehousing and industrial environments. Based on real-time visual perception data, the system achieves efficient operation of equipment through dynamic modeling, path planning and scheduling optimization, and realizes closed-loop optimization through feedback mechanism.

[0081] like Figure 1 As shown, the vision-based operation management system may include the following modules:

[0082] Visual perception module: responsible for real-time perception of device dynamic information.

[0083] Traffic modeling module: converts equipment status data into a dynamic weighted graph model of the traffic network.

[0084] Path optimization module: calculates the optimal travel path for devices in the transportation network.

[0085] Scheduling optimization module: optimizes the passage sequence and speed of equipment.

[0086] Feedback adjustment module: monitor the equipment operation status in real time and dynamically adjust traffic modeling and optimization results.

[0087] The various modules of the system of the present invention are described in detail below.

[0088] In this embodiment, the visual perception module is used to obtain image or video data in the operating area and extract the dynamic state of the target device in real time, including position, speed, direction and target position. This module is implemented based on industrial cameras and computer vision algorithms, and can provide accurate input data for subsequent traffic modeling, path optimization and scheduling optimization.

[0089] As an option, the visual perception module acquires panoramic images or video streams by arranging multiple industrial-grade high-precision cameras in the operating area. These cameras can be fixedly installed in key areas, such as traffic intersections, loading and unloading points, or above the work area to cover the main activity range of the operating equipment. The cameras are connected to the central processing unit via the network to transmit and process image data in real time.

[0090] Specifically, the visual perception module uses a deep learning target detection algorithm (such as YOLOv8) to detect and track the target device in the video stream. In some embodiments, the target device can be an AGV forklift, an unmanned transport vehicle, or other mobile robots. The algorithm can extract key parameters of the target device from the image, including:

[0091] Position, specifically the two-dimensional coordinates (x, y) of the device, obtained by geometrically transforming the pixel position of the device in the image;

[0092] Speed, specifically the current speed v of the device, which is determined by calculating the rate of change of the device's position in consecutive frames;

[0093] Direction, specifically, the device's travel direction angle θ, determined by analyzing the device's motion trajectory in the image;

[0094] The target location, specifically, the target node location d of the current task of the device. This information is usually obtained by interacting with the task scheduling system of the device.

[0095] It should be noted that in order to improve the accuracy of target detection, the visual perception module can combine the background modeling method to eliminate the influence of static obstacles or fixed facilities, so as to more accurately identify dynamic target devices.

[0096] In one possible implementation, the visual perception module performs preprocessing operations on the original image after acquiring it. The preprocessing steps include grayscale, denoising, and scale normalization to improve the algorithm's robustness to different lighting and device colors. The preprocessed image is input into the object detection algorithm to generate a detection box for each device and its category information.

[0097] As an option, target detection is subsequently combined with a device tracking algorithm (such as a Kalman filter or SORT algorithm) to continuously track the dynamic changes of the device. Through the tracking algorithm, the visual perception module can associate the detection results of the same device in different frames, reducing the detection loss caused by occlusion or rapid movement of the device.

[0098] It should be noted that the visual perception module is not limited to target detection and tracking, but can also perform extended analysis on the acquired image data. For example, in some embodiments, the environment around the target device can be analyzed through an instance segmentation algorithm to extract the obstacle distribution or road status in the area, thereby providing more information for subsequent traffic modeling.

[0099] Exemplarily, the output of the visual perception module can be represented as the following device state set:

[0100] S={(x i ,y i , v i ,θ i , d i )|i=1, 2, ..., n};

[0101] in:

[0102] x i ,y i Represents the two-dimensional coordinates of the i-th device;

[0103] v i Indicates the speed of the i-th device;

[0104] θ i Represents the direction angle of the i-th device;

[0105] d i represents the target position of the i-th device;

[0106] n represents the total number of devices in the current operating area.

[0107] It is understandable that the parameters in the device status set are updated in real time through the visual perception module, and the update frequency depends on the frame rate of the camera and the processing power of the algorithm. As an optimization, the module can be combined with edge computing technology to complete preliminary data processing on the camera side to reduce the computing burden of the central processing unit.

[0108] In one possible implementation, the visual perception module also supports marking abnormal states of detection and tracking results, such as device stagnation, abnormal motion trajectory, or device loss. In this case, the module will pass the abnormal state information to subsequent modules to trigger the system's adjustment mechanism.

[0109] It should be noted that in order to adapt to a variety of operating environments, the visual perception module can integrate multi-spectral cameras (such as infrared cameras and RGB cameras) to enhance detection performance under low light or complex background conditions. In low light environments, the infrared camera can provide thermal images of the device, while the RGB camera is used for target detection under normal lighting. The module combines the outputs of the two to improve detection reliability.

[0110] In some embodiments, the visual perception module can support the management capabilities of a multi-target environment. For example, when there are multiple equipment targets in an operating area, the module can output a set of equipment states in groups and assign a unique identification code to each group of equipment to facilitate traffic modeling and path planning operations in subsequent modules.

[0111] In this embodiment, the visual perception module also supports real-time data storage to save historical detection data. These historical data can be used for retrospective analysis of abnormal conditions or provide training data for continuous optimization of machine learning models.

[0112] It should be further explained that the camera layout and algorithm selection in the visual perception module can be adjusted according to the actual scenario. For example, at a traffic intersection, the camera can cover multiple directions of the intersection, while in a narrow passage, the camera can be placed at both ends of the passage to ensure that the device is visible throughout the entire process.

[0113] To sum up, the visual perception module in this embodiment uses high-precision cameras and deep learning algorithms to perceive the status of target equipment in the operating area in real time, and provides the results to subsequent modules in the form of structured data, laying a data foundation for realizing dynamic optimization and closed-loop adjustment of the system.

[0114] In this embodiment, the traffic modeling module constructs a dynamic weighted graph model in the operation area based on the device status data provided by the visual perception module to describe the traffic status of the target device in the traffic network. Through this dynamic weighted graph model, it can provide structured data support for the subsequent path optimization module and dynamically reflect the real-time status of the equipment and traffic flow changes in the operation area.

[0115] It should be noted that the dynamic weighted graph model G constructed by the traffic modeling module consists of a node set V, an edge set E, and a weight set W, specifically G = (V, E, W). Nodes are used to represent the current location of the device and its target location, edges are used to represent possible paths between nodes, and weights are used to quantify the cost of the path.

[0116] As an option, the node set V includes the current position and target position of the device. The current position of the device is provided in real time by the visual perception module and is expressed as a two-dimensional coordinate (xi, yi), and the target position di is obtained through the task scheduling system or the device local task data. Each node in the node set can be marked with a unique identifier to facilitate the traffic modeling module to track and update the status of the node.

[0117] Specifically, the edge set E is used to describe possible paths between nodes. The definition of the edge is based on the feasible movement range of the equipment in the operating area. For example, in a warehousing and logistics scenario, an edge can represent the passable relationship between two channels. In a dynamic environment, the edge set will be updated in real time as the equipment status or environment changes (such as new obstacles, occupied paths, etc.).

[0118] In one possible implementation, the edge weight w ij represents the travel cost from node i to node j, which is mainly composed of travel time and priority. ij The calculation formula is as follows:

[0119] w ij =α·t ij +β·p i ;

[0120] in:

[0121] represents the travel time of the device from node i to node j;

[0122] d ij is the Euclidean distance from node i to node j, according to the formula calculate;

[0123] v i is the current speed of the device;

[0124] p i Indicates the priority of the equipment, determined by task urgency or equipment type;

[0125] α and β are adjustable weight coefficients, satisfying α+β=1.

[0126] For example, in the modeling of a traffic intersection, if device A moves from the current position (2,3) to the target node (5,7), the device speed v A =2m / s, priority p A =0.8, then the travel time of the path is:

[0127]

[0128] Assuming the weight coefficient α = 0.7, β = 0.3, the weight w ij for:

[0129] w ij =0.7·2.5+0.3·0.8=1.75+0.24=1.99;

[0130] It is understandable that the weight w ij The calculation can dynamically reflect the operating status and task requirements of the equipment. i or priority p i When changes occur, weights are adjusted in real time, ensuring that modeling results accurately reflect current traffic conditions.

[0131] In one possible implementation, the traffic modeling module stores the dynamic weighted graph model in the form of an adjacency matrix to quickly query the path relationship between nodes. The element A[i][j] of the adjacency matrix corresponds to the weight w between node i and node j. ij When the running environment changes (such as adding new nodes or edges), the adjacency matrix will be dynamically updated.

[0132] As an option, the traffic modeling module can be extended to support obstacle modeling. For example, by analyzing the image data provided by the visual perception module, the traffic modeling module can identify fixed obstacles (such as shelves) or temporary obstacles (such as stacked goods) on the equipment's driving path. These obstacles are represented as non-traffic nodes in the graph model, effectively preventing the path optimization module from generating an infeasible path.

[0133] It should be noted that the real-time and accuracy of the traffic modeling module are crucial to the overall performance of the system. In some embodiments, the module implements rapid modeling of large traffic networks through distributed computing. For example, in an operating area containing hundreds of devices, the module can distribute traffic modeling tasks to multiple computing units based on regional divisions, and maintain the consistency of modeling results through boundary data sharing.

[0134] In one possible implementation, the traffic modeling module supports a mechanism for handling abnormal conditions to adapt to dynamic environments. For example, when a path is occupied or the speed of a device drops suddenly, the module dynamically adjusts the weight of the corresponding edge or marks the edge as inaccessible to reflect the change in a timely manner. This mechanism ensures that the path generated by the path optimization module is always feasible and efficient.

[0135] As a technical extension, the traffic modeling module also supports the accumulation and analysis of historical data. For example, the module can record statistical information such as the frequency of use of each edge and the distribution of travel time. These historical data can not only be used to optimize future path planning, but also help identify potential traffic bottleneck areas.

[0136] In summary, the traffic modeling module in this embodiment provides a structured traffic network description for the system by constructing a dynamic weighted graph model. By updating the status of nodes and edge weights in real time, the module can dynamically reflect the traffic conditions in the operating area and provide reliable data support for subsequent path optimization and scheduling optimization modules.

[0137] In this embodiment, the path optimization module is used to calculate the optimal path from the starting point to the target node for the target device based on the dynamic weighted graph model G = (V, E, W) generated by the traffic modeling module. The core goal of path optimization is to minimize the total weight of the device passing through the traffic network, ensuring that the path selection not only meets the device operation requirements, but also can adapt to traffic flow changes in a dynamic environment.

[0138] It should be noted that the path optimization module is based on the dynamic Hamiltonian path problem (HPP) and optimizes the device path in real time through a dynamic programming algorithm. The goal of the dynamic Hamiltonian path problem is to find the optimal path that traverses all nodes in the graph so that the total weight of the path is minimized.

[0139] In a possible implementation, the path optimization module describes the path optimization problem through the following mathematical model. Let the path P represent the node access sequence of the target device, and the total weight C(P) of the path is defined as:

[0140]

[0141] in:

[0142] P = {v1, v2, ..., v k} represents a path node sequence;

[0143] v i represents the iiith node in the path;

[0144] Represents the slave node vi To node v i+1 The edge weight of .

[0145] It is understandable that the path optimization module needs to minimize the total weight C(P) of the path while satisfying the node access constraints. This optimization goal can ensure that the device completes the task along the most economical path.

[0146] Specifically, in order to solve the dynamic Hamiltonian path problem, the path optimization module in this embodiment adopts a dynamic programming algorithm. The dynamic programming algorithm gradually builds a global optimal solution by recursively solving sub-problems. The following are the specific implementation steps of the dynamic programming algorithm:

[0147] As an alternative, define the subproblem D(S,v i ), which means starting from the starting point, passing through the node set S and ending at node v i is the shortest path length to the end point. The recursive relation can be expressed as:

[0148]

[0149] in:

[0150] S is the set of nodes visited by the current path;

[0151] v i is the current endpoint node;

[0152] v j is a node in the set S;

[0153] For slave node v j To node v i The edge weight of .

[0154] The initial conditions are defined as:

[0155] D({v1}, v1) = 0;

[0156] Among them, {v1} represents a node set that only contains the starting point, and D({v1}, v1) indicates that the path length from the starting point to itself is zero.

[0157] For example, the total weight of the optimal path C * It can be calculated by the following formula:

[0158]

[0159] in:

[0160] V represents the set of all nodes in the dynamic weighted graph model;

[0161] From the last node v i Returns the edge weight of the starting point v1.

[0162] In one possible implementation, the path optimization module solves the above recursive relationship in real time and dynamically updates the optimal path during the operation of the device. For example, when a new node is added to the operation area or the weight of an edge changes, the module recalculates the optimal path in the dynamic weighted graph model G.

[0163] It should be noted that in order to improve computing efficiency, the path optimization module can combine pruning strategies to pre-empt nodes or paths that do not meet constraints. For example, if the weight of an edge exceeds the current task time budget of the device, the module will directly discard the path branch containing the edge. This pruning strategy can significantly reduce the amount of calculation and improve the real-time performance of path optimization.

[0164] As an option, the path optimization module supports parallel path optimization for multiple devices. In this case, the path optimization problem for each device is modeled as an independent dynamic Hamiltonian path problem, and the optimal path for each device is calculated in parallel. In addition, to avoid path conflicts, the module can introduce path mutual exclusion constraints, that is, restrict multiple devices from accessing the same node or edge at the same time, thereby ensuring the consistency of path optimization results on a global scale.

[0165] In some embodiments, the path optimization module also supports a dynamic weight adjustment function. For example, for a device with a higher priority, the module can dynamically increase the weight coefficient α of the device-related edge, thereby giving priority to planning a better path for it. Conversely, for a device with a lower priority, the module can appropriately postpone the priority of its path optimization to avoid blocking the path of the high-priority device.

[0166] For example, in a traffic intersection scenario, assume that device A and device B need to pass through the same path, and the task priority of device A is higher than that of device B. The path optimization module will prioritize the optimal path for device A and appropriately adjust the path planning result of device B to ensure the smooth execution of the high-priority task.

[0167] As a technical extension, the route optimization module can combine historical route data to verify the current route planning. For example, the module can analyze the travel time distribution of a certain route in the historical data and use the result to correct the current route weight to avoid generating unreasonable route planning.

[0168] In summary, the path optimization module in this embodiment realizes real-time optimization of the target device path through the dynamic programming algorithm and the dynamic weighted graph model. In a dynamic environment, the module can adjust the path planning results in time according to the changes of nodes and edges, thereby providing an efficient passage path for the device.

[0169] In this embodiment, the scheduling optimization module optimizes the passage order and speed of the target equipment based on the optimal path provided by the path optimization module, combined with the game theory model and multi-objective optimization method, so as to ensure that the equipment can complete the task in an efficient and safe manner within the operation area. The core goal of this module is to avoid path conflicts, reduce equipment waiting time and energy consumption, and meet the requirements of equipment task priority.

[0170] It should be noted that the scheduling optimization module uses a non-cooperative game model to describe the travel decision-making problem of multiple target devices in a shared transportation network. Specifically, the target devices are regarded as participants in the game, and the travel order and speed that each device can choose constitute its strategy set S i The utility function u of the device i Used to quantify the benefits of its passage, specifically defined as:

[0171] u i (s i ,s -i )=-(α·t i +β·e i );

[0172] in:

[0173] s i is the strategy of device i, indicating its passage order or speed selection;

[0174] s -i A collection of policies for other devices;

[0175] t i is the travel time of device i;

[0176] e i is the energy consumption of device i;

[0177] α and β are weight parameters, satisfying α+β=1.

[0178] As an option, the scheduling optimization module aims to solve the Nash equilibrium point s of the game by * , determine the optimal strategy for each device The Nash equilibrium is defined as:

[0179]

[0180] It can be understood that this equilibrium point means that when the strategies of other devices are fixed, the benefit of each device reaches the optimal value under its selected strategy, thereby achieving global scheduling optimization.

[0181] In a possible implementation, the scheduling optimization module solves the Nash equilibrium by using an iterative best response algorithm (IBRA). The specific implementation includes the following steps:

[0182] Initialize the policy set S of all devices i ;

[0183] For each device i, calculate the optimal response of its current strategy when the strategies of other devices are fixed; update the strategy set of the device, and repeat the above calculation until the strategies of all devices converge to the equilibrium point.

[0184] It should be noted that in order to further improve the scheduling efficiency, the scheduling optimization module combines the multi-objective optimization method to comprehensively optimize the operating status of the equipment. In some embodiments, the module constructs the following three objective functions:

[0185]

[0186] in:

[0187] F1(x) represents the total travel time of all equipment;

[0188] F2(x) represents the total energy consumption of all devices;

[0189] F3(x) represents the total wear of all equipment;

[0190] t i 、e i 、m i are the travel time, energy consumption and equipment wear of equipment i respectively;

[0191] n is the total number of devices in the operating area.

[0192] In order to transform the multi-objective problem into a single-objective optimization problem, the scheduling optimization module constructs a comprehensive optimization objective function through the weight normalization method:

[0193] F(x)=λ1·F1(x)+λ2·F2(x)+λ3·F3(x);

[0194] in:

[0195] λ1, λ2, and λ3 are weight parameters, satisfying λ1+λ2+λ3=1.

[0196] As an option, the scheduling optimization module uses a genetic algorithm (GA) to solve the above comprehensive optimization problem. The specific implementation includes the following steps:

[0197] Initialize the device strategy population and randomly generate different travel orders and speed combinations;

[0198] Calculate the fitness value of each strategy combination, and the fitness function is F(x);

[0199] Based on the fitness value, the population is selected, crossed and mutated to generate a new generation of strategy combinations; the above iterative process is repeated until the fitness value converges to the optimal value.

[0200] Specifically, in a traffic intersection scenario, assume that there are three devices A, B, and C, whose initial priorities are 0.8, 0.6, and 0.5, respectively. The path optimization module provides the following path for each device:

[0201] Device A: Path P A , with a weight of 5;

[0202] Device B: Path P B , with a weight of 6;

[0203] Device C: Path P C , with a weight of 7.

[0204] The scheduling optimization module calculates the passage order and speed of each device based on the above weights and priorities, and ensures that the tasks of device A are executed first through game theory and multi-objective optimization, while balancing the energy consumption and passage time of devices B and C.

[0205] It should be noted that the scheduling optimization module can adjust the order and speed of equipment in real time in a dynamic environment. For example, when equipment A is stuck due to a fault, the module will immediately recalculate the optimal traffic strategy for the remaining equipment to ensure smooth traffic. In addition, the module supports dynamic adjustment of priority rules, such as reallocating the priority weight p of equipment according to the urgency of the task. i .

[0206] As a technical extension, the scheduling optimization module also supports path exclusivity constraints, which restricts multiple devices from accessing the same edge or node at the same time. This mechanism can effectively avoid conflicts in path planning among multiple devices, thereby improving the overall efficiency of the system.

[0207] In summary, the scheduling optimization module in this embodiment uses a game theory model and a multi-objective optimization method to comprehensively consider the travel time, energy consumption, and task priority of the equipment to achieve efficient scheduling optimization in a dynamic environment. The module has strong adaptability and can quickly respond to environmental changes in multi-device scenarios, providing reliable scheduling decisions for the efficient operation of the equipment.

[0208] In this embodiment, the feedback adjustment module is used to monitor the operating status of the target device in real time, and adjust the input data and optimization results of the traffic modeling module according to the dynamic changes of the operating environment to ensure the stability and adaptability of the system. This module compares the actual operation of the device with the optimization results through a closed-loop control mechanism, and updates the modeling and optimization parameters in a timely manner when deviations are found, thereby achieving efficient response to the dynamic environment.

[0209] It should be noted that the feedback adjustment module mainly includes three functional parts: real-time monitoring, anomaly detection and dynamic adjustment. Through these functions, it can effectively respond to unexpected situations in equipment operation, such as path blockage, new equipment addition or equipment failure.

[0210] As an option, the feedback adjustment module obtains the real-time position, speed, direction and other operating status of the target device through the visual perception module. Based on this data, the module can compare the actual path of the device with the optimal path P generated by the path optimization module in real time. When it is detected that the actual operating status is inconsistent with the planned result, the feedback adjustment module will trigger the anomaly detection mechanism to determine the cause of the deviation and take corresponding adjustment measures.

[0211] Specifically, the anomaly detection mechanism of the feedback adjustment module includes the following:

[0212] 1. Path blocking detection:

[0213] The module analyzes whether there are static or dynamic obstacles on the current path based on the device status data provided by the visual perception module. For example, if the occupancy time t of a node or edge is occupy Exceeding the estimated travel time of the equipment t plan , then it is considered that the path may be blocked. The detection formula is as follows:

[0214] t occupy >t plan +Δt;

[0215] Among them, Δt is the allowable time error range.

[0216] 2. Equipment fault detection:

[0217] The module compares the estimated speed of the device v plan and the actual speed v actual , to determine whether the device is faulty or stagnant. When the actual speed is lower than the preset threshold v threshold When the device is abnormal, the detection conditions are as follows:

[0218] v actual <v threshold ;

[0219] 3. New equipment detection:

[0220] When the visual perception module detects a new target device in the operating area, the feedback adjustment module will automatically add the information of the device to the node set V of the traffic modeling module and recalculate the dynamic weighted graph model.

[0221] In one possible implementation, the dynamic adjustment function of the feedback adjustment module will take corresponding optimization measures based on the abnormal detection results. For example, in the case of path blocking, the module will adjust the weight w of the blocked path. ij Dynamically set to infinity, so that the path optimization module regenerates a feasible path. In the event of equipment failure, the module removes the faulty equipment from the current optimization calculation and notifies relevant personnel for maintenance.

[0222] As an option, the feedback adjustment module can also dynamically adjust the weight parameters in the traffic modeling module. For example, when the equipment priority changes, the module recalculates the edge weight w ij To ensure that the path planning and scheduling results can meet the latest task requirements. The weight adjustment formula is as follows:

[0223] w ij =α′·t ij +β′·p i ;

[0224] It should be noted that the feedback adjustment module is not only applicable to the dynamic adjustment of a single device, but also can handle conflicts in the parallel optimization of multiple devices. In some embodiments, the module monitors the path conflicts of multiple devices in real time. For example, when two devices try to access the same node or edge at the same time, the module adjusts the access order or path of one of the devices to avoid conflicts.

[0225] For example, in a traffic intersection scenario, assume that device A and device B plan to pass through node v at the same time. k , the feedback adjustment module will give priority to adjusting the path planning of low-priority device B, or reduce the travel speed v of device B B to ensure the normal passage of high-priority equipment A.

[0226] It is understandable that the real-time performance of the feedback adjustment module is critical to system performance. In some embodiments, the module achieves rapid response to large-scale traffic networks through parallel computing and edge computing technology. For example, the module can directly complete preliminary path occupancy detection on the camera side, thereby reducing the delay of data transmission.

[0227] As a technical extension, the feedback adjustment module also supports the analysis and use of historical data. For example, the module can record the frequency, location distribution and impact range of abnormal events, and optimize the initial weight of the traffic modeling module based on these historical data, thereby reducing the probability of abnormal events in the future.

[0228] In summary, the feedback adjustment module in this embodiment ensures the stability and adaptability of the system in a dynamic environment through real-time monitoring, anomaly detection and dynamic adjustment functions. The module can not only quickly respond to various unexpected situations in the operation of the equipment, but also optimize future operation strategies through historical data, thereby providing reliable guarantee for the efficient operation of the system.

[0229] In general, the present invention obtains the dynamic status of the equipment in the operating area in real time through the visual perception module, and constructs a dynamic weighted graph model in combination with the traffic modeling module to describe the equipment's passage path and status. The system calculates the optimal path based on the dynamic Hamiltonian path problem through the path optimization module, and optimizes the passage sequence and speed of the equipment by using the game theory model and multi-objective optimization method in combination with the scheduling optimization module. Finally, the feedback adjustment module monitors the equipment operation in real time, and dynamically adjusts the modeling and optimization results. The system can effectively solve the problems of path conflicts, traffic congestion and uneven resource allocation in the collaborative operation of multiple devices. It is suitable for various scenarios such as industrial logistics and warehouse management, and has the characteristics of high efficiency, dynamics and adaptability.

[0230] Please see attached Figure 2 The present invention also provides a vision-based operation management method for dynamic management and optimization of multi-target devices in the operation area. The method comprises the following steps: obtaining the dynamic state of the device through a visual device, optimizing the path and scheduling based on a dynamic weighted graph model, and realizing closed-loop optimization of the system through real-time monitoring and feedback mechanism. The specific implementation of this method is described in detail below in conjunction with the workflow of the aforementioned system.

[0231] S1. Obtain image data of the operating area through visual equipment and extract the dynamic state of the target device using target detection algorithm;

[0232] In step S1, image or video data is acquired through visual devices deployed in the operation area, and the target device is identified and tracked using the target detection algorithm. The extracted dynamic state includes the current position, speed, direction and task target node of the target device. The device state data is the basis for subsequent modeling and optimization, and can be updated in real time to reflect the latest operation status of the device.

[0233] S2. constructing a dynamic weighted graph model according to the device status data;

[0234] In step S2, a dynamic weighted graph model is constructed based on the extracted device status data. The model uses nodes to represent the current location and target location of the device, and edges to represent the traversable paths between nodes, and assigns weights based on travel time and priority to the edges. The dynamic weighted graph model can describe the traffic conditions in the operating area and provide structured data support for subsequent path optimization. When the operating environment or device status changes, the model will be updated in real time to maintain its accuracy and effectiveness.

[0235] S3, based on the dynamic weighted graph model, solve the dynamic Hamiltonian path problem and generate the optimal path to the target device;

[0236] In step S3, the optimal path of the target device is generated by solving the dynamic Hamiltonian path problem. This path is based on the dynamic weighted graph model and combines the task requirements and operating environment of the device to calculate the minimum weight path from the starting point to the target node. The path optimization result will be used as the input of subsequent scheduling optimization to provide an efficient path planning solution for the operation of the device.

[0237] S4. Based on the game theory model, optimize the passage order and speed of the target equipment and generate equipment passage instructions;

[0238] In step S4, the passage order and speed of the equipment are optimized based on the game theory model. Each device generates the best passage strategy according to its own task requirements, path weight and priority. The global optimal strategy is achieved through game theory solution among multiple devices to ensure that path conflicts are avoided and efficiency is improved in the collaborative operation environment of multiple devices. At the same time, combined with the multi-objective optimization method, multiple operating indicators such as the equipment's passage time, energy consumption and wear are balanced, so as to further optimize the equipment's operating status.

[0239] S5. Real-time monitoring of equipment operation status, detection of abnormal conditions during operation, and dynamic adjustment of traffic modeling and optimization results;

[0240] In step S5, real-time monitoring and dynamic adjustment of the operating status are achieved through the feedback adjustment mechanism. The feedback adjustment module compares the actual operating status of the equipment with the path optimization and scheduling optimization results. If an abnormal situation is detected (such as path blocking, equipment failure, or new equipment added), it triggers the dynamic update of modeling and optimization results. For example, when the path is blocked, the system will adjust the edge weights of the dynamic weighted graph model or regenerate the optimal path; when the equipment fails, the faulty equipment will be removed and the tasks will be reallocated to ensure the continuity and stability of the system operation.

[0241] It should be noted that this method adopts a closed-loop control mode, and a dynamic cycle of data flow is formed between each step. When the environment or device status changes, the system can adjust the input and output of each module in time through the feedback mechanism to adapt to the changing operating conditions and maintain the efficiency and adaptability of the system.

[0242] In summary, the vision-based operation management method in this embodiment can effectively solve the path conflict and traffic congestion problems in the operation of multiple devices by implementing dynamic modeling, path optimization and scheduling optimization of equipment in steps, and combining real-time monitoring and feedback adjustment. This method is applicable to a variety of complex operation scenarios and has strong scalability and robustness.

[0243] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A vision-based operation management system, characterized in that: include: Visual perception module, used to obtain image or video data in the operating area and identify the real-time dynamic status of the target device, including position, speed, direction and target position; The traffic modeling module is used to construct a dynamic weighted graph model according to the equipment status data provided by the visual perception module, wherein the dynamic weighted graph model includes a plurality of nodes and edge weights, wherein: Nodes are used to represent the current and target locations of the device; Edges are used to represent possible paths between devices. The weight of an edge is determined based on the device's travel time and priority. The weight of an edge is calculated using the following formula: w ij =α·t ij +β·p i ; Among them, w ij represents the edge weight from node i to node j; where t ij represents the travel time of the device from node i to node j, d ij is the distance from node i to node j, v i is the current speed of the device; p i Indicates the priority of the device; α and β are weight coefficients, satisfying α+β=1; The traffic modeling module updates the status of nodes and edges in real time, including the addition of new equipment, path congestion, or changes in equipment speed, and recalculates the weights in the dynamic weighted graph model based on the updated equipment status; The path optimization module is used to solve the optimal path of the device from the starting point to the target node based on the dynamic weighted graph model. The path optimization module describes the path optimization problem through the following mathematical model: Assume that path P represents the node access sequence of the target device, and the total weight C(P) of the path is defined as: in: P = {v1, v2, ..., v k } represents a path node sequence; v i represents the iiith node in the path; Represents the slave node v i To node v i+1 The edge weight of The scheduling optimization module is used to optimize the passage sequence and speed of the equipment according to the optimal path provided by the path optimization module. The scheduling optimization module optimizes the passage sequence and speed of the target equipment based on a non-cooperative game model, wherein: Scheduling optimization is based on a non-cooperative game model, with the target device as the participant in the game. Each device has an optional strategy set S i , indicating the passage sequence or speed selection of the equipment; Define the utility function u i (s i ,s -i ), indicating that the target device i is selecting strategy s i And other devices select strategies -i The benefit when , the utility function is: you i (s i ,s -i )=-(α·t i +β·e i ); Among them, t i represents the travel time of device i; e i represents the energy consumption of device i; α and β are adjustable weight parameters, satisfying α+β=1; s i is the selection strategy for device i; s -i A selection policy set for other devices; The scheduling optimization module solves the Nash equilibrium point s of the game * , determine the optimal strategy of the device, the Nash equilibrium satisfies the following conditions: in, represents the optimal strategy of device i; represents the optimal strategy set for other devices; The feedback adjustment module is used to monitor the execution status of the equipment in real time and dynamically adjust the input data and optimization results of the traffic modeling module.

2. The vision-based operation management system according to claim 1, characterized in that: The visual perception module identifies the dynamic state of the target device based on a deep learning target detection algorithm, wherein: The target equipment includes AGV forklifts, unmanned guided vehicles and mobile robots; The dynamic state includes the spatial coordinates, driving speed, driving direction and target node of the device.

3. The vision-based operation management system according to claim 1, characterized in that: The path optimization module is used to dynamically optimize the passage path of the equipment based on the dynamic weighted graph model and solve the dynamic Hamiltonian path problem, where: The dynamic Hamiltonian path problem aims to minimize the total weight of all nodes in the dynamic weighted graph model. The total weight is calculated by the following formula: Where C(P) represents the total weight of path P; P = {v1, v2, ..., v k } represents the node sequence in the path; v i represents the i-th node in the path; Represents the slave node v i To node v i+1 The edge weight of Dynamic path optimization is solved based on a dynamic programming method, which includes: Define the subproblem, let D(S,v i ) means starting from the starting point, passing through the node set S and ending at node v i is the shortest path length to the end point; Construct a recursive relation: Among them, D(S,v i ) represents the shortest path length passing through the node set S and ending at node vi; v j represents a node in the node set S; S\{v i } means removing node v from set S i The collection after Represents the slave node v j To node v i The edge weight of Initial conditions: D({v1}, v1) = 0; Where {v1} represents the node set including the starting node v1; D({v1},v1) represents the shortest path length starting from the starting node v1 and passing only through the node v1; The optimal solution is: Where V represents the set of all nodes in the dynamic weighted graph model; D(V, v i ) means starting from the starting point, passing through all nodes V and ending at node v i is the shortest path length to the end point; Represents the slave node v i Return the edge weight of the starting point v1; The path optimization module updates the path planning results in real time according to the changes in the dynamic weighted graph model, and re-solves the dynamic Hamiltonian path problem when new nodes are added or the path is blocked to ensure the dynamic optimality of the passage path.

4. The vision-based operation management system according to claim 1, characterized in that: The scheduling optimization module optimizes the passage order and speed of the target equipment based on a non-cooperative game model, wherein: Scheduling optimization is based on a non-cooperative game model, with the target device as the participant in the game. Each device has an optional strategy set S i , indicating the passage sequence or speed selection of the equipment; Define the utility function u i (s i ,s -i ), indicating that the target device i is selecting strategy s i And other devices select strategies -i The benefit when , the utility function is: you i (s i ,s -i )=-(α·t i +β·e i ); Among them, t i represents the travel time of device i; e i represents the energy consumption of device i; α and β are adjustable weight parameters, satisfying α+β=1; s i is the selection strategy for device i; s -i A selection policy set for other devices; The scheduling optimization module solves the Nash equilibrium point s of the game * , determine the optimal strategy of the device, the Nash equilibrium satisfies the following conditions: in, represents the optimal strategy of device i; Represents the optimal strategy set for other devices.

5. The vision-based operation management system according to claim 4, characterized in that: The scheduling optimization module uses an iterative optimal response algorithm to solve the Nash equilibrium, which specifically includes the following steps: Initialize the policy set for all target devices; For each device, calculate its optimal strategy when the strategies of other devices are fixed; Repeatedly update the strategy of each device until it converges to the Nash equilibrium point s * .

6. The vision-based operation management system according to claim 1, characterized in that: The scheduling optimization module combines a multi-objective optimization method to comprehensively optimize the operating efficiency of the target equipment, where: The goal of scheduling optimization is to simultaneously minimize the following objective functions: Among them, F1(x) represents the total travel time of all target devices; F2(x) represents the total energy consumption of all target devices; F3(x) represents the total wear of all target devices; t i 、e i 、m i They represent the travel time, energy consumption and equipment wear of target equipment i respectively; n represents the number of target equipment; The multi-objective optimization is converted into a single-objective optimization problem by normalizing the weights, and its optimization function is: F(x)=λ1·F1(x)+λ2·F2(x)+λ3·F3(x); Wherein, F(x) represents the comprehensive optimization objective function; λ1, λ2, and λ3 are weight parameters of the objective function, satisfying λ1+λ2+λ3=1.

7. The vision-based operation management system according to claim 6, characterized in that: The scheduling optimization module solves the optimization problem through a genetic algorithm, which specifically includes the following steps: Initialize the population and randomly generate the passage order and speed strategy of the equipment; Select the population according to the fitness function F(x); Generate new strategy populations through crossover and mutation operations; Repeat the iteration until the fitness function converges to obtain the optimal strategy for comprehensive optimization.

8. The vision-based operation management system according to claim 1, characterized in that: The feedback adjustment module detects the following abnormal conditions and triggers dynamic adjustment by visually monitoring the operating status of the equipment: path occupation or blockage; The addition of new equipment; Equipment operation failure.

9. A vision-based operation management method, applied to a system as claimed in any one of claims 1 to 8, characterized in that: The following steps are involved: The image data of the operating area is obtained through visual equipment, and the dynamic state of the target device, including position, speed, direction and target node, is extracted using the target detection algorithm; Based on the device status data, a dynamic weighted graph model is constructed, wherein nodes represent the current location and target location of the device, and weights are based on the travel time and priority of the device; Based on the dynamic weighted graph model, the dynamic Hamiltonian path problem is solved to generate the optimal path to the target device; Based on the game theory model, the passage order and speed of the target equipment are optimized and the equipment passage instructions are generated; Monitor equipment operating status in real time, detect abnormal conditions during operation, and dynamically adjust traffic modeling and optimization results.

Images