Vision-based operation management system

Abstract

The invention relates to the field of industrial automation and intelligent logistics, and discloses a vision-based operation management system, which comprises a visual perception module, a traffic modeling module, a path optimization module, a scheduling optimization module and a feedback adjustment module, the visual perception module obtains image or video data in the operation area and extracts the dynamic state of the target equipment; the traffic modeling module constructs a dynamic weighted graph model based on the equipment state data; the path optimization module generates an optimal path of the equipment by solving a dynamic Hamiltonian path problem; the scheduling optimization module optimizes the passing sequence and speed of the equipment based on a game theory model; and the feedback adjustment module monitors the running state of the equipment in real time, detects abnormal conditions and dynamically adjusts modeling and optimization results. The method can effectively solve the problems of path conflict and traffic jam in multi-device collaborative operation, improves the operation efficiency and the resource utilization rate, has high real-time performance and adaptability, and is suitable for various scenes such as logistics storage, industrial production and intelligent transportation.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial automation and intelligent logistics, and in particular to a vision-based operation management system. Background Art

[0002] With the development of industrial automation and intelligence, unmanned handling equipment (such as AGV forklifts, unmanned transport vehicles and logistics robots) has been widely used in logistics warehousing, smart factories and other fields. These devices can efficiently complete operations such as material transportation, loading and unloading and inventory management through automatic navigation, path planning and task scheduling. However, in the complex environment of multi-device collaborative operation, the traditional operation management system still has many shortcomings in dynamic adaptability, real-time optimization and multi-objective collaboration.

[0003] In the existing technology, most operation management systems rely on path planning and task scheduling solutions based on static rules. This method usually assumes that the equipment operation environment is static, that is, the node location, the passage path and the equipment status do not change during the planning cycle. However, in actual operation scenarios, the equipment status and environmental conditions are often dynamically changing. For example, path blocking, new equipment joining or equipment failure will cause the established path and scheduling strategy to be unable to be effectively executed. The existing technology lacks the ability to respond quickly to such dynamic environments, resulting in low equipment operation efficiency and even large-scale congestion or task delays.

[0004] In addition, traditional path planning methods are mostly optimized with a single device as the center, ignoring the resource contention problem in the coordinated operation of multiple devices. For example, when multiple devices share the same path or node, the lack of an effective coordination mechanism can easily lead to path conflicts or unfair priority allocation. Especially in scenarios with high-density device operation, this defect can significantly reduce the system's operating efficiency and device utilization.

[0005] On the other hand, existing technologies also have limitations in multi-objective optimization. Most systems focus only on a single objective (such as the shortest path or the lowest energy consumption), while ignoring the multiple requirements of equipment operation, such as the balance of comprehensive indicators such as travel time, task priority, energy consumption, and equipment wear. In addition, due to the lack of a closed-loop feedback mechanism, it is difficult for existing systems to monitor equipment status in real time and adjust optimization strategies during operation, and it is impossible to maintain the stability and adaptability of the system in a changing operating environment. Summary of the invention

[0006] In view of the deficiencies of the prior art, the present invention provides a vision-based operation management system, which solves the problems of path conflicts, insufficient adaptability to dynamic environments, and difficulty in balancing multi-objective optimization in a multi-device collaborative operation environment.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a vision-based operation management system, comprising: Visual perception module, used to obtain image or video data in the operating area and identify the real-time dynamic status of the target device, including position, speed, direction and target position; A traffic modeling module, used to construct a dynamic weighted graph model based on the equipment status data provided by the visual perception module; A path optimization module, used to solve the optimal path of the device from the starting point to the target node based on the dynamic weighted graph model; A scheduling optimization module, used to optimize the passage sequence and speed of the equipment according to the optimal path provided by the path optimization module; The feedback adjustment module is used to monitor the execution status of the equipment in real time and dynamically adjust the input data and optimization results of the traffic modeling module.

[0008] Preferably, the visual perception module identifies the dynamic state of the target device based on a deep learning target detection algorithm, wherein: The target equipment includes AGV forklifts, unmanned guided vehicles and mobile robots; The dynamic state includes the spatial coordinates, driving speed, driving direction and target node of the device.

[0009] Preferably, the dynamic weighted graph model includes a plurality of nodes, edges and weights, wherein: Nodes are used to represent the current and target locations of the device; Edges are used to represent possible paths between devices, and the weight of the edge is determined based on the device's travel time and priority; The edge weight is calculated by the following formula: ; in, Represents a slave node To Node The edge weight of ,in Indicates the device slave node To Node The travel time, For device slave nodes To Node The distance is the current speed of the device; Indicates the priority of the device; and is the weight coefficient, satisfying ; The traffic modeling module updates the status of nodes and edges in real time, including the addition of new equipment, path congestion, or changes in equipment speed, and recalculates the weights in the dynamic weighted graph model based on the updated equipment status.

[0010] Preferably, the path optimization module is used to dynamically optimize the passage path of the equipment based on a dynamic weighted graph model and solve the dynamic Hamiltonian path problem, wherein: The dynamic Hamiltonian path problem aims to minimize the total weight of all nodes in the dynamic weighted graph model. The total weight is calculated by the following formula: ; in, Indicates the path The total weight of Represents the sequence of nodes in a path; Indicates the first nodes; Represents a slave node To Node The edge weight of Dynamic path optimization is solved based on a dynamic programming method, which includes: Define the subproblem, let Indicates that the node set is passed from the starting point And with node is the shortest path length to the end point; Construct a recursive relation: ; in, Represents a set of nodes And with node is the shortest path length to the end point; Represents a node collection A node in Represents from the set Remove a node The collection after Represents a slave node To Node The edge weight of Initial conditions: ; in, Indicates that the starting point node is included A collection of nodes; From the starting point Departure and only pass through nodes The shortest path length; The optimal solution is: ; in, Represents the set of all nodes in the dynamic weighted graph model; It means starting from the starting point and passing through all nodes And with node is the shortest path length to the end point; Represents a slave node Return to starting point The edge weight of The path optimization module updates the path planning results in real time according to the changes in the dynamic weighted graph model, and re-solves the dynamic Hamiltonian path problem when new nodes are added or the path is blocked to ensure the dynamic optimality of the passage path.

[0011] Preferably, the scheduling optimization module optimizes the passage order and speed of the target equipment based on a non-cooperative game model, wherein: Scheduling optimization is based on a non-cooperative game model, with the target device as the participant in the game, and each device has an optional set of strategies. , indicating the passage sequence or speed selection of the equipment; Defining the utility function , indicating the target device In choosing strategy And other device selection strategies The benefit when , the utility function is: ; in, Indicates the device The travel time; Indicates the device Energy consumption; and is an adjustable weight parameter, satisfying ; For equipment selection strategy; A selection policy set for other devices; The scheduling optimization module solves the Nash equilibrium point of the game , determine the optimal strategy of the device, the Nash equilibrium satisfies the following conditions: ; in, Indicates the device The optimal strategy for Represents the optimal strategy set for other devices.

[0012] Preferably, the scheduling optimization module adopts an iterative optimal response algorithm to solve the Nash equilibrium, which specifically includes the following steps: Initialize the policy set for all target devices; For each device, calculate its optimal strategy when the strategies of other devices are fixed; Repeatedly update the strategy of each device until it converges to the Nash equilibrium point .

[0013] Preferably, the scheduling optimization module combines a multi-objective optimization method to comprehensively optimize the operating efficiency of the target equipment, wherein: The goal of scheduling optimization is to simultaneously minimize the following objective functions: ; in, Indicates the total transit time of all target devices; Represents the total energy consumption of all target devices; Represents the total wear of all target devices; , , Respectively represent the target device travel time, energy consumption and equipment wear; Indicates the number of target devices; The multi-objective optimization is converted into a single-objective optimization problem by normalizing the weights, and its optimization function is: ; in, represents the comprehensive optimization objective function; , , is the weight parameter of the objective function, satisfying .

[0014] Preferably, the scheduling optimization module solves the optimization problem by means of a genetic algorithm, specifically comprising the following steps: Initialize the population and randomly generate the passage order and speed strategy of the equipment; According to the fitness function Selecting a population; Generate new strategy populations through crossover and mutation operations; Repeat the iteration until the fitness function converges to obtain the optimal strategy for comprehensive optimization.

[0015] Preferably, the feedback adjustment module detects the following abnormal conditions and triggers dynamic adjustment by visually monitoring the operating status of the device: Path occupation or blockage; The addition of new equipment; Equipment operation failure.

[0016] The present invention also provides a vision-based operation management method, comprising the following steps: The image data of the operating area is obtained through visual equipment, and the dynamic state of the target device, including position, speed, direction and target node, is extracted using the target detection algorithm; Based on the device status data, a dynamic weighted graph model is constructed, wherein nodes represent the current location and target location of the device, and weights are based on the travel time and priority of the device; Based on the dynamic weighted graph model, the dynamic Hamiltonian path problem is solved to generate the optimal path to the target device; Based on the game theory model, the passage order and speed of the target equipment are optimized and the equipment passage instructions are generated; Monitor equipment operating status in real time, detect abnormal conditions during operation, and dynamically adjust traffic modeling and optimization results.

[0017] The present invention provides a vision-based operation management system, which has the following beneficial effects: 1. The present invention realizes the global optimization of target equipment from path planning to traffic scheduling by solving dynamic weighted graph model and dynamic Hamiltonian path problem, combined with game theory model and multi-objective optimization method. The equipment can perform tasks according to the optimal path and traffic sequence in a complex dynamic environment, effectively reducing path conflicts, equipment waiting time and operating energy consumption, and improving overall operating efficiency.

[0018] 2. The feedback adjustment module of the present invention can monitor the operating status of the equipment in real time, and dynamically adjust the traffic modeling and optimization results by detecting abnormal situations (such as path blockage, equipment failure or new equipment), thereby ensuring the stability and continuity of the system in a changing environment. Compared with the traditional static scheduling method, the present invention can respond to environmental changes more quickly and ensure the normal operation of the equipment.

[0019] 3. The present invention solves the traffic conflict problem of multiple devices in a shared transportation network by introducing a game theory model in the scheduling optimization module. The system can optimize the traffic order according to the device priority and task requirements, avoiding resource contention and path congestion, thereby improving the efficiency and fairness of multi-device collaborative operations.

[0020] 4. The architecture design and method flow of the present invention can adapt to a variety of operating scenarios, including industrial logistics, warehouse management, intelligent transportation, etc. By flexibly adjusting model parameters (such as weight coefficients, priority rules) and supporting multi-objective optimization, the present invention can be extended to operating environments of different scales and complexities, and has good versatility and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the system architecture of the present invention; Figure 2 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Please see attached Figure 1 The vision-based operation management system of the present invention is used to dynamically manage and optimize the operation status of target equipment such as AGV forklifts and mobile robots in logistics, warehousing and industrial environments. Based on real-time visual perception data, the system achieves efficient operation of equipment through dynamic modeling, path planning and scheduling optimization, and realizes closed-loop optimization through feedback mechanism.

[0024] like Figure 1 As shown, the vision-based operation management system may include the following modules: Visual perception module: responsible for real-time perception of device dynamic information.

[0025] Traffic modeling module: converts equipment status data into a dynamic weighted graph model of the traffic network.

[0026] Path optimization module: calculates the optimal travel path for devices in the transportation network.

[0027] Scheduling optimization module: optimizes the passage sequence and speed of equipment.

[0028] Feedback adjustment module: monitor the equipment operation status in real time and dynamically adjust traffic modeling and optimization results.

[0029] The various modules of the system of the present invention are described in detail below.

[0030] In this embodiment, the visual perception module is used to obtain image or video data in the operating area and extract the dynamic state of the target device in real time, including position, speed, direction and target position. This module is implemented based on industrial cameras and computer vision algorithms, and can provide accurate input data for subsequent traffic modeling, path optimization and scheduling optimization.

[0031] As an option, the visual perception module acquires panoramic images or video streams by arranging multiple industrial-grade high-precision cameras in the operating area. These cameras can be fixedly installed in key areas, such as traffic intersections, loading and unloading points, or above the work area to cover the main activity range of the operating equipment. The cameras are connected to the central processing unit via the network to transmit and process image data in real time.

[0032] Specifically, the visual perception module uses a deep learning target detection algorithm (such as YOLOv8) to detect and track the target device in the video stream. In some embodiments, the target device can be an AGV forklift, an unmanned transport vehicle, or other mobile robots. The algorithm can extract key parameters of the target device from the image, including: Position, specifically the two-dimensional coordinates of the device , obtained by geometrically transforming the pixel position of the device in the image; Speed, specifically the current speed of the device , determined by calculating the rate of change of the device position in consecutive frames; Direction, specifically the device's driving direction angle , determined by analyzing the motion trajectory of the device in the image; Target location, specifically the target node location of the device's current task ,This information is usually obtained by interacting with the device's task scheduling system.

[0033] It should be noted that in order to improve the accuracy of target detection, the visual perception module can combine the background modeling method to eliminate the influence of static obstacles or fixed facilities, so as to more accurately identify dynamic target devices.

[0034] In one possible implementation, the visual perception module performs preprocessing operations on the original image after acquiring it. The preprocessing steps include grayscale, denoising, and scale normalization to improve the algorithm's robustness to different lighting and device colors. The preprocessed image is input into the object detection algorithm to generate a detection box for each device and its category information.

[0035] As an option, target detection is subsequently combined with a device tracking algorithm (such as a Kalman filter or SORT algorithm) to continuously track the dynamic changes of the device. Through the tracking algorithm, the visual perception module can associate the detection results of the same device in different frames, reducing the detection loss caused by occlusion or rapid movement of the device.

[0036] It should be noted that the visual perception module is not limited to target detection and tracking, but can also perform extended analysis on the acquired image data. For example, in some embodiments, the environment around the target device can be analyzed through an instance segmentation algorithm to extract the obstacle distribution or road status in the area, thereby providing more information for subsequent traffic modeling.

[0037] Exemplarily, the output of the visual perception module can be represented as the following device state set: ; in: Indicates The two-dimensional coordinates of a device; Indicates The speed of each device; Indicates The orientation angle of each device; Indicates The target location of each device; Indicates the total number of devices in the current operating area.

[0038] It is understandable that the parameters in the device status set are updated in real time through the visual perception module, and the update frequency depends on the frame rate of the camera and the processing power of the algorithm. As an optimization, the module can be combined with edge computing technology to complete preliminary data processing on the camera side to reduce the computing burden of the central processing unit.

[0039] In one possible implementation, the visual perception module also supports marking abnormal states of detection and tracking results, such as device stagnation, abnormal motion trajectory, or device loss. In this case, the module will pass the abnormal state information to subsequent modules to trigger the system's adjustment mechanism.

[0040] It should be noted that in order to adapt to a variety of operating environments, the visual perception module can integrate multi-spectral cameras (such as infrared cameras and RGB cameras) to enhance detection performance under low light or complex background conditions. In low light environments, the infrared camera can provide thermal images of the device, while the RGB camera is used for target detection under normal lighting. The module combines the outputs of the two to improve detection reliability.

[0041] In some embodiments, the visual perception module can support the management capabilities of a multi-target environment. For example, when there are multiple equipment targets in an operating area, the module can output a set of equipment states in groups and assign a unique identification code to each group of equipment to facilitate traffic modeling and path planning operations in subsequent modules.

[0042] In this embodiment, the visual perception module also supports real-time data storage to save historical detection data. These historical data can be used for retrospective analysis of abnormal conditions or provide training data for continuous optimization of machine learning models.

[0043] It should be further explained that the camera layout and algorithm selection in the visual perception module can be adjusted according to the actual scenario. For example, at a traffic intersection, the camera can cover multiple directions of the intersection, while in a narrow passage, the camera can be placed at both ends of the passage to ensure that the device is visible throughout the entire process.

[0044] To sum up, the visual perception module in this embodiment uses high-precision cameras and deep learning algorithms to perceive the status of target equipment in the operating area in real time, and provides the results to subsequent modules in the form of structured data, laying a data foundation for realizing dynamic optimization and closed-loop adjustment of the system.

[0045] In this embodiment, the traffic modeling module constructs a dynamic weighted graph model in the operation area based on the device status data provided by the visual perception module to describe the traffic status of the target device in the traffic network. Through this dynamic weighted graph model, it can provide structured data support for the subsequent path optimization module and dynamically reflect the real-time status of the equipment and traffic flow changes in the operation area.

[0046] It should be noted that the dynamic weighted graph model constructed by the traffic modeling module By node collection , edge set and weight set Composition, specifically Nodes are used to represent the current location of a device and its target location, edges are used to represent possible paths between nodes, and weights are used to quantify the cost of the paths.

[0047] Alternatively, the node collection Including the current position of the device and the target position. The current position of the device is provided in real time by the visual perception module and expressed as a two-dimensional coordinate , target location The task data is obtained through the task scheduling system or the local task data of the device. Each node in the node set can be marked with a unique identifier to facilitate the traffic modeling module to track and update the status of the node.

[0048] Specifically, the edge set Used to describe possible paths between nodes. The definition of an edge is based on the feasible movement range of the equipment in the operating area. For example, in a warehousing and logistics scenario, an edge can represent the passable relationship between two channels. In a dynamic environment, the edge set will be updated in real time as the equipment status or environment changes (such as new obstacles, occupied paths, etc.).

[0049] In one possible implementation, the edge weights Represents a slave node To Node The travel cost is mainly composed of travel time and priority. The calculation formula is as follows: ; in: , indicating that the device is a slave node To Node The travel time; For Node To Node The Euclidean distance is given by the formula calculate; is the current speed of the device; Indicates the priority of the equipment, determined by task urgency or equipment type; and is an adjustable weight coefficient, satisfying .

[0050] For example, in the modeling of a traffic intersection, if device A moves from the current position (2,3) to the target node (5,7), the device speed , priority , then the travel time of the path is: ; Assuming weight coefficient , , then the weight for: ; It is understandable that the weight The calculation can dynamically reflect the operating status and task requirements of the equipment. or priority When changes occur, weights are adjusted in real time, ensuring that modeling results accurately reflect current traffic conditions.

[0051] In one possible implementation, the traffic modeling module stores the dynamic weighted graph model in the form of an adjacency matrix to quickly query the path relationship between nodes. Corresponding to the node and nodes The weight between When the running environment changes (such as adding new nodes or edges), the adjacency matrix will be dynamically updated.

[0052] As an option, the traffic modeling module can be extended to support obstacle modeling. For example, by analyzing the image data provided by the visual perception module, the traffic modeling module can identify fixed obstacles (such as shelves) or temporary obstacles (such as stacked goods) on the equipment's driving path. These obstacles are represented as non-traversable nodes in the graph model, effectively preventing the path optimization module from generating an infeasible path.

[0053] It should be noted that the real-time and accuracy of the traffic modeling module are crucial to the overall performance of the system. In some embodiments, the module implements rapid modeling of large traffic networks through distributed computing. For example, in an operating area containing hundreds of devices, the module can distribute traffic modeling tasks to multiple computing units based on regional divisions, and maintain the consistency of modeling results through boundary data sharing.

[0054] In one possible implementation, the traffic modeling module supports a mechanism for handling abnormal conditions to adapt to dynamic environments. For example, when a path is occupied or the speed of a device drops suddenly, the module dynamically adjusts the weight of the corresponding edge or marks the edge as inaccessible to reflect the change in a timely manner. This mechanism ensures that the path generated by the path optimization module is always feasible and efficient.

[0055] As a technical extension, the traffic modeling module also supports the accumulation and analysis of historical data. For example, the module can record statistical information such as the frequency of use of each edge and the distribution of travel time. These historical data can not only be used to optimize future path planning, but also help identify potential traffic bottleneck areas.

[0056] In summary, the traffic modeling module in this embodiment provides a structured traffic network description for the system by constructing a dynamic weighted graph model. By updating the status of nodes, edges and weights in real time, the module can dynamically reflect the traffic conditions in the operating area and provide reliable data support for subsequent path optimization and scheduling optimization modules.

[0057] In this embodiment, the path optimization module is used to generate a dynamic weighted graph model based on the traffic modeling module. , calculate the optimal path from the starting point to the target node for the target device. The core goal of path optimization is to minimize the total weight of the device passing through the traffic network, ensuring that the path selection not only meets the device operation requirements, but also can adapt to traffic flow changes in a dynamic environment.

[0058] It should be noted that the path optimization module is based on the dynamic Hamiltonian path problem (HPP) and uses a dynamic programming algorithm to optimize the device path in real time. The goal of the dynamic Hamiltonian path problem is to find the optimal path that traverses all nodes in the graph so that the total weight of the path is minimized.

[0059] In a possible implementation, the path optimization module describes the path optimization problem through the following mathematical model. Indicates the node access sequence of the target device and the total weight of the path Defined as: ; in: Represents a sequence of path nodes; represents the iiith node in the path; Represents a slave node To Node The edge weight of .

[0060] It is understandable that the path optimization module needs to make the total weight of the path Minimize. This optimization goal ensures that the equipment completes the task in the most economical way.

[0061] Specifically, in order to solve the dynamic Hamiltonian path problem, the path optimization module in this embodiment adopts a dynamic programming algorithm. The dynamic programming algorithm gradually builds a global optimal solution by recursively solving sub-problems. The following are the specific implementation steps of the dynamic programming algorithm: As an option, define subproblems , which means starting from the starting point and passing through the node set And the node is the shortest path length to the end point. The recursive relation can be expressed as: ; in: The node set visited by the current path; is the current endpoint node; For collection A node in For slave nodes To Node The edge weight of .

[0062] The initial conditions are defined as: ; in, represents a node set containing only the starting point, Indicates that the path length from the starting point to itself is zero.

[0063] For example, the total weight of the optimal path is It can be calculated by the following formula: ; in: Represents the set of all nodes in the dynamic weighted graph model; From the last node Return to starting point The edge weight of .

[0064] In one possible implementation, the path optimization module solves the above recursive relationship in real time and dynamically updates the optimal path during the operation of the device. For example, when a new node is added to the operation area or the weight of an edge changes, the module recalculates the dynamic weighted graph model. The optimal path in .

[0065] It should be noted that in order to improve computing efficiency, the path optimization module can combine pruning strategies to pre-empt nodes or paths that do not meet constraints. For example, if the weight of an edge exceeds the current task time budget of the device, the module will directly discard the path branch containing the edge. This pruning strategy can significantly reduce the amount of calculation and improve the real-time performance of path optimization.

[0066] As an option, the path optimization module supports parallel path optimization for multiple devices. In this case, the path optimization problem for each device is modeled as an independent dynamic Hamiltonian path problem, and the optimal path for each device is calculated in parallel. In addition, to avoid path conflicts, the module can introduce path mutual exclusion constraints, that is, restrict multiple devices from accessing the same node or edge at the same time, thereby ensuring the consistency of path optimization results on a global scale.

[0067] In some embodiments, the path optimization module also supports dynamic weight adjustment. For example, for a device with a higher priority, the module can dynamically increase the weight coefficient of the edge associated with the device. , so as to prioritize the planning of better paths for them. Conversely, for devices with lower priority, the module can appropriately postpone the priority of their path optimization to avoid blocking the paths of high-priority devices.

[0068] For example, in a traffic intersection scenario, assume that device A and device B need to pass through the same path, and the task priority of device A is higher than that of device B. The path optimization module will prioritize the optimal path for device A and appropriately adjust the path planning result of device B to ensure the smooth execution of the high-priority task.

[0069] As a technical extension, the route optimization module can combine historical route data to verify the current route planning. For example, the module can analyze the travel time distribution of a certain route in the historical data and use the result to correct the current route weight to avoid generating unreasonable route planning.

[0070] In summary, the path optimization module in this embodiment realizes real-time optimization of the target device path through the dynamic programming algorithm and the dynamic weighted graph model. In a dynamic environment, the module can adjust the path planning results in time according to the changes of nodes and edges, thereby providing an efficient passage path for the device.

[0071] In this embodiment, the scheduling optimization module optimizes the passage order and speed of the target equipment based on the optimal path provided by the path optimization module, combined with the game theory model and multi-objective optimization method, so as to ensure that the equipment can complete the task in an efficient and safe manner within the operation area. The core goal of this module is to avoid path conflicts, reduce equipment waiting time and energy consumption, and meet the requirements of equipment task priority.

[0072] It should be noted that the scheduling optimization module uses a non-cooperative game model to describe the travel decision-making problem of multiple target devices in a shared transportation network. Specifically, the target devices are regarded as participants in the game, and the travel order and speed that each device can choose constitute its strategy set. The utility function of the device Used to quantify the benefits of its passage, specifically defined as: ; in: For equipment The strategy of , which indicates its passage sequence or speed selection; A collection of policies for other devices; For equipment The travel time; For equipment Energy consumption; and is the weight parameter, satisfying .

[0073] As an alternative, the scheduling optimization module aims to solve the Nash equilibrium point of the game , determine the optimal strategy for each device . The Nash equilibrium is defined as: ; It can be understood that this equilibrium point means that when the strategies of other devices are fixed, the benefit of each device reaches the optimal value under its selected strategy, thereby achieving global scheduling optimization.

[0074] In one possible implementation, the scheduling optimization module solves the Nash equilibrium through an iterative best response algorithm (IBRA). The specific implementation includes the following steps: Initialize the policy set for all devices ; For each device , when the strategies of other devices are fixed, calculate the optimal response of its current strategy; Update the device's strategy set and repeat the above calculation until the strategies of all devices converge to the equilibrium point.

[0075] It should be noted that in order to further improve the scheduling efficiency, the scheduling optimization module combines the multi-objective optimization method to comprehensively optimize the operating status of the equipment. In some embodiments, the module constructs the following three objective functions: ; in: Indicates the total travel time of all devices; Represents the total energy consumption of all devices; Represents the total wear of all equipment; , , Equipment travel time, energy consumption and equipment wear; is the total number of devices in the operating area.

[0076] In order to transform the multi-objective problem into a single-objective optimization problem, the scheduling optimization module constructs a comprehensive optimization objective function through the weight normalization method: ; in: , , is the weight parameter, satisfying .

[0077] As an option, the scheduling optimization module uses a genetic algorithm (GA) to solve the above comprehensive optimization problem. The specific implementation includes the following steps: Initialize the device strategy population and randomly generate different travel orders and speed combinations; Calculate the fitness value of each strategy combination, and the fitness function is ; Perform selection, crossover and mutation operations on the population based on fitness values ​​to generate a new generation of strategy combinations; Repeat the above iterative process until the fitness value converges to the optimal value.

[0078] Specifically, in a traffic intersection scenario, assume that there are three devices A, B, and C, whose initial priorities are 0.8, 0.6, and 0.5, respectively. The path optimization module provides the following path for each device: Device A: Path , with a weight of 5; Device B: Path , with a weight of 6; Device C: Path , with a weight of 7.

[0079] The scheduling optimization module calculates the passage order and speed of each device based on the above weights and priorities, and ensures the equipment is running smoothly through game theory and multi-objective optimization. The tasks are executed first, while balancing the equipment and equipment energy consumption and travel time.

[0080] It should be noted that the scheduling optimization module can adjust the order and speed of equipment in real time in a dynamic environment. When a fault occurs, the module will immediately recalculate the optimal traffic strategy for the remaining devices to ensure smooth traffic. In addition, the module supports dynamic adjustment of priority rules, such as reallocating the priority weight of devices according to the urgency of the task. .

[0081] As a technical extension, the scheduling optimization module also supports path exclusivity constraints, which restricts multiple devices from accessing the same edge or node at the same time. This mechanism can effectively avoid conflicts in path planning among multiple devices, thereby improving the overall efficiency of the system.

[0082] In summary, the scheduling optimization module in this embodiment uses a game theory model and a multi-objective optimization method to comprehensively consider the travel time, energy consumption, and task priority of the equipment to achieve efficient scheduling optimization in a dynamic environment. The module has strong adaptability and can quickly respond to environmental changes in multi-device scenarios, providing reliable scheduling decisions for the efficient operation of the equipment.

[0083] In this embodiment, the feedback adjustment module is used to monitor the operating status of the target device in real time, and adjust the input data and optimization results of the traffic modeling module according to the dynamic changes of the operating environment to ensure the stability and adaptability of the system. This module compares the actual operation of the device with the optimization results through a closed-loop control mechanism, and updates the modeling and optimization parameters in a timely manner when deviations are found, thereby achieving efficient response to the dynamic environment.

[0084] It should be noted that the feedback adjustment module mainly includes three functional parts: real-time monitoring, anomaly detection and dynamic adjustment. Through these functions, it can effectively respond to unexpected situations in equipment operation, such as path blockage, new equipment addition or equipment failure.

[0085] As an option, the feedback adjustment module obtains the real-time position, speed, direction and other operating status of the target device through the visual perception module. Based on this data, the module can compare the actual path of the device with the optimal path generated by the path optimization module in real time. When it is detected that the actual operating status is inconsistent with the planned results, the feedback adjustment module triggers the anomaly detection mechanism to determine the cause of the deviation and take corresponding adjustment measures.

[0086] Specifically, the anomaly detection mechanism of the feedback adjustment module includes the following: 1. Path blocking detection: The module analyzes whether there are static or dynamic obstacles on the current path based on the device status data provided by the visual perception module. For example, if the occupancy time of a node or edge is Exceeding the estimated travel time of the equipment , then it is considered that the path may be blocked. The detection formula is as follows: ; in, is the allowable time error range.

[0087] 2. Equipment fault detection: The module compares the expected speed of the device and actual speed , to determine whether the device is faulty or stagnant. When the actual speed is lower than the preset threshold When the device is abnormal, the detection conditions are as follows: ; 3. New equipment detection: When the visual perception module detects a new target device in the operating area, the feedback adjustment module will automatically add the information of the device to the node set of the traffic modeling module. and recalculate the dynamic weighted graph model.

[0088] In one possible implementation, the dynamic adjustment function of the feedback adjustment module will take corresponding optimization measures based on the abnormal detection results. For example, in the case of path blocking, the module will adjust the weight of the blocked path. Dynamically set to infinity, so that the path optimization module regenerates a feasible path. In the event of equipment failure, the module removes the faulty equipment from the current optimization calculation and notifies relevant personnel for maintenance.

[0089] As an option, the feedback adjustment module can also dynamically adjust the weight parameters in the traffic modeling module. For example, when the equipment priority changes, the module recalculates the edge weights. To ensure that the path planning and scheduling results can meet the latest task requirements. The weight adjustment formula is as follows: ; It should be noted that the feedback adjustment module is not only applicable to the dynamic adjustment of a single device, but also can handle conflicts in the parallel optimization of multiple devices. In some embodiments, the module monitors the path conflicts of multiple devices in real time. For example, when two devices try to access the same node or edge at the same time, the module adjusts the access order or path of one of the devices to avoid conflicts.

[0090] For example, in a traffic intersection scenario, assume that device A and device B plan to pass through node , the feedback adjustment module will give priority to adjusting low-priority devices Path planning, or lowering the equipment The speed of travel To ensure that high priority devices Normal passage.

[0091] It is understandable that the real-time performance of the feedback adjustment module is critical to system performance. In some embodiments, the module achieves rapid response to large-scale traffic networks through parallel computing and edge computing technology. For example, the module can directly complete preliminary path occupancy detection on the camera side, thereby reducing the delay of data transmission.

[0092] As a technical extension, the feedback adjustment module also supports the analysis and use of historical data. For example, the module can record the frequency, location distribution and impact range of abnormal events, and optimize the initial weight of the traffic modeling module based on these historical data, thereby reducing the probability of abnormal events in the future.

[0093] In summary, the feedback adjustment module in this embodiment ensures the stability and adaptability of the system in a dynamic environment through real-time monitoring, anomaly detection and dynamic adjustment functions. The module can not only quickly respond to various unexpected situations in the operation of the equipment, but also optimize future operation strategies through historical data, thereby providing reliable guarantee for the efficient operation of the system.

[0094] In general, the present invention obtains the dynamic status of the equipment in the operating area in real time through the visual perception module, and constructs a dynamic weighted graph model in combination with the traffic modeling module to describe the equipment's passage path and status. The system calculates the optimal path based on the dynamic Hamiltonian path problem through the path optimization module, and optimizes the passage sequence and speed of the equipment by using the game theory model and multi-objective optimization method in combination with the scheduling optimization module. Finally, the feedback adjustment module monitors the equipment operation in real time, and dynamically adjusts the modeling and optimization results. The system can effectively solve the problems of path conflicts, traffic congestion and uneven resource allocation in the collaborative operation of multiple devices. It is suitable for various scenarios such as industrial logistics and warehouse management, and has the characteristics of high efficiency, dynamics and adaptability.

[0095] Please see attached Figure 2 The present invention also provides a vision-based operation management method for dynamic management and optimization of multi-target devices in the operation area. The method comprises the following steps: obtaining the dynamic state of the device through a visual device, optimizing the path and scheduling based on a dynamic weighted graph model, and realizing closed-loop optimization of the system through real-time monitoring and feedback mechanism. The specific implementation of this method is described in detail below in conjunction with the workflow of the aforementioned system.

[0096] S1. Obtain image data of the operating area through visual equipment and extract the dynamic state of the target device using target detection algorithm; In step S1, image or video data is acquired through visual devices deployed in the operation area, and the target device is identified and tracked using the target detection algorithm. The extracted dynamic state includes the current position, speed, direction and task target node of the target device. The device state data is the basis for subsequent modeling and optimization, and can be updated in real time to reflect the latest operation status of the device.

[0097] S2. constructing a dynamic weighted graph model according to the device status data; In step S2, a dynamic weighted graph model is constructed based on the extracted device status data. The model uses nodes to represent the current location and target location of the device, and edges to represent the traversable paths between nodes, and assigns weights based on travel time and priority to the edges. The dynamic weighted graph model can describe the traffic conditions in the operating area and provide structured data support for subsequent path optimization. When the operating environment or device status changes, the model will be updated in real time to maintain its accuracy and effectiveness.

[0098] S3, based on the dynamic weighted graph model, solve the dynamic Hamiltonian path problem and generate the optimal path to the target device; In step S3, the optimal path of the target device is generated by solving the dynamic Hamiltonian path problem. This path is based on the dynamic weighted graph model and combines the task requirements and operating environment of the device to calculate the minimum weight path from the starting point to the target node. The path optimization result will be used as the input of subsequent scheduling optimization to provide an efficient path planning solution for the operation of the device.

[0099] S4. Based on the game theory model, optimize the passage order and speed of the target equipment and generate equipment passage instructions; In step S4, the passage order and speed of the equipment are optimized based on the game theory model. Each device generates the best passage strategy according to its own task requirements, path weight and priority. The global optimal strategy is achieved through game theory solution among multiple devices to ensure that path conflicts are avoided and efficiency is improved in the collaborative operation environment of multiple devices. At the same time, combined with the multi-objective optimization method, multiple operating indicators such as the equipment's passage time, energy consumption and wear are balanced, so as to further optimize the equipment's operating status.

[0100] S5. Real-time monitoring of equipment operation status, detection of abnormal conditions during operation, and dynamic adjustment of traffic modeling and optimization results; In step S5, real-time monitoring and dynamic adjustment of the operating status are achieved through the feedback adjustment mechanism. The feedback adjustment module compares the actual operating status of the equipment with the path optimization and scheduling optimization results. If an abnormal situation is detected (such as path blocking, equipment failure, or new equipment added), it triggers the dynamic update of modeling and optimization results. For example, when the path is blocked, the system will adjust the edge weights of the dynamic weighted graph model or regenerate the optimal path; when the equipment fails, the faulty equipment will be removed and the tasks will be reallocated to ensure the continuity and stability of the system operation.

[0101] It should be noted that this method adopts a closed-loop control mode, and a dynamic cycle of data flow is formed between each step. When the environment or device status changes, the system can adjust the input and output of each module in time through the feedback mechanism to adapt to the changing operating conditions and maintain the efficiency and adaptability of the system.

[0102] In summary, the vision-based operation management method in this embodiment can effectively solve the path conflict and traffic congestion problems in the operation of multiple devices by implementing dynamic modeling, path optimization and scheduling optimization of equipment in steps, and combining real-time monitoring and feedback adjustment. This method is applicable to a variety of complex operation scenarios and has strong scalability and robustness.

[0103] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A vision-based operation management system, characterized in that: include: Visual perception module, used to obtain image or video data in the operating area and identify the real-time dynamic status of the target device, including position, speed, direction and target position; A traffic modeling module, used to construct a dynamic weighted graph model based on the equipment status data provided by the visual perception module; A path optimization module, used to solve the optimal path of the device from the starting point to the target node based on the dynamic weighted graph model; A scheduling optimization module, used to optimize the passage sequence and speed of the equipment according to the optimal path provided by the path optimization module; The feedback adjustment module is used to monitor the execution status of the equipment in real time and dynamically adjust the input data and optimization results of the traffic modeling module.

2. The vision-based operation management system according to claim 1, characterized in that: The visual perception module identifies the dynamic state of the target device based on a deep learning target detection algorithm, wherein: The target equipment includes AGV forklifts, unmanned guided vehicles and mobile robots; The dynamic state includes the spatial coordinates, driving speed, driving direction and target node of the device.

3. The vision-based operation management system according to claim 1, characterized in that: The dynamic weighted graph model includes multiple nodes, edges and weights, wherein: Nodes are used to represent the current and target locations of the device; Edges are used to represent possible paths between devices, and the weight of the edge is determined based on the device's travel time and priority; The edge weight is calculated by the following formula: ； in, Represents a slave node To Node The edge weight of ,in Indicates the device slave node To Node The travel time, For device slave nodes To Node The distance is the current speed of the device; Indicates the priority of the device; and is the weight coefficient, satisfying ; The traffic modeling module updates the status of nodes and edges in real time, including the addition of new equipment, path congestion, or changes in equipment speed, and recalculates the weights in the dynamic weighted graph model based on the updated equipment status.

4. The vision-based operation management system according to claim 1, characterized in that: The path optimization module is used to dynamically optimize the passage path of the equipment based on the dynamic weighted graph model and solve the dynamic Hamiltonian path problem, where: The dynamic Hamiltonian path problem aims to minimize the total weight of all nodes in the dynamic weighted graph model. The total weight is calculated by the following formula: ； in, Indicates the path The total weight of Represents the sequence of nodes in a path; Indicates the first nodes; Represents a slave node To Node The edge weight of Dynamic path optimization is solved based on a dynamic programming method, which includes: Define the subproblem, let Indicates that the node set is passed from the starting point And with node is the shortest path length to the end point; Construct a recursive relation: ； in, Represents a set of nodes And with node is the shortest path length to the end point; Represents a node collection A node in Represents from the set Remove a node The collection after Represents a slave node To Node The edge weight of Initial conditions: ； in, Indicates that the starting point node is included A collection of nodes; From the starting point Departure and only pass through nodes The shortest path length; The optimal solution is: ； in, Represents the set of all nodes in the dynamic weighted graph model; It means starting from the starting point and passing through all nodes And with node is the shortest path length to the end point; Represents a slave node Return to starting point The edge weight of The path optimization module updates the path planning results in real time according to the changes in the dynamic weighted graph model, and re-solves the dynamic Hamiltonian path problem when new nodes are added or the path is blocked to ensure the dynamic optimality of the passage path.

5. The vision-based operation management system according to claim 1, characterized in that: The scheduling optimization module optimizes the passage order and speed of the target equipment based on a non-cooperative game model, wherein: Scheduling optimization is based on a non-cooperative game model, with the target device as the participant in the game, and each device has an optional set of strategies. , indicating the passage sequence or speed selection of the equipment; Defining the utility function , indicating the target device In choosing strategy And other device selection strategies The benefit when , the utility function is: ； in, Indicates the device The travel time; Indicates the device Energy consumption; and is an adjustable weight parameter, satisfying ; For equipment selection strategy; A selection policy set for other devices; The scheduling optimization module solves the Nash equilibrium point of the game , determine the optimal strategy of the device, the Nash equilibrium satisfies the following conditions: ； in, Indicates the device The optimal strategy for Represents the optimal strategy set for other devices.

6. The vision-based operation management system according to claim 5, characterized in that: The scheduling optimization module uses an iterative optimal response algorithm to solve the Nash equilibrium, which specifically includes the following steps: Initialize the policy set for all target devices; For each device, calculate its optimal strategy when the strategies of other devices are fixed; Repeatedly update the strategy of each device until it converges to the Nash equilibrium point .

7. The vision-based operation management system according to claim 1, characterized in that: The scheduling optimization module combines a multi-objective optimization method to comprehensively optimize the operating efficiency of the target equipment, where: The goal of scheduling optimization is to simultaneously minimize the following objective functions: ； in, Indicates the total transit time of all target devices; Represents the total energy consumption of all target devices; Represents the total wear of all target devices; , , Respectively represent the target device travel time, energy consumption and equipment wear; Indicates the number of target devices; The multi-objective optimization is converted into a single-objective optimization problem by normalizing the weights, and the optimization function is: ； in, represents the comprehensive optimization objective function; , , is the weight parameter of the objective function, satisfying .

8. The vision-based operation management system according to claim 7, characterized in that: The scheduling optimization module solves the optimization problem through a genetic algorithm, which specifically includes the following steps: Initialize the population and randomly generate the passage order and speed strategy of the equipment; According to the fitness function Selecting a population; Generate new strategy populations through crossover and mutation operations; Repeat the iteration until the fitness function converges to obtain the optimal strategy for comprehensive optimization.

9. The vision-based operation management system according to claim 1, characterized in that: The feedback adjustment module detects the following abnormal conditions and triggers dynamic adjustment by visually monitoring the operating status of the equipment: Path occupation or blockage; The addition of new equipment; Equipment operation failure.

10. A vision-based operation management method, applied to a system as claimed in any one of claims 1 to 9, characterized in that: The following steps are involved: The image data of the operating area is obtained through visual equipment, and the dynamic state of the target device, including position, speed, direction and target node, is extracted using the target detection algorithm; Based on the device status data, a dynamic weighted graph model is constructed, wherein nodes represent the current location and target location of the device, and weights are based on the travel time and priority of the device; Based on the dynamic weighted graph model, the dynamic Hamiltonian path problem is solved to generate the optimal path to the target device; Based on the game theory model, the passage order and speed of the target equipment are optimized and the equipment passage instructions are generated; Monitor equipment operating status in real time, detect abnormal conditions during operation, and dynamically adjust traffic modeling and optimization results.

Images