Patrol vehicle path planning and dynamic obstacle avoidance method in combination with virtual-real hybrid positioning

Abstract

The invention relates to the technical field of path planning, in particular to a patrol vehicle path planning and dynamic obstacle avoidance method combined with virtual-real hybrid positioning, which comprises the following steps: acquiring a real-time camera video stream in a patrol area, processing the real-time camera video stream by adopting a pre-trained deep learning model, and identifying a target object in a picture; pixel coordinates of the target object in the camera picture are converted into three-dimensional coordinates in the real world; constructing a virtual scene consistent with the geometric structure of the real environment based on Unity 3D, and mapping the recognized target object and the three-dimensional coordinates in the real world obtained by converting the recognized target object into the virtual scene; and calculating an optimal path according to a preset patrol target point in the virtual scene, and controlling the target object to perform local obstacle avoidance or re-planning a route in real time in response to an obstacle recognition operation. According to the invention, the patrol vehicle can keep continuous, safe and efficient patrol operation in a complex and frequently-changing environment.

Description

Technical Field

[0001] This application relates to the field of path planning technology, and in particular to a method for path planning and dynamic obstacle avoidance of patrol vehicles that combines virtual and real positioning. Background Technology

[0002] With the increasing demand for automation in urban management, industrial park security, and industrial logistics, the use of unmanned patrol vehicles for area patrols and mission execution has become an important development direction in the field of intelligent transportation and automatic control. Existing patrol vehicles typically complete fixed-point patrols based on preset routes, using onboard sensors or navigation devices to travel within fixed routes, thereby achieving basic automated patrol functions.

[0003] However, in real-world applications, patrol areas often encounter dynamic obstacles such as temporarily parked vehicles and pedestrians crossing the road. Because traditional patrol vehicles rely heavily on fixed path planning and static environment modeling, their path control lacks the ability to respond to real-time environmental changes and cannot effectively perceive and flexibly avoid dynamic obstacles. When the patrol path is temporarily blocked, the vehicle typically requires manual intervention or route resetting, leading to patrol interruptions, rigid obstacle avoidance actions, and impacting the system's operational continuity and safety. Therefore, how to ensure continuous, safe, and efficient patrol operations for patrol vehicles in complex and frequently changing environments remains a significant challenge. Summary of the Invention

[0004] This application provides a method for patrol vehicle path planning and dynamic obstacle avoidance that combines virtual and real hybrid positioning, enabling patrol vehicles to maintain continuous, safe, and efficient patrol operations in complex and frequently changing environments. This application provides the following technical solution: Firstly, this application provides a method for patrol vehicle path planning and dynamic obstacle avoidance that combines virtual and real hybrid positioning, the method comprising: The system acquires real-time camera video streams within the patrol area, processes the real-time camera video streams using a pre-trained deep learning model, and identifies target objects in the images. Convert the pixel coordinates of the target object in the camera image into three-dimensional coordinates in the real world; Based on Unity3D, a virtual scene with the same geometry as the real environment is constructed, and the identified target objects and their converted three-dimensional coordinates in the real world are mapped to the virtual scene; In the virtual scene, the optimal path is calculated based on the preset patrol target points, and in response to the obstacle recognition operation, the target object is controlled in real time to perform local obstacle avoidance or replan the route.

[0005] In one specific implementation, the process of processing the real-time camera video stream using a pre-trained deep learning model further includes: Video frames are preprocessed using OpenCV or third-party visual recognition libraries to extract each frame of the real-time video stream and adjust all frames to a uniform resolution. Gaussian blur or median filtering algorithms are used to remove sensor noise and compression artifacts, enhancing the sharpness of target edges. The image is converted to a uniform RGB or HSV color space, and histogram equalization or Gamma correction is performed on unevenly lit areas to improve the visibility of targets in dark areas.

[0006] In one specific implementation, the step of using a pre-trained deep learning model to process the real-time camera video stream and identify target objects in the image includes: The target objects include the patrol vehicle itself, other vehicles, and pedestrians. The deep learning model performs feature analysis and target inference on the input image frames through a convolutional feature extraction network. The model generates three types of detection results for each video image frame: category label, bounding box coordinates, and confidence score. Among them, the category label is used to distinguish different types of objects, the bounding box coordinates are used to mark the position and range of the target in the image coordinate system, and the confidence score is used to characterize the reliability of the model for the recognition result. By comprehensively judging the three types of information, the target object is identified and distinguished in a single frame image.

[0007] In one specific implementation, the step of using a pre-trained deep learning model to process the real-time camera video stream and identify target objects in the image further includes: The deep learning model tracks the center point of the target pixel block in the convolutional feature map, combines the time series information of the video frame, calculates the motion displacement of the target through inter-frame difference, and infers the speed and direction of the target in the image coordinate system. For patrol vehicles and moving pedestrians, the velocity vector of the target is calculated by using the trajectory of the center point of the bounding box detected in consecutive frames; at the same time, the major axis direction of the bounding box is combined with the direction of the motion trajectory to estimate the current orientation of the target.

[0008] In one specific implementation, converting the pixel coordinates of the target object in the camera image into three-dimensional coordinates in the real world includes: Acquire image data of the same target from different perspectives through multi-view vision imaging or specific marker recognition; For multi-view vision, a projection model is constructed using the intrinsic and extrinsic parameter matrices of the camera. Depth information is calculated based on the pixel position difference of the same target in two images, and the position coordinates of the target in the real-world coordinate system are solved by triangulation. When using specific markers, the homography matrix between image coordinates and world coordinates is obtained by identifying the pose and scale of the markers in the image, and the target pixel coordinates are converted into real-world three-dimensional coordinate points using the matrix mapping relationship.

[0009] In one specific implementation, the construction of a virtual scene based on Unity3D that conforms to the geometry of the real environment, and the mapping of the identified target objects and their converted real-world 3D coordinates to the virtual scene, includes: In the Unity3D platform, a 3D model is built based on the actual structure of the patrol area, and a navigable area is generated using NavMesh baking. Determine the correspondence between the real-world coordinate system and the Unity world coordinate system, set the origin, axial direction, and scale parameters of both, and establish a mapping function to map the three-dimensional coordinates of the real world to the Unity virtual space; The identified target object is instantiated into the Unity scene as a virtual model, and its position, orientation, and motion state are updated in real time based on the mapping data.

[0010] In one specific implementation scheme, the step of calculating the optimal path based on preset patrol target points in the virtual scene, and responding to obstacle recognition operations by controlling the target object to perform local obstacle avoidance or replanning the route in real time includes: Several patrol target points are set in the virtual scene. The NavMeshAgent component built into the virtual patrol vehicle model is used to calculate the optimal path from the current vehicle position to the next target point based on the baked NavMesh data. During operation, it continuously receives the recognition data and position change information of the target object. When an obstacle is detected to enter the patrol path range, a dynamic NavMeshObstacle object is generated at the corresponding position in the Unity scene. For obstacles that are moving, the RVO algorithm is called to correct the speed vector and direction of travel of the virtual patrol vehicle, so that it can smoothly bypass the obstacle by relying on the local obstacle avoidance capability of NavMeshAgent. For stationary or long-term immobilized obstacles, the Carving function of NavMeshObstacle is triggered, temporarily emptying the occupied area in the navigation grid, prompting NavMeshAgent to perform global path replanning and recalculate drivable alternative routes.

[0011] Secondly, this application provides a patrol vehicle path planning and dynamic obstacle avoidance system that combines virtual and real hybrid positioning, employing the following technical solution: A patrol vehicle path planning and dynamic obstacle avoidance system combining virtual and real positioning includes: The object perception module is used to acquire real-time camera video streams within the patrol area, and to process the real-time camera video streams using a pre-trained deep learning model to identify target objects in the image. The coordinate transformation module is used to convert the pixel coordinates of the target object in the camera image into three-dimensional coordinates in the real world; The Virtual Mapping module is used to build virtual scenes based on Unity3D that are consistent with the geometry of the real environment, mapping the identified target objects and their converted 3D coordinates in the real world to the virtual scene; The decision control module is used to calculate the optimal path in the virtual scene based on the preset patrol target point, and in response to the obstacle recognition operation, to control the target object to perform local obstacle avoidance or replan the route in real time.

[0012] Thirdly, this application provides an electronic device, the device including a processor and a memory; the memory stores a program, the program being loaded and executed by the processor to implement a patrol vehicle path planning and dynamic obstacle avoidance method combining virtual and real hybrid positioning as described in the first aspect.

[0013] Fourthly, this application provides a computer-readable storage medium storing a program that, when executed by a processor, is used to implement a patrol vehicle path planning and dynamic obstacle avoidance method combining virtual and real hybrid positioning as described in the first aspect.

[0014] By acquiring real-time camera video streams within the patrol area and processing each frame using a pre-trained deep learning model, the system identifies target objects such as patrol vehicles, other vehicles, and pedestrians. It also extracts category, location, speed, and direction information to achieve comprehensive perception of various targets. Subsequently, the identified target pixel coordinates are converted into real-world 3D coordinates, establishing a correspondence between image space and physical space, providing accurate data support for subsequent virtual mapping. Based on this, a virtual scene with a geometric structure consistent with the real environment is constructed using Unity3D, mapping target objects and their 3D coordinates to the virtual space, enabling the virtual environment to dynamically reflect the state of each target in the real-world scene. Finally, the system calculates the optimal path in the virtual scene based on preset patrol target points and, combined with real-time obstacle identification and position change information, controls the movement of the virtual patrol vehicle through local obstacle avoidance and path replanning mechanisms, achieving continuous path optimization and dynamic adjustment. Through this series of operations, the solution forms a closed-loop processing flow from target recognition and spatial positioning to virtual simulation and dynamic path adjustment, enabling patrol vehicles to maintain continuous and smooth patrol behavior in dynamically changing environments.

[0015] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the patrol vehicle path planning and dynamic obstacle avoidance method that combines virtual and real hybrid positioning in the embodiments of this application.

[0017] Figure 2 This is a schematic diagram of the overall process of the patrol vehicle path planning and dynamic obstacle avoidance method that combines virtual and real positioning in the embodiments of this application.

[0018] Figure 3 This is a structural block diagram of a patrol vehicle path planning and dynamic obstacle avoidance system that combines virtual and real positioning in an embodiment of this application.

[0019] Figure 4 This is a block diagram of an electronic device for patrol vehicle path planning and dynamic obstacle avoidance that combines virtual and real hybrid positioning in an embodiment of this application. Detailed Implementation

[0020] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.

[0021] Optionally, this application uses the method for patrol vehicle path planning and dynamic obstacle avoidance that combines virtual and real positioning provided in various embodiments as an example for application in an electronic device. The electronic device is a terminal or a server. The terminal can be a mobile phone, computer, tablet computer, etc. This embodiment does not limit the type of electronic device.

[0022] Reference Figure 1 This is a flowchart illustrating a method for patrol vehicle path planning and dynamic obstacle avoidance combining virtual and real-world positioning, provided in an embodiment of this application. The method includes at least the following steps: Step S101: Obtain real-time camera video streams within the patrol area, and use a pre-trained deep learning model to process the real-time camera video streams to identify target objects in the image.

[0023] In step S101, fixed cameras deployed within the patrol area capture video streams in real time to obtain image data containing patrol vehicles, other vehicles, and pedestrians. The purpose of this step is to perform visual recognition of dynamic targets in the patrol environment, thereby obtaining initial perception information of various targets.

[0024] Specifically, before performing deep learning processing on the real-time camera video stream, the video frames are first preprocessed using OpenCV or a third-party visual recognition library (such as MediaPipe) to eliminate the impact of ambient lighting, camera viewpoint, and video encoding differences on the recognition results. Preprocessing includes, but is not limited to, the following operations: extracting each frame of the real-time video stream and adjusting all frames to a uniform resolution; using Gaussian blur or median filtering algorithms to remove sensor noise and compression artifacts, enhancing target edge sharpness; converting the image to a uniform RGB or HSV color space; and performing histogram equalization or Gamma correction on unevenly lit areas to improve target visibility in dark areas. After preprocessing, the image data is input into a pre-trained deep learning object detection model for recognition.

[0025] In implementation, the target objects identified in this application include the patrol vehicle itself, other vehicles, and pedestrians. During the detection process of the deep learning model, a convolutional feature extraction network is used to perform feature analysis and target inference on the input image frames. The model generates three types of detection results for each video image frame: category label, bounding box coordinates, and confidence score. The category label is used to distinguish different types of objects, such as patrol vehicles, other vehicles, and pedestrians; the bounding box coordinates are used to determine the position and range of the target in the image coordinate system; and the confidence score characterizes the reliability of the model's recognition result. By comprehensively judging these three types of information, the model can complete the identification and differentiation of target objects in a single frame image. Furthermore, in this application, the identified target objects not only include category and position, but also their real-time speed and direction information, all of which are directly generated by the recognition process. The model tracks the center point of the target pixel block in the convolutional feature map, combines it with the time-series information of the video frames, and calculates the target's motion displacement through inter-frame difference calculation, thereby inferring the target's speed and direction in the image coordinate system. For patrol vehicles and moving pedestrians, the target's velocity vector can be calculated from the trajectory of the bounding box center point detected in consecutive frames. Simultaneously, the major axis direction of the bounding box is combined with the direction of the motion trajectory to estimate the target's current orientation. Stationary targets have zero velocity, and their orientation angle remains constant.

[0026] Furthermore, as a preferred embodiment, in this application, to further enhance the motion perception capability of the target object, a velocity and direction extraction method based on pixel distribution centroid and boundary morphological gradient is introduced.

[0027] Specifically, for each target object identified in step S101, its pixel density centroid is first defined as: ; in, Indicates the first The set of pixels representing the bounding box of the target object in the frame. Indicates belonging to the target bounding box The pixel coordinates within the image are used to mark the spatial distribution of each pixel within the target area on the image plane. Their horizontal and vertical coordinates correspond to the column and row positions of the image pixel matrix, respectively. This represents the weighted intensity of the corresponding pixel, which can be either the grayscale value or the weighted sum of the color channels. For the goal in the The weighted pixel centroid coordinates of the frame. Based on the continuous frame changes of the pixel centroid, the instantaneous velocity vector of the target is defined as: ; in, The continuous inter-frame time interval Indicates the first The gradient covariance matrix of the target boundary pixels in the frame is used to describe the variation characteristics of the target edge distribution, and it is defined as follows: ; in, and These are the gradient values ​​of the pixel in the horizontal and vertical directions, respectively. It is a matrix determinant used to characterize the complexity of the target pixel shape. This is an empirical adjustment coefficient. By coupling the velocity vector with the target shape gradient change, the influence of target shape expansion or contraction can be considered in velocity estimation, thereby obtaining more accurate velocity information.

[0028] Regarding the direction of motion of the target, this application proposes the following calculation formula: ; in, and These are the components of the velocity vector in the pixel coordinate system. Represents the gradient matrix of the target pixel The direction of the largest eigenvector, This is an adjustment coefficient used to balance the contributions of the displacement direction and the principal axis direction of the target shape, thereby providing stable direction prediction when the target rotates or deforms. To suppress the influence of sudden noise on velocity calculations, this application also introduces weighted smoothing processing: ; in, For smoothing coefficients, For velocity modulus, This represents the smoothed velocity magnitude. This method combines pixel centroid, boundary morphological gradient, and continuous frame information to achieve a novel extraction of target velocity and direction, making the identification of patrol vehicles, other vehicles, and pedestrians in dynamic environments more accurate, stable, and robust.

[0029] In the speed calculation formula, Indicates the first The weighted centroid of a frame's pixels, which takes into account the spatial average center of gravity of different pixel brightness weights in the target region, can reflect the true positional change of the target's overall centroid in the image coordinate system. Traditional velocity calculation methods rely solely on the displacement change of the bounding box center point between adjacent frames, ignoring the non-uniformity of pixel distribution and morphological changes within the target, making them prone to velocity deviations when the target rotates, experiences partial occlusion, or changes in illumination. In this application, however, a pixel intensity weight is introduced... The true center of gravity coordinates, which are affected by the brightness distribution inside the target, are calculated so that the velocity estimation can reflect the overall motion trend of the target, rather than relying solely on the geometric center of the outer contour.

[0030] Furthermore, the gradient covariance matrix is ​​introduced into the velocity vector calculation. Its determinant, the gradient covariance matrix, is used to describe the statistical distribution of the edge direction and gradient magnitude of a target pixel within a local region. This invention quantifies the complexity and extent of the target's morphological structure. By comparing the ratio of changes in the determinant in consecutive frames, this application adds a morphological change correction term to the velocity vector. This allows for dynamic adjustment of the velocity amplitude when the target undergoes stretching, rotation, or deformation, ensuring that velocity calculations reflect not only spatial displacement but also the effective motion components resulting from morphological changes. This design overcomes the limitation of existing methods that only estimate velocity through changes in the bounding box center point, enabling the model to stably estimate the true motion velocity even under complex visual conditions.

[0031] For direction calculation, this application introduces the direction of the principal eigenvector of the gradient matrix into the traditional velocity direction angle calculation. This refers to the direction of the target's main edge distribution. The theoretical basis of this design is that when a target rotates or changes its attitude during movement, the main direction of its shape often correlates with the actual direction of travel. This is achieved by weighted fusion of the main shape axis direction and the velocity direction (by parameters). (Control), which can significantly improve the robustness of orientation estimation under local motion instability, partial occlusion, or target turning. Unlike existing orientation estimation based solely on velocity vector angles, the orientation extraction in this application considers both target shape and trajectory information, resulting in higher stability and anti-interference capabilities.

[0032] Optionally, this application preferably uses the YOLOv8 model as a pre-trained deep learning model, which has high detection speed and target recognition accuracy. However, other models with similar feature extraction and detection capabilities (such as SSD, Faster R-CNN, etc.) can also be used. This application does not limit this.

[0033] Step S102: Convert the pixel coordinates of the target object in the camera image into three-dimensional coordinates in the real world.

[0034] In step S102, the target object identified in step S101 is spatially located by calculation, converting its pixel coordinates in the image coordinate system into three-dimensional coordinates in the real-world coordinate system. The purpose of this step is to establish the correspondence between image space and physical space, so that the identified patrol vehicles, other vehicles, and pedestrians can obtain accurate location representations in the real-world scene.

[0035] Specifically, image data of the same target from different viewpoints is acquired through multi-view vision imaging or specific marker recognition. For multi-view vision, a projection model is constructed using the camera's intrinsic and extrinsic parameter matrices. Depth information is calculated based on the pixel position difference (parallax) of the same target in two images, and the target's position coordinates in the real-world coordinate system are determined using triangulation. For specific markers (such as AR markers), the homography matrix between image coordinates and world coordinates is obtained by identifying the marker's pose and scale in the image. The matrix mapping relationship is then used to convert the target's pixel coordinates into three-dimensional coordinates in the real world.

[0036] Step S103: Construct a virtual scene with the same geometric structure as the real environment based on Unity3D, and map the identified target objects and their converted three-dimensional coordinates in the real world onto the virtual scene.

[0037] In step S103, the position, orientation, and velocity information of the target object identified in step S101, and the three-dimensional coordinates of the target object obtained in step S102, are imported into the Unity3D simulation platform to construct a virtual patrol scene consistent with the geometry of the real environment. The target object is then mapped and visualized within this virtual scene. The purpose of this step is to unify the coordinates between real and virtual spaces, enabling the system to synchronously display the dynamic distribution of patrol vehicles and obstacles in the virtual environment, and providing an operational simulation basis for the path planning and navigation calculations of the virtual patrol vehicle.

[0038] Specifically, firstly, a 3D model of the patrol area is created in the Unity3D platform based on its actual structure, including basic elements such as terrain, roads, buildings, and obstacles. NavMesh baking is then used to generate a navigable area, defining the range within which the patrol vehicle can travel. Next, the correspondence between the real-world coordinate system and the Unity world coordinate system is determined, setting the origin, axial direction, and scale parameters for both. A mapping function is established to accurately map the real-world 3D coordinate data to the Unity virtual space. After coordinate transformation, the identified patrol vehicle, other vehicles, and pedestrians are instantiated into the Unity scene as virtual models, with their positions, orientations, and motion states updated in real-time based on the mapping data. To achieve path planning and dynamic motion control of the virtual patrol vehicle, a virtual patrol vehicle model containing the NavMeshAgent component is created, enabling path planning and movement control on the baked NavMesh, thus achieving navigable movement of the virtual patrol vehicle within the virtual scene. Meanwhile, the real-time position, speed and direction information generated in steps S101 and S102 is received via a network communication interface (such as Socket) in JSON or other lightweight formats to ensure that the object state in the virtual environment is synchronized with the real environment.

[0039] Step S104: Calculate the optimal path in the virtual scene based on the preset patrol target points, and respond to the obstacle recognition operation to control the target object to perform local obstacle avoidance or replan the route in real time.

[0040] In step S104, based on the virtual scene and virtual patrol vehicle model constructed in step S103, multiple preset patrol target points are set within the patrol area, and the optimal driving path of the virtual patrol vehicle between each target point is calculated using a path planning algorithm. When the system detects the appearance, movement, or change in the stationary state of an obstacle, it controls the virtual patrol vehicle to perform local obstacle avoidance or path replanning operations according to the type and location characteristics of the obstacle. The purpose of this step is to enable the patrol vehicle to have intelligent path adjustment capabilities in a dynamically changing virtual environment, thereby ensuring the continuity and safety of patrol missions.

[0041] Specifically, firstly, several patrol target points (Waypoints) are set in the virtual scene. Using the NavMeshAgent component of the virtual patrol vehicle model, the optimal path from the current vehicle position to the next target point is calculated based on the baked NavMesh data. During operation, the system continuously receives obstacle recognition data and position change information from step S101. When an obstacle is detected entering the patrol path range, a dynamic NavMeshObstacle object is generated at the corresponding position in the Unity scene. For moving obstacles, the RVO (Reciprocal VelocityObstacles) algorithm is called to correct the velocity vector and direction of travel of the virtual patrol vehicle, enabling it to smoothly bypass obstacles using the local obstacle avoidance capabilities of the NavMeshAgent. For stationary or long-term immobile obstacles, the Carving function of the NavMeshObstacle is triggered, temporarily emptying the occupied area in the navigation mesh, thereby prompting the NavMeshAgent to perform global path replanning and recalculate drivable alternative routes. At the same time, dynamic obstacle avoidance priorities and safe distance thresholds are set based on the speed, direction and distance parameters of obstacles to ensure that vehicles will not enter potential collision zones under any circumstances.

[0042] Through this step, the virtual patrol vehicle can automatically plan and adjust its route in real time in dynamically changing scenarios. When unexpected obstacles or environmental changes occur, the system can maintain the feasibility of the route and the continuity of the task through local obstacle avoidance and global replanning mechanisms, thereby achieving smooth and safe intelligent patrol operations.

[0043] In summary, combining Figure 2 This application addresses the problem that traditional patrol vehicles rely on fixed path planning and cannot perceive and flexibly avoid dynamic obstacles in real time. It constructs a complete technical process from real-world environment perception to virtual simulation synchronization, and then to intelligent path planning and obstacle avoidance control, thereby enabling patrol vehicles to conduct continuous, safe and efficient patrol operations in complex and frequently changing environments.

[0044] This application first acquires real-time video streams using fixed cameras deployed within the patrol area. A pre-trained deep learning model (preferably YOLOv8) is then used to detect targets in the image frames, identifying various dynamic targets including the patrol vehicle itself, other vehicles, and pedestrians. Real-time position, velocity, and orientation information for each target are obtained based on continuous frame difference calculations. Subsequently, using multi-view vision or specific markers (such as AR markers), triangulation or homography matrix transformation is employed to convert the identified pixel coordinates into 3D coordinates in the real world, achieving a precise mapping from image space to physical space. Based on this, the 3D position, velocity, and orientation data of the identified and located target objects are imported into the Unity3D simulation platform to construct a 3D virtual patrol scene consistent with the geometry of the real environment. A one-to-one mapping relationship between real-world coordinates and Unity coordinates is established, enabling real-time synchronization of dynamic changes in the real environment within the virtual scene.

[0045] In a virtual environment, a virtual patrol vehicle model containing the NavMeshAgent component is established by setting patrol target points and generating navigable areas using NavMesh baking. This model can automatically calculate the optimal path based on preset patrol target points and, upon receiving obstacle recognition signals, execute corresponding obstacle avoidance strategies based on the obstacle's motion state and distance information. For fast-moving obstacles, the system uses the RVO (Reciprocal VelocityObstacles) algorithm to dynamically adjust the vehicle's velocity vector and driving direction, achieving smooth local avoidance. For obstacles that remain stationary for extended periods, the NavMeshObstacle's Carving function is triggered, dynamically "carving out" occupied areas on the navigation grid, thus prompting the system to perform global path replanning and regenerate a drivable path. By dynamically adjusting obstacle avoidance priorities and safe distance thresholds, the patrol vehicle can maintain a stable and safe patrol state under different types of obstacle interference.

[0046] Through the above technical solution, this application achieves high-precision mapping and data linkage between the real environment and the virtual simulation environment, enabling patrol vehicles to execute path planning and dynamic obstacle avoidance strategies in real time within the virtual scene. Compared with traditional static path patrol methods, this solution can dynamically update the path based on visual recognition results, intelligently respond to environmental changes, and significantly improve the autonomous decision-making ability and environmental adaptability of patrol vehicles in complex scenarios. This effectively solves the problem of existing technologies being unable to achieve real-time perception and flexible avoidance of dynamic obstacles, ensuring the continuity, safety, and efficiency of the patrol process.

[0047] Figure 3This is a structural block diagram of a patrol vehicle path planning and dynamic obstacle avoidance system combining virtual and real positioning, provided in one embodiment of this application. The system includes at least the following modules: The object perception module is used to acquire real-time camera video streams within the patrol area. It uses a pre-trained deep learning model to process the real-time camera video streams and identify target objects in the image. The coordinate transformation module is used to convert the pixel coordinates of the target object in the camera image into three-dimensional coordinates in the real world; The Virtual Mapping module is used to build virtual scenes based on Unity3D that are consistent with the geometry of the real environment, mapping the identified target objects and their converted 3D coordinates in the real world to the virtual scene; The decision control module is used to calculate the optimal path in the virtual scene based on the preset patrol target points, and respond to the obstacle recognition operation to control the target object to perform local obstacle avoidance or replan the route in real time.

[0048] For relevant details, please refer to the above method implementation examples.

[0049] Figure 4 This is a block diagram of an electronic device provided in one embodiment of this application. The device includes at least a processor 401 and a memory 402.

[0050] Processor 401 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 401 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 401 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 401 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0051] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 402 is used to store at least one instruction, which is executed by the processor 401 to implement the patrol vehicle path planning and dynamic obstacle avoidance method combining virtual and real hybrid positioning provided in the method embodiments of this application.

[0052] In some embodiments, the electronic device may also optionally include: a peripheral device interface and at least one peripheral device. The processor 401, memory 402, and peripheral device interface can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface via a bus, signal line, or circuit board. Indicatively, peripheral devices include, but are not limited to: radio frequency circuits, touch displays, audio circuits, and power supplies.

[0053] Of course, electronic devices may also include fewer or more components, and this embodiment does not limit this.

[0054] Optionally, this application also provides a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the patrol vehicle path planning and dynamic obstacle avoidance method combining virtual and real hybrid positioning described in the above method embodiments.

[0055] Optionally, this application also provides a computer product including a computer-readable storage medium storing a program, which is loaded and executed by a processor to implement the patrol vehicle path planning and dynamic obstacle avoidance method combining virtual and real hybrid positioning described in the above method embodiments.

[0056] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0057] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for patrol vehicle path planning and dynamic obstacle avoidance combined with virtual-real hybrid positioning, characterized in that, The method comprises: acquiring a real-time camera video stream in a patrol area, processing the real-time camera video stream using a pre-trained deep learning model, and identifying a target object in the picture; converting the pixel coordinates of the target object in the camera picture into three-dimensional coordinates in the real world; based on Unity3D, a virtual scene consistent with the geometric structure of the real environment is constructed, and the identified target object and its converted three-dimensional coordinates in the real world are mapped into the virtual scene; in the virtual scene, an optimal path is calculated according to a preset patrol target point, and in response to an obstacle identification operation, the target object is controlled to perform local obstacle avoidance or re-plan a route.

2. The method of claim 1, wherein, Before the processing of the real-time camera video stream using the pre-trained deep learning model, the method further comprises: using OpenCV or a third-party visual recognition library to preprocess the video frames, extracting the real-time video stream frame by frame, and adjusting all frames to a uniform resolution; using a Gaussian blur or median filter algorithm to remove sensor noise and compression artifacts, and enhancing target edge definition; converting the image to a uniform RGB or HSV color space, performing histogram equalization or Gamma correction on the uneven illumination area to improve dark area target visibility.

3. The method of claim 1, wherein, The processing of the real-time camera video stream using the pre-trained deep learning model to identify the target object in the picture comprises: The target object includes the patrol vehicle itself, other vehicles, and pedestrians. The deep learning model analyzes the features of the input image frames and infers the target through a convolution feature extraction network. The model generates three types of detection results for each video image: class label, bounding box coordinates, and confidence score. The class label is used to distinguish different types of objects, the bounding box coordinates are used to determine the position and range of the target in the image coordinate system, and the confidence score is used to represent the credibility of the model for the identification result. Through comprehensive judgment of the three types of information, the identification and differentiation of the target object in a single image are completed.

4. The method of claim 3, wherein, The processing of the real-time camera video stream using the pre-trained deep learning model to identify the target object in the picture further comprises: The deep learning model tracks the center point of the target pixel block in the convolution feature map, combines the time sequence information of the video frames, calculates the motion displacement of the target through inter-frame difference, and estimates the speed and direction of the target in the image coordinate system. For the patrol vehicle and moving pedestrians, the speed vector of the target is calculated by the trajectory of the center points of the detected bounding boxes in consecutive frames. At the same time, the long axis direction of the bounding box and the motion trajectory direction are combined to estimate the current orientation of the target.

5. The method of claim 1, wherein, The conversion of the pixel coordinates of the target object in the camera picture into three-dimensional coordinates in the real world comprises: acquiring image data of the same target under different angles through multi-view imaging or specific marker recognition; for the multi-view vision case, a projection model is constructed using the intrinsic and extrinsic parameter matrices of the camera, the depth information is calculated according to the pixel position difference of the same target in two images, and the position coordinates of the target in the real world coordinate system are solved through triangulation method; For the case of using a specific marker, by recognizing the pose and scale of the marker in the picture, a homography matrix between image coordinates and world coordinates is obtained, and the target pixel coordinates are converted into real-world three-dimensional coordinate points using the matrix mapping relationship.

6. The method of claim 1, wherein, The virtual scene consistent with the geometric structure of the real environment is constructed based on Unity3D, and the recognized target object and the three-dimensional coordinates of the real world converted by the target object are mapped into the virtual scene, which includes: A three-dimensional model is established in the Unity3D platform according to the actual structure of the patrol area, and a navigable area is generated by using NavMesh baking; The correspondence between the real coordinate system and the Unity world coordinate system is determined, the origins, axial directions and scale parameters of the two are set, and a mapping function is established to map the three-dimensional coordinates of the real world to the Unity virtual space; The recognized target object is instantiated in the form of a virtual model in the Unity scene, and its position, orientation and motion state are updated in real time according to the mapping data.

7. The method of claim 1, wherein, The optimal path is calculated in the virtual scene according to the preset patrol target point, and the target object is controlled to perform local obstacle avoidance or re-plan the route in response to the identification operation of the obstacle, which includes: A plurality of patrol target points are set in the virtual scene, and the NavMeshAgent component built-in the virtual patrol vehicle model is used to calculate the optimal path from the current vehicle position to the next target point according to the baked NavMesh data; During running, the identification data and position change information of the target object are continuously received, and when an obstacle is detected to enter the patrol path range, a dynamic NavMeshObstacle object is generated at the corresponding position in the Unity scene; For the obstacle in the moving state, the RVO algorithm is called to correct the speed vector and advancing direction of the virtual patrol vehicle, so that it relies on the local obstacle avoidance capability of the NavMeshAgent to realize smooth detouring; For the static or long-time non-moving obstacle, the Carving function of the NavMeshObstacle is triggered to temporarily hollow out the occupied area in the navigation mesh, so as to make the NavMeshAgent execute global path re-planning and recalculate the alternative route that can be driven.

8. A system for patrol vehicle path planning and dynamic obstacle avoidance combined with virtual-real hybrid positioning, characterized in that, It includes: An object perception module is configured to acquire real-time camera video stream in a patrol area, process the real-time camera video stream using a pre-trained deep learning model, and identify target objects in the picture; A coordinate conversion module is configured to convert pixel coordinates of the target objects in the camera picture into three-dimensional coordinates in the real world; A virtual mapping module is configured to construct a virtual scene consistent with the geometric structure of the real environment based on Unity3D, and map the recognized target objects and the three-dimensional coordinates of the real world converted by the target objects into the virtual scene; A decision control module is configured to calculate an optimal path in the virtual scene according to a preset patrol target point, and control the target object to perform local obstacle avoidance or re-plan the route in response to the identification operation of the obstacle.

9. An electronic device, comprising: The device comprises a processor and a memory; the memory stores a program, which is loaded and executed by the processor to implement the method for patrol vehicle path planning and dynamic obstacle avoidance in combination with virtual-real hybrid positioning according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program, which is executed by the processor to implement the method for patrol vehicle path planning and dynamic obstacle avoidance in combination with virtual-real hybrid positioning according to any one of claims 1 to 7.

Images