Obstacle avoidance method for new energy vehicle unmanned AGV based on computer vision

Abstract

This invention belongs to the field of intelligent vehicle and automated guided vehicle (AGV) control technology, specifically disclosing an obstacle avoidance method for unmanned AGVs in new energy vehicles based on computer vision. This method acquires panoramic visual data streams through an onboard multi-view vision sensor array, combines a dynamic visual attention model simulating human megalocellular pathways to identify high-risk dynamic targets, and analyzes their posture and gait to generate intention representation vectors. Then, a social force model is introduced to predict the target trajectory based on the intention, constructing a dynamic feasible region with social constraints. Within this region, a local obstacle avoidance path that balances safety distance and passage efficiency is planned, and finally smoothly integrated with the global path and executed in a closed loop. This invention, by integrating neuroscience mechanisms and social interaction models, enables AGVs to possess human-like obstacle avoidance behavior, improving safety, smoothness, and logistics efficiency in human-machine coexistence environments.

Description

Technical Field

[0001] This invention belongs to the field of intelligent vehicle and automated guided vehicle (AGV) control technology, specifically relating to an obstacle avoidance method for unmanned AGVs of new energy vehicles based on computer vision. Background Technology

[0002] With the rapid development of the new energy vehicle industry, unmanned automated guided vehicles (AGVs) are increasingly being used in smart factories and warehousing logistics, becoming a core carrier for achieving production line automation and intelligent logistics. Computer vision-based obstacle avoidance technology utilizes image sensors to acquire rich scene semantic information and constructs dynamic environment models to achieve real-time adjustments to the AGV's trajectory, becoming an important means to improve the system's environmental perception. In highly dynamic industrial scenarios where humans and machines coexist, AGVs need to frequently respond to pedestrians changing direction or sudden behavioral interference, which places high demands on the system's data processing efficiency, target recognition accuracy, and ability to predict movement intentions.

[0003] Traditional feature extraction models often treat dynamic obstacles as simple geometric shapes, making it difficult to efficiently analyze behavioral semantic information in complex scenes. This leads to automated guided vehicles (AGVs) easily adopting over-obstacle avoidance or frequent abrupt stops when facing pedestrian traffic, resulting in system deadlock and reduced logistics efficiency. Existing visual processing mechanisms lack the ability to quickly lock onto high-risk targets and cannot simulate the salient feature extraction mechanism in biological visual cortex processing, leading to uneven allocation of computational resources and increased response latency in complex backgrounds. Furthermore, the obstacle avoidance decision-making process heavily relies on linear logic analysis, making it difficult to capture the non-linear intention changes behind pedestrian posture and gait, and failing to achieve path planning with social attributes, resulting in a discrepancy between visual obstacle avoidance performance and the real human-machine collaboration experience.

[0004] This invention aims to develop an obstacle avoidance method for autonomous guided vehicles (AGVs) in the new energy vehicle industry, based on computer vision. (Summary of the Invention) The purpose of this invention is to provide an obstacle avoidance method for unmanned AGVs of new energy vehicles based on computer vision, which can solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: an obstacle avoidance method for unmanned AGVs of new energy vehicles based on computer vision, comprising the following specific steps: Step 1: Real-time acquisition of continuous image sequences of the surrounding environment through an on-board multi-view vision sensor array, and spatiotemporal synchronization and geometric correction of the image sequences to generate a panoramic vision data stream with a unified coordinate system; Step 2: Construct a dynamic visual attention model based on the macrocellular pathway mechanism in neuroscience, extract salient regions from the panoramic visual data stream, identify dynamic targets with high motion risk or behavioral uncertainty in the scene, and generate corresponding target attention weight maps. Step 3: Perform posture and gait feature analysis on the dynamic target, extract behavioral semantic information such as body orientation, limb movement trend and head gaze direction, and form a structured intent representation vector; Step 4: Treat the dynamic target as a particle with social attributes, combine it with the social force model, and calculate the psychological repulsion and attraction of the target to other moving entities based on the intention representation vector, so as to predict its trajectory within a predetermined time in the future. Step 5: Based on the predicted motion trajectory and the current state of the AGV, construct a dynamic feasible region that includes social constraints, and plan a local obstacle avoidance path that satisfies both safety distance and passage efficiency constraints within the feasible region; Step 6: Smoothly integrate the local obstacle avoidance path with the global task path to generate the final driving command and drive the AGV to execute it, while continuously monitoring environmental changes to achieve closed-loop feedback adjustment.

[0006] Preferably, in step 1, the vehicle-mounted multi-view vision sensor array consists of at least three wide-angle cameras facing forward, to the left front, and to the right front, respectively. Each camera has the same frame rate and its timestamps are strictly aligned. The original image is transformed and stitched using the calibrated intrinsic and extrinsic parameter matrix to eliminate parallax interference and ensure the spatial continuity and temporal consistency of the panoramic vision data stream.

[0007] Preferably, in step 2, the dynamic visual attention model simulates the information processing mechanism of the human primary visual cortex, prioritizing responses to high-contrast edges, fast-moving regions, and non-periodic texture changes. Through multi-scale convolution kernels and spatiotemporal gradient joint analysis, it outputs the risk activation value of each pixel, and then through non-maximum suppression and connected region merging, it finally determines several high-risk dynamic targets and their spatial locations in the image.

[0008] Preferably, in step 3, the posture and gait feature analysis adopts a deep neural network based on keypoint regression. During the training phase, the network uses a large amount of labeled human skeleton data for supervised learning, which can stably estimate the trunk axis, shoulder-hip line angle, leg swing phase and head yaw angle from single or multiple frames of images. The behavioral semantic information is encoded into a fixed-dimensional vector for subsequent intent modeling.

[0009] Preferably, in step 4, the magnitude of the psychological repulsion force is related to the relative speed, approach angle, and gaze direction between the dynamic target and the AGV. When the target's gaze is not directed toward the AGV, the repulsion force is enhanced. The target attraction force is determined by the deviation between the target's current direction of movement and its preset destination. The smaller the deviation, the stronger the attraction force. The two work together to form a resultant force field, which is used to deduce the probability distribution of the target's position at the next moment.

[0010] Preferably, the construction of the dynamic feasible domain in step 5 takes into account pedestrian traffic habits and industrial site traffic rules, sets different traffic priorities for different areas, and introduces a buffer zone mechanism to automatically expand the safety margin in high-density pedestrian areas, ensuring that the AGV does not intrude into the passage space of others during the avoidance process and avoids triggering a chain reaction of obstacle avoidance.

[0011] Preferably, in step 6, the fusion of the local obstacle avoidance path and the global task path adopts a spline interpolation and curvature constraint optimization algorithm. Under the premise of ensuring the continuity and differentiability of the path, it minimizes the sudden change in direction and acceleration fluctuation, so that the AGV runs smoothly and retains the ability to respond quickly to sudden obstacles, thus achieving a balance between efficiency and safety.

[0012] Compared with the prior art, the present invention has the following beneficial effects: This invention integrates neuroscience visual attention mechanisms with social force models, enabling AGVs to quickly identify high-risk targets in complex dynamic environments. Based on a deep understanding of pedestrian postures and gaits, it accurately predicts their potential movement intentions and generates obstacle avoidance strategies that conform to human social habits.

[0013] This method avoids the over-braking or ineffective detour problems caused by traditional obstacle avoidance systems that simplify pedestrians into geometric obstacles, reducing the probability of system deadlock and improving logistics efficiency. By constructing a dynamic feasible domain with social attributes, the AGV exhibits human-like navigation behavior during obstacle avoidance, enhancing the smoothness and safety of collaboration in human-machine coexistence environments, and providing reliable technical support for high-density human-machine collaboration in the intelligent manufacturing scenario of new energy vehicles. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the overall technical solution architecture of the present invention; Figure 2 This is a schematic diagram of the core principle framework of obstacle avoidance in this invention, which combines a dynamic visual attention model and a social force model. Figure 3 This is a flowchart illustrating the logical process of extracting salient regions from panoramic visual data streams to identify high-risk dynamic targets in this invention. Figure 4This is a schematic diagram of the principle framework of the present invention, which uses the posture and gait feature analysis of dynamic targets to perform structured intent representation and combines the resultant force field to predict the motion trajectory. Figure 5 This is a schematic diagram of the multi-level interaction relationship and data flow in this invention, which constructs a dynamic feasible domain with social constraints based on predicted trajectories and generates local obstacle avoidance paths. Detailed Implementation

[0015] Example 1: Please refer to the appendix Figure 1 To be continued Figure 5 To make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments.

[0016] In the obstacle avoidance method for unmanned AGVs in new energy vehicles based on computer vision, the entire control system is deployed on an onboard embedded computing platform with high-performance computing capabilities. This platform achieves high-speed data interaction with the chassis actuators, sensor arrays, and power management system through the onboard bus system. In specific implementation, the method of the present invention is implemented through the following steps: In step 1, a continuous sequence of images of the surrounding environment is acquired in real time by an on-board multi-view vision sensor array, and the image sequence is spatiotemporally synchronized and geometrically corrected to generate a panoramic visual data stream with a unified coordinate system.

[0017] In practical implementation, the vehicle-mounted multi-view vision sensor array consists of at least three wide-angle cameras, which are respectively installed at the front, left front, and right front of the autonomous AGV. The installation height, pitch angle, and horizontal deflection angle of each camera are pre-calibrated according to the AGV's dimensions. The frame rate of each camera remains consistent, typically set to 30 or 60 frames per second to ensure capture accuracy in dynamic environments. To achieve nanosecond-level spatiotemporal synchronization, a hardware synchronization trigger mechanism is introduced, whereby the master clock sends pulse signals to all cameras to ensure that each camera performs photoelectric conversion and sampling at the same moment.

[0018] After acquiring the original image sequence, the system calls the intrinsic and extrinsic parameter matrices stored in non-volatile memory. The intrinsic parameter matrix contains the focal length, principal point coordinates, and radial and tangential distortion coefficients; the extrinsic parameter matrix defines the rotation matrix and translation vector of each camera relative to the coordinate system of the AGV body center. Using a perspective transformation algorithm, the system performs distortion correction on the original image and projects it onto a unified virtual sphere. Through feature matching and image fusion techniques, parallax interference in overlapping areas between different viewpoints is eliminated. Specifically, the system uses a weighted average fusion algorithm in overlapping areas, assigning weights based on the radial distance of each pixel from the image center, with pixels closer to the center receiving higher weights. This ensures a smooth transition of the panoramic visual data stream at the stitching points, possessing spatial continuity and temporal consistency. The processed panoramic visual data stream is stored in high-bit-depth bitmap form in high-speed random access memory, providing a data foundation for subsequent real-time processing.

[0019] Step 2 involves constructing a dynamic visual attention model based on the macrocellular pathway mechanism in neuroscience, extracting salient regions from the panoramic visual data stream, identifying dynamic targets with high motion risk or behavioral uncertainty in the scene, and generating corresponding target attention weight maps.

[0020] In this step, the dynamic visual attention model simulates the parallel processing capability of the human primary visual cortex for spatial frequency and motion information. The macrocellular pathway is highly sensitive to signals with low spatial frequency and high temporal frequency. Therefore, the system first performs multi-scale Gaussian pyramid decomposition on the panoramic visual data stream to extract image features of different frequency components. In the feature extraction layer, the model uses a set of directionally selective convolutional kernels to simulate the receptive fields of simple cells, identifying high-contrast edges in the image.

[0021] The system calculates pixel displacement between two consecutive frames using a spatiotemporal gradient joint analysis method. For each pixel location, its partial derivatives in the horizontal and vertical directions are calculated, and combined with the pixel value change rate over time, the optical flow vector at that point is derived. The system prioritizes responding to regions with larger motion vector magnitudes and non-periodic changes; these regions typically represent pedestrians, inspectors, or other moving vehicles. By introducing a nonlinear competition mechanism, the system suppresses static textures in the background while enhancing features of rapidly moving regions or suddenly appearing objects.

[0022] The risk activation values ​​of each output pixel constitute a saliency distribution map. The system binarizes the saliency distribution map by setting a dynamically adjusted threshold, and then uses a connected component merging algorithm to aggregate scattered high-activation points into complete candidate target regions. During this process, a non-maximum suppression algorithm is used to eliminate overlapping candidate boxes, ensuring that each physical entity corresponds to only one recognition result. Finally, the system identifies several high-risk dynamic targets and their spatial locations in the image coordinate system and the physical world coordinate system, and generates a target attention weight map, where the weight represents the proportion of resources allocated to that target by the AGV vision system.

[0023] In step 3, the dynamic target is subjected to posture and gait feature analysis to extract behavioral semantic information such as body orientation, limb movement trend and head gaze direction, forming a structured intent representation vector.

[0024] In practice, posture and gait feature parsing relies on a deep convolutional neural network structure, which has been pre-trained on a large-scale human skeleton annotation dataset. For each dynamic target identified in step 2, the system crops and scales its corresponding image sub-region to a fixed resolution before inputting it into the neural network. This network preserves fine-grained features at the lower level through residual connection structures and uses heatmap regression technology to accurately locate 17 key points of the human body, including the tip of the nose, both eyes, both ears, both shoulders, both elbows, both wrists, both hips, both knees, and both ankles.

[0025] Based on these key points, the system further calculates high-level features reflecting behavioral semantics. The torso axis is determined by the line connecting the midpoints of the shoulders and the midpoints of the hips, representing the basic orientation of the body; the dynamic change in the angle of the shoulder-hip line reflects the body's torsional tendency. Gait feature analysis is achieved through a sequence of skeletons across multiple consecutive frames, where the system extracts the phase difference of leg swings, stride length, and the time interval between foot strikes. For example, when an increased frequency of leg crossings and a larger stride are detected, the semantic encoder labels this state as "accelerated walking."

[0026] Crucially, head gaze direction is determined by identifying facial key points and combining this with a head pose estimation algorithm. The system constructs a gaze vector based on a 3D sphere, and semantic weights are adjusted accordingly when the gaze vector points to the AGV's travel path. All this information, including body orientation angle, limb movement rate, head deflection angle, and gait stability indices, is encoded into a fixed-length, for example, 128-dimensional floating-point vector—the intent representation vector. This vector is stored in a structured form and can be efficiently accessed by the subsequent logical reasoning module.

[0027] In step 4, the dynamic target is regarded as a particle with social attributes. Based on the social force model and the intention representation vector, the psychological repulsion and attraction of the target to other moving entities are calculated, and its trajectory within a predetermined time is predicted.

[0028] In the detailed implementation of this step, the social force model abstracts complex pedestrian behavior into the motion of a point mass under the influence of a force field. Psychological repulsion simulates the human tendency to maintain personal space during movement. The magnitude of this force depends not only on the geometric distance between the dynamic target and the AGV, but also significantly on the relative motion state. Specifically, the magnitude of the psychological repulsion is directly proportional to the relative velocity between them; as the relative velocities point towards each other and the approach angle decreases, the amplitude of the repulsion increases exponentially.

[0029] Furthermore, the system adjusts the force field distribution based on the gaze direction obtained in step 3. If the dynamic target's gaze is not directed towards the AGV, it indicates that the target is unaware of the AGV's presence. In this case, the system automatically increases the weighting coefficient of the psychological repulsion force to reserve more safety margin. The target attraction reflects the target's dynamics towards its predetermined destination. By analyzing the target's trajectory over the past 2 to 3 seconds, the system uses the least squares method to fit its main direction of motion, treating it as an attraction vector pointing towards the virtual destination. The smaller the deviation between the target's current direction of motion and this attraction vector, the stronger the correction effect of the attraction on the trajectory.

[0030] The system vector-sums the psychological repulsion vector with the target attraction vector to obtain the resultant force acting on the dynamic particle. Based on the second-order differential equation of Newton's laws of motion (which is described as: the resultant force drives the change in acceleration, thereby changing velocity and position), the system performs stepwise recursive calculations in the time dimension. Through Markov chain Monte Carlo sampling or Gaussian process regression, the system not only predicts a mean trajectory but also generates a probability distribution cloud map of the target's position over the next 3 to 5 seconds. This prediction mechanism fully considers social attributes and can effectively predict social interaction behaviors such as "sidestepping to avoid," "accelerating through," or "stopping and waiting."

[0031] In step 5, based on the predicted motion trajectory and the current state of the AGV, a dynamic feasible region containing social constraints is constructed, and a local obstacle avoidance path that satisfies both safety distance and passage efficiency constraints is planned within the feasible region.

[0032] In practical implementation, the construction of the dynamic feasible region is a process of solving multi-dimensional constraints. First, the system reads the AGV's current linear velocity, angular velocity, braking performance parameters, and physical contour dimensions. The feasible region is defined as a continuous space in a spatiotemporal coordinate system. To introduce social constraints, the system sets up a hierarchical safety buffer zone. Around pedestrians, a core restricted area and a social transition zone are set based on a probability cloud map predicted according to their movement intentions. The radius of the core restricted area is determined by the AGV's emergency braking distance, while the range of the social transition zone is referenced to human comfort standards in industrial settings.

[0033] In high-density pedestrian areas, the system automatically increases the level of social constraints, expanding the safety margin and preventing AGVs from coming into close contact with pedestrians. Traffic efficiency constraints require the AGVs to maintain their rated operating speed as close as possible to the rated speed without collisions, and to minimize path deviation. The planning algorithm finds an optimal path within the feasible region. This algorithm evaluates each candidate path point using a cost function, which includes the reciprocal of the distance to obstacles, the square of the path curvature, and the deviation from the global path.

[0034] The system employs an optimization strategy based on either a dynamic window method or a spatiotemporal search tree. Within each control cycle, the system searches for velocity pairs (linear velocity and angular velocity) that meet dynamic constraints in the velocity space, simulating multiple candidate trajectories and eliminating those that would intrude into the boundary of the dynamic feasible region. The final selected path must ensure that the AGV, when avoiding pedestrians, does not encroach on the passage space of other personnel or vehicles, preventing a domino-like chain reaction of obstacle avoidance and maintaining the orderliness of the entire logistics environment.

[0035] In step 6, the local obstacle avoidance path is smoothly integrated with the global task path to generate the final driving command and drive the AGV to execute it, while continuously monitoring environmental changes to achieve closed-loop feedback adjustment.

[0036] In the specific implementation of this step, the fusion algorithm is responsible for handling the continuity issue between the local and global paths at the connection points. The global task path is usually a polyline or large-radius curve issued by the scheduling center. The local obstacle avoidance path is a fine-tuning process generated when dealing with sudden obstacles. The system uses a fifth-order spline interpolation algorithm to smoothly transition with the global path at the start and end points of the local path. The selection of spline curve parameters ensures the continuity of the path's position, first derivative (velocity direction), and second derivative (curvature) at the connection points.

[0037] This continuity minimizes abrupt changes in direction and acceleration fluctuations during AGV operation, reducing the impact on onboard cargo caused by inertia. The generated driving commands, including the target linear velocity, target steering angle, and corresponding execution timestamps, are sent to the chassis controller via a CAN bus or Ethernet interface. The chassis controller employs proportional-integral-derivative (PID) control logic or model predictive control (MPC) algorithms to drive the AGV to precisely track the command path by controlling the speed of the drive motor and the angle of the steering mechanism.

[0038] Meanwhile, the system maintains a high-frequency closed-loop feedback mechanism. Each time a control command is issued, the system immediately re-triggers the cycle from steps 1 to 5. If a dynamic target in the environment suddenly changes direction or a new obstacle enters the visual field of view, the prediction model immediately updates the probability cloud map and replans the local path. This millisecond-level response capability ensures that the AGV can achieve efficient material transportation while exhibiting extremely high levels of social safety and navigation intelligence within the complex and ever-changing new energy vehicle production workshop.

[0039] To further illustrate the technical details of this invention, specific application scenarios are provided below. In a new energy vehicle power battery assembly workshop, an AGV equipped with the obstacle avoidance system of this invention is performing a transport task from the cell warehouse to the module assembly line.

[0040] In this scenario, the vehicle-mounted multi-view vision sensor array in step 1 captures panoramic images at a frequency of 60Hz. Due to the complex lighting conditions in the workshop, including direct sunlight from strong LED overhead lights and specular reflections from the metal frame, the system's geometric correction procedure not only handles perspective distortion but also ensures that material boxes in the shadow areas and pedestrians in the highlight areas have clear outlines in the panoramic vision data stream through automatic white balance and high dynamic range image processing technology.

[0041] In step 2, when the AGV reaches the intersection, the dynamic visual attention model detects a rapidly moving salient region at the edge of the right-side visual blind spot. Although this target occupies only a tiny percentage of pixels in the original image, its significant lateral displacement relative to the static background prompts the model's large cell pathway response layer to immediately generate a high-intensity risk activation signal, prioritizing visual processing resources for this region.

[0042] Step 3's posture analysis module identified the dynamic target as a workshop worker pushing a manual hydraulic pallet trolley. The neural network accurately located the worker's head and torso key points. Semantic analysis showed that the worker's head was tilted to the left, seemingly observing a shelf on the other side, not focusing on the path ahead; simultaneously, his gait characteristics showed a forward-leaning body and heavy steps, consistent with the dynamics of pushing a heavy object. This information was encapsulated in the intent representation vector.

[0043] In the social force model calculation in step 4, the system identified that due to the worker's deviated line of sight, the psychological repulsive force field exerted on the AGV exhibited an asymmetrical distribution, with the repulsive force slope towards the AGV being steeper. Based on this force model, the trajectory prediction module determined that the worker had a very high probability of maintaining their original straight-line movement, and due to the inertia of carrying heavy objects, the possibility of them suddenly stopping was low. The predicted probability distribution map showed that the worker's trajectory would cross the AGV's predetermined global path in the next 2.5 seconds.

[0044] In step 5, the AGV system did not employ the emergency braking strategy used in traditional algorithms, as this would cause congestion for other following AGVs. Instead, the system calculated a local path within the dynamic feasible region that slightly shifted to the left and reduced speed. This path utilized a social buffer zone mechanism, ensuring a safe social distance of 1.5 meters from the workers while using the remaining space at intersections to detour.

[0045] Step 6, the smooth fusion module, transforms this left-leaning detour path into a smooth Ackerman steering control sequence. The AGV gracefully passes behind the worker while maintaining a speed of approximately 0.8 meters per second. The worker does not need to stop the transport vehicle, and the AGV avoids the risk of deadlock. Throughout the process, the closed-loop monitoring system continuously detects the worker's turning intentions. Once a shift in the worker's center of gravity is detected, the system immediately recalculates the trajectory and adjusts the commands within the next millisecond cycle.

[0046] This implementation fully demonstrates the superiority of the present invention in handling complex and dynamic environments. By simulating human visual attention and social perception, it significantly improves the operating efficiency and safety of unmanned AGVs in industrial scenarios.

[0047] Example 2: Based on Example 1, Example 2 further refines the robust implementation of the computer vision-based obstacle avoidance method for unmanned AGVs in new energy vehicles under extreme lighting and high-density human-machine collaboration environments.

[0048] In step 1, considering the unique environment of new energy vehicle painting workshops with numerous glass curtain walls and highly reflective metal surfaces, polarization filtering preprocessing is introduced into the image sequence acquisition process of the visual sensor array. Each camera lens is equipped with an automatically rotating polarization filter. The system dynamically adjusts the polarization angle of the filter by detecting the proportion of overexposed pixels in the image to eliminate strong glare interference from the metal vehicle body.

[0049] For geometric correction, the system employs a deep learning-based feature point descriptor algorithm. Traditional feature matching methods are prone to failure under varying lighting intensities. This embodiment extracts semantic features with rotation and brightness invariance to ensure that the generated panoramic visual data stream maintains sub-pixel registration accuracy even during image stitching, despite drastic fluctuations in ambient lighting. Furthermore, the generated unified coordinate system not only covers the horizontal plane but also integrates data from an attitude sensor (IMU) to achieve real-time compensation for AGV bumps and vibrations, making the panoramic video stream as smooth as if it were mounted on a stable gimbal.

[0050] In step 2, the construction of the dynamic visual attention model incorporates a feedback adjustment mechanism. When too many salient regions are identified, causing the computational load to exceed a preset threshold, the system automatically reduces the search radius based on the current driving speed. At low speeds, the model focuses more on subtle movements at close range; at high speeds, it prioritizes responding to distant, high-speed moving targets by changing the receptive field size of the convolutional kernel.

[0051] To improve recognition accuracy, a dynamic background model pool is established within the model. Utilizing the statistical characteristics of multiple consecutive frames, the system can distinguish between periodic disturbances in the environment (such as rotating exhaust fans or reciprocating mechanisms on assembly lines) and truly risky non-periodic dynamic targets. The values ​​of these disturbance terms in the target attention weight map are forcibly reduced to zero, focusing computational resources on dynamic entities that may collide or conflict.

[0052] In step 3, for situations where workers are wearing heavy work clothes or carrying obstructions (such as cardboard boxes), a partially occluded robust network structure is used for posture and gait feature analysis. When the lower limbs are occluded, the network analyzes the swing rhythm of the upper limbs and the tilt angle of the torso, utilizing the temporal memory capability of a recurrent neural network (RNN) to infer the motion state of the occluded part. This virtual keypoint completion technique allows the intent representation vector to maintain high representation dimensionality and accuracy even with missing information.

[0053] Furthermore, the extraction of head gaze direction enhances adaptability to features such as workwear like goggles or safety helmets. The system uses the orientation of the helmet brim as a supplementary reference for head posture, ensuring reliable behavioral semantic information is still obtained even when the eye gaze point is not clearly discernible. This refined information is further subdivided into multiple semantic labels such as "active interaction," "unconscious movement," and "emergency avoidance," and incorporated into the intent representation vector.

[0054] In step 4, the computational logic of the social force model is further extended to a multi-agent game model. When there are multiple pedestrians and multiple AGVs in the scene, each dynamic target will also generate mutual attraction and repulsion forces on other targets. The system constructs a global interaction force field matrix.

[0055] The calculation of psychological repulsion is no longer simply a function of distance between two points, but incorporates a penalty term for relative acceleration. If predictions indicate that two objects will approach each other with high relative acceleration in the future, the repulsion increases dramatically. The calculation of target attraction, on the other hand, incorporates a knowledge base of the workshop's technological processes. For example, when a worker is detected moving towards a material rack, the system focuses the probability distribution of their destination in front of the rack, generating a strong attractive vector pointing towards it. In this way, the predicted trajectory more closely aligns with the actual logic of an industrial setting.

[0056] In step 5, the construction of the dynamic feasible region takes into account the impact of ground friction coefficient and load weight on obstacle avoidance capability. The system acquires real-time data from onboard load cells and Electronic Stability Program (ESP). Under full load, due to the increased braking distance, the system automatically expands the radius of the core no-entry zone within the feasible region boundary.

[0057] The local obstacle avoidance path planning incorporates obstacle avoidance logic based on the potential field method. The system assigns a value to each grid point within the feasible region. A high-potential-energy zone forms around the predicted trajectory of the dynamic target, while a low-potential-energy valley forms along the global path direction. The planner, like a small ball rolling in the undulating potential energy field, automatically seeks the path of least resistance. To ensure traffic efficiency, the system sets a minimum path deviation threshold; only when the predicted collision risk exceeds a certain probability is a significant path change triggered, avoiding frequent turning caused by minor pedestrian movements.

[0058] In step 6, the closed-loop feedback adjustment mechanism adds predictive compensation for execution deviations. Due to wear on the chassis mechanical structure or uneven ground, the actual execution path may deviate slightly from the commanded path. The system calculates the AGV's actual pose in real time using a visual odometry system and compares it with the ideal path.

[0059] If the deviation value continues to increase, the smoothing fusion module dynamically adjusts the tension coefficient of the spline curve, quickly correcting the deviation by increasing the steering gain. Simultaneously, the generation of driving commands employs a feedforward control strategy, adjusting the motor torque output in advance based on the predicted road conditions for the next moment. Throughout the entire operation, the system log records all obstacle avoidance decision-making processes and raw sensor data, uploading them in real-time to the cloud monitoring center via the workshop wireless network for offline iterative optimization of the obstacle avoidance model.

[0060] Example 3: In this example, the safety assurance mechanism and multi-sensor fusion enhancement strategy of the computer vision-based new energy vehicle unmanned AGV obstacle avoidance method in response to sudden emergency situations are described in detail.

[0061] In step 1, to prevent single points of failure, the vehicle-mounted multi-view vision sensor array incorporates a redundancy backup mechanism. The system includes not only visible light cameras but also a set of long-wave infrared thermal imaging sensors. In the event of a sudden power outage or excessive smoke concentration in the workshop, the system automatically switches to the infrared vision channel. The infrared image sequence is spatiotemporally aligned with the visible light images, generating a panoramic visual data stream that includes temperature dimension features. This allows the system to accurately locate personnel in a scene even in darkness using thermal characteristics.

[0062] The geometric correction process simultaneously addresses the non-uniformity correction of the thermal imaging sensor, ensuring that the panoramic image retains high spatial resolution after multispectral fusion. The generated panoramic visual data stream is transmitted to the processor in the form of a multi-channel tensor, which includes three color channels (red, green, and blue) and one thermal radiation intensity channel.

[0063] In step 2, the dynamic visual attention model introduces a "fright trigger" mechanism to simulate the rapid defensive response of the human visual system to high-speed approaching objects. When the pixel change rate of a certain area in the panoramic data stream exceeds a set safety threshold (e.g., representing an object approaching head-on at a speed exceeding 3 seconds per second), the model will skip the complex large cell pathway analysis and directly mark the area as the highest-level salience area, triggering the emergency braking of the underlying safety controller.

[0064] Under normal conditions, the saliency region extraction process incorporates depth information. The system utilizes a disparity map generated by multi-view vision, using the depth value of pixels as a saliency adjustment factor. Objects closer to the AGV and with more dramatic depth changes receive higher weights in the target attention weight map. This 3D attention mechanism effectively filters out irrelevant dynamic interference from the distant background.

[0065] In step 3, the construction of the intent representation vector adds environmental context awareness. The system visually identifies landmark facilities within the workshop, such as charging stations, automatic roller shutters, or emergency exits. If a dynamic target is near these facilities, its behavioral semantic information will be assigned specific logical meanings. For example, when a worker approaches an emergency exit, the probability component of "leave quickly" in their intent vector will automatically increase.

[0066] The gait analysis module also includes the ability to identify abnormal behaviors. If it detects specific postures such as slipping, prolonged lingering, or waving for help, the system encodes these abnormal features into an intent vector and simultaneously triggers an alarm signal. This semantic analysis not only serves obstacle avoidance but has also become part of the safety monitoring system for new energy vehicle factories.

[0067] In step 4, the social force model introduces an energy cost term. When calculating the predicted trajectory, the system not only seeks a force field equilibrium point but also attempts to minimize the "energy expenditure" required for the pedestrian to change their motion. According to biomechanical principles, pedestrians typically tend to avoid obstacles by navigating smooth curves rather than turning in place.

[0068] Therefore, when calculating the target's attractiveness, the system assigns a significant inertial weight to the original direction of motion. The predicted trajectory generated within a predetermined future timeframe is a set of weighted multimodal paths. Each path corresponds to a different social intent hypothesis, such as "the pedestrian chooses to stop and give way" or "the pedestrian chooses to speed up and rush across." The system updates the probability weights of these hypotheses in real time through Bayesian inference and selects the set of trajectories with the highest probability as the basis for obstacle avoidance.

[0069] In step 5, the construction of the dynamic feasible region introduces a four-dimensional spatiotemporal tunnel model that includes a time dimension. The feasible region defines where the AGV cannot appear at a certain moment and its safe location for a certain period of time in the future. Social constraints are quantified as a kind of "repulsive potential field." In high-density crowds, this potential field will spread like ripples, guiding the AGV to actively seek gaps in the crowd when planning its path.

[0070] The safety distance setting is dynamically adaptive. At high speeds, the safety distance increases with the square of the speed; however, in confined spaces such as narrow passages, the system compresses the safety distance to the centimeter level by increasing the sensor sampling frequency and reducing vehicle speed. Traffic efficiency constraints are implemented through a multi-objective planner, utilizing the Pareto optimality principle to achieve a balance between obstacle avoidance risk and travel path length.

[0071] In step 6, the smoothly fused driving commands are sent to a redundant execution module with functional safety standards. The driving commands include not only motion parameters but also heartbeat detection signals. If the chassis controller does not receive a closed-loop feedback command within a specified period, it will automatically execute a safety shutdown.

[0072] The closed-loop feedback adjustment process integrates the prediction of obstacle occlusion risk. If the prediction indicates that the AGV's detour path would obstruct the sensor's observation of another dynamic target, the system will adjust the detour radius to maintain the optimal observation angle. Simultaneously, the system continuously monitors the quality of the panoramic visual data stream. If a camera is found to be obstructed by dirt or experiences a hardware failure, the fusion algorithm will immediately switch to a degraded operating mode, utilizing the overlapping field of view of the remaining cameras for coverage, or ensuring safety by reducing the maximum travel speed.

[0073] The generated driving commands not only drive the motor's movement but also coordinate with the AGV's audible and visual warning system. When avoiding obstacles, the AGV projects directional lights or emits gentle voice prompts towards pedestrians based on the predicted intent, achieving two-way social communication. This comprehensive implementation ensures the deep integration and efficient operation of new energy vehicle-driven AGVs within complex industrial ecosystems.

[0074] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. An obstacle avoidance method for unmanned AGVs (Automated Guided Vehicles) of new energy vehicles based on computer vision, characterized in that, The specific steps include the following: Step 1: Real-time acquisition of continuous image sequences of the surrounding environment through an on-board multi-view vision sensor array, and spatiotemporal synchronization and geometric correction of the image sequences to generate a panoramic vision data stream with a unified coordinate system; Step 2: Construct a dynamic visual attention model based on the macrocellular pathway mechanism in neuroscience, extract salient regions from the panoramic visual data stream, identify dynamic targets with high motion risk or behavioral uncertainty in the scene, and generate corresponding target attention weight maps. Step 3: Perform posture and gait feature analysis on the dynamic target, extract its body orientation, limb movement trend and head gaze direction behavioral semantic information, and form a structured intent representation vector; Step 4: Treat the dynamic target as a particle with social attributes, combine it with the social force model, and calculate the psychological repulsion and attraction of the target to other moving entities based on the intention representation vector, so as to predict its trajectory within a predetermined time in the future. Step 5: Based on the predicted motion trajectory and the current state of the AGV, construct a dynamic feasible region that includes social constraints, and plan a local obstacle avoidance path that satisfies both safety distance and passage efficiency constraints within the feasible region; Step 6: Smoothly integrate the local obstacle avoidance path with the global task path to generate the final driving command and drive the AGV to execute it, while continuously monitoring environmental changes to achieve closed-loop feedback adjustment.

2. The obstacle avoidance method for unmanned AGVs of new energy vehicles based on computer vision according to claim 1, characterized in that, In step 1, the vehicle-mounted multi-view vision sensor array consists of multiple wide-angle cameras distributed in front, left front, and right front of the AGV; The spatiotemporal synchronization process sends pulse signals to all cameras through a hardware synchronization triggering mechanism to ensure that each camera performs photoelectric conversion and sampling at the same time. The geometric correction process performs distortion correction on the original image sequence by calling the intrinsic parameter matrix and extrinsic parameter matrix stored in the memory. The intrinsic parameter matrix includes focal length, principal point coordinates, and radial and tangential distortion coefficients. The extrinsic parameter matrix defines the rotation matrix and translation vector of each camera relative to the coordinate system of the AGV body center. The distortion-free image is projected onto a unified virtual sphere using a perspective transformation algorithm. A weighted average fusion algorithm is then applied to the overlapping areas of the images, with weights assigned based on the radial distance of each pixel from the image center. Pixels closer to the center have higher weights, thus eliminating parallax interference and generating a panoramic visual data stream with spatial continuity and temporal consistency.

3. The obstacle avoidance method for unmanned AGVs of new energy vehicles based on computer vision according to claim 1, characterized in that, In step 2, the dynamic visual attention model extracts salient regions through the following sub-steps: Multi-scale Gaussian pyramid decomposition is performed on the panoramic visual data stream to extract image features of different frequency components; direction-selective convolution kernels are used to simulate the receptive field of simple cells to identify high-contrast edges in the image. The pixel displacement between consecutive image frames is calculated by the spatiotemporal gradient joint analysis method. For each pixel position, its partial derivatives in the horizontal and vertical directions are calculated, and the optical flow vector of the point is derived by combining the pixel value change rate in the time dimension. In response to regions with large and non-periodic changes in motion vector magnitude, static textures in the background are suppressed, while features of fast-moving regions or suddenly appearing objects are enhanced. The risk activation value of each pixel is output and a saliency distribution map is constructed. The saliency distribution map is binarized using a dynamically adjusted threshold, and high-risk dynamic targets and their spatial locations in the physical world coordinate system are determined by a connected component merging algorithm and a non-maximum suppression algorithm, ultimately generating the target attention weight map.

4. The obstacle avoidance method for unmanned AGVs of new energy vehicles based on computer vision according to claim 1, characterized in that, In step 3, key point regression is performed on the image sub-region of the dynamic target using a deep convolutional neural network. The neural network includes a residual connection structure and a key point localization layer based on heatmap regression, which is used to locate human key points. The key points include the tip of the nose, both eyes, both ears, both shoulders, both elbows, both wrists, both hips, both knees, and both ankles. The process of calculating the behavioral semantic information based on the key points includes: determining the torso axis by connecting the midpoints of the shoulders and the midpoints of the hips to represent the body orientation; The body's torsional tendency is identified by the dynamic changes in the angle between the shoulder and hip lines; the phase difference of the leg swing, stride length, and the time interval between the feet touching the ground are extracted from a series of multiple frames of skeletal sequences to identify gait characteristics. A gaze vector based on a three-dimensional sphere is constructed by identifying facial key points and combining them with a head pose estimation algorithm to determine the head gaze direction. The body orientation, limb movement rate, head yaw angle, and gait stability indices are encoded as fixed-dimensional floating-point vectors, i.e., the intent representation vectors.

5. The obstacle avoidance method for unmanned AGVs of new energy vehicles based on computer vision according to claim 1, characterized in that, In step 4, the psychological repulsion force simulates the psychological tendency of a dynamic target to maintain personal space during movement. Its magnitude is proportional to the relative speed between the dynamic target and the AGV, and the magnitude of the repulsion force increases exponentially when the relative speed directions point towards each other and the approach angle becomes smaller. Simultaneously, the force field distribution is adjusted according to the gaze direction in the intent representation vector. When the gaze of the dynamic target is not directed toward the AGV, the weighting coefficient of the psychological repulsion force is increased. The target attraction is determined by analyzing the trajectory of the dynamic target over a predetermined historical period and fitting its main motion direction using the least squares method. The main motion direction is regarded as the attraction vector pointing to the virtual destination. The resultant force acting on the dynamic target is obtained by vector summing the psychological repulsion vector and the target attraction vector. Based on the second-order differential equation of Newton's laws of motion, the resultant force drives the change in acceleration, thereby changing the velocity and position of the dynamic target. The probability distribution cloud map of the position of the dynamic target in the future within a predetermined time is predicted by sampling calculation.

6. The obstacle avoidance method for unmanned AGVs of new energy vehicles based on computer vision according to claim 1, characterized in that, In step 5, the dynamic feasible region is defined as a continuous space in a spatiotemporal coordinate system; When constructing the dynamic feasible domain, the current linear velocity, angular velocity, braking performance parameters and physical contour dimensions of the AGV are first read, and social constraints are introduced to set a hierarchical safety buffer, which includes a core restricted area and a social transition area. The radius of the core restricted area is set according to the emergency braking distance of the AGV, and the range of the social transition zone is set with reference to the human comfort standards of the industrial site; in high-density pedestrian areas, the safety margin is expanded by increasing the social restraint level. When planning local obstacle avoidance paths within the dynamic feasible domain, a cost function is used to evaluate candidate path points. The cost function includes the reciprocal of the distance to the obstacle, the square of the path curvature, and the deviation from the global path. By using dynamic windowing or spatiotemporal search tree optimization strategies, velocity pairs that meet dynamic constraints are searched in the velocity space. Multiple candidate trajectories are simulated and trajectories that intrude into the boundary of the dynamic feasible domain are eliminated to ensure that the passage space of other people or vehicles is not intruded during the avoidance process.

7. The obstacle avoidance method for unmanned AGVs of new energy vehicles based on computer vision according to claim 1, characterized in that, In step 6, the fusion of the local obstacle avoidance path and the global task path adopts a quintic spline interpolation algorithm. The quintic spline interpolation algorithm ensures that the path is continuous in position at the connection point, continuous in velocity direction represented by the first derivative, and continuous in curvature represented by the second derivative, so as to reduce abrupt changes in direction and acceleration fluctuations. The generated driving command is sent to the chassis controller through the vehicle bus system. The chassis controller uses proportional-integral-derivative control logic or model predictive control algorithm to track the command path by controlling the speed of the drive motor and the angle of the steering mechanism. The closed-loop feedback adjustment process calculates the actual pose of the AGV in real time using a visual odometer and compares the actual pose with the ideal path. When the deviation value continues to increase, the tension coefficient of the spline curve is dynamically adjusted and the steering gain is increased to correct the deviation.

8. The obstacle avoidance method for unmanned AGVs of new energy vehicles based on computer vision according to claim 1, characterized in that, For high-reflection environments, during the image sequence acquisition process in step 1, an automatically rotating polarizing filter is installed in front of the camera. The system dynamically adjusts the polarization angle of the filter by detecting the proportion of overexposed pixels in the image to eliminate glare interference. During geometric correction, semantic features with rotation and brightness invariance are extracted as feature point descriptors to ensure registration accuracy under drastic fluctuations in ambient light. Simultaneously, data from the integrated attitude sensor is used to compensate for the AGV's own bumps and vibrations in real time, maintaining the stability of the panoramic vision data stream. In step 2, a feedback adjustment mechanism is introduced. When the number of salient regions exceeds a preset threshold, the search radius is adjusted according to the driving speed, and a dynamic background model pool is established. The statistical characteristics of multiple consecutive frames are used to distinguish between periodic interference and non-periodic dynamic targets in the environment. The values ​​of interference items in the target attention weight map are reduced to zero, and the focus is on the real physical conflict objects.

9. The obstacle avoidance method for unmanned AGVs of new energy vehicles based on computer vision according to claim 1, characterized in that, To improve safety, a multi-spectral redundancy backup mechanism is introduced in step 1. The vehicle-mounted multi-view vision sensor array also integrates a long-wave infrared thermal imaging sensor. When visible light vision is blocked, it automatically switches to the infrared vision channel and spatiotemporally aligns the infrared image sequence with the visible light image to generate a multi-channel tensor panoramic vision data stream containing temperature dimension features. In step 2, a panic triggering mechanism is introduced. When the pixel change rate of a local area in the panoramic visual data stream exceeds a preset safety limit threshold, the area is directly marked as the highest level of salience and emergency braking is triggered. In the construction of the intent representation vector, environmental context awareness is added, and the probability components in the intent vector are adjusted by identifying the iconic facilities in the workshop and their geographical locations. If a person is detected to have slipped or to be waving for help, the abnormal features are encoded into the intent vector and an alarm signal is triggered.

10. The obstacle avoidance method for unmanned AGVs of new energy vehicles based on computer vision according to claim 1, characterized in that, In step 4, an energy cost term is introduced when calculating the predicted motion trajectory. Based on biomechanical principles, the original motion direction of the dynamic target is given a large inertial weight. Bayesian inference is used to update multiple hypothetical probabilities about the pedestrian's social intentions in real time, and the trajectory with the highest probability is selected as the basis for obstacle avoidance. In step 5, the dynamic feasible domain is extended to a four-dimensional spatiotemporal tunnel model, and the social constraints are quantified into a repulsive potential field that changes over time, guiding the AGV to actively seek gaps in pedestrian flow when planning its path. The safety distance is dynamically and adaptively adjusted according to the AGV speed. In high-speed sections, the safety distance increases with the square of the speed, while in confined spaces, the safety distance is compressed by increasing the sampling frequency and reducing the vehicle speed. In step 6, the driving command includes a signal for heartbeat detection. If the chassis controller does not receive a valid closed-loop feedback command within a specified period, a safety shutdown is performed. Simultaneously monitor the risk of sensor obstruction by the detour path, and adjust the detour radius to maintain the observation view of other dynamic targets.

Images