Image recognition-based intelligent control method and system for sprinkler truck to avoid pedestrians

Abstract

The application provides a kind of image recognition-based intelligent control method and system for water sprinkler to avoid pedestrians, relating to image recognition control technical field, the application obtains pedestrian skeleton points and carrying object profile through multi-view image, maps to vehicle coordinate system and calculates motion trend vector according to time sequence change, combines spray arm state to predict nozzle trajectory.Then calculate the spatial intersection of each motion trend and predicted trajectory, generate avoidance priority according to volume and time.Input priority, skeleton point time sequence and carrying object profile into fusion model for spatio-temporal correlation analysis, get the risk distribution map of pedestrian future danger probability.Finally, based on the risk distribution map, under the condition of maximizing the spray coverage area and avoiding pedestrians, plan the joint trajectory of the spray arm and the nozzle start-stop time sequence, issue control instructions, which can accurately predict the direction of pedestrians using image recognition, realize intelligent avoidance of water sprinkler, and ensure safe and efficient operation.

Description

Technical Field

[0001] This application relates to the field of image recognition control technology, and in particular to an intelligent control method and system for a sprinkler truck to avoid pedestrians based on image recognition. Background Technology

[0002] Urban road watering and cleaning is a core part of sanitation operations. An intelligent control method for water trucks to avoid pedestrians, based on image recognition, is a key technological direction for upgrading smart sanitation. This technology relies on visual perception to achieve autonomous avoidance, improving operational safety and humanization, adapting to densely populated urban environments, and possessing broad prospects for practical application.

[0003] Current mainstream intelligent control methods for pedestrian avoidance on sprinkler trucks mostly use a monocular camera to collect surrounding images and calculate the safe distance simply by identifying the pedestrian's position, triggering a spray pause or water pressure adjustment command. This solution relies on a single visual dimension for shallow distance determination and is currently the standard application method for intelligent avoidance on sanitation sprinkler trucks.

[0004] These conventional avoidance methods cannot accurately predict the spatiotemporal conflicts between pedestrian movements and spraying operations, easily leading to problems such as delayed avoidance, mis-avoidance, or insufficient spray coverage, making it difficult to achieve both safety and operational efficiency. Furthermore, they fail to consider interference factors such as pedestrians carrying items and the movement of the spraying mechanism, significantly reducing the overall avoidance effectiveness. Therefore, existing technologies suffer from insufficient avoidance accuracy and an inability to balance safety and operational efficiency. Summary of the Invention

[0005] The purpose of this application is to provide an intelligent control method and system for water sprinkler trucks to avoid pedestrians based on image recognition, so as to solve the problems of insufficient avoidance accuracy and inability to balance safety and operation efficiency in the existing technology.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides an intelligent control method for a sprinkler truck to avoid pedestrians based on image recognition, comprising: Collect multi-view images of the environment surrounding the sprinkler truck, obtain key skeletal point data of pedestrians from the multi-view images through human pose estimation, and extract the contour features of objects carried by pedestrians from the multi-view images. The key skeletal point data is mapped to a coordinate system centered on the sprinkler truck. The pedestrian's movement trend vector is calculated based on the temporal changes of the key skeletal point data. The predicted movement trajectory of the nozzle at the end of the sprinkler arm is calculated by combining the real-time movement state of the sprinkler truck's spray arm. Calculate the spatial intersection region between each of the motion trend vectors and the predicted motion trajectory, and generate an avoidance priority list based on the size of the spatial intersection region and the order of the intersection time; The avoidance priority list, the temporal changes of the key skeletal point data, and the contour features of the pedestrian's carried objects are input into a preset fusion model for spatiotemporal feature correlation analysis to generate a risk distribution map. The risk distribution map is used to characterize the probability of danger for the pedestrian occupying a spatial position in the future. Based on the risk distribution map, under the conditions of maximizing the spray coverage area and avoiding pedestrians, the joint motion trajectory of the spray arm and the start-stop sequence of the nozzle are calculated, and corresponding motion control commands and valve control commands are generated and issued.

[0007] Optionally, the step of inputting the avoidance priority list, the temporal changes of the key skeletal point data, and the contour features of the pedestrian's carried objects into a preset fusion model for spatiotemporal feature correlation analysis to generate a risk distribution map includes: The avoidance priority list, the temporal changes of the key skeletal point data, and the contour features of the pedestrian's carried objects are all input into a preset fusion model; The graph convolutional network in the fusion model is used to perform graph convolution operations on the key skeletal point data to extract the spatial interaction relationships between different pedestrians and between different key points within the same pedestrian. The long short-term memory network in the fusion model is used to perform temporal feature extraction on the temporal changes of the key skeletal point data in order to capture changes in the pedestrian's movement pattern; Based on the avoidance priority list, a weight coefficient is assigned to each pedestrian; By combining the weighting coefficients, the spatial interaction relationships, the changes in motion patterns, and the contour features of the pedestrian's carried items, a potential spatial range that each pedestrian may occupy at a future time point is defined in the three-dimensional spatial coordinate system. A risk value is assigned to each spatial location point within the potential spatial range, and the risk values ​​of all pedestrians are superimposed in the three-dimensional spatial coordinate system to generate a risk distribution map, which is used to characterize the spatial distribution probability of pedestrians at future times.

[0008] Optionally, based on the risk distribution map, and under the conditions of maximizing spray coverage area and avoiding pedestrians, the joint motion trajectory of the spraying arm and the start-stop sequence of the nozzles are calculated, and corresponding motion control commands and valve control commands are generated and issued, including: Within the preset kinematic constraints of the spraying arm, multiple candidate joint motion trajectories are planned; For each candidate joint motion trajectory, calculate the risk cost and spray coverage benefit. The risk cost is used to characterize the degree of overlap between the candidate joint motion trajectory and the high-risk area in the risk distribution map, and the spray coverage benefit is used to characterize the total spray coverage area corresponding to the candidate joint motion trajectory. Based on the risk cost and the spray coverage benefit, the joint movement trajectory with the lowest risk cost and the highest spray coverage benefit is selected from multiple candidate joint movement trajectories and used as the target joint movement trajectory. By comparing the target joint movement trajectory with the risk distribution map, the time period in which the spraying area overlaps with the high-risk area is determined, and the start-stop sequence of the nozzle is generated accordingly. The target joint motion trajectory and the start / stop timing sequence are converted into motion control commands and valve control commands, respectively, and sent to the corresponding actuators.

[0009] Optionally, the step of calculating the spatial intersection region between each of the motion trend vectors and the predicted motion trajectory, and generating an avoidance priority list based on the size of the spatial intersection region and the order of intersection time, includes: Based on the movement trend vector of each pedestrian, the current spatial position of the pedestrian is extrapolated to define the estimated space occupied by the pedestrian in the future time period. Based on the predicted motion trajectory and combined with the preset spraying range of the nozzle, the spraying operation space to be sprayed and covered by the nozzle at the end of the spraying arm in the future time period is determined. Calculate the spatial intersection area between the estimated space occupied by each pedestrian and the spraying operation space, and determine the volume of the spatial intersection area and the first occurrence time of the spatial intersection area; Sort all pedestrians in the spatial intersection area: sort them in ascending order based on the first appearance time as the first sorting criterion; when the first appearance times are the same, sort them in descending order based on the volume of the spatial intersection area as the second sorting criterion. A priority list for yielding is generated based on the sorting results. This priority list is used to indicate the order in which pedestrians should yield to the sprinkler truck.

[0010] Optionally, the step of combining the weighting coefficients, the spatial interaction relationships, the changes in motion patterns, and the contour features of the pedestrian's carried objects to define a potential spatial range for each pedestrian in a three-dimensional spatial coordinate system at a future point in time includes: The items carried by pedestrians are classified according to their outline characteristics, and corresponding spatial expansion coefficients are set for different types of items. For each pedestrian, based on the current spatial location, spatial extrapolation is performed by combining the motion trend vector and the change in the motion pattern to generate an initial spatial range; The initial spatial range is adjusted using the spatial interaction relationship to generate a first intermediate spatial range, which reflects the interaction between pedestrians. The first intermediate space range is expanded using the aforementioned spatial expansion coefficient to generate a second intermediate space range, which takes into account the additional space occupied by the carried items. The second intermediate spatial range is scaled in combination with the weighting coefficient to determine the final potential spatial range of the pedestrian.

[0011] Optionally, the step of mapping the key skeletal point data to a coordinate system centered on the sprinkler truck, calculating the pedestrian's motion trend vector based on the temporal changes of the key skeletal point data, and calculating the predicted motion trajectory of the nozzle at the end of the sprinkler arm in conjunction with the real-time motion state of the sprinkler truck's spray arm includes: The key body position points in the key skeletal point data are transformed into a three-dimensional spatial coordinate system with the center of the sprinkler truck as the origin to obtain the spatial position of the skeletal points. Multiple sets of the spatial locations of the skeletal points are obtained in a continuous time series to form a time sequence of changes in key points of the pedestrian's body. Based on the change sequence, the direction and speed of movement of the pedestrian's body representative point are determined, and a motion trend vector is generated. The motion trend vector is used to characterize the pedestrian's future movement trend. Meanwhile, based on the current joint motion parameters and mechanical connection structure of the sprinkler arm, the predicted motion trajectory of the nozzle at the end of the sprinkler arm within a preset time period is calculated.

[0012] Optionally, before determining the direction and speed of movement of the pedestrian's body representative point based on the change sequence and generating the motion trend vector, the method further includes: Obtain the real-time driving speed and real-time driving direction of the sprinkler truck; Based on the real-time driving speed and the real-time driving direction, motion compensation is performed on the change sequence to eliminate the influence of the water truck's own movement on the relative position change of pedestrians; The motion trend vector is calculated based on the compensated change sequence.

[0013] Secondly, this application provides an image recognition-based intelligent control system for sprinkler trucks to avoid pedestrians, comprising: The acquisition module is used to acquire multi-view images of the environment surrounding the sprinkler truck, obtain key skeletal point data of pedestrians from the multi-view images through human pose estimation, and extract the contour features of objects carried by pedestrians from the multi-view images. The calculation module is used to map the key skeletal point data to a coordinate system centered on the sprinkler truck, calculate the pedestrian's motion trend vector based on the temporal changes of the key skeletal point data, and calculate the predicted motion trajectory of the nozzle at the end of the sprinkler arm in combination with the real-time motion state of the sprinkler truck's spray arm. The first generation module is used to calculate the spatial intersection region between each of the motion trend vectors and the predicted motion trajectory, and generate an avoidance priority list based on the volume of the spatial intersection region and the order of the intersection time. The analysis module is used to input the avoidance priority list, the temporal changes of the key skeletal point data and the contour features of the pedestrian's carried objects into a preset fusion model for spatiotemporal feature correlation analysis, and generate a risk distribution map. The risk distribution map is used to characterize the probability of danger of the pedestrian occupying a spatial position in the future. The second generation module is used to calculate the joint motion trajectory of the spraying arm and the start-stop sequence of the nozzle based on the risk distribution map, under the conditions of maximizing the spray coverage area and avoiding pedestrians, and to generate and issue corresponding motion control commands and valve control commands.

[0014] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the steps of the image recognition-based intelligent control method for pedestrian avoidance by a sprinkler truck as described in the first aspect above.

[0015] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the image recognition-based intelligent control method for water trucks to avoid pedestrians as described in the first aspect above.

[0016] The image recognition-based intelligent control method for pedestrian avoidance of sprinkler trucks provided in this application can eliminate blind spots and comprehensively acquire information about pedestrians and surrounding obstacles by collecting multi-view images and extracting key skeletal points of pedestrians and the contour features of their carried objects. By mapping the skeletal points to the sprinkler truck's central coordinate system and calculating the pedestrian's movement trend and the nozzle trajectory, it can unify the spatial reference and accurately predict the dynamic movements of pedestrians and the spraying mechanism. By calculating the spatial intersection and generating an avoidance priority list, it can quantify the conflict risk, clarify the avoidance order, and avoid disorderly avoidance. By performing spatiotemporal feature analysis through a fusion model to generate a risk distribution map, it can intuitively quantify the future danger probability of pedestrians and provide data support for decision-making. Based on the risk distribution map, it can plan the spray arm trajectory and the nozzle start-stop sequence and issue commands, which can maximize the spray coverage area while ensuring pedestrian safety, and balance operational safety and efficiency.

[0017] Furthermore, the avoidance priority list, temporal changes of key pedestrian skeletal points, and the outline features of carried objects are simultaneously input into a pre-defined fusion model. First, a graph convolutional network is used to extract the spatial interaction relationships between pedestrians and between skeletal points. Then, a long short-term memory network is used to capture changes in pedestrian movement patterns. Weight coefficients are assigned to pedestrians based on the avoidance priority, thereby defining the potential future spatial range of pedestrians and assigning risk values. Finally, all pedestrian risk values ​​are superimposed to generate a risk distribution map representing the probability of future pedestrian spatial distribution. This step achieves deep mining of spatiotemporal features through graph convolution and long short-term memory networks, accurately capturing pedestrian interaction patterns and dynamic movement trends. Combining priority weights and carried object features refines the risk judgment dimensions, making the potential space delineation more closely aligned with actual scenarios. The accuracy and reliability of the risk distribution map are significantly improved, providing a more reliable basis for subsequent avoidance control. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating an intelligent control method for a sprinkler truck to avoid pedestrians based on image recognition, provided in an embodiment of this application; Figure 2 A flowchart illustrating another intelligent control method for water trucks to avoid pedestrians based on image recognition, provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an intelligent control system for a sprinkler truck to avoid pedestrians based on image recognition, provided in an embodiment of this application. Detailed Implementation

[0020] Existing intelligent sanitation sprinkler truck pedestrian avoidance technologies mostly rely on monocular cameras, which can only superficially determine distance and cannot predict pedestrian dynamic trajectories. Furthermore, they ignore factors such as carried items and spray arm status, easily leading to problems like improper avoidance and poor spraying effects, making it difficult to balance safety and operational efficiency. This application proposes an image recognition-based intelligent control method for sprinkler truck pedestrian avoidance. Through multi-view image acquisition, dynamic trajectory prediction, conflict risk quantification, and trajectory planning control, it achieves full-process optimization, fundamentally improving avoidance accuracy and balancing pedestrian safety with sprinkler operation efficiency.

[0021] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] The core of this application is to provide an intelligent control method for water trucks to avoid pedestrians based on image recognition. A flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes: S101. Collect multi-view images of the surrounding environment of the sprinkler truck, obtain key skeletal point data of pedestrians from the multi-view images through human pose estimation, and extract the contour features of objects carried by pedestrians from the multi-view images.

[0023] Multi-view images refer to image data covering the area surrounding the vehicle, simultaneously collected by multiple cameras installed at different locations on the sprinkler truck body, used to eliminate blind spots that may occur from a single viewpoint. Human pose estimation is a deep learning-based image processing technique used to identify and locate key parts of the human body from images. Key skeletal point data refers to the coordinate information of key points representing the skeletal structure of pedestrians, obtained through human pose estimation techniques, such as the spatial positions of joints like the head, shoulders, elbows, wrists, hips, and knees. Contour features of items carried by pedestrians refer to the edge information of the external shape of items carried by pedestrians, segmented from the image, such as the outline shape of umbrellas, backpacks, and handcarts.

[0024] The purpose of this step is to comprehensively acquire static and dynamic information about pedestrians around the sprinkler truck through multi-view visual perception technology, providing basic data for subsequent avoidance decisions.

[0025] In practical applications, a high-definition camera is installed on each of the four sides of the sprinkler truck, forming a multi-view image acquisition system. When the sprinkler truck is operating on urban roads, these four cameras capture video streams of the surrounding environment in real time. Assuming that a camera captures a frame of image on a certain road section, the image is processed using a human pose estimation model to identify two pedestrians and obtain the key skeletal point data of the first pedestrian, including the pixel coordinates of seventeen key points such as head coordinates, left shoulder coordinates, right shoulder coordinates, left hip coordinates, and right hip coordinates. Simultaneously, an image segmentation algorithm extracts the contour features of the umbrella carried by the first pedestrian from the same frame; this contour feature consists of a set of pixels describing the edge of the umbrella. Similarly, the key skeletal point data of the second pedestrian and the contour features of their backpack are also extracted simultaneously. All of this data is transmitted in real time to the sprinkler truck's central control system.

[0026] S102 maps key skeletal point data to a coordinate system centered on the sprinkler truck, calculates the pedestrian's motion trend vector based on the temporal changes of the key skeletal point data, and calculates the predicted motion trajectory of the nozzle at the end of the sprinkler arm by combining the real-time motion state of the sprinkler truck's spray arm.

[0027] S102 specifically includes: S1021. Transform each key body position point in the key skeletal point data into a three-dimensional spatial coordinate system with the center of the sprinkler truck as the origin to obtain the spatial position of the skeletal points.

[0028] Among them, the coordinate system centered on the sprinkler truck refers to a three-dimensional rectangular coordinate system with the geometric center of the sprinkler truck body or a specific reference point as the origin and the direction of the truck's front as the positive direction of a certain coordinate axis. It is used to uniformly describe the positional relationship between pedestrians, obstacles, and the sprinkler truck's own components.

[0029] In this embodiment of the application, the pixel coordinates of key skeleton points in the two-dimensional image obtained in step S101 are first converted into three-dimensional coordinate points in a three-dimensional spatial coordinate system centered on the sprinkler truck, based on the internal and external parameters of the camera. These coordinate points are the spatial positions of the skeleton points.

[0030] In practical applications, a three-dimensional coordinate system is established with the center of the sprinkler truck's roof as the origin. The direction of the truck's front is the positive X-axis, the left side of the truck is the positive Y-axis, and the vertical upward direction is the positive Z-axis. Using camera calibration parameters, the pixel coordinates of the first pedestrian's head in step S101 are mapped to this coordinate system, resulting in the spatial position of the head's skeletal points as (2.5m, 1.2m, 1.6m). Similarly, the other sixteen key points of the pedestrian, such as the left and right shoulders, are also converted into corresponding skeletal point spatial positions; for example, the coordinates of the left shoulder are (2.4m, 1.1m, 1.4m). All key skeletal points of the second pedestrian undergo the same coordinate transformation.

[0031] S1022. Obtain the spatial positions of multiple sets of skeletal points in a continuous time series to form a time sequence of changes in key points of a pedestrian's body.

[0032] In this embodiment of the application, the central control system continuously receives and stores the spatial positions of the skeletal points output in step S1021 at fixed time intervals, and arranges the spatial positions of the skeletal points of the same pedestrian at different times in chronological order to form a change sequence of the pedestrian. This sequence reflects the displacement process of the pedestrian's key body points over time.

[0033] In practical applications, the central control system processes image data at a frequency of ten frames per second. For the first pedestrian, a set of skeletal point spatial positions is acquired at time T0, a second set is acquired at T0+0.1 seconds, and a third set is acquired at T0+0.2 seconds. Arranging these three sets of data in chronological order constitutes the change sequence of the first pedestrian, where each element is a set of seventeen skeletal point spatial positions at that moment.

[0034] S1023. Based on the change sequence, determine the direction and speed of movement of the pedestrian's body representative point, and generate a motion trend vector.

[0035] Prior to S1023, it also includes: The system obtains the real-time driving speed and direction of the sprinkler truck; based on the real-time driving speed and direction, it performs motion compensation on the change sequence to eliminate the influence of the sprinkler truck's own movement on the relative position changes of pedestrians; and calculates the motion trend vector based on the compensated change sequence.

[0036] Motion compensation refers to the process of reconstructing the absolute motion of a pedestrian relative to the ground from the relative motion of the sprinkler truck by subtracting the sprinkler truck's own motion component. The motion trend vector is a three-dimensional vector containing information about the direction and magnitude of the pedestrian's motion, used to characterize the pedestrian's future movement trend.

[0037] In this embodiment, the real-time driving speed and direction of the sprinkler truck are first obtained by the vehicle speed sensor and the steering wheel angle sensor; then, the spatial position of each skeletal point in the change sequence obtained in step S1022 is corrected using the speed and direction to remove the coordinate change caused by the movement of the sprinkler truck; finally, based on the compensated change sequence, the movement direction and speed of representative points of the pedestrian's body, such as the center point of the hip, are calculated by a fitting algorithm to generate a three-dimensional motion trend vector.

[0038] In practical application, the sprinkler truck is currently traveling along the positive X-axis at a speed of two meters per second. Based on the change sequence of the first pedestrian in step S1022, the position of its hip center point at time T0 is calculated to be (2.3m, 1.0m, 0.9m), and the position at time T0 plus 0.2 seconds is (2.5m, 0.9m, 0.9m). After deducting the 0.4m movement of the sprinkler truck itself within 0.2 seconds, the compensated displacement of the hip center point is obtained as an increase of -0.2 meters in the X direction, a decrease of 0.1 meters in the Y direction, and no change in the Z direction. Thus, the motion trend vector of the first pedestrian is calculated to be one meter per second along the negative X-axis and 0.5 meters per second along the negative Y-axis, i.e., the motion trend vector is [-1.0, -0.5, 0.0], in meters per second.

[0039] S1024. Simultaneously, based on the current joint motion parameters and mechanical connection structure of the sprinkler arm, the predicted motion trajectory of the nozzle at the end of the sprinkler arm within a preset time period is calculated.

[0040] The current joint motion parameters of the spray arm include data such as the current rotation angle, angular velocity, and extension / retraction of each joint. The mechanical connection structure refers to the geometric connection relationships and dimensional parameters between the links that make up the spray arm. The predicted motion trajectory refers to the set of position points that the spray head at the end of the spray arm will sequentially pass through in three-dimensional space over a future period.

[0041] In this embodiment, the central control system reads the encoder data of each joint of the sprinkler arm in real time to obtain the current joint motion parameters. Then, combined with the pre-stored mechanical structure dimensions of the sprinkler arm, it calculates a series of spatial position points that the end nozzle will pass through in a future preset time period under the condition of continuous movement according to the current control command. These points are connected in chronological order to form a predicted motion trajectory.

[0042] In practical applications, the sprinkler arm of a water truck consists of a base, a boom, a forearm, and nozzles. Currently, the boom joint has a rotation angle of 30 degrees and extends at a rate of 5 degrees per second, while the forearm joint has a rotation angle of -10 degrees and retracts at a rate of 3 degrees per second. Combining the lengths of each link, the spatial sequence of points the end-point nozzle will pass through in the next two seconds is calculated using forward kinematics formulas. For example, at T0 + 0.5 seconds, the nozzle is located at (0.4m, 1.5m, 2.0m); at T0 + 1.0 seconds, the nozzle is located at (4.3m, 1.7m, 2.1m), and so on, thus forming the predicted motion trajectory.

[0043] Through the above steps, this application converts the pedestrian's visual information into a motion trend vector in the water truck coordinate system and simultaneously predicts the motion trajectory of the spray arm itself, laying the foundation for determining whether the two will conflict in time and space.

[0044] S103. Calculate the spatial intersection region between each motion trend vector and the predicted motion trajectory, and generate an avoidance priority list based on the size of the spatial intersection region and the order of intersection time.

[0045] S103 specifically includes: S1031. Based on the movement trend vector of each pedestrian, extrapolate the current spatial position of the pedestrian to define the estimated space occupied by the pedestrian in the future time period.

[0046] Pedestrian-estimated space refers to the three-dimensional spatial area that a pedestrian will occupy over a future period of time, based on the pedestrian's current movement trend. This area is usually approximated by a time-varying geometric shape, such as a sphere or cube, whose size takes into account the pedestrian's own body shape.

[0047] In this embodiment, for each pedestrian, the spatial position of a representative point, such as the center point of the hip, is used as a reference. Combined with the pedestrian's movement trend vector, linear extrapolation is performed along the vector direction to predict the position of the representative point at consecutive future time points. Then, with each predicted position point as the center and a preset typical body size of the pedestrian as the radius, a series of spheres or cubes are generated. These geometric shapes are connected continuously on the time axis to form the estimated space occupied by the pedestrian in the future time period.

[0048] In practical applications, the current hip center point of the first pedestrian is located at (2.3m, 1.0m, 0.9m), and its motion trend vector is [-1.0, -0.5, 0.0]. The estimated space occupied by the pedestrian is represented by a sphere with a radius of 0.3 meters. Based on the motion trend vector, it is predicted that at T0 + 0.5 seconds, the hip center point will move to (1.8m, 0.75m, 0.9m), and the sphere centered at this point represents the estimated space occupied at that moment. At T0 + 1.0 seconds, the hip center point will move to (1.3m, 0.5m, 0.9m), and the sphere centered at this point represents the estimated space occupied at that moment. This process continues, with the spheres at all times collectively constituting the estimated space occupied by the first pedestrian from T0 to the next two seconds.

[0049] S1032. Based on the predicted motion trajectory and the preset spraying range of the nozzle, determine the spraying operation space to be sprayed and covered by the nozzle at the end of the spraying arm in the future time period.

[0050] The preset spray range of a nozzle refers to the shape and size of the spatial area that the water mist sprayed by a single nozzle can cover when it is turned on, such as a cone or a fan-shaped area. The spraying work space refers to the collection of all areas swept by the spray range of the end nozzle as it moves along the predicted motion trajectory.

[0051] In this embodiment of the application, the preset spraying range parameters of the nozzle are first obtained, such as the spraying cone angle and range; then, each trajectory point on the predicted motion trajectory obtained in step S1024 is used as the nozzle center, and the preset spraying range geometry is attached to the trajectory point; finally, all the spraying range geometries attached to the trajectory points are merged in space to form an overall spraying operation space that changes continuously with time.

[0052] In practical applications, the preset spraying range of the sprinkler head is a cone with a 60-degree apex angle and a range of two meters. The predicted trajectory point in step S1024 includes the sprinkler head being located at (4.0m, 1.5m, 2.0m) at T0 + 0.5 seconds. Using this point as the vertex, the axis of the cone along the direction the sprinkler head points forms the spraying range at that moment. At T0 + 1.0 seconds, the sprinkler head is located at (4.3m, 1.7m, 2.1m), similarly forming the spraying range at that moment. By combining all the spatial points covered by the cone at these different times, the spraying operation space for the next two seconds is obtained.

[0053] S1033. Calculate the spatial intersection area between the estimated space occupied by each pedestrian and the spraying operation space, and determine the volume of the spatial intersection area and the first occurrence time of the spatial intersection area.

[0054] The spatial intersection area refers to the portion of the space where the estimated space occupied by pedestrians and the spraying operation space overlap at the same moment. The first occurrence time refers to the moment on the future timeline when the spatial intersection area first appears, starting from the current moment.

[0055] In this embodiment of the application, the estimated pedestrian space occupied by each pedestrian obtained in step S1031 is synchronously compared with the spraying operation space obtained in step S1032 in the time dimension. For each future time point or time period, it is determined whether the two spatial regions overlap. If there is an overlap, the geometric volume of the overlapping part and the time point when this overlap phenomenon first occurs are recorded.

[0056] In practical applications, the estimated spherical sequence of the first pedestrian's space occupation is compared time-by-time with the conical sequence of the spraying operation space. It is found that at T0 + 0.8 seconds, the center of the first pedestrian's sphere is located at (1.5m, 0.6m, 0.9m), and its sphere partially overlaps with a conical shape in the spraying operation space at this time. The volume of this overlapping portion is calculated, assumed to be approximately 0.1 cubic meters. This T0 + 0.8 seconds is the first appearance time. The same calculation is performed for the second pedestrian, assuming its first appearance time is T0 + 1.2 seconds, and the volume of the spatial intersection region is 0.05 cubic meters.

[0057] S1034. Sort all pedestrians with spatial intersection regions: sort them in ascending order based on the first appearance time; when the first appearance times are the same, sort them in descending order based on the size of the spatial intersection region.

[0058] In this embodiment of the application, pedestrians who have spatial intersections with the spraying operation space in the future time period are first screened out; then these pedestrians are sorted from early to late according to the first appearance time calculated in step S1033; if the first appearance time of two or more pedestrians is exactly the same, they are then sorted from large to small according to the volume of their respective spatial intersection areas.

[0059] In practical applications, continuing the previous example, the first pedestrian's first appearance time is T0 + 0.8 seconds, the second pedestrian's is T0 + 1.2 seconds, and assuming there are three pedestrians in the scene, the third pedestrian's first appearance time is T0 + 0.8 seconds, and the volume of their spatial intersection region is 0.15 cubic meters. According to the sorting rules, firstly, the first and third pedestrians are compared; both are T0 + 0.8 seconds, earlier than the second. Then, the volumes of the first and third pedestrians with the same first appearance time are compared; the third pedestrian's volume of 0.15 cubic meters is greater than the first pedestrian's volume of 0.1 cubic meters. Therefore, the sorting result is: the third pedestrian first, the first second, and the second last.

[0060] S1035. Generate an avoidance priority list based on the sorting results.

[0061] The yield priority list is used to indicate the order in which pedestrians should yield to water trucks.

[0062] In this embodiment of the application, the order of pedestrians after sorting in step S1034 is recorded in the form of a list. The order in this list is the order of pedestrians that the sprinkler truck control system needs to prioritize and avoid when making avoidance decisions. The earlier a pedestrian is listed, the more urgent or severe the conflict with the sprinkler truck's spraying operation is.

[0063] In practical applications, an avoidance priority list is generated based on the above sorting results. The order in the list is: pedestrian three, highest priority; pedestrian one, next; pedestrian two, lowest priority. The central control system will allocate computing resources and formulate avoidance strategies based on this list in subsequent steps.

[0064] Through the steps described above, this application quantifies the urgency of the conflict between each pedestrian and the spraying operation in both time and space dimensions, and generates a clear avoidance sequence, providing a basis for decision-making in subsequent complex risk analysis.

[0065] S104. Input the avoidance priority list, the temporal changes of key skeletal point data, and the contour features of pedestrians' carried objects into the preset fusion model to perform spatiotemporal feature correlation analysis and generate a risk distribution map.

[0066] like Figure 2 As shown, S104 specifically includes: S1041. Input the avoidance priority list, the temporal changes of key skeletal point data, and the contour features of the pedestrian's carried objects into the preset fusion model.

[0067] The preset fusion model is a pre-trained deep learning network model used to comprehensively process multi-source heterogeneous data and predict the probability of pedestrians occupying space in the future.

[0068] In this embodiment of the application, the central control system takes the avoidance priority list generated in step S1035, the temporal changes of the key skeletal point data of each pedestrian recorded in step S1022, and the contour features of the items carried by each pedestrian extracted in step S101 as input data and sends them together to the pre-trained fusion model for processing.

[0069] In practical applications, a priority list for avoiding pedestrians in the order of pedestrian 3, pedestrian 1, and pedestrian 2, along with a sequence of coordinate changes of seventeen key skeletal points of pedestrian 3 over the past 0.5 seconds, and the contour feature data of the umbrella carried by pedestrian 3, are input into the fusion model. Similarly, the corresponding data for pedestrians 1 and 2 are also input.

[0070] S1042. Utilize the graph convolutional network in the fusion model to perform graph convolution operations on the key skeletal point data in order to extract the spatial interaction relationships between different pedestrians and between different key points within the same pedestrian.

[0071] Graph convolutional networks are a type of neural network capable of processing graph-structured data. In this scenario, spatial interaction relationships refer to the mutual influence of the relative positions and postures of pedestrians, as well as the motion coordination and constraint relationships between various body parts of a single pedestrian.

[0072] In this embodiment, the fusion model first constructs a graph structure from the input key skeletal point data. Nodes in the graph represent each key skeletal point, and edges are divided into two categories: one connecting different key points of the same pedestrian, such as the torso and limbs; and the other connecting nearby key points of different pedestrians. Then, the graph convolutional network learns and extracts the spatial interaction relationships inherent within these nodes and edges through convolution operations, such as whether two pedestrians are walking side-by-side, or the correlation between a pedestrian's arm swing and leg steps.

[0073] In practical applications, for pedestrian three, its seventeen key points form a graph. The graph convolutional network learns the connection weights between these key points and extracts, for example, the strong correlation between the movements of the left elbow key point and the left wrist key point, reflecting the natural swing of the arm. Meanwhile, pedestrian three's hip key point and pedestrian one's shoulder key point are spatially close, and the graph convolutional network also learns the interaction relationship between these two nodes, which may indicate that the two pedestrians are about to pass each other.

[0074] S1043. Utilize the long short-term memory network in the fusion model to perform temporal feature extraction on the temporal changes of key skeletal point data in order to capture changes in pedestrian movement patterns.

[0075] Long Short-Term Memory (LSTM) networks are recurrent neural networks that excel at processing time-series data and learning long-term dependencies. Changes in movement patterns refer to alterations in a pedestrian's motion state, such as dynamic characteristics like walking at a constant speed, sudden acceleration, deceleration, turning, or stopping.

[0076] In this embodiment, the fusion model feeds the temporal changes of key skeletal point data for each pedestrian into a Long Short-Term Memory (LSTM) network. This network, through its internal memory units and gating mechanisms, analyzes the patterns of skeletal point position changes over time, thereby capturing the unique movement pattern changes of each pedestrian, such as whether it is a regular periodic swaying or an irregular abrupt change.

[0077] In practical applications, the sequence of changes in the X-coordinate of pedestrian three's hip center point over the past 0.5 seconds was input into a long short-term memory network. The network learned that the coordinate value was steadily decreasing at a rate of one meter per second, thus determining that pedestrian three's current movement pattern was uniform linear walking. For another pedestrian, the rate of decrease in the X-coordinate of their hip center point accelerated in the last 0.2 seconds. The long short-term memory network captured this change and determined that their movement pattern was transitioning from uniform speed to acceleration.

[0078] S1044. Based on the avoidance priority list, assign a weight coefficient to each pedestrian.

[0079] The weighting coefficient is a value used to quantify the importance of pedestrians in risk prediction; the higher the priority, the larger the weighting coefficient.

[0080] In this embodiment, the fusion model maps the order of the pedestrians in the input avoidance priority list to specific numerical weight coefficients. For example, a mapping function can be set such that the pedestrian ranked first receives the largest weight coefficient, the second ranked second, and so on. This weight coefficient will be used to adjust the pedestrian's contribution to the final risk distribution map when generating the potential spatial extent.

[0081] In practical applications, according to the yield priority list, pedestrian three has the highest priority and is assigned a weight coefficient of 0.6; pedestrian one is assigned a weight coefficient of 0.3; and pedestrian two has the lowest priority and is assigned a weight coefficient of 0.1. The sum of all weight coefficients is 1.0. The weight coefficients can be set according to the actual situation.

[0082] S1045. Combining weighting coefficients, spatial interaction relationships, changes in motion patterns, and the contour features of objects carried by pedestrians, a potential spatial range that each pedestrian may occupy at a future time point is defined in the three-dimensional spatial coordinate system.

[0083] S1045 specifically includes: The system categorizes the items carried by pedestrians based on their outline features and assigns corresponding spatial expansion coefficients to different types of items. For each pedestrian, an initial spatial range is generated by extrapolating the current spatial position, combining the motion trend vector and motion pattern changes. The initial spatial range is then adjusted using spatial interaction relationships to generate a first intermediate spatial range. The first intermediate spatial range is expanded using the spatial expansion coefficient to generate a second intermediate spatial range. Finally, the second intermediate spatial range is scaled in conjunction with a weighting coefficient to determine the pedestrian's final potential spatial range.

[0084] The spatial expansion coefficient is a multiplier factor used to amplify the basic spatial range of pedestrians. Its magnitude depends on the type of carrying; for example, an open umbrella has a larger expansion coefficient than a backpack. The initial spatial range is a basic spatial area predicted based on the pedestrian's own movement. The first intermediate spatial range is the spatial area corrected for the interaction between pedestrians, reflecting their interactions. The second intermediate spatial range further considers the additional space occupied by carrying items. The final potential spatial range is a spatial area that integrates all the above factors and priority weights, used for subsequent risk superposition.

[0085] In this embodiment of the application, this is a multi-level progressive calculation process, as detailed below: First, the fusion model identifies the type of the carried item, such as an umbrella, backpack, or trolley, based on the contour features of the input item using a classifier, and then looks up the corresponding spatial expansion coefficient from a pre-set parameter library.

[0086] Secondly, for each pedestrian, based on the spatial position of their current hip center point, and combined with the motion trend vector obtained in step S1023 and the motion pattern changes captured in step S1043, such as whether they are accelerating, a dynamic prediction model is used to extrapolate the pedestrian's center position at a series of future time points. A sphere or ellipsoid is generated with these positions as the center and a basic radius, which is the initial spatial range.

[0087] Then, the initial spatial range is adjusted using the spatial interaction relationships extracted in step S1042. For example, if the initial spatial ranges of two pedestrians are very close and their movement trends are opposite, the model will appropriately increase their respective ranges to reflect the higher uncertainty caused by their proximity, and the adjusted range is the first intermediate spatial range.

[0088] Then, based on the spatial expansion coefficient obtained in step S1045, each dimension of the first intermediate space range is multiplied by the coefficient to obtain the second intermediate space range that takes into account the size of the carried items. This means that pedestrians carrying large items will be considered to occupy more space.

[0089] Finally, the second intermediate spatial range is combined with the weighting coefficients assigned in step S1044 to scale the spatial range. The larger the weighting coefficient, the greater the influence of the range in subsequent overlays. This can be understood as the potential spatial range of high-priority pedestrians being given a higher "concentration" or "intensity" in risk calculation, rather than just a geometric enlargement. After this step, the potential spatial range for each pedestrian in subsequent risk mapping is finally determined.

[0090] In practical application, based on contour features, pedestrian three is identified as carrying an open umbrella. A pre-defined query database shows that the spatial expansion coefficient for an umbrella is 1.2. Pedestrian three's current hip center is located at (1.3m, 0.5m, 0.9m), with a motion trend vector of [-1.0, -0.5, 0.0], and the motion pattern shows a uniform speed. Extrapolation yields an initial spatial range of a sphere with a radius of 0.4 meters at T0 + 0.5 seconds, centered at (0.8m, 0.25m, 0.9m). Since the spatial interaction relationship shows a significant distance between pedestrian three and pedestrian one, no adjustment is needed; the first intermediate spatial range is this sphere. Applying the spatial expansion coefficient of 1.2 expands the sphere's radius to 0.48 meters, resulting in the second intermediate spatial range. Finally, combined with a weighting coefficient of 0.6, this range will be multiplied by a risk intensity coefficient of 0.6 in subsequent steps for aggregation, rather than being geometrically scaled. For pedestrian 1 carrying a backpack, the expansion coefficient might be 1.1 and the weighting coefficient 0.3, similarly generating its final potential spatial extent.

[0091] S1046. Assign a risk value to each spatial location point within the potential spatial range, and overlay the risk values ​​of all pedestrians in a three-dimensional spatial coordinate system to generate a risk distribution map.

[0092] Here, the risk value is a quantitative measure of the danger of a point in space being occupied by pedestrians at a specific time in the future. The risk distribution map is a data field in three-dimensional space where each point corresponds to a risk value. It can be viewed as a heatmap of the probability of pedestrians appearing at future times, used to characterize the spatial distribution probability of pedestrians at future times.

[0093] In this embodiment, the fusion model maps the final potential spatial range of each pedestrian determined in step S1045 to each grid point in the three-dimensional spatial coordinate system. For a grid point falling within the potential spatial range of a pedestrian, a non-zero risk value is assigned based on factors such as its distance from the center of the range and the weight coefficient of the pedestrian to which it belongs. The closer to the center, the higher the risk value; the larger the weight coefficient, the higher the risk value at the same location. Then, the risk values ​​generated by all pedestrians at the same grid point are accumulated. By traversing all grid points within the entire three-dimensional spatial region of interest, a complete risk distribution map can be obtained.

[0094] In practical applications, the space around the sprinkler truck is divided into 0.1-meter square grids in the X direction (from -5 meters to +10 meters), the Y direction (from -5 meters to +5 meters), and the Z direction (from 0 meters to 3 meters). For each grid point, its potential space range relative to pedestrian zone 3 is determined. If a grid point is located near the center of pedestrian zone 3's potential space range and has a weighting coefficient of 0.6, it is assigned a higher risk value, such as 0.8. If another grid point is located at the edge of pedestrian zone 1's potential space range and has a weighting coefficient of 0.3, it is assigned a lower risk value, such as 0.1. If a grid point is located within the potential space range of multiple pedestrian zones, the risk values ​​are added together. For example, if a grid point is located within both pedestrian zone 3 and pedestrian zone 1's potential space range, the risk value is 0.8 plus 0.1, which equals 0.9. The risk values ​​of all grid points together constitute a three-dimensional risk distribution map.

[0095] This application uses a fusion model to comprehensively consider various factors such as pedestrian movement, mutual influence, carried items, and urgency of avoidance, generating a comprehensive and dynamic risk distribution map, which provides a refined decision-making basis for the subsequent precise control of the spraying arm.

[0096] S105. Based on the risk distribution map, under the conditions of maximizing the spray coverage area and avoiding pedestrians, calculate the joint motion trajectory of the spray arm and the start-stop sequence of the nozzle, and generate and issue the corresponding motion control commands and valve control commands.

[0097] S105 specifically includes: S1051. Within the preset kinematic constraints of the spraying arm, plan multiple candidate joint motion trajectories.

[0098] Among them, the preset kinematic constraints refer to the joint motion limit parameters determined by the mechanical structure of the spray arm, such as the maximum rotation angle, maximum angular velocity, maximum angular acceleration of each joint, and interference limits between links. The candidate joint motion trajectory refers to multiple alternative paths that satisfy the above kinematic constraints and can guide the nozzle's movement in the future, starting from the current state.

[0099] In this embodiment, the central control system first reads the current joint state of the spray arm, and then, within a predefined kinematic constraint space, uses path planning algorithms such as fast expanding random tree or probabilistic route map methods to generate multiple joint motion trajectories that can guide the nozzle to different target positions or maintain different motion modes, and satisfy joint limits and speed limits throughout the process, as candidate schemes to be evaluated.

[0100] In practical applications, the allowable rotation range of the spray arm's upper joint is 0 to 90 degrees, with a maximum angular velocity of 10 degrees per second. The control system, starting from the upper arm's current 30 degrees and the forearm's current -10 degrees, plans candidate trajectory one, which causes the upper arm to extend uniformly to 34 degrees at a speed of 2 degrees per second over the next two seconds, while the forearm remains unchanged; candidate trajectory two causes the upper arm to rapidly extend to 40 degrees at a speed of 5 degrees per second, then retract to 35 degrees; candidate trajectory three keeps the upper arm stationary while only the forearm retracts to -12 degrees at a speed of 1 degree per second. All these trajectories satisfy preset kinematic constraints.

[0101] S1052. For each candidate joint motion trajectory, calculate the risk cost and spray coverage benefit.

[0102] Among these, risk cost is a quantifiable value used to characterize the degree of overlap between the candidate joint motion trajectory and the high-risk area in the risk distribution map; the greater the overlap, the higher the risk cost. Spray coverage benefit is a quantifiable value used to characterize the total spray coverage area corresponding to the candidate joint motion trajectory.

[0103] In this embodiment, for each candidate joint motion trajectory, the control system first calculates the future position and attitude of the nozzle based on the trajectory, and then, combined with the preset spraying range of the nozzle, calculates its corresponding spraying operation space. Next, this spraying operation space is compared with the risk distribution map generated in step S1046. The volume or weighted sum of the overlapping portion of the spraying operation space and the area in the risk distribution map where the risk value exceeds a certain threshold is the risk cost of that trajectory. The larger the overlapping portion, or the higher the risk value of the overlapping area, the greater the risk cost. Simultaneously, the control system calculates the total effective sprayed area based on the area covered by the nozzle under that trajectory, combined with a preset operation area map; this is the spraying coverage benefit of that trajectory.

[0104] In practical applications, for candidate trajectory one, its corresponding spraying operation space is overlaid with the risk distribution map for calculation. It was found that a portion of this operation space overlaps with a high-risk area (risk value greater than 0.7), with an overlap volume of 0.05 cubic meters, resulting in a calculated risk cost of 0.05. Simultaneously, this trajectory covers a 10-square-meter area on the road surface, thus providing a spraying coverage benefit of 10 square meters. For candidate trajectory two, its spraying operation space overlaps with the high-risk area by 0.02 cubic meters, with a risk cost of 0.02, but the coverage area is only 8 square meters. For candidate trajectory three, its spraying operation space does not overlap with the high-risk area, resulting in a zero risk cost, but the coverage area is also small, only 5 square meters.

[0105] S1053. Based on risk cost and spray coverage benefit, select the joint movement trajectory with the lowest risk cost and the highest spray coverage benefit from multiple candidate joint movement trajectories as the target joint movement trajectory.

[0106] In this embodiment, the control system employs a multi-objective optimization decision-making method to comprehensively evaluate the risk cost and spray coverage benefits of all candidate trajectories calculated in step S1052. The typical objective is to select a trajectory with the lowest possible risk cost and the highest possible spray coverage benefits. Weighted summation or Pareto optimality methods can be used for selection and ranking, ultimately choosing the optimal trajectory as the target joint motion trajectory.

[0107] In practical applications, three candidate trajectories are compared: Trajectory 3 has a risk cost of zero but a gain of only five square meters; Trajectory 2 has a risk cost of 0.02 and a gain of eight square meters; Trajectory 1 has a risk cost of 0.05 and a gain of ten square meters. The system sets a trade-off rule, for example, prioritizing the option with the highest gain when the risk cost is below 0.03. Trajectory 3 and Trajectory 2 satisfy the risk cost condition, with Trajectory 2 offering a greater gain; therefore, candidate Trajectory 2 is selected as the target joint motion trajectory.

[0108] S1054. By comparing the target joint movement trajectory with the risk distribution map, determine the time period when the spraying area and the high-risk area overlap, and generate the start-stop sequence of the nozzles accordingly.

[0109] The start-stop sequence refers to the sequence of time points in which the nozzle valve needs to perform opening and closing operations within a certain period of time.

[0110] In this embodiment, after determining the target joint's motion trajectory, the control system compares the spraying workspace corresponding to that trajectory at each future moment with the risk distribution map in detail. This precisely identifies the time periods during which the spraying workspace overlaps with high-risk areas where the risk value exceeds a preset threshold. During these time periods, the nozzle valves need to be closed to prevent pedestrians from being sprayed; during other time periods, the nozzle valves can be opened to ensure operational efficiency. Recording these time points forms the nozzle start-stop sequence.

[0111] In practical applications, the target joint's motion trajectory is candidate trajectory two. A time-by-time comparison of the spraying operation space with the risk distribution map reveals that between T0+0.7 seconds and T0+1.1 seconds, the spraying operation space overlaps with the high-risk area where pedestrians are located. Therefore, the generated start-stop sequence is as follows: the nozzle is open from the current time T0, the valve is closed at T0+0.7 seconds, and the valve is reopened at T0+1.1 seconds.

[0112] S1055. Convert the target joint motion trajectory and start / stop sequence into motion control commands and valve control commands respectively, and send them to the corresponding actuators.

[0113] The motion control commands are control signals sent to the actuators of each joint of the spray arm, containing parameters such as target position, speed, or torque. The valve control commands are switching signals sent to the solenoid valves of the spray heads, containing opening or closing actions. The actuators include servo motors or hydraulic cylinders that drive the spray arm movement, and valves that control the on / off flow of water.

[0114] In this embodiment, the control system decomposes the target joint motion trajectory selected in step S1053 into position or velocity commands for each joint at various times according to the control cycle, generating standard-format motion control commands. Simultaneously, the start / stop sequence generated in step S1054 is converted into a series of valve switching signals bound to timestamps, i.e., valve control commands. Finally, these commands are sent in real-time to the joint actuators of the spray arm and the solenoid valves of the spray head via fieldbus or industrial Ethernet.

[0115] In practical applications, the candidate trajectory is decomposed into a sequence of angular velocity commands for the upper arm and forearm joints. For example, within the next 0.1 second, the upper arm rotates at 5 degrees per second, and the forearm retracts at 1 degree per second. Corresponding motion control commands are generated and sent to the joint motor driver via the CAN bus. Simultaneously, the timing sequence of closing the valve at T0 + 0.7 seconds and opening the valve at T0 + 1.1 seconds is converted into high and low level signals and sent to the solenoid valve controller of the nozzle. The motor and valve then execute the commands accordingly.

[0116] Through the above steps, this application, based on a full understanding of the dynamic risks of surrounding pedestrians, plans a spray arm movement trajectory and nozzle control sequence that can balance operational efficiency and obstacle avoidance safety, thus achieving intelligent, safe, and efficient watering operations.

[0117] Figure 3 This is a schematic diagram illustrating a specific implementation of an image recognition-based intelligent control system for sprinkler trucks to avoid pedestrians, provided in this application. (Refer to...) Figure 3 The system may include: The acquisition module 31 is used to acquire multi-view images of the environment surrounding the sprinkler truck, obtain key skeletal point data of pedestrians from the multi-view images through human pose estimation, and extract the contour features of objects carried by pedestrians from the multi-view images. The calculation module 32 is used to map the key skeleton point data to a coordinate system centered on the sprinkler truck, calculate the pedestrian's motion trend vector based on the temporal changes of the key skeleton point data, and calculate the predicted motion trajectory of the nozzle at the end of the sprinkler arm in combination with the real-time motion state of the sprinkler truck's spray arm. The first generation module 33 is used to calculate the spatial intersection region between each of the motion trend vectors and the predicted motion trajectory, and generate an avoidance priority list based on the volume of the spatial intersection region and the order of the intersection time. Analysis module 34 is used to input the avoidance priority list, the temporal changes of the key skeletal point data and the contour features of the pedestrian's carried objects into a preset fusion model for spatiotemporal feature correlation analysis, and generate a risk distribution map. The risk distribution map is used to characterize the probability of danger of the pedestrian occupying a spatial position in the future. The second generation module 35 is used to calculate the joint movement trajectory of the spraying arm and the start-stop sequence of the nozzle based on the risk distribution map, under the conditions of maximizing the spray coverage area and avoiding pedestrians, and to generate and issue corresponding motion control commands and valve control commands.

[0118] The image recognition-based intelligent control system for water trucks to avoid pedestrians in this application embodiment is used to implement the aforementioned image recognition-based intelligent control method for water trucks to avoid pedestrians. Therefore, the specific implementation of the image recognition-based intelligent control system for water trucks to avoid pedestrians can be found in the embodiment section of the image recognition-based intelligent control method for water trucks to avoid pedestrians in the preceding text. The specific implementation can be referred to the description of the corresponding embodiments, and will not be repeated here.

[0119] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the image recognition-based intelligent control method for water trucks to avoid pedestrians as described above.

[0120] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described intelligent control methods for water trucks avoiding pedestrians based on image recognition.

[0121] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0122] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the intelligent control method for water trucks to avoid pedestrians based on image recognition.

[0123] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0124] The above provides a detailed description of the intelligent control method and system for water trucks to avoid pedestrians based on image recognition, as provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for intelligent control of a sprinkler truck avoiding pedestrians based on image recognition, characterized in that, include: Collect multi-view images of the environment surrounding the sprinkler truck, obtain key skeletal point data of pedestrians from the multi-view images through human pose estimation, and extract the contour features of objects carried by pedestrians from the multi-view images. The key skeletal point data is mapped to a coordinate system centered on the sprinkler truck. The pedestrian's movement trend vector is calculated based on the temporal changes of the key skeletal point data. The predicted movement trajectory of the nozzle at the end of the sprinkler arm is calculated by combining the real-time movement state of the sprinkler truck's spray arm. Calculate the spatial intersection region between each of the motion trend vectors and the predicted motion trajectory, and generate an avoidance priority list based on the size of the spatial intersection region and the order of the intersection time; The avoidance priority list, the temporal changes of the key skeletal point data, and the contour features of the pedestrian's carried objects are input into a preset fusion model for spatiotemporal feature correlation analysis to generate a risk distribution map. The risk distribution map is used to characterize the probability of danger for the pedestrian occupying a spatial position in the future. Based on the risk distribution map, under the conditions of maximizing the spray coverage area and avoiding pedestrians, the joint motion trajectory of the spray arm and the start-stop sequence of the nozzle are calculated, and corresponding motion control commands and valve control commands are generated and issued.

2. The method according to claim 1, characterized in that, The step of inputting the avoidance priority list, the temporal changes of the key skeletal point data, and the contour features of the pedestrian's carried objects into a preset fusion model for spatiotemporal feature correlation analysis to generate a risk distribution map includes: The avoidance priority list, the temporal changes of the key skeletal point data, and the contour features of the pedestrian's carried objects are all input into a preset fusion model; The graph convolutional network in the fusion model is used to perform graph convolution operations on the key skeletal point data to extract the spatial interaction relationships between different pedestrians and between different key points within the same pedestrian. The long short-term memory network in the fusion model is used to perform temporal feature extraction on the temporal changes of the key skeletal point data in order to capture changes in the pedestrian's movement pattern; Based on the avoidance priority list, a weight coefficient is assigned to each pedestrian; By combining the weighting coefficients, the spatial interaction relationships, the changes in motion patterns, and the contour features of the pedestrian's carried items, a potential spatial range that each pedestrian may occupy at a future time point is defined in the three-dimensional spatial coordinate system. A risk value is assigned to each spatial location point within the potential spatial range, and the risk values ​​of all pedestrians are superimposed in the three-dimensional spatial coordinate system to generate a risk distribution map, which is used to characterize the spatial distribution probability of pedestrians at future times.

3. The method according to claim 1, characterized in that, Based on the risk distribution map, and under the conditions of maximizing spray coverage area and avoiding pedestrians, the joint motion trajectory of the spray arm and the start-stop sequence of the nozzles are calculated, and corresponding motion control commands and valve control commands are generated and issued, including: Within the preset kinematic constraints of the spraying arm, multiple candidate joint motion trajectories are planned; For each candidate joint motion trajectory, calculate the risk cost and spray coverage benefit. The risk cost is used to characterize the degree of overlap between the candidate joint motion trajectory and the high-risk area in the risk distribution map, and the spray coverage benefit is used to characterize the total spray coverage area corresponding to the candidate joint motion trajectory. Based on the risk cost and the spray coverage benefit, the joint movement trajectory with the lowest risk cost and the highest spray coverage benefit is selected from multiple candidate joint movement trajectories and used as the target joint movement trajectory. By comparing the target joint movement trajectory with the risk distribution map, the time period in which the spraying area overlaps with the high-risk area is determined, and the start-stop sequence of the nozzle is generated accordingly. The target joint motion trajectory and the start / stop timing sequence are converted into motion control commands and valve control commands, respectively, and sent to the corresponding actuators.

4. The method according to claim 1, characterized in that, The calculation of the spatial intersection region between each of the motion trend vectors and the predicted motion trajectory, and the generation of an avoidance priority list based on the size of the spatial intersection region and the order of intersection time, includes: Based on the movement trend vector of each pedestrian, the current spatial position of the pedestrian is extrapolated to define the estimated space occupied by the pedestrian in the future time period. Based on the predicted motion trajectory and combined with the preset spraying range of the nozzle, the spraying operation space to be sprayed and covered by the nozzle at the end of the spraying arm in the future time period is determined. Calculate the spatial intersection area between the estimated space occupied by each pedestrian and the spraying operation space, and determine the volume of the spatial intersection area and the first occurrence time of the spatial intersection area; Sort all pedestrians in the spatial intersection area: sort them in ascending order based on the first appearance time as the first sorting criterion; when the first appearance times are the same, sort them in descending order based on the volume of the spatial intersection area as the second sorting criterion. A priority list for yielding is generated based on the sorting results. This priority list is used to indicate the order in which pedestrians should yield to the sprinkler truck.

5. The method according to claim 2, characterized in that, The method, which combines the weighting coefficients, spatial interaction relationships, changes in motion patterns, and contour features of the pedestrian's belongings, defines a potential spatial range for each pedestrian in a three-dimensional coordinate system, including: The items carried by pedestrians are classified according to their outline characteristics, and corresponding spatial expansion coefficients are set for different types of items. For each pedestrian, based on the current spatial location, spatial extrapolation is performed by combining the motion trend vector and the change in the motion pattern to generate an initial spatial range; The initial spatial range is adjusted using the spatial interaction relationship to generate a first intermediate spatial range, which reflects the interaction between pedestrians. The first intermediate space range is expanded using the aforementioned spatial expansion coefficient to generate a second intermediate space range, which takes into account the additional space occupied by the carried items. The second intermediate spatial range is scaled in combination with the weighting coefficient to determine the final potential spatial range of the pedestrian.

6. The method according to claim 1, characterized in that, The process of mapping the key skeletal point data to a coordinate system centered on the water truck, calculating the pedestrian's movement trend vector based on the temporal changes of the key skeletal point data, and calculating the predicted movement trajectory of the nozzle at the end of the water truck's spray arm in conjunction with the real-time movement state of the spray arm includes: The key body position points in the key skeletal point data are transformed into a three-dimensional spatial coordinate system with the center of the sprinkler truck as the origin to obtain the spatial position of the skeletal points. Multiple sets of the spatial locations of the skeletal points are obtained in a continuous time series to form a time sequence of changes in key points of the pedestrian's body. Based on the change sequence, the direction and speed of movement of the pedestrian's body representative point are determined, and a motion trend vector is generated. The motion trend vector is used to characterize the pedestrian's future movement trend. Meanwhile, based on the current joint motion parameters and mechanical connection structure of the sprinkler arm, the predicted motion trajectory of the nozzle at the end of the sprinkler arm within a preset time period is calculated.

7. The method according to claim 6, characterized in that, Before determining the direction and speed of movement of the pedestrian's body representative points based on the aforementioned change sequence and generating the motion trend vector, the process further includes: Obtain the real-time driving speed and real-time driving direction of the sprinkler truck; Based on the real-time driving speed and the real-time driving direction, motion compensation is performed on the change sequence to eliminate the influence of the water truck's own movement on the relative position change of pedestrians; The motion trend vector is calculated based on the compensated change sequence.

8. A smart control system for a sprinkler truck to avoid pedestrians based on image recognition, characterized in that, include: The acquisition module is used to acquire multi-view images of the environment surrounding the sprinkler truck, obtain key skeletal point data of pedestrians from the multi-view images through human pose estimation, and extract the contour features of objects carried by pedestrians from the multi-view images. The calculation module is used to map the key skeletal point data to a coordinate system centered on the sprinkler truck, calculate the pedestrian's motion trend vector based on the temporal changes of the key skeletal point data, and calculate the predicted motion trajectory of the nozzle at the end of the sprinkler arm in combination with the real-time motion state of the sprinkler truck's spray arm. The first generation module is used to calculate the spatial intersection region between each of the motion trend vectors and the predicted motion trajectory, and generate an avoidance priority list based on the volume of the spatial intersection region and the order of the intersection time. The analysis module is used to input the avoidance priority list, the temporal changes of the key skeletal point data and the contour features of the pedestrian's carried objects into a preset fusion model for spatiotemporal feature correlation analysis, and generate a risk distribution map. The risk distribution map is used to characterize the probability of danger of the pedestrian occupying a spatial position in the future. The second generation module is used to calculate the joint motion trajectory of the spraying arm and the start-stop sequence of the nozzle based on the risk distribution map, under the conditions of maximizing the spray coverage area and avoiding pedestrians, and to generate and issue corresponding motion control commands and valve control commands.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the steps of the image recognition-based intelligent control method for water trucks to avoid pedestrians as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the image recognition-based intelligent control method for water trucks to avoid pedestrians as described in any one of claims 1 to 7.

Images