Graded collision early warning system and method for cooperative operation of unmanned agricultural machinery

Abstract

The invention discloses a hierarchical collision early warning system and method for cooperative operation of unmanned agricultural machinery, and the method comprises the steps: building a TGAP neural network model, carrying out the multi-modal trajectory prediction of the driving trajectory of the unmanned agricultural machinery in the future several seconds, firstly, roughly detecting whether the collision occurs or not through employing an AABB according to the predicted trajectory point, and if the collision occurs, carrying out the early warning of the collision of the unmanned agricultural machinery; and if the collision is determined to occur, detecting again by using an OBB algorithm, and if the collision is determined to occur, returning to the first time point when the collision occurs, and comparing the first time point with the safe collision time of the unmanned agricultural machine so as to carry out graded collision early warning. According to the invention, trajectory prediction is carried out by using the neural network model, and collision early warning of the unmanned agricultural machine cluster in the agricultural operation process can be carried out in real time in combination with the bounding box collision detection algorithm, and the early warning is timely and accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural machinery equipment, and in particular to a hierarchical collision warning system and method for cooperative operation of unmanned agricultural machinery. Background Art

[0002] As an important focus of modern agricultural development, smart agriculture is an important means to realize agricultural industry modernization and improve agricultural production efficiency. The promotion of smart agriculture can not only improve the efficiency and sustainability of agricultural production, but also optimize resource allocation and improve the level of green agricultural development, providing strong support for the realization of the strategy of building a strong agricultural country.

[0003] In this context, unmanned agricultural machinery, as one of the key equipment in the field of smart agriculture, has gradually replaced traditional manual labor with its efficient, precise and stable operating capabilities, and significantly improved the level of automation in agricultural production, especially during the busy farming season. In order to cope with challenges such as weather changes and short crop planting windows, rushing to plant and harvest has become a common demand. Therefore, the collaborative operation of multiple tractors has become an important trend in the future development of agriculture. In the face of dynamic obstacles (such as other tractors, pedestrians, animals, etc.) and emergencies such as agricultural machinery failures, how to ensure the safety of unmanned agricultural machinery in collaborative operations has become a technical problem that needs to be solved urgently. Especially in the scenario of multiple tractor cluster operations, as the operation density increases, the risk of collision between tractors also increases.

[0004] At present, some multi-machine collaborative operations regard farmland as a static scene, upload the plot information of farmland and the real-time posture information of unmanned agricultural machinery to the cloud server, calculate the deviation between the actual driving path of the agricultural machinery and the global planning path on the cloud server, and judge whether there is a path conflict. The specific implementation is to generate a rectangular bounding box based on the external dimensions of the agricultural machinery and the implements, and use the azimuth and real-time speed of the agricultural machinery to calculate the future bounding box coordinates and steering parameters every 0.5 seconds, and use the bounding boxes of multiple agricultural machinery at a certain moment in the future according to the coordinates and steering parameters, using geometric algorithms to judge whether the paths conflict, if a conflict occurs, adjust the steering and speed of the agricultural machinery. There are related studies in the automotive field that use vehicle-to-vehicle communication technology to send the lane change intention of the driver of the target vehicle in the adjacent lane in front to the self-vehicle, use neural networks to predict the trajectory of the switched vehicle, and then use the bounding box theory (OBB) and the separating axis theorem for collision warning. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a hierarchical collision warning system for cooperative operation of unmanned agricultural machinery, which can improve the safety of multiple unmanned agricultural machinery in complex farmland operation environments, realize real-time and intelligent collision warning of unmanned agricultural machinery, ensure efficient and safe operation of agricultural machinery, avoid potential operation accidents, reduce damage to agricultural equipment, and improve agricultural production efficiency.

[0006] The present invention also proposes a hierarchical collision warning method for cooperative operation of unmanned agricultural machinery based on the above system.

[0007] According to the first aspect of the present invention, a collision warning system for cooperative operation of unmanned agricultural machinery includes a data acquisition module, a communication module and a data processing module. The data acquisition module is used to collect the posture information of the unmanned agricultural machinery in the scene and the status information of the obstacles; the communication module is used to transmit the data information collected by the acquisition module to the data processing module; the data processing module is used to receive the trajectory data of all unmanned agricultural machinery in the scene for a historical N seconds and the status information of all obstacles, and analyze and process the received data to determine whether the unmanned agricultural machinery in the scene will collide with other unmanned agricultural machinery in the next M seconds. If a collision occurs, a warning signal or an emergency braking signal is sent to the unmanned driving system of the unmanned agricultural machinery, so that the unmanned driving system of the unmanned agricultural machinery performs a collision warning or emergency braking on the agricultural machinery, thereby realizing a graded warning for unmanned agricultural machinery.

[0008] In some embodiments, the data processing module of the present invention includes a data preprocessing module, a trajectory prediction module and a collision warning module. The data preprocessing module is used to receive the posture information of all unmanned agricultural machinery in the scene and the status information of obstacles, and analyze and process the received data, and transmit the analyzed and processed data to the trajectory prediction module for trajectory prediction. The trajectory prediction module transmits the prediction result to the collision warning module. The collision warning module processes the prediction result to determine whether the unmanned agricultural machinery in the scene will collide with the other unmanned agricultural machinery in the next M seconds. If a collision occurs, a warning signal or an emergency braking signal is sent to the unmanned driving system of the unmanned agricultural machinery, so that the unmanned driving system of the unmanned agricultural machinery performs a collision warning or emergency braking on the agricultural machinery, thereby realizing a graded warning for unmanned agricultural machinery.

[0009] According to the second aspect of the present invention, the collision warning method for unmanned agricultural machinery cooperative operation comprises the following steps:

[0010] S1 obtains the posture information of agricultural machinery: builds a self-organizing network communication system for the unmanned agricultural machinery cluster. The unmanned agricultural machinery in the scene collects the posture information of the vehicle every a seconds, and then sends the posture information of the vehicle for N seconds to other unmanned agricultural machinery through broadcast wireless communication. At the same time, it receives the posture information of other unmanned agricultural machinery in the scene for N seconds. Finally, the posture information of all unmanned agricultural machinery in the scene is obtained and transmitted to the data preprocessing module of the vehicle.

[0011] S2 Obtaining obstacle status information: The unmanned agricultural machinery uses the vehicle as the center and collects point cloud data within a set range through the laser radar on the vehicle. Data processing is performed based on the acquired point cloud data to obtain the status information of all obstacles in the scene. The obstacle status information includes the center position and influence range of each obstacle.

[0012] S3 data preprocessing: Process the posture information of all agricultural machinery and the state information of obstacles obtained in step S1 and step S2, count all objects in the scene in the past N seconds, assign a unique ID to each object, assign different type identifiers to agricultural machinery and obstacles, and arrange the state data of all objects in the scene according to the time frame to obtain the input data for trajectory prediction;

[0013] S4 trajectory prediction: After data enhancement, the input data obtained in step S3 is input into the neural network model to predict the multi-modal driving trajectories of all unmanned agricultural machinery in the field scene in the future M seconds in parallel. The data of each driving trajectory is a frame every a seconds, and a total of W frames of position points are included;

[0014] S5 Collision warning: Based on the position information of the unmanned agricultural machinery cluster in each frame of the predicted future driving trajectory, the axis-aligned bounding box algorithm is used to determine in parallel whether the future driving trajectory of the agricultural machinery may collide with other agricultural machinery. If it is determined that a collision may occur, the directed bounding box algorithm is used to perform collision detection on the future driving trajectory of the agricultural machinery. If the detection result shows that a collision will indeed occur, the first time point T at which the collision occurs is returned. c That is, the collision time, and the collision time T c The collision safety time with the agricultural machinery is T s For comparison:

[0015] If T c ≥T s When the unmanned agricultural machinery is in motion, a warning signal of possible collision is given to the unmanned driving system of the unmanned agricultural machinery;

[0016] If the unmanned driving system of the agricultural machinery does not change its driving route after receiving the danger warning signal, when T c <T s When the unmanned driving system of the agricultural machinery is in emergency braking, an emergency braking signal is given to the unmanned driving system of the agricultural machinery, causing the unmanned driving system of the agricultural machinery to perform emergency braking.

[0017] Compared with the prior art, the present invention builds an agricultural machinery self-organizing network communication system, so that unmanned agricultural machinery can automatically establish a network connection and can adapt to the dynamic changes of network topology in the farmland operation environment; it combines the graph structure network with the unmanned agricultural machinery, uses the graph structure to model the spatial relationship between the unmanned agricultural machinery and obstacles, and uses the TGAP neural network model to perform multimodal trajectory prediction on the agricultural machinery, further improving the accuracy of identifying potential collision risks, improving the safety and operating efficiency of unmanned agricultural machinery in complex farmland environments, and realizing intelligent and efficient agricultural production. The present invention realizes collision warning of unmanned agricultural machinery in dynamic and complex farmland environments.

[0018] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram of an early warning system for a hierarchical collision early warning method for cooperative operation of unmanned agricultural machinery according to an embodiment of the present invention;

[0020] Figure 2 is a collision warning strategy flow chart of a hierarchical collision warning method for cooperative operation of unmanned agricultural machinery according to an embodiment of the present invention;

[0021] Figure 3 It is a diagram of the warning information transmission process of a hierarchical collision warning method for cooperative operation of unmanned agricultural machinery according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0023] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplicity and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed. In addition, the present invention provides various specific examples of processes and materials, but those of ordinary skill in the art can be aware of the applicability of the remaining processes and / or the use of the remaining materials.

[0024] Reference below Figure 1 A collision warning system for cooperative operation of unmanned agricultural machinery according to an embodiment of the present invention is described, including a data acquisition module, a wireless communication module and a data processing module. The data acquisition module includes a GPS module and a laser radar installed on the unmanned agricultural machinery. The GPS module is used to collect the historical posture information of the unmanned agricultural machinery for 5 seconds. Each unmanned agricultural machinery transmits the collected posture information to the remaining agricultural machinery in the scene except the self-vehicle through the xbee wireless communication module. The laser radar is used to collect the status information of obstacles in the scene. The data processing module is used to receive the historical posture information of all unmanned agricultural machinery in the scene for 5 seconds, as well as the status information of all obstacles, and analyze and process the received data to determine whether the unmanned agricultural machinery self-vehicle will collide with the remaining unmanned agricultural machinery in the scene within the next 5 seconds. If a collision occurs, a warning signal or an emergency braking signal is sent to the unmanned driving system of the unmanned agricultural machinery, so that the unmanned driving system performs a collision warning or emergency braking on the unmanned agricultural machinery, thereby realizing a graded warning of the unmanned agricultural machinery. The collision warning system of this embodiment is installed on the unmanned agricultural machinery, and the system is electrically connected to the unmanned driving system of the unmanned agricultural machinery.

[0025] Specifically, the data processing module of this embodiment includes a data preprocessing module, a trajectory prediction module and a collision warning module. The data preprocessing module is used to receive the posture information of all unmanned agricultural machinery in the scene, as well as the status information of obstacles, and analyze and process the received data, and transmit the analyzed and processed data to the trajectory prediction module for trajectory prediction. The trajectory prediction module transmits the prediction result to the collision warning module. The collision warning module processes the prediction result to determine whether the unmanned agricultural machinery will collide with the other unmanned agricultural machinery in the scene within the next 5 seconds. If a collision occurs, a warning signal or an emergency braking signal is sent to the unmanned driving system of the unmanned agricultural machinery, so that the unmanned driving system of the unmanned agricultural machinery performs a collision warning or emergency braking on the unmanned agricultural machinery, thereby realizing a graded warning for unmanned agricultural machinery.

[0026] Based on the above system, the collision warning method used in this embodiment is as follows:

[0027] S1 obtains the posture information of agricultural machinery: builds a self-organizing network communication system for a cluster of unmanned agricultural machinery (hereinafter referred to as "agricultural machinery"). The unmanned agricultural machinery in the scene collects the posture information of its own vehicle every 0.5 seconds through the GPS module on its own vehicle, and then transmits the posture information of the own vehicle for 5 seconds in history to the remaining agricultural machinery in the scene except the own vehicle through broadcast wireless communication. At the same time, the posture information of the remaining agricultural machinery in the scene for 5 seconds in history is received, and finally the posture information of all agricultural machinery in the scene is obtained and transmitted to the data preprocessing module of the own vehicle. The posture information includes global coordinates, heading angle and current driving speed; specifically, the posture information collected by the agricultural machinery self-vehicle is transmitted to the data preprocessing module through the CAN bus, and the posture information collected by the remaining agricultural machinery is transmitted to the data preprocessing module of the agricultural machinery self-vehicle through broadcast wireless communication.

[0028] S2 Obtain obstacle status information: Each agricultural machine takes its own vehicle as the center, collects point cloud data within a set range through the laser radar on its own vehicle, and performs data processing based on the acquired point cloud data to obtain the status information of all obstacles in the scene. The obstacle status information includes the center position (center of the circle) and the affected range (radius) of each obstacle.

[0029] Specifically, the point cloud data is processed as follows: in order to model the spatial relationship of the field environment, it is necessary to accurately identify and model the position and size of the field obstacles. According to the acquired point cloud data, the dense point cloud area is automatically divided into multiple clusters through a point cloud clustering algorithm, such as DBSCAN. Each cluster represents an independent obstacle cluster. The minimum circumscribed circle algorithm is further applied to calculate the minimum circle covering each obstacle cluster, so as to determine the center position (center of the circle) and the influence range (radius) of the obstacle, and then obtain the status information of all obstacles, and pass the status information of all obstacles to the data preprocessing module.

[0030] S3 data preprocessing: Process the posture information of all agricultural machinery and the state information of obstacles obtained in step S1 and step S2, count all objects (including agricultural machinery and obstacles) in the scene in the past 5 seconds, and assign a unique ID to each object, and assign different type identifiers to agricultural machinery and obstacles, such as 1 for agricultural machinery and 2 for obstacles, and arrange the state data of all objects in the scene according to the time frame to obtain the input data for trajectory prediction. Specifically, all data are sorted according to the following three levels: the first level is the number of time frames, the second level is the type ID, and the third level is the type identifier.

[0031] S4 trajectory prediction: After data enhancement is performed on the input data obtained in step S3, it is input into the neural network model, and the multi-modal driving trajectories of all unmanned agricultural machinery in the field scene in the next 5 seconds are predicted in parallel (for example, the number of modes is 6, indicating that 6 possible driving trajectories are predicted for each agricultural machinery). The data of each driving trajectory is a frame every 0.5 seconds, and a total of 10 frames of position points are included. The specific method is as follows:

[0032] S4.1 Graph structure construction: Construct the input data in the neural network model TGAP (Spatiotemporal Graph Attention Prediction Network), namely the graph data G. The graph data G is a network composed of node and edge connection relationships. In the field scene, all objects (including agricultural machinery and obstacles) are regarded as nodes i in the graph data G in the neural network model TGAP. Each node i has its own feature vector, which includes the agricultural machinery feature vector and the obstacle feature vector. The agricultural machinery feature vector is expressed as [1, x, y, θ, v, 0] T , the obstacle feature vector is represented as [2,x,y,0,0,r] T , where 1 and 2 represent the categories of agricultural machinery and obstacles, i.e., type indicators, x and y represent the current positions, θ represents the heading angle, v represents the current speed, and r represents the radius of the obstacle. Due to the interaction between objects in the scene, the edge connection relationship between nodes is constructed. The edge connection relationship includes the relationship between agricultural machinery and the interaction relationship between agricultural machinery and obstacles. That is, for any agricultural machinery node N i ,i=1,2,…,n v , its neighbor nodes in the neural network graph G are N j ,j=0,1,2,…,n v +n j , where i≠j;

[0033] Therefore, the shape of trajectory prediction input data is (V, C, T), where V represents all objects in the scene, C = 6 represents the original feature channel (i.e., the feature vectors of agricultural machinery and obstacles), T represents the time step of the historical trajectory, and the shape of the edge connection relationship is (2, E), where E represents the number of edge connections in the neural network graph G;

[0034] S4.2 Temporal feature extraction: The trajectory prediction input data (V, C, T) is input into the trajectory prediction model. The trajectory prediction model uses the TCN convolutional network (temporal convolutional network) as an encoder to extract the temporal features of the input sequence. Each TCN layer contains two one-dimensional causal convolutional layers and residual connections. By stacking TCN layers, the long-term dependent feature information of the time series is extracted and the features are mapped to a higher-dimensional feature space. The shape of the feature information output data is (V, C out ,T), where C outIt represents the number of high-dimensional feature channels extracted by the TCN convolutional network. After the input data passes through the TCN convolutional network, the features (V, C out ,T) is further processed to obtain the time features of all nodes by compressing and aggregating information. The data shape of the time feature is (V,C out ).

[0035] S4.3 Obtaining spatiotemporal features: The extracted time features are passed together with the edge connection relationship constructed in step S4.1 as input to the graph convolutional network to obtain spatiotemporal feature information.

[0036] First, the feature difference between the node and its neighboring nodes is calculated based on the constructed edge connection relationship, and the original feature of the node and the difference feature are spliced ​​along the second channel (feature channel). The spliced ​​features are processed in two ways. One is to use the normalized output as the query vector (Query) of the attention mechanism, and the other is to generate the key vector (Key) and the value vector (Value) through a three-layer encoder-decoder structure. Each encoder-decoder structure is composed of a fully connected layer, a normalization layer, and an activation function. The similarity (i.e., attention weight) between the (Query) query vector and the (Key) key vector is calculated through a dot product operation and the weight is normalized based on the graph structure. The normalized attention weight is weighted summed on the value vector (Value), and finally the aggregated spatiotemporal features are obtained. Its data shape is (V, C out ), the data is split to obtain the spatiotemporal characteristics of each agricultural machine, and the spatiotemporal characteristics are input into the decoder module respectively to predict the multimodal driving trajectory of each agricultural machine in the next M seconds in parallel.

[0037] S4.4 Result prediction: The spatiotemporal features of each agricultural machine aggregated in S4.3 are input into the decoder module, which is composed of multiple sub-network layers. Each sub-network layer is constructed with a multi-layer fully connected network and a residual network structure. During the model training process, for example, if the number of modes is 6 (six future trajectories of the agricultural machine are predicted each time), only the first sub-network layer is trained first. After the model converges, the feature extraction module and the first sub-network layer are frozen, and the remaining five sub-network layers are trained to generate multi-modal trajectory prediction results, improve the accuracy of trajectory prediction, and predict and output six future 5-second driving trajectories for each agricultural machine. Each trajectory data is a frame every 0.5 seconds, and contains a total of 10 frames of position points;

[0038] This embodiment regards agricultural machinery and obstacles as nodes in the graph structure, and constructs the edge connection relationship between agricultural machinery and obstacles. The neural network model TGAP is used to process the information of these nodes and edges, thereby realizing the multi-modal trajectory prediction of each agricultural machinery. This method fully considers the dynamic interaction characteristics in the field scene, provides a more reliable guarantee for the collision warning of unmanned agricultural machinery, and helps to improve the safety and efficiency of agricultural machinery collaborative operations.

[0039] S5 Collision warning: Based on the position information of the agricultural machinery cluster in each frame of the predicted future driving trajectory, the axis-aligned bounding box (AABB) algorithm is used to determine in parallel whether the six future driving trajectories of the agricultural machinery may collide with the other agricultural machinery. If it is determined that a collision may occur, the directed bounding box (OBB) algorithm is used to perform collision detection on the six future driving trajectories of the agricultural machinery in parallel. If the detection result shows that a collision will indeed occur, the first time point T at which the collision occurs is returned. c That is, the collision time, and the collision time T c Collision safety time with agricultural machinery T s For comparison:

[0040] If T c ≥T s When the vehicle is in motion, it will give the unmanned driving system of the agricultural machinery a warning signal of possible collision danger;

[0041] With T c Gradually approaching T s , the risk of collision also gradually increases. If the unmanned driving system of the agricultural machinery does not change its driving route after receiving the danger warning signal, when T c <T s When the unmanned driving system of the agricultural machinery is given an emergency braking signal, the unmanned driving system of the agricultural machinery performs emergency braking to ensure the safety of the unmanned agricultural machinery operation. Figure 2 As shown, the specific method is as follows:

[0042] S5.1 The agricultural machinery is regarded as a rectangular model according to the top view of the agricultural machinery. The length and width of the agricultural machinery are the length and width of the rectangular model respectively. Whether a collision will occur is determined frame by frame through time frames. The center position of the agricultural machinery is the position information of the trajectory prediction at that time point. The heading angle is calculated by the position of the current time frame and the previous time frame, thereby obtaining the position and angle of the rectangular model of the agricultural machinery in the current time frame;

[0043] S5.2 Based on the multimodal trajectory generated by each agricultural machine, the multimodal trajectory of the self-vehicle and each other agricultural machine is detected in parallel to see whether there is a collision. For each possible trajectory of any agricultural machine in the self-vehicle, the AABB bounding box algorithm is used to check whether the projections of the rectangular model of the agricultural machine on the x-axis and y-axis overlap. If the projections of the two agricultural machines do not overlap on any coordinate axis, it is determined that the two agricultural machines will not collide. Otherwise, it is determined that the two agricultural machines will collide, and the process goes to step S5.3;

[0044] S5.3 uses the OBB bounding box algorithm and the Separating Axis Theorem (SAT) for collision detection to determine whether the projections on the four axes parallel to the rectangular sides overlap. If the projections on all axes overlap, it means a collision has occurred, and the first time point of the collision T is returned. c , i.e., collision time;

[0045] S5.4 Set the collision time T c Collision safety time with agricultural machinery T s For comparison:

[0046] If T c ≥T s When the vehicle is in collision, a warning signal of possible collision is sent to the unmanned driving system of the agricultural machinery to remind the agricultural machinery to take action to avoid collision with the adjacent vehicle;

[0047] If the unmanned driving system does not take corresponding measures after receiving the danger warning signal, that is, does not change the driving route of the agricultural machinery, as T c Gradually approaching T s , the risk of collision also increases gradually. When T c <T s When the warning information is transmitted, an emergency braking signal is sent to the unmanned driving system of the agricultural machinery, so that the unmanned driving system performs emergency braking on the agricultural machinery to ensure the operation safety of the unmanned agricultural machinery. The warning information transmission process refers to Figure 3 shown.

[0048] This implementation of the collision safety time T s It is obtained by the following method:

[0049] Vehicle in t 2 Speed ​​at the end of time for:

[0050]

[0051] Where: v 0 is the initial velocity; a 0 is the initial acceleration, a max is the maximum braking deceleration;

[0052] In t3 During the time period, maintain the maximum braking deceleration a max , t 3 The avoidance time TTA is the time required for the vehicle to take avoidance measures (such as steering or braking) to avoid collision when the vehicle is in danger. 3 It is expressed as:

[0053]

[0054] Among them, v t2 is the speed of the target agricultural machinery, a max is the maximum deceleration, a 0 is the initial acceleration;

[0055] Therefore, the minimum time interval T from detecting an impending collision to completing braking is:

[0056] T=t 0 +t 1 +t t +t r +t 2 +t 3

[0057] Among them, t 0 is the communication delay time, t 1 represents the running time of trajectory prediction module, t t is the time when the warning information is generated and transmitted, t r is the reaction time of the unmanned driving system, t 2 is the time required for the braking force to increase from 0 to the maximum braking force, t 3 is the braking duration;

[0058] Therefore, the collision safety time T of agricultural machinery is set s =T+Δt, where Δt is a time constant, for example, Δt=0.5s.

[0059] The remaining components of the collision warning system for cooperative operation of unmanned agricultural machinery according to an embodiment of the present invention, such as an axisymmetric bounding box (AABB), an directed bounding box (OBB), a TGAP neural network model, etc., as well as operations are known to ordinary technicians in the field and will not be described in detail here.

[0060] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0061] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0062] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or a communication; it can be a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0063] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0064] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0065] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A collision warning system for cooperative operation of unmanned agricultural machinery, characterized in that: include A data acquisition module, which is used to collect the posture information of the unmanned agricultural machinery and the status information of obstacles in the scene; A communication module, wherein the communication module is used to transmit the data information collected by the collection module to the data processing module; A data processing module is used to receive the trajectory data of all unmanned agricultural machinery in the scene in the past N seconds and the status information of all obstacles, and analyze and process the received data to determine whether the unmanned agricultural machinery in the scene will collide with other unmanned agricultural machinery in the next M seconds. If a collision occurs, a warning signal or an emergency braking signal is sent to the unmanned driving system of the unmanned agricultural machinery, so that the unmanned driving system of the unmanned agricultural machinery performs a collision warning or emergency braking on the agricultural machinery, thereby realizing a graded warning for the unmanned agricultural machinery.

2. A collision warning system for unmanned agricultural machinery coordinated operation according to claim 1, characterized in that: The data processing module includes a data preprocessing module, a trajectory prediction module and a collision warning module. The data preprocessing module is used to receive the posture information of all unmanned agricultural machinery in the scene and the status information of obstacles, and analyze and process the received data, and transmit the analyzed and processed data to the trajectory prediction module for trajectory prediction. The trajectory prediction module transmits the prediction result to the collision warning module. The collision warning module processes the prediction result to determine whether the unmanned agricultural machinery in the scene will collide with the other unmanned agricultural machinery in the next M seconds. If a collision occurs, a warning signal or an emergency braking signal is sent to the unmanned driving system of the unmanned agricultural machinery, so that the unmanned driving system of the unmanned agricultural machinery performs a collision warning or emergency braking on the agricultural machinery, thereby realizing a graded warning for unmanned agricultural machinery.

3. A collision warning method for cooperative operation of unmanned agricultural machinery, characterized in that: The following steps are involved: S1 obtains the posture information of agricultural machinery: builds a self-organizing network communication system for the unmanned agricultural machinery cluster. The unmanned agricultural machinery in the scene collects the posture information of the vehicle every a seconds, and then sends the posture information of the vehicle for N seconds to other unmanned agricultural machinery through broadcast wireless communication. At the same time, it receives the posture information of other unmanned agricultural machinery in the scene for N seconds. Finally, the posture information of all unmanned agricultural machinery in the scene is obtained and transmitted to the data preprocessing module of the vehicle. S2 Obtaining obstacle status information: The unmanned agricultural machinery uses the vehicle as the center and collects point cloud data within a set range through the laser radar on the vehicle. Data processing is performed based on the acquired point cloud data to obtain the status information of all obstacles in the scene. The obstacle status information includes the center position and influence range of each obstacle. S3 data preprocessing: Process the posture information of all agricultural machinery and the state information of obstacles obtained in step S1 and step S2, count all objects in the scene in the past N seconds, assign a unique ID to each object, assign different type identifiers to agricultural machinery and obstacles, and arrange the state data of all objects in the scene according to the time frame to obtain the input data for trajectory prediction; S4 trajectory prediction: After data enhancement, the input data obtained in step S3 is input into the neural network model to predict the multi-modal driving trajectories of all unmanned agricultural machinery in the field scene in the future M seconds in parallel. The data of each driving trajectory is a frame every a seconds, and a total of W frames of position points are included; S5 Collision warning: Based on the position information of the unmanned agricultural machinery cluster in each frame of the predicted future driving trajectory, the axis-aligned bounding box algorithm is used to determine in parallel whether the future driving trajectory of the agricultural machinery may collide with other agricultural machinery. If it is determined that a collision may occur, the directed bounding box algorithm is used to perform collision detection on the future driving trajectory of the agricultural machinery. If the detection result shows that a collision will indeed occur, the first time point T at which the collision occurs is returned. c That is, the collision time, and the collision time T c The collision safety time with the agricultural machinery is T s For comparison: If T c ≥T s When the unmanned agricultural machinery is in motion, a warning signal of possible collision is given to the unmanned driving system of the unmanned agricultural machinery; If the unmanned driving system of the agricultural machinery does not change its driving route after receiving the danger warning signal, when T c <T s When the unmanned driving system of the agricultural machinery is in emergency braking, an emergency braking signal is given to the unmanned driving system of the agricultural machinery, causing the unmanned driving system of the agricultural machinery to perform emergency braking.

4. The collision warning method for cooperative operation of unmanned agricultural machinery according to claim 3, characterized in that: The processing method of the point cloud data in step S2 is as follows: in order to model the spatial relationship of the field environment, it is necessary to accurately identify and model the position and size of the field obstacles. According to the acquired point cloud data, a point cloud clustering algorithm, such as DBSCAN, is used to automatically divide the dense point cloud area into multiple clusters, each cluster represents an independent obstacle cluster, and the minimum circumscribed circle algorithm is further applied to calculate the minimum circle covering each obstacle cluster, so as to determine the center position and influence range of the obstacle, and then obtain the status information of all obstacles, and pass the status information of all obstacles to the data preprocessing module.

5. The collision warning method for cooperative operation of unmanned agricultural machinery according to claim 3, characterized in that: In step S3, all status data are sorted according to the following three levels: the first level is the time frame number, the second level is the type ID, and the third level is the type identifier.

6. The collision warning method for cooperative operation of unmanned agricultural machinery according to claim 3, characterized in that: The step S4 specifically comprises the following steps: S4.1 Graph structure construction: Construct the input data in the neural network model TGAP, namely the graph data G. The graph data G is a network composed of node and edge connection relationships. In the scene, all objects are regarded as nodes i in the graph data G in the neural network model TGAP. Each node i has its own feature vector, which includes the agricultural machinery feature vector and the obstacle feature vector. The agricultural machinery feature vector is expressed as [1, x, y, θ, v, 0] T , the obstacle feature vector is represented as [2,x,y,0,0,r] T , where 1 and 2 represent the categories of agricultural machinery and obstacles, i.e., type indicators, x and y represent the current position, θ represents the heading angle, v represents the current speed, and r represents the radius of the obstacle; for any agricultural machinery node N i ,i=1,2,…,n v , its neighbor nodes in the neural network graph G are N j ,j=0,1,2,…,n v +n j , where i≠j; Therefore, the shape of the trajectory prediction input data is (V, C, T), where V represents all objects in the scene, C = 6 represents the original feature channel, T represents the time step of the historical trajectory, and the shape of the edge connection relationship is (2, E), where E represents the number of edge connections in the neural network graph G; S4.2 Temporal feature extraction: The trajectory prediction input data (V, C, T) is input into the trajectory prediction model. The trajectory prediction model uses the TCN convolutional network as an encoder to extract the temporal features of the input sequence. Each TCN layer contains two one-dimensional causal convolutional layers and residual connections. By stacking TCN layers, the long-term dependent feature information of the time series is extracted and the features are mapped to a higher-dimensional feature space. The shape of the feature information output data is (V, C out ,T), where C out It represents the number of high-dimensional feature channels extracted by the TCN convolutional network. After the input data passes through the TCN convolutional network, the features (V, C out ,T) is further processed to obtain the time features of all nodes by compressing and aggregating information. The data shape of the time feature is (V,C out ); S4.3 Obtaining spatiotemporal features: passing the extracted time features together with the edge connection relationship constructed in step S4.1 as input to the graph convolutional network to obtain spatiotemporal feature information; S4.4 Result prediction: The spatiotemporal features of each unmanned agricultural machinery aggregated in step S4.3 are input into the decoder module. The module consists of multiple sub-network layers. Each sub-network layer is constructed using a multi-layer fully connected network and a residual network structure. During the model training process, the number of modes is set to X. First, only the first sub-network layer is trained. After the model converges, the feature extraction module and the first sub-network layer are frozen, and the remaining sub-network layers are trained to generate multi-modal trajectory prediction results. The predicted output is X driving trajectories of each agricultural machinery in the next M seconds. Each trajectory data is a frame every a seconds, and a total of W frames of position points are included.

7. A collision warning method for cooperative operation of unmanned agricultural machinery according to claim 6, characterized in that: The specific steps of step S4.3 are as follows: First, the feature difference between the node and its neighbor nodes is calculated based on the constructed edge connection relationship, and the original feature of the node and the difference feature are spliced ​​along the second channel. The spliced ​​features are processed in two ways. One is to use the normalized output as the query vector of the attention mechanism, and the other is to generate the key vector and value vector through a three-layer encoder-decoder structure. Each encoder-decoder structure is composed of a fully connected layer, a normalization layer, and an activation function. The similarity between the query vector and the key vector is calculated through a dot product operation and the weight is normalized based on the graph structure. The normalized attention weight is weighted summed on the value vector (Value), and finally the aggregated spatiotemporal feature is obtained. Its data shape is (V, C out ), the data is split to obtain the spatiotemporal characteristics of each agricultural machine, and the spatiotemporal characteristics are input into the decoder module respectively to predict the multimodal driving trajectory of each agricultural machine in the next M seconds in parallel.

8. The collision warning method for cooperative operation of unmanned agricultural machinery according to claim 3, characterized in that: The specific steps of step S5 are as follows: S5.1 The agricultural machinery is regarded as a rectangular model according to the top view of the agricultural machinery. The length and width of the agricultural machinery are the length and width of the rectangular model respectively. Whether a collision will occur is determined frame by frame through time frames. The center position of the agricultural machinery is the position information of the trajectory prediction at that time point. The heading angle is calculated by the position of the current time frame and the previous time frame, thereby obtaining the position and angle of the rectangular model of the agricultural machinery in the current time frame; S5.2 Based on the multimodal trajectory generated by the unmanned agricultural machinery, it is detected in parallel whether there is a collision between the self-vehicle and the multimodal trajectory of each other agricultural machinery. For each possible trajectory of the agricultural machinery self-vehicle and any agricultural machinery, the axis-aligned bounding box algorithm is used to check whether the projections of the rectangular models of the agricultural machinery on the x-axis and y-axis overlap. If the projections of the two agricultural machinery do not overlap on any coordinate axis, it is determined that the two agricultural machinery will not collide. Otherwise, it is determined that the two agricultural machinery will collide, and the process goes to step S5.3; S5.3 uses the directed bounding box algorithm and the separating axis theorem to perform collision detection, and determines whether the projections on the four axes parallel to the rectangular sides overlap. If the projections on all axes overlap, it means a collision has occurred, and the first time point of the collision is returned. c , i.e., collision time; S5.4 Set the collision time T c Collision safety time with agricultural machinery T s For comparison: If T c ≥T s When the unmanned agricultural machinery is in motion, a warning signal of possible collision is given to the unmanned driving system of the unmanned agricultural machinery; If the unmanned driving system of the agricultural machinery does not change its driving route after receiving the danger warning signal, when T c <T s When the unmanned driving system of the agricultural machinery is in emergency, an emergency braking signal is given to the unmanned driving system of the agricultural machinery, causing the unmanned driving system of the agricultural machinery to perform emergency braking.

9. A collision warning method for cooperative operation of unmanned agricultural machinery according to claim 3 or 8, characterized in that: The collision safety time T in step S5 s It is obtained by the following method: The speed of the vehicle at the end of time t2 for: Where: v0 is the initial velocity; a0 is the initial acceleration, a max is the maximum braking deceleration; During the t3 period, the maximum braking deceleration a is maintained. max , t3 is the risk avoidance time of unmanned agricultural machinery operation, t3 is expressed as: in, is the speed of the target agricultural machinery, a max is the maximum deceleration, a0 is the initial acceleration; Therefore, the minimum time interval T from detecting an impending collision to completing braking is: T=t0+t1+t t +t r +t2+t3 Among them, t0 is the communication delay time, t1 is the running time of the trajectory prediction module, and t t is the time when the warning information is generated and transmitted, t r is the reaction time of the unmanned driving system, t2 is the time required for the braking force to increase from 0 to the maximum braking force, and t3 is the braking duration; Therefore, the collision safety time T of agricultural machinery is set s =T+Δt, where Δt is a time constant, for example, Δt=0.5s.

Images