A UAV embodied intelligent safety decision-making method based on feedback learning

Abstract

The present invention discloses a drone embodied intelligent safety decision-making method based on mutual feedback learning, namely, obtaining drone images taken at multiple continuous time points; obtaining multiple basic environmental information based on the drone images taken at multiple continuous time points; judging whether the environmental object is stationary or in motion according to the position of the environmental object at multiple continuous time points; if the environmental object is in motion, obtaining the environmental object movement speed, environmental object movement direction and the distance between the environmental objects through an environmental matching network and a time discrimination network according to the drone images taken at multiple continuous time points and the basic environmental information; making a safety decision according to the environmental object movement speed, environmental object movement direction and the distance between the environmental objects. This decision-making method improves the accuracy of the drone embodied intelligent perception environment data, and on this basis, judges whether the drone will collide to control the drone to stop, thereby improving the accuracy of high-intelligence safety decisions.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an embodied intelligent safety decision-making method for a drone based on mutual feedback learning. Background Art

[0002] At present, embodied intelligence systems perceive various signals and data in the environment, convert them into understanding of the environment, and make intelligent decisions through thinking and learning. Unlike traditional rule-based or symbol-based artificial intelligence, embodied intelligence emphasizes the combination of perception and action, so that the intelligent agent can better understand its surrounding environment and interact with the environment. However, when interacting with the surrounding environment, there are different decision-making methods for objects moving toward the drone and stationary objects. At the same time, it is necessary to overcome the differences in shape and size of objects in the environment at different time points in motion, and more accurate matching is required. Summary of the invention

[0003] In order to solve the problems of the prior art, the present invention provides a UAV embodied intelligent safety decision-making method based on feedback learning, which specifically includes:

[0004] In a first aspect, an embodiment of the present invention provides a drone embodied intelligent safety decision-making method based on feedback learning, comprising:

[0005] Acquire images taken by a drone at multiple consecutive time points; the images taken by the drone include multiple environmental objects; the environmental objects represent objects in the images taken by the drone of the surrounding environment;

[0006] Based on the drone images taken at multiple consecutive time points, multiple basic environmental information are obtained; one basic environmental information corresponds to one drone image; the basic environmental information includes the category of environmental objects, the contour of environmental objects, the position of environmental objects and the contour of environmental background; the position of environmental objects represents the center point, width and height of the detected object in the drone image;

[0007] Judging whether an environmental object is stationary or moving according to the position of the environmental object at multiple consecutive time points;

[0008] If the environmental objects are moving, the speed, direction and distance of the environmental objects are obtained based on the images taken by the drone at multiple consecutive time points and the basic environmental information through the environmental matching network and the time discrimination network;

[0009] Making safety decisions based on the speed of the environmental object, the direction of the environmental object and the distance between the environmental objects;

[0010] The environment detection network performs feedback learning through a temporal discrimination network.

[0011] Optionally, judging whether an environmental object is stationary or moving according to the positions of the environmental object at multiple consecutive time points includes:

[0012] Based on the positions of environmental objects at multiple consecutive time points, a three-dimensional environmental point set is obtained; and multiple three-dimensional environmental point sets are obtained corresponding to multiple environmental objects;

[0013] Clustering the multiple three-dimensional environmental point sets to obtain curve clusters and cluster center curves; the curve clusters include movement routes of multiple environmental objects; the cluster center curves represent movement routes of environmental objects that are stationary during the movement of the drone in the curve clusters;

[0014] The environmental objects corresponding to the three-dimensional environmental point set in the curve cluster are set to be stationary;

[0015] The environmental objects corresponding to the set of three-dimensional environmental points outside the curve cluster are set as moving.

[0016] Optionally, if the environmental object is moving, the moving speed, moving direction and distance between the environmental objects are obtained through an environmental matching network and a time discrimination network according to the images taken by the drone at multiple consecutive time points and basic environmental information, including:

[0017] Obtaining an initial object position; the initial object position is the position of an environmental object to be detected;

[0018] Based on the drone images and the initial object positions at multiple consecutive time points, two matching object positions are obtained by matching through an environment matching network and a time discrimination network; the matching object positions are the positions of the same environment object at other time points except the initial object position;

[0019] Obtaining a size ratio; the size ratio represents a ratio of an actual distance to a distance in the image;

[0020] The actual distance is obtained by multiplying the difference between the initial object position and the matching object position by the size ratio;

[0021] Subtracting the time point corresponding to the position of the matched object from the time point corresponding to the initial object position to obtain the actual time length;

[0022] According to the positions of two matching objects, the movement direction of the environmental object is obtained;

[0023] The speed of environmental objects is obtained by dividing the actual distance by the actual time length.

[0024] Optionally, the images taken by the drone at multiple consecutive time points and the initial object positions are matched through an environment matching network and a time discrimination network to obtain two matching object positions, including:

[0025] Obtaining an initial environmental object category and an initial object contour; the initial environmental object category is the environmental object category corresponding to the initial object position; the initial object contour is the contour of the environmental object corresponding to the initial object position;

[0026] Construct a detection set by using the contours and positions of multiple environmental objects of the same category as the initial environmental objects in the image taken by the drone; obtain multiple detection sets corresponding to multiple consecutive time points;

[0027] Inputting the image corresponding to the environmental object position of the detection set and the initial object image into the environmental matching network for matching to obtain a plurality of first matching object positions;

[0028] The first matching object position is subjected to a contour detection algorithm to obtain a first matching object contour; a plurality of first matching object positions are correspondingly obtained to obtain a plurality of first matching object contours;

[0029] Calculate similarity values ​​between the multiple first matching object contours and the initial object contour, and use the first matching object position corresponding to the first matching object contour with a greater similarity value as the second matching object position; and obtain multiple second matching object positions corresponding to multiple consecutive time points;

[0030] According to the time points from far to near, the images corresponding to the second matching object positions in the image taken by the drone are input into the time discrimination network to obtain two matching object positions.

[0031] Optionally, the images corresponding to the second matching object positions in the image taken by the drone are input into a time discrimination network according to the time points from far to near to obtain two matching object positions, including:

[0032] A test image and annotated position are obtained; the annotated position is a position of an environmental object corresponding to a time point greater than other time points among a plurality of continuous time points; the test image is one of the images corresponding to a plurality of second matching object positions except the annotated position; and n-1 test images are obtained corresponding to n second matching object positions;

[0033] Construct a test image set from the images corresponding to the multiple second matching object positions except the test image and the image corresponding to the annotated position; obtain n-2 test image sets corresponding to n second matching object positions;

[0034] The images in the test image set are sequentially input into the time discrimination network according to the time points from far to near, and the predicted environmental object positions are obtained; the predicted environmental object positions represent the object positions at the time points corresponding to the predicted annotated positions; n-2 test image sets correspond to obtaining n-2 predicted environmental object positions;

[0035] The test image corresponding to the value where the similarity value between the predicted environmental object position and the marked position is less than the similarity threshold is set as the optimized image;

[0036] Two second matching object positions corresponding to the optimized images whose sizes are larger than those of the other optimized images among the multiple optimized images are set as the matching object positions.

[0037] Optionally, the step of inputting the image corresponding to the environmental object position of the detection set and the initial object image into an environmental matching network for matching to obtain a plurality of first matching object positions includes:

[0038] Inputting the image of the position of the environmental object into a first matching convolutional network, extracting features, and obtaining a first feature;

[0039] Inputting the initial object image into a second matching convolutional network to extract features to obtain second features;

[0040] Inputting the first feature and the second feature into a matching discriminant network to determine whether the two images are similar and obtain a discriminant value;

[0041] If the discrimination value is greater than the discrimination threshold, the position of the environmental object corresponding to the first feature is set as the first matching object position.

[0042] Optionally, the environment detection network performs feedback learning through the temporal discrimination network, including:

[0043] The environment detection network includes a first matching convolutional network, a second matching convolutional network and a matching discriminant network;

[0044] Obtaining a training image, marking an initial object contour, marking a first matching object position, and marking a matching object position;

[0045] Inputting the training image into an environment detection network to obtain a plurality of training first matching object positions;

[0046] The first training matching object position and the marked object position are subjected to a first loss function to obtain a position loss value;

[0047] Train the environment detection network by backpropagating the position loss value;

[0048] Calculating similarity values ​​between the plurality of first training matching object positions and the initial object contour annotated in the training image, and taking the first training matching object position having a larger similarity value than other similarity values ​​as the second training matching object position;

[0049] Inputting a plurality of training second matching object positions into a temporal discriminant network to obtain training matching object positions;

[0050] By training the matching object position and marking the matching object position, a matching loss value is obtained through the second loss function;

[0051] By backpropagating the matching loss values, we train the temporal discriminator network and then train the environment detection network.

[0052] Optionally, the performing safety decision learning according to the movement speed of the environmental object, the movement direction of the environmental object and the distance between the environmental objects includes:

[0053] According to the movement direction of the environmental objects, the distance between the environmental objects is divided into an environmental vertical distance, an environmental height distance and an environmental horizontal distance;

[0054] According to the moving direction of the environmental object, the moving speed of the environmental object is divided into the environmental vertical speed, the environmental height speed and the environmental horizontal speed;

[0055] Get the vertical speed, altitude speed and horizontal speed of the drone at the current time point;

[0056] Add the environmental horizontal distance to the drone's horizontal speed to obtain a horizontal relative speed;

[0057] Add the vertical distance of the environment to the vertical speed of the drone to obtain the vertical relative speed;

[0058] Add the environmental height distance to the UAV height speed to obtain the height relative speed;

[0059] If the quotient of the environmental horizontal distance divided by the horizontal relative speed, the quotient of the environmental height distance divided by the height relative speed, and the quotient of the environmental vertical distance divided by the vertical relative speed are equal, it is determined that the UAV will collide, and a stop flight signal is sent to the UAV to control the UAV to stop.

[0060] Optionally, obtaining a three-dimensional environmental point set based on the positions of environmental objects at multiple consecutive time points includes:

[0061] Construct a three-dimensional environment matrix; the number of rows of the three-dimensional environment matrix is ​​equal to the width of the image taken by the drone; the number of columns of the three-dimensional environment matrix is ​​equal to the height of the image taken by the drone; the number of pages of the three-dimensional environment matrix is ​​equal to the number of time points at which the image taken by the drone is taken;

[0062] According to the center point and time point of the environmental object, the value of the corresponding position in the three-dimensional environmental matrix is ​​set to 1;

[0063] Subtract the adjacent positions of the time points that are 1 in the three-dimensional environment matrix to obtain multiple three-dimensional environment points;

[0064] Construct multiple 3D environment points into a 3D environment point set.

[0065] In a second aspect, an embodiment of the present invention provides a drone embodied intelligent safety decision-making system based on feedback learning, comprising:

[0066] Acquisition module: acquires drone images taken at multiple consecutive time points; the drone images include multiple environmental objects; the environmental objects represent objects in the images taken by the drone of the surrounding environment;

[0067] Basic information acquisition module: based on the drone images taken at multiple consecutive time points, multiple basic environmental information are obtained; one basic environmental information corresponds to one drone image; the basic environmental information includes the category of environmental objects, the contour of environmental objects, the position of environmental objects and the contour of environmental background; the position of environmental objects represents the center point, width and height of the detected object in the drone image;

[0068] Motion judgment module: judges whether an environmental object is stationary or moving based on the position of the environmental object at multiple consecutive time points;

[0069] Environmental object judgment module: If the environmental object is moving, the moving speed and distance of the environmental object are obtained based on the images taken by the drone at multiple consecutive time points and the basic environmental information through the environmental matching network and the time discrimination network;

[0070] UAV control module: Make safety decisions based on the movement speed of the environmental objects and the distance between the environmental objects; the environmental detection network performs feedback learning through the time discrimination network.

[0071] Compared with the prior art, the embodiments of the present invention achieve the following beneficial effects:

[0072] 1. The method for embodied intelligent safety decision-making of unmanned aerial vehicles based on mutual feedback learning provided by the present invention determines the changes of all objects in the environment during the movement of the unmanned aerial vehicle. According to the different changes in the objects that are in motion and still, it is determined whether the surrounding environmental objects are moving or still. By matching the different positions of the same environmental object at multiple time points, the purpose is to obtain the optimal position of the environmental object at the time point with both accurate detection and the largest bounding box, so as to obtain the speed of movement. According to the image of the surrounding environment of the unmanned aerial vehicle and the flight status of the unmanned aerial vehicle, the changes of the unmanned aerial vehicle are controlled. At the same time, through mutual feedback learning, the unmanned aerial vehicle is controlled to make safe decisions. And the system is applied in the direction of the unmanned aerial vehicle. Accurate embodied intelligent safety decision-making of the unmanned aerial vehicle in mutual feedback learning is achieved.

[0073] 2. When the present invention interacts with the surrounding environment, it does not need to use different decision-making methods for objects moving toward the drone and stationary objects. It also does not need to overcome the differences in shapes and sizes of objects in the environment at different time points during movement. The decision-making method is simpler.

[0074] 3. The drone embodied intelligent safety decision-making method of the present invention takes into account the influence of environmental data based on embodied intelligence, calculates the position loss value based on the first matching object position and the marked object position, trains the environment detection network through back propagation, calculates the matching loss value by training the matching object position and the marked matching object position, trains the time discrimination network through back propagation, and then trains the environment detection network to achieve mutual feedback learning between the time discrimination network and the environment detection network, thereby improving the accuracy of the drone embodied intelligent perception of environmental data. On this basis, it is judged whether the drone will collide to control the drone to stop, thereby improving the accuracy of high-intelligence safety decision-making. That is, based on embodied intelligence, mutual feedback learning is performed to make safe flight decisions for drones, and perception and action are combined to make safety decisions, so that the drone intelligent body can better understand the environment around it and interact with the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The present invention has the following accompanying drawings:

[0076] Figure 1 This is a flow chart of a method for embodied intelligent safety decision-making of a drone based on mutual feedback learning provided by an embodiment of the present invention;

[0077] Figure 2 It is a structural schematic diagram of a method for embodied intelligent safety decision-making of a drone based on mutual feedback learning provided by an embodiment of the present invention, in which an environment detection network performs mutual feedback learning through a time discrimination network. DETAILED DESCRIPTION

[0078] The following is combined with Figures 1-2 , the present invention is described in detail.

[0079] Example 1

[0080] like Figure 1 As shown, an embodiment of the present invention provides a UAV embodied intelligent safety decision-making method based on feedback learning, and the method includes:

[0081] S101: Obtaining drone images taken at multiple consecutive time points. The drone images include multiple environmental objects. The environmental objects refer to objects in the images taken by the drone of the surrounding environment.

[0082] In this embodiment, the image taken by the drone is an image taken by a camera on the front of the drone.

[0083] S102: Based on the drone images taken at multiple consecutive time points, multiple basic environmental information are obtained. One basic environmental information corresponds to one drone image. The basic environmental information includes the category of environmental objects, the outline of environmental objects, and the position of environmental objects. The position of environmental objects represents the center point, width, and height of the detected object in the drone image.

[0084] S103: judging whether the environmental object is stationary or moving according to the positions of the environmental object at a plurality of consecutive time points.

[0085] S104: If the environmental object is moving, the moving speed, moving direction and distance between the environmental objects are obtained based on the drone images and basic environmental information at multiple consecutive time points through an environmental matching network and a time discrimination network.

[0086] S105: Making a safety decision based on the moving speed of the environmental object, the moving direction of the environmental object and the distance between the environmental objects.

[0087] The environment detection network performs feedback learning through a temporal discrimination network.

[0088] The UAV flight status includes the speed and direction of the UAV flight at the current time point.

[0089] The speed and direction of the UAV flight at the current time point and the speed of the environmental object are used to determine whether a collision will occur. If a collision will occur, the collision time length is obtained by dividing the sum of the environmental object speed and the UAV flight speed at the current time point by the distance between the environmental objects. The length of the collision event is used to determine whether it is within the time threshold for the occurrence of a collision, thereby controlling the adjustment state of the UAV. In this embodiment, the adjustment state is to stop the UAV.

[0090] Judging whether an environmental object is stationary or moving based on the position of the environmental object at multiple consecutive time points includes:

[0091] Based on the environmental object categories and environmental object positions at multiple consecutive time points, a three-dimensional environmental point set is obtained. Multiple environmental object categories correspond to multiple three-dimensional environmental point sets.

[0092] The plurality of three-dimensional environmental point sets are clustered to obtain curve clusters and cluster center curves. The curve clusters include movement routes of a plurality of environmental objects. The cluster center curves represent movement routes of environmental objects that are stationary during the movement of the drone in the curve clusters.

[0093] In this embodiment, the clustering algorithm is the k-means algorithm.

[0094] The environmental objects corresponding to the three-dimensional environmental point set in the curve cluster are set to be stationary.

[0095] The environmental objects corresponding to the set of three-dimensional environmental points outside the curve cluster are set as moving.

[0096] Through the above method, the changes of all objects in the environment are judged during the movement of the drone. Because the static environmental objects are moving relative to the moving drone, but the changes in the movement routes are almost the same, while the changes in multiple moving objects are different, so the method of judging the changes is used to judge whether the surrounding environmental objects are moving or static. Optionally, if the environmental objects are moving, the environmental object movement speed, environmental object movement direction and the distance between environmental objects are obtained through the environmental matching network and the time discrimination network based on the drone images and basic environmental information at multiple consecutive time points, including:

[0097] Obtain an initial object position, which is the position of an environmental object at a distance to be detected.

[0098] Based on the drone images and initial object positions at multiple consecutive time points, two matching object positions are obtained by matching through the environment matching network and the time discrimination network. The matching object position is the position of the same environment object at other time points except the initial object position.

[0099] The matching object position includes the horizontal coordinate and the vertical coordinate of the center point, and the width and height of the matching environment object.

[0100] A size ratio is obtained, wherein the size ratio indicates a ratio between an actual distance and a distance in the image.

[0101] In this embodiment, the size ratio is calculated based on the focal length of the camera. The actual distance is obtained by multiplying the difference between the initial object position and the matching object position by the size ratio.

[0102] The actual time length is obtained by subtracting the time point corresponding to the matched object position from the time point corresponding to the initial object position.

[0103] According to the positions of the two matching objects, the movement direction of the environmental object is obtained.

[0104] The moving direction of the environmental object is the vertical direction with the moving direction of the drone, the horizontal direction is perpendicular to the vertical direction when looking down, and the height direction is perpendicular to the horizontal and vertical directions. The angle between the straight line connecting the positions of the two matching objects and the vertical direction, the horizontal direction, and the height direction is taken as the moving direction of the environmental object.

[0105] The movement speed is obtained by dividing the actual distance by the actual time length.

[0106] Through the above method, the environment of the drone will change during operation, the distant scenery will become the current scenery, and the current scenery will disappear. Therefore, by matching the same environmental object at different time points, the movement of the environmental object can be obtained.

[0107] Optionally, the images taken by the drone at multiple consecutive time points and the initial object positions are matched through an environment matching network and a time discrimination network to obtain two matching object positions, including:

[0108] An initial environmental object category and an initial object contour are obtained. The initial environmental object category is the environmental object category corresponding to the initial object position. The initial object contour is the contour of the environmental object corresponding to the initial object position.

[0109] A detection set is constructed by using multiple environmental object contours and environmental object positions of the same environmental object category as the initial object position. Multiple detection sets are obtained corresponding to multiple continuous time points. The image corresponding to the environmental object position of the detection set is input into the environmental matching network with the initial object image for matching to obtain multiple first matching object positions.

[0110] The image corresponding to the position of the environmental object in the detection set is an image obtained by cutting a boundary box corresponding to the position of the environmental object in the detection set.

[0111] The first matching object position includes a center point, a horizontal coordinate, a vertical coordinate, and a width and a height of a bounding box.

[0112] The first matching object position is subjected to a contour detection algorithm to obtain a first matching object contour. Multiple first matching object positions correspond to multiple first matching object contours. In this embodiment, the contour detection algorithm is a canny edge detection algorithm.

[0113] The similarity values ​​of the multiple first matching object contours and the initial object contour are calculated, and the first matching object position corresponding to the first matching object contour having a larger similarity value than other first matching object contours is used as the second matching object position. Multiple second matching object positions are obtained corresponding to multiple consecutive time points.

[0114] In this embodiment, the similarity value of the contour is calculated by Euclidean distance.

[0115] According to the time points from far to near, the images corresponding to the second matching object positions in the image taken by the drone are input into the time discrimination network to obtain two matching object positions.

[0116] Through the above method, the categories that can be matched are obtained, and the same environmental objects at different time points are found through the environmental matching network. Through contour detection at multiple time points, the environmental objects that best match the initial object position are found.

[0117] Optionally, the images corresponding to the second matching object positions in the image taken by the drone are input into a time discrimination network according to the time points from far to near to obtain two matching object positions, including:

[0118] A test image and annotated position are obtained. The annotated position is the position of an environmental object corresponding to a time point greater than other time points among multiple continuous time points. The test image is one of the images corresponding to the position other than the annotated position of multiple second matching object positions. n-1 test images are obtained corresponding to n second matching object positions.

[0119] The images corresponding to the plurality of second matching object positions, except the test image and the image corresponding to the annotated position, are used to construct a test image set. n-2 test image sets are obtained corresponding to n second matching object positions.

[0120] The images in the test image set are input into the time discrimination network in order from far to near according to the time point, and the predicted environmental object position is obtained. The predicted environmental object position represents the object position at the time point corresponding to the predicted annotated position. The n-2 test image sets correspond to n-2 predicted environmental object positions.

[0121] Wherein, the temporal discrimination network is a trained temporal convolutional network (Temporal Convolutional Network, TCN).

[0122] The test image corresponding to the value where the similarity value between the predicted environmental object position and the marked position is less than the similarity threshold is set as the optimized image.

[0123] In this embodiment, the Euclidean distance is used to calculate the similarity value. The similarity threshold is 2.

[0124] Two second matching object positions corresponding to the optimized images whose sizes are larger than those of the other optimized images among the multiple optimized images are set as the matching object positions.

[0125] Through the above method, a series of detected positions and image information are obtained, with the aim of obtaining the optimal position of the environmental object at two time points with both accurate detection and the largest bounding box.

[0126] The image corresponding to the environmental object position of the detection set and the initial object image are input into the environmental matching network for matching to obtain multiple first matching object positions, including:

[0127] The image of the position of the environmental object is input into a first matching convolutional network to extract features to obtain a first feature.

[0128] Wherein, the first matching convolutional network is a convolutional neural network (Convolutional Neural Networks, CNN).

[0129] The initial object image is input into a second matching convolutional network to extract features to obtain second features.

[0130] The second matching convolutional network is a convolutional neural network (CNN). The first matching convolutional network and the second matching convolutional network have different parameters and structures.

[0131] The first feature and the second feature are input into a matching discriminant network to determine whether the two images are similar and obtain a discriminant value.

[0132] The matching discrimination network is a discriminator network (Discriminator Network, D). If the discrimination value is greater than the discrimination threshold, the position of the environmental object corresponding to the first feature is set as the first matching object position.

[0133] In this embodiment, the discrimination threshold is 0.9.

[0134] The environment detection network performs feedback learning through the temporal discrimination network, including:

[0135] The environment detection network performs feedback learning through the temporal discrimination network. Figure 2 shown.

[0136] The environment detection network includes a first matching convolutional network, a second matching convolutional network and a matching discriminant network.

[0137] The first matching convolutional network is a convolutional neural network (CNN), and the second matching convolutional network is a convolutional neural network (CNN) having a different structure from the first matching convolutional network. The matching discriminator network is a discriminator network (D).

[0138] A training image is obtained, an initial object contour is marked, a first matching object position is marked, and a matching object position is marked.

[0139] The marked first matching object position includes multiple positions, and the marked matching object position includes two positions.

[0140] The training image represents an image corresponding to the historical position of an environmental object used for training. The annotated initial object contour is the contour of the historical environmental object used for training. The annotated first matching object position is the position of the environmental object corresponding to multiple manually annotated time points. The annotated matching object position is the position corresponding to the time point that is the best match with the environmental object position used for training.

[0141] The training image is input into the environment detection network to obtain a plurality of training first matching object positions. The training first matching object positions and the marked object positions are compared with each other through a first loss function to obtain a position loss value.

[0142] Wherein, the loss function is a cross entropy loss function.

[0143] Train the environment detection network by backpropagating the position loss values.

[0144] Similarity values ​​are calculated between the plurality of first training matching object positions and the initial object contour annotated in the training image, and the first training matching object position having a larger similarity value than other similarity values ​​is used as the second training matching object position.

[0145] A plurality of training second matching object positions are input into a temporal discrimination network to obtain training matching object positions.

[0146] The training matching object position output by the temporal discrimination network includes two positions. The matching loss value is obtained by training the matching object position and marking the matching object position through the second loss function.

[0147] The larger labeled matching object position is lost with the larger training matching object position. The smaller labeled matching object position is lost with the smaller training matching object position.

[0148] In this embodiment, the matching loss value is obtained by the cross entropy loss function. The matching loss value is back-propagated to train the time discrimination network, and then the environment detection network is trained. Through the above method, the environment matching network is trained by marking the position of the first matching object, and the environment matching network is trained by back-propagating the time discrimination network.

[0149] According to the moving speed of the environmental object, the moving direction of the environmental object and the distance between the environmental objects, safety decision learning is performed, including:

[0150] According to the moving direction of the environmental objects, the distances between the environmental objects are divided into environmental vertical distance, environmental height distance and environmental horizontal distance.

[0151] Among them, s×cosθ=y represents the vertical distance of the environment, s×sinθ=x represents the horizontal distance of the environment, and s×cosα=z represents the height distance of the environment.

[0152] Among them, s represents the distance between environmental objects.

[0153] According to the moving direction of the environmental object, the moving speed of the environmental object is divided into the environmental vertical speed, the environmental height speed and the environmental horizontal speed.

[0154] Get the vertical speed, altitude speed and horizontal speed of the drone at the current time point.

[0155] The horizontal distance of the environment is added to the horizontal speed of the drone to obtain the horizontal relative speed.

[0156] The vertical relative speed is obtained by adding the horizontal distance of the environment to the vertical speed of the drone.

[0157] Add the environmental height distance to the UAV height speed to obtain the height relative speed.

[0158] If the quotient of the environmental horizontal distance divided by the horizontal relative speed, the quotient of the environmental height distance divided by the height relative speed, and the quotient of the environmental vertical distance divided by the vertical relative speed are equal, it is determined that the UAV will collide, and a stop flight signal is sent to the UAV to control the UAV to stop.

[0159] In this embodiment, because the camera is in front of the drone, the drone's motion direction is the vertical direction, and the environmental pollution's motion direction is the direction relative to the object. The following formula is used to determine whether the drone has collided:

[0160] V1×t1×sinθ=x

[0161] V1×t2×cosθ+V2×t2=y

[0162] V1×t3×cosα+V3×t3=z

[0163] Wherein, t1 represents the length of time of collision in the horizontal direction. t2 represents the length of time of collision in the vertical direction. t3 represents the length of time of collision in the height direction. V1 represents the speed of movement of environmental objects. V2 represents the vertical speed of the drone at the current time. V3 represents the height speed of the drone at the current time. x represents the horizontal distance of the environment, y represents the vertical distance of the environment, and z represents the height distance of the environment. θ represents the angle between the moving direction of the environmental object and the vertical direction, and α represents the angle between the moving direction of the environmental object and the height direction. That is, the angle when the environmental object moves toward the drone.

[0164] If t1 = t2, it is assumed that the drones collide.

[0165] Based on the categories and positions of environmental objects at multiple consecutive time points, a three-dimensional environmental curve is obtained, including:

[0166] Construct a three-dimensional environment matrix. The number of rows of the three-dimensional environment matrix is ​​equal to the width of the image taken by the drone. The number of columns of the three-dimensional environment matrix is ​​equal to the height of the image taken by the drone. The number of pages of the three-dimensional environment matrix is ​​equal to the number of time points at which the image taken by the drone is taken.

[0167] Among them, the values ​​in the three-dimensional environment matrix are all 0 initially.

[0168] According to the center point of the environmental object position and the corresponding time point, the value of the corresponding position in the three-dimensional environmental matrix is ​​set to 1.

[0169] Among them, the number of rows of the center point of the corresponding environmental object in the three-dimensional matrix is ​​obtained according to the horizontal coordinate of the center point of the environmental object position. The number of columns of the corresponding center point of the environmental object in the three-dimensional matrix is ​​obtained according to the vertical coordinate of the center point of the environmental object position. The number of pages of the corresponding center point of the environmental object in the three-dimensional matrix is ​​obtained according to the time point corresponding to the environmental object position. The values ​​of the positions corresponding to the number of rows, columns and pages are set to 1.

[0170] The adjacent positions of the time points that are 1 in the three-dimensional environment matrix are subtracted to obtain multiple three-dimensional environment points.

[0171] The three-dimensional environment point is a set of the number of rows, the number of columns and the number of pages in the three-dimensional environment matrix.

[0172] Construct multiple 3D environment points into a 3D environment point set.

[0173] Optionally, the basic environmental information is obtained based on the drone images taken at multiple consecutive time points, including:

[0174] According to the images taken by the drone, the category and position of the environmental objects are obtained through the target detection network.

[0175] In this embodiment, 64 categories, such as birds, walls, trees and other common objects, are used to train the target detection network. The target detection network used in this embodiment is the yolov5 model.

[0176] According to the environmental object category and the environmental object position, the environmental object contour is obtained through a contour detection network.

[0177] The contour detection network is a contour detection network for detecting objects. In this embodiment, the contour detection network adopts the Canny edge detection algorithm.

[0178] Example 2

[0179] Based on the above-mentioned UAV embodied intelligent safety decision-making method based on mutual feedback learning, an embodiment of the present invention also provides a UAV embodied intelligent safety decision-making system based on mutual feedback learning, the system includes an acquisition module, a basic information acquisition module, a motion judgment module, an environmental object judgment module and a UAV control module.

[0180] The acquisition module is used to obtain drone images taken at multiple consecutive time points. The drone images include multiple environmental objects. The environmental objects represent objects in the images taken by the drone of the surrounding environment.

[0181] The basic information acquisition module is used to obtain multiple basic environmental information based on the drone images taken at multiple consecutive time points. One basic environmental information corresponds to one drone image. The basic environmental information includes the environmental object category, environmental object outline, environmental object position and environmental background outline. The environmental object position represents the center point, width and height of the detected object in the drone image.

[0182] The motion judgment module is used to judge whether an environmental object is stationary or moving according to the position of the environmental object at multiple consecutive time points.

[0183] The environmental object judgment module is used to obtain the environmental object's speed, direction and distance between environmental objects based on the drone's images and basic environmental information at multiple consecutive time points, if the environmental object is moving. The drone control module is used to make safety decisions based on the environmental object's speed, direction and distance between environmental objects. The environmental detection network performs feedback learning through the time discrimination network.

[0184] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0185] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0186] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0187] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0188] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components of the apparatus according to the embodiments of the present invention. The present invention may also be implemented as a device or device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form. It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be constructed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, and third etc. do not indicate any order. These words may be interpreted as names.

Claims

1. A UAV embodied intelligent safety decision-making method based on feedback learning, characterized in that: include: Obtaining drone images taken at multiple consecutive time points; the drone images include multiple environmental objects; The environmental objects refer to objects in the image captured by the drone of the surrounding environment; Based on the drone images taken at multiple consecutive time points, multiple basic environmental information is obtained; One basic environmental information corresponds to one image taken by a drone; the basic environmental information includes the category of environmental objects, the contour of environmental objects, the position of environmental objects and the contour of environmental background; The environmental object position represents the center point, width and height of the detected object in the image taken by the drone; Judging whether an environmental object is stationary or moving according to the position of the environmental object at multiple consecutive time points; The step of judging whether an environmental object is stationary or moving according to the positions of the environmental object at multiple consecutive time points includes: Based on the positions of environmental objects at multiple consecutive time points, a three-dimensional environmental point set is obtained; and multiple three-dimensional environmental point sets are obtained corresponding to multiple environmental objects; Clustering the multiple three-dimensional environmental point sets to obtain curve clusters and cluster center curves; the curve clusters include movement routes of multiple environmental objects; the cluster center curves represent movement routes of environmental objects that are stationary during the movement of the drone in the curve clusters; The environmental objects corresponding to the three-dimensional environmental point set in the curve cluster are set to be stationary; The environmental objects corresponding to the set of three-dimensional environmental points outside the curve cluster are set as motion; If the environmental objects are moving, the speed, direction and distance of the environmental objects are obtained based on the images taken by the drone at multiple consecutive time points and the basic environmental information through the environmental matching network and the time discrimination network; If the environmental object is moving, the moving speed, moving direction and distance between the environmental objects are obtained through the environmental matching network and the time discrimination network according to the images taken by the drone at multiple consecutive time points and the basic environmental information, including: Obtaining an initial object position; the initial object position is the position of an environmental object to be detected; Based on the drone images and the initial object positions at multiple consecutive time points, two matching object positions are obtained by matching through an environment matching network and a time discrimination network; the matching object positions are the positions of the same environment object at other time points except the initial object position; Obtaining a size ratio; the size ratio represents a ratio of an actual distance to a distance in the image; The actual distance is obtained by multiplying the difference between the initial object position and the matching object position by the size ratio; The actual time length is obtained by subtracting the time point corresponding to the position of the matched object from the time point corresponding to the initial object position; According to the positions of two matching objects, the movement direction of the environmental object is obtained; According to the actual distance divided by the actual time length, the movement speed of the environmental object is obtained; Making safety decisions based on the speed of the environmental object, the direction of the environmental object and the distance between the environmental objects; The environment detection network performs feedback learning through the temporal discrimination network; The environment detection network performs feedback learning through the temporal discrimination network, including: The environment detection network includes a first matching convolutional network, a second matching convolutional network and a matching discriminant network; Obtaining a training image, marking an initial object contour, marking a first matching object position, and marking a matching object position; Inputting the training image into an environment detection network to obtain a plurality of training first matching object positions; The first training matching object position and the marked object position are subjected to a first loss function to obtain a position loss value; Train the environment detection network by backpropagating the position loss value; Calculating similarity values ​​between the plurality of first training matching object positions and the initial object contour annotated in the training image, and taking the first training matching object position having a larger similarity value than other similarity values ​​as the second training matching object position; Inputting a plurality of training second matching object positions into a temporal discriminant network to obtain training matching object positions; By training the matching object position and marking the matching object position, a matching loss value is obtained through the second loss function; By backpropagating the matching loss values, we train the temporal discriminator network and then train the environment detection network.

2. The method for embodied intelligent safety decision-making of unmanned aerial vehicles based on mutual feedback learning according to claim 1 is characterized in that: The images taken by the drone at multiple consecutive time points and the initial object positions are matched through the environment matching network and the time discrimination network to obtain two matching object positions, including: Obtaining an initial environmental object category and an initial object contour; the initial environmental object category is the environmental object category corresponding to the initial object position; the initial object contour is the contour of the environmental object corresponding to the initial object position; Construct a detection set by using the contours and positions of multiple environmental objects of the same category as the initial environmental objects in the image taken by the drone; obtain multiple detection sets corresponding to multiple consecutive time points; Inputting the image corresponding to the environmental object position of the detection set and the initial object image into the environmental matching network for matching to obtain a plurality of first matching object positions; The first matching object position is subjected to a contour detection algorithm to obtain a first matching object contour; a plurality of first matching object positions are correspondingly obtained to obtain a plurality of first matching object contours; Calculate similarity values ​​between the multiple first matching object contours and the initial object contour, and use the first matching object position corresponding to the first matching object contour with a greater similarity value as the second matching object position; and obtain multiple second matching object positions corresponding to multiple consecutive time points; According to the time points from far to near, the images corresponding to the second matching object positions in the image taken by the drone are input into the time discrimination network to obtain two matching object positions.

3. The method for embodied intelligent safety decision-making of unmanned aerial vehicles based on mutual feedback learning according to claim 2 is characterized in that: According to the time points from far to near, the images corresponding to the second matching object positions in the image taken by the drone are input into the time discrimination network to obtain two matching object positions, including: A test image and annotated position are obtained; the annotated position is a position of an environmental object corresponding to a time point greater than other time points among a plurality of continuous time points; the test image is one of the images corresponding to a plurality of second matching object positions except the annotated position; and n-1 test images are obtained corresponding to n second matching object positions; Construct a test image set from the images corresponding to the multiple second matching object positions except the test image and the image corresponding to the annotated position; obtain n-2 test image sets corresponding to n second matching object positions; The images in the test image set are sequentially input into the time discrimination network according to the time points from far to near, and the predicted environmental object positions are obtained; the predicted environmental object positions represent the object positions at the time points corresponding to the predicted annotated positions; n-2 test image sets correspond to obtaining n-2 predicted environmental object positions; The test image corresponding to the value where the similarity value between the predicted environmental object position and the marked position is less than the similarity threshold is set as the optimized image; Two second matching object positions corresponding to the optimized images whose sizes are larger than those of the other optimized images among the multiple optimized images are set as the matching object positions.

4. The method for embodied intelligent safety decision-making of unmanned aerial vehicles based on mutual feedback learning according to claim 2 is characterized in that: The step of inputting the image corresponding to the environmental object position of the detection set and the initial object image into the environmental matching network for matching to obtain a plurality of first matching object positions comprises: Inputting the image of the position of the environmental object into a first matching convolutional network, extracting features, and obtaining a first feature; Inputting the initial object image into a second matching convolutional network to extract features to obtain second features; The first feature and the second feature are input into a matching discrimination network to determine whether the two images are similar and obtain a discrimination value; if the discrimination value is greater than a discrimination threshold, the position of the environmental object corresponding to the first feature is set as the first matching object position.

5. The method for embodied intelligent safety decision-making of unmanned aerial vehicles based on mutual feedback learning according to claim 1 is characterized in that: The safety decision learning is performed according to the movement speed of the environmental object, the movement direction of the environmental object and the distance between the environmental objects, including: According to the movement direction of the environmental objects, the distance between the environmental objects is divided into an environmental vertical distance, an environmental height distance and an environmental horizontal distance; According to the moving direction of the environmental object, the moving speed of the environmental object is divided into the environmental vertical speed, the environmental height speed and the environmental horizontal speed; Get the vertical speed, altitude speed and horizontal speed of the drone at the current time point; Add the environmental horizontal distance to the drone's horizontal speed to obtain a horizontal relative speed; Add the vertical distance of the environment to the vertical speed of the drone to obtain the vertical relative speed; Add the environmental height distance to the UAV height speed to obtain the height relative speed; If the quotient of the environmental horizontal distance divided by the horizontal relative speed, the quotient of the environmental height distance divided by the height relative speed, and the quotient of the environmental vertical distance divided by the vertical relative speed are equal, it is determined that the UAV will collide and the UAV is controlled to stop.

6. The method for embodied intelligent safety decision-making of unmanned aerial vehicles based on mutual feedback learning according to claim 1, characterized in that: The three-dimensional environment point set is obtained based on the positions of the environment objects at multiple consecutive time points, including: Construct a three-dimensional environment matrix; the number of rows of the three-dimensional environment matrix is ​​equal to the width of the image taken by the drone; the number of columns of the three-dimensional environment matrix is ​​equal to the height of the image taken by the drone; the number of pages of the three-dimensional environment matrix is ​​equal to the number of time points at which the image taken by the drone is taken; According to the center point and time point of the environmental object position, the value of the corresponding position in the three-dimensional environmental matrix is ​​set to 1; Subtract the adjacent positions of the time points that are 1 in the three-dimensional environment matrix to obtain multiple three-dimensional environment points; Construct multiple 3D environment points into a 3D environment point set.

7. A UAV embodied intelligent safety decision-making system based on mutual feedback learning, characterized in that: include: Acquisition module: obtains drone images taken at multiple consecutive time points; the drone images include multiple environmental objects; the environmental objects represent objects in the images taken by the drone of the surrounding environment; Basic information acquisition module: Based on the drone images taken at multiple consecutive time points, multiple basic environmental information is obtained; One basic environmental information corresponds to one drone-photographed image; the basic environmental information includes environmental object category, environmental object outline, environmental object position and environmental background outline; the environmental object position represents the center point, width and height of the detected object in the drone-photographed image; Motion judgment module: judges whether an environmental object is stationary or moving based on the position of the environmental object at multiple consecutive time points; The step of judging whether an environmental object is stationary or moving according to the positions of the environmental object at multiple consecutive time points includes: Based on the positions of environmental objects at multiple consecutive time points, a three-dimensional environmental point set is obtained; and multiple three-dimensional environmental point sets are obtained corresponding to multiple environmental objects; Clustering the multiple three-dimensional environmental point sets to obtain curve clusters and cluster center curves; the curve clusters include movement routes of multiple environmental objects; the cluster center curves represent movement routes of environmental objects that are stationary during the movement of the drone in the curve clusters; The environmental objects corresponding to the three-dimensional environmental point set in the curve cluster are set to be stationary; The environmental objects corresponding to the set of three-dimensional environmental points outside the curve cluster are set as motion; Environmental object judgment module: If the environmental object is moving, the moving speed and distance of the environmental object are obtained based on the images taken by the drone at multiple consecutive time points and the basic environmental information through the environmental matching network and the time discrimination network; If the environmental object is moving, the moving speed, moving direction and distance between the environmental objects are obtained through the environmental matching network and the time discrimination network according to the images taken by the drone at multiple consecutive time points and the basic environmental information, including: Obtaining an initial object position; the initial object position is the position of an environmental object to be detected; Based on the drone images and the initial object positions at multiple consecutive time points, two matching object positions are obtained by matching through an environment matching network and a time discrimination network; the matching object positions are the positions of the same environment object at other time points except the initial object position; Obtaining a size ratio; the size ratio represents a ratio of an actual distance to a distance in the image; The actual distance is obtained by multiplying the difference between the initial object position and the matching object position by the size ratio; The actual time length is obtained by subtracting the time point corresponding to the position of the matched object from the time point corresponding to the initial object position; According to the positions of two matching objects, the movement direction of the environmental object is obtained; According to the actual distance divided by the actual time length, the movement speed of the environmental object is obtained; UAV control module: Make safety decisions based on the speed of the environmental objects and the distance between them; the environmental detection network performs feedback learning through the time discrimination network; The environment detection network performs feedback learning through the temporal discrimination network, including: The environment detection network includes a first matching convolutional network, a second matching convolutional network and a matching discriminant network; Obtaining a training image, marking an initial object contour, marking a first matching object position, and marking a matching object position; Inputting the training image into an environment detection network to obtain a plurality of training first matching object positions; The first training matching object position and the marked object position are subjected to a first loss function to obtain a position loss value; Train the environment detection network by backpropagating the position loss value; Calculating similarity values ​​between the plurality of first training matching object positions and the initial object contour annotated in the training image, and taking the first training matching object position having a larger similarity value than other similarity values ​​as the second training matching object position; Inputting a plurality of training second matching object positions into a temporal discriminant network to obtain training matching object positions; By training the matching object position and marking the matching object position, a matching loss value is obtained through the second loss function; By backpropagating the matching loss values, we train the temporal discriminator network and then train the environment detection network.

Images