Virtual pipeline generation method, generation device, electronic equipment and storage medium

Abstract

The application is suitable for the field of artificial intelligence technology, and provides a virtual pipeline generation method, a generation device, an electronic device and a storage medium. The method comprises the following steps: acquiring an environment sample image set and a virtual pipeline sample image set; constructing a target deep learning model based on an adversarial generation neural network and a variational auto-encoding mechanism, wherein the target deep learning model comprises a target generator and a target discriminator; based on the environment sample image set and the virtual pipeline sample image set, training the target discriminator by using a first loss function, training the target generator by using a second loss function, and obtaining a virtual pipeline generation model, wherein the second loss function comprises a variable regularization term, a first regularization term correction value of the variable regularization term is obtained based on a training method of an image conversion model, and a second regularization term correction value is obtained based on a penalty function method; acquiring a target environment image and inputting the target environment image into the virtual pipeline generation model, and obtaining a target virtual pipeline image, thereby improving the generation effect of the virtual pipeline.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a virtual pipeline generation method and device, electronic equipment and a storage medium. BACKGROUND

[0002] The virtual pipeline is a two-dimensional strip or a three-dimensional tubular region similar to a relatively safe flight corridor, which can provide a clear path planning framework for a UAV, and can also provide a smooth, feasible and safe flight space for a UAV group in an obstacle-dense environment.

[0003] At present, a virtual pipeline is usually generated by using a machine learning-based image generation network, specifically, an image generation technology is used to perform supervised learning on an existing virtual pipeline, so that the generation network has the ability to generate a virtual pipeline. However, the effect of generating a virtual pipeline by using this method depends on the data range of the training set, and the reaction capability of the method to a new scene is poor. After a large amount of training, there may still be a case that the generation effect of the virtual pipeline is poor, for example, the edge of the generated virtual pipeline is not smooth, the image is discontinuous, and there may be unexpected branches, and the like, which may cause an accident of the UAV group when flying by using the virtual pipeline.

[0004] Therefore, how to improve the generation effect of the virtual pipeline has become a problem to be solved. SUMMARY

[0005] Embodiments of the application provide a virtual pipeline generation method and device, electronic equipment and a storage medium, which are aimed at improving the generation effect of the virtual pipeline.

[0006] In a first aspect, an embodiment of the present application provides a virtual pipeline generation method, the method comprising: obtaining an environment sample image set and a virtual pipeline sample image set, the environment sample image set comprising a plurality of environment sample images, each of the environment sample images comprising an obstacle sample image and a start / endpoint sample image, the virtual pipeline sample image set comprising a virtual pipeline sample image corresponding to each of the environment sample images; constructing a target deep learning model based on a generative adversarial network and a variational autoencoder, the target deep learning model comprising a target generator and a target discriminator; training the target discriminator using a first loss function and training the target generator using a second loss function based on the environment sample image set and the virtual pipeline sample image set to obtain a virtual pipeline generation model, the second loss function comprising a variable regularization term, a first regularization term correction value of the variable regularization term being obtained based on a training method of an image conversion model, and a second regularization term correction value being obtained based on a penalty function method; obtaining a target environment image and inputting the target environment image into the virtual pipeline generation model to obtain a target virtual pipeline image, the target environment image comprising a start point and an endpoint of a flight of a UAV in a target region and obstacle information in the target region.

[0007] In a possible implementation, the target generator comprises, in sequence, a first down-sampling convolutional layer, a first double-block residual network layer, a second down-sampling convolutional layer, a second double-block residual network layer, a third down-sampling convolutional layer, a third double-block residual network layer, a fourth down-sampling convolutional layer, a latent vector space, a fourth double-block residual network layer, a first up-sampling convolutional layer, a fifth double-block residual network layer, a second up-sampling convolutional layer, a sixth double-block residual network layer, a third up-sampling convolutional layer, a seventh double-block residual network layer, a fourth up-sampling convolutional layer, and a convolutional output layer; and the target discriminator comprises, in sequence, a fifth down-sampling convolutional layer, a sixth down-sampling convolutional layer, a seventh down-sampling convolutional layer, an eighth down-sampling convolutional layer, a ninth down-sampling convolutional layer, and a tenth down-sampling convolutional layer.

[0008] In a possible implementation, after convolution operation is performed based on each down-sampling convolutional layer or up-sampling convolutional layer, batch normalization is used for data processing, and a rectified linear unit is used as an excitation function; a convolution kernel of each of the up-sampling convolutional layers and each of the down-sampling convolutional layers is 4x4, a step length is 2, a convolution kernel of each of the double-block residual network layers and the convolutional output layer is 3x3, a step length is 1, and padding is 1.

[0009] In a possible implementation, the training of the target discriminator based on the set of environment sample images and the set of virtual pipeline sample images adopts a first loss function, and the training of the target generator adopts a second loss function, to obtain a virtual pipeline generation model, including: inputting the set of environment sample images into the target generator to obtain a set of virtual pipeline prediction images; inputting the set of virtual pipeline sample images and the set of virtual pipeline prediction images into the target discriminator to obtain a set of probability prediction values; fixing the target generator, and based on the set of environment sample images, the set of virtual pipeline sample images, the set of virtual pipeline prediction images and the set of probability prediction values, training the target discriminator by using the first loss function; fixing the target discriminator, and based on the set of environment sample images, the set of virtual pipeline sample images and the set of virtual pipeline prediction images, training the target generator by using, in sequence, a second loss function in which a variable regular term is a second regular term correction value, and a second loss function in which a variable regular term is a first regular term correction value; and determining the virtual pipeline generation model based on the trained target discriminator and target generator.

[0010] In a possible implementation, the process of obtaining the first regular term correction value based on the training of the image conversion model includes: calculating an L1 loss function based on the set of virtual pipeline sample images and the set of virtual pipeline prediction images; and determining the first regular term correction value based on the L1 loss function.

[0011] In a possible implementation, the process of obtaining the second regular term correction value based on the penalty function method includes: generating a set of ideal path curves based on the set of virtual pipeline sample images; determining a set of backbone pipelines and a set of block pipelines based on the set of ideal path curves, and establishing a constraint equation based on the set of backbone pipelines, the set of block pipelines and the set of virtual pipeline prediction images; determining a penalty function based on the constraint equation; and determining the second regular term correction value based on the penalty function.

[0012] In a possible implementation, the obtaining of the set of environment sample images and the set of virtual pipeline sample images includes: randomly generating a plurality of obstacle sample images and a plurality of start-end sample images corresponding to the plurality of obstacle sample images respectively; determining the set of environment sample images based on the plurality of obstacle sample images and the plurality of start-end sample images corresponding to the plurality of obstacle sample images respectively; and based on the set of environment sample images, generating a virtual pipeline sample image corresponding to each environment sample image by using a traditional heuristic algorithm with strict mathematical constraints, to construct the set of virtual pipeline sample images.

[0013] In a second aspect, an embodiment of the present application provides a virtual pipeline generation apparatus, the apparatus comprising: a sample image acquisition module configured to acquire an environment sample image set and a virtual pipeline sample image set, the environment sample image set comprising a plurality of environment sample images, each of the environment sample images comprising an obstacle sample image and a start / endpoint sample image, the virtual pipeline sample image set comprising a virtual pipeline sample image corresponding to each of the environment sample images; a model construction module configured to construct a target deep learning model based on a generative adversarial network and a variational autoencoder, the target deep learning model comprising a target generator and a target discriminator; a model training module configured to train the target discriminator using a first loss function and train the target generator using a second loss function based on the environment sample image set and the virtual pipeline sample image set, to obtain a virtual pipeline generation model, the second loss function comprising a variable regularization term, a first regularization term correction value of the variable regularization term being obtained based on a training method of an image conversion model, and a second regularization term correction value being obtained based on a penalty function method; and a virtual pipeline generation module configured to acquire a target environment image and input the target environment image into the virtual pipeline generation model to obtain a target virtual pipeline image, the target environment image comprising a start point and an endpoint of a flight of a UAV in a target region and obstacle information in the target region.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the method according to the first aspect or any one of the implementation manners thereof when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement the method according to the first aspect or any one of the implementation manners thereof.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising a computer program, and the computer program is executed by a processor to implement the steps of the method according to the first aspect or any one of the implementation manners thereof.

[0017] Compared with the prior art, the embodiment of the present application has the beneficial effects that: an adversarial generative neural network and a variational autoencoder are used to build a target deep learning model, the target deep learning model includes a target generator and a target discriminator; based on the obtained environment sample image set and the virtual pipe sample image set, the target discriminator is trained by using a first loss function, and the target generator is trained by using a second loss function, to obtain a virtual pipe generation model, wherein the second loss function includes a variable regularization term, a first regularization term correction value in the variable regularization term is obtained based on a training method of an image conversion model, and a second regularization term correction value is obtained based on a penalty function method; and then the obtained target environment image is input into the trained virtual pipe generation model to obtain a target virtual pipe image. When training the target generator in the virtual pipe generation model, different values of the variable regularization term in the second loss function are obtained by using a two-stage training method based on the training method of the image conversion model and the penalty function method, and the obtained different regularization term correction values are used to train the target generator in two stages based on the second loss function, wherein the training method based on the image conversion model is used to help the target generator quickly learn the related knowledge of the virtual pipe in the sample data set, and the penalty function method is used to inspire the target generator to learn the specified geometric topological properties, so as to realize accurate control of the model generation result by adding the required geometric topological constraints as a penalty function into the loss function, effectively improve the training effect of the target generator, improve the accuracy of the virtual pipe generation model, and improve the generation effect of the virtual pipe when the virtual pipe generation model is used to generate the virtual pipe, and significantly improve the smoothness, connectivity and branch generation of the virtual pipe.

[0018] It can be understood that the virtual pipe generation device, the electronic equipment, the computer readable storage medium and the computer program product provided by the embodiment of the present application have the same beneficial effects as the virtual pipe generation method described above, and will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creating laborious work.

[0020] Figure 1 is a structural schematic diagram of a virtual pipe;

[0021] Figure 2 is a schematic diagram of a two-dimensional virtual pipe generated by a machine learning based image generation network;

[0022] Figure 3A flowchart of a virtual pipeline generation method provided by an embodiment of the present application is shown in FIG. 1.

[0023] Figure 4 A structural diagram of a target generator provided by an embodiment of the present application is shown in FIG. 2.

[0024] Figure 5 A structural diagram of a target discriminator provided by an embodiment of the present application is shown in FIG. 3.

[0025] Figure 6 A structural diagram of a double-block residual network layer provided by an embodiment of the present application is shown in FIG. 4.

[0026] Figure 7 A diagram of a backbone pipeline and a block pipeline obtained based on a penalty function heuristic provided by an embodiment of the present application is shown in FIG. 5.

[0027] Figure 8 A comparison diagram of virtual pipeline generation results provided by an embodiment of the present application is shown in FIG. 6.

[0028] Figure 9 A structural block diagram of a virtual pipeline generation apparatus provided by an embodiment of the present application is shown in FIG. 7.

[0029] Figure 10 A structural diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 8. DETAILED DESCRIPTION

[0030] In the following description, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, persons skilled in the art will understand that the present application can be practiced without these specific details. In other instances, well-known structures, devices, circuits, and methods have not been described in detail in order to avoid obscuring the description of the present application.

[0031] It should be understood that the term "comprises" as used in the specification and in the claims indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0032] It should also be understood that the term "and / or" as used herein refers to any one of the associated listed items, combinations thereof, and all possible combinations thereof, and includes these combinations.

[0033] As used in the specification and the appended claims herein, the term “if’ can be interpreted as meaning “when” or “upon” or “in response to determining” or “in response to detecting” depending on the context. Similarly, the phrase “if it is determined” or “if [the described condition or event] is detected” can be interpreted as meaning “upon determining” or “in response to determining” or “upon detecting [the described condition or event]” or “in response to detecting [the described condition or event]” depending on the context.

[0034] In addition, in the description of the present application and the appended claims, the terms “first”, “second”, “third”, etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0035] In the present application, the reference “one embodiment” or “some embodiments” and the like means that the specific features, structures or characteristics described in connection with the embodiment are included in one or more embodiments of the present application. Therefore, the statements “in one embodiment”, “in some embodiments”, “in other some embodiments”, “in further some embodiments” and the like appearing in different places in the specification are not necessarily all referring to the same embodiment, but mean “one or more but not all embodiments”, unless otherwise specifically emphasized. The terms “include”, “contain”, “have” and their variants mean “include but not limited to”, unless otherwise specifically emphasized.

[0036] Virtual pipeline is a two-dimensional bar or three-dimensional tubular region similar to a relatively safe flight corridor. It can provide a clear path planning framework for UAVs and also provide smooth, feasible and safe flight space for UAV groups in obstacle-dense environments.

[0037] Existing virtual pipeline generation techniques can be roughly divided into the following two types:

[0038] One is a traditional heuristic algorithm with strict mathematical constraints. This method relies on numerical solution based on mathematical constraints and can provide accurate solutions. The process is as follows: a generating curve is solved by a path planning algorithm, and then a pipeline surface that meets the requirements is solved according to the generating curve, thereby obtaining a virtual pipeline. Figure 1 shows the basic structure of a virtual pipeline, in which the red dashed part is a generating curve and the blue area is a virtual pipeline.

[0039] The other way is to use a machine learning-based image generation network to generate a virtual pipeline. By using image generation technology, existing virtual pipelines are supervised to learn, so that the generation network has the ability to generate virtual pipelines.

[0040] The biggest disadvantage of the traditional heuristic algorithm with strict mathematical constraints is the long calculation time. It takes a lot of time and computing resources to solve such a complex mathematical problem. Table 1 shows the calculation time comparison of the above two methods, which shows that the traditional heuristic algorithm needs a lot of calculation time.

[0041] Table 1

[0042] Solving method Single scene time consumption Traditional heuristic algorithm with strict mathematical constraints 18.547s Machine learning based image generation method 0.025s

[0043] For the image generation method based on machine learning, the generation effect is highly dependent on the data range of the training set, and the reaction ability for new scenes is poor. After a large amount of training, there may still be problems such as rough edges of the generated virtual pipeline, discontinuous images, and possible unexpected branches, which may cause unexpected flight of the UAV group when using the virtual pipeline. Figure 2 Schematic diagram of a two-dimensional virtual pipeline generated by a machine learning-based image generation network, where the red part is the generation result and the blue part is the ideal virtual pipeline.

[0044] To solve the above technical problems, the present application provides a virtual pipeline generation method, which acquires an environment sample image set and a virtual pipeline sample image set, the environment sample image set includes multiple environment sample images, each environment sample image includes an obstacle sample image and a start and end sample image, and the virtual pipeline sample image set includes a virtual pipeline sample image corresponding to each environment sample image; a target deep learning model is constructed based on a generative adversarial network and a variational autoencoder, the target deep learning model includes a target generator and a target discriminator; based on the environment sample image set and the virtual pipeline sample image set, the target discriminator is trained using a first loss function, and the target generator is trained using a second loss function, to obtain a virtual pipeline generation model, the second loss function includes a variable regularization term, a first regularization term correction value of the variable regularization term is obtained based on a training method of an image conversion model, and a second regularization term correction value is obtained based on a penalty function method; a target environment image is acquired and input into the virtual pipeline generation model to obtain a target virtual pipeline image, the target environment image includes a start point and an end point of a UAV flying in a target area, and obstacle information in the target area, and the generation effect of the virtual pipeline is improved.

[0045] The virtual pipeline generation method provided by the embodiments of the present application can be executed by a processor of an electronic device when running a corresponding computer program in the scenario of generating a virtual pipeline according to an environment map to guide the flight of a UAV cluster.

[0046] For ease of understanding, the technical solutions of the present application will be described in detail below with reference to the drawings.

[0047] Figure 3 A flowchart of a virtual pipeline generation method provided by an embodiment of the present application is shown. For ease of illustration, only the parts related to the embodiment are shown. The method provided by the embodiment includes the following steps:

[0048] S310, obtain an environment sample image set and a virtual pipeline sample image set, the environment sample image set including multiple environment sample images, each environment sample image including an obstacle sample image and a start / endpoint sample image, and the virtual pipeline sample image set including a virtual pipeline sample image corresponding to each environment sample image.

[0049] In a specific implementation, environment sample images are randomly generated based on different environments and scenarios, each environment sample image containing a corresponding obstacle sample image and start / endpoint sample image, and an existing virtual pipeline generation method such as the traditional heuristic algorithm using strict mathematical constraints or the image generation method based on machine learning is used to generate a corresponding virtual pipeline sample image for each environment sample image to construct the virtual pipeline sample image set.

[0050] Preferably, step S310 can optionally but not limited to include the following steps: randomly generating multiple obstacle sample images and start / endpoint sample images corresponding to each obstacle sample image; determining the environment sample image set based on the multiple obstacle sample images and the start / endpoint sample images corresponding to each obstacle sample image; and generating a virtual pipeline sample image corresponding to each environment sample image based on the environment sample image set using the traditional heuristic algorithm with strict mathematical constraints to construct the virtual pipeline sample image set.

[0051] S320, construct a target deep learning model using a generative adversarial network and a variational auto encoder, the target deep learning model including a target generator and a target discriminator.

[0052] Specifically, the generative adversarial network (GAN) includes a generator and a discriminator, and the variational auto encoder (VAE) includes an encoder and a decoder. The target generator is obtained by combining the generator in the VAE and the generator in the GAN, and the target discriminator of the target deep learning model is obtained by using the discriminator in the GAN.

[0053] In a possible implementation, the target generator comprises a first down-sampling convolutional layer, a first double-block residual network layer, a second down-sampling convolutional layer, a second double-block residual network layer, a third down-sampling convolutional layer, a third double-block residual network layer, a fourth down-sampling convolutional layer, a latent vector space, a fourth double-block residual network layer, a first up-sampling convolutional layer, a fifth double-block residual network layer, a second up-sampling convolutional layer, a sixth double-block residual network layer, a third up-sampling convolutional layer, a seventh double-block residual network layer, a fourth up-sampling convolutional layer, and a convolutional output layer, which are sequentially connected; and the target discriminator comprises a fifth down-sampling convolutional layer, a sixth down-sampling convolutional layer, a seventh down-sampling convolutional layer, an eighth down-sampling convolutional layer, a ninth down-sampling convolutional layer, and a tenth down-sampling convolutional layer, which are sequentially connected.

[0054] Further, after the convolution operation based on each down-sampling convolutional layer or up-sampling convolutional layer, batch normalization (BN) is used to normalize the data, and then a rectified linear unit (ReLU) is used as an activation function. In the up-sampling convolutional layer and the down-sampling convolutional layer, a 4*4 convolution kernel is used, and the step is 2; in the residual network block and the convolutional output layer, a 3*3 convolution kernel is used, the step is 1, and the padding is 1.

[0055] As an example, as shown in FIG. 2, the target generator comprises a first down-sampling convolutional layer, a first double-block residual network layer, a second down-sampling convolutional layer, a second double-block residual network layer, a third down-sampling convolutional layer, a third double-block residual network layer, a fourth down-sampling convolutional layer, a latent vector space, a fourth double-block residual network layer, a first up-sampling convolutional layer, a fifth double-block residual network layer, a second up-sampling convolutional layer, a sixth double-block residual network layer, a third up-sampling convolutional layer, a seventh double-block residual network layer, a fourth up-sampling convolutional layer, and a convolutional output layer, which are sequentially connected. Figure 4As shown, the target generator takes an obstacle map and a start & goal point image as input, both 256×256×1. After processing through a first downsampling convolutional layer (Conv), Batch Normalization (BN), and ReLU, two images of size 128×128×32 are obtained. These two images are then concatenated and processed through a first double-block residual network layer, a second downsampling convolutional layer, BN, and ReLU to obtain a 64×64×64 image. After processing through a second double-block residual network layer, a third downsampling convolutional layer, BN, and ReLU, a 32×32×128 image is obtained. After processing through a third double-block residual network layer, a fourth downsampling convolutional layer, BN, and ReLU, a 16×16×256 image is obtained. Finally, after processing through the latent vector space, a fourth double-block residual network layer, and a first upsampling convolutional layer (Transposed... After processing with Conv, BN, and ReLU, an image of size 32×32×128 is obtained. After processing with the fifth double-block residual network layer, the second upsampling convolutional layer, BN, and ReLU, an image of size 64×64×64 is obtained. After processing with the sixth double-block residual network layer, the third upsampling convolutional layer, BN, and ReLU, an image of size 128×128×32 is obtained. After processing with the seventh double-block residual network layer, the fourth upsampling convolutional layer, BN, and ReLU, an image of size 256×256×16 is obtained. After passing through the convolutional output layer (including downsampling convolutional layer and hyperbolic tangent function Tanh), a virtual tube image of size 256×256×16 is obtained.

[0056] As an example, such as Figure 5 As shown, the input to the target discriminator includes an obstacle map, a start and goal point map, and a virtual tube map. After processing by the fifth downsampling convolutional layer, batch normalization (BN), and ReLU, three images of size 128×128×64 are obtained. These three images are concatenated and processed by the sixth downsampling convolutional layer, batch normalization (BN), and ReLU to obtain an image of size 64×64×128. After processing by the seventh downsampling convolutional layer, batch normalization (BN), and ReLU, an image of size 32×32×256 is obtained. After processing by the eighth downsampling convolutional layer, batch normalization (BN), and ReLU, an image of size 16×16×512 is obtained. After processing by the ninth downsampling convolutional layer, batch normalization (BN), and ReLU, an image of size 8×8×1024 is obtained. After processing by the tenth downsampling convolutional layer, batch normalization (BN), and ReLU, an image of size 1×1×1 is obtained.

[0057] As an example, as shown in Figure 6 The double-block residual network layer includes two connected residual network layers, each of which includes a down-sampling convolution layer, a BN, and a ReLU connected in sequence.

[0058] S330, based on the environment sample image set and the virtual pipeline sample image set, training the target discriminator using a first loss function and training the target generator using a second loss function to obtain a virtual pipeline generation model, the second loss function including a variable regularization term, a first regularization term correction value in the variable regularization term being obtained based on a training method of the image conversion model, and a second regularization term correction value being obtained based on a penalty function method.

[0059] Preferably, step S330, optionally but not limited to, includes: inputting the environment sample image set into the target generator to obtain a virtual pipeline predicted image set; inputting the virtual pipeline sample image set and the virtual pipeline predicted image set into the target discriminator to obtain a probability prediction value set; fixing the target generator, and based on the environment sample image set, the virtual pipeline sample image set, the virtual pipeline predicted image set, and the probability prediction value set, training the target discriminator using the first loss function; fixing the target discriminator, and based on the environment sample image set, the virtual pipeline sample image set, and the virtual pipeline predicted image set, training the target generator using the second loss function with the variable regularization term being the first regularization term correction value and the second loss function with the variable regularization term being the second regularization term correction value in sequence; and determining the virtual pipeline generation model based on the trained target discriminator and the target generator.

[0060] In a specific implementation, for a batch of sample data , the virtual pipeline sample image is , the encoder in the target generator will generate a random hidden space vector , which is then decoded by the decoder in the target generator to obtain a virtual pipeline predicted image , and then the target discriminator receives the virtual pipeline sample image and the virtual pipeline predicted image , respectively, and attempts to distinguish between the two. In the GAN training process, the target generator needs to be fixed first, and the target discriminator is optimized using the first loss function. The calculation formula of the first loss function is:

[0061] ,

[0062] where L D represents the first loss function, e represents the obstacle sample image, p represents the start-end sample image, and r represents the virtual pipeline sample image. D represents a target discriminator, represents a probability prediction value.

[0063] Subsequently, the target discriminator is fixed, and the target generator is optimized. In the two stages, the second loss function is composed of a GAN Loss, a KL divergence, and other loss functions determined by the stage. Among them, the GAN Loss is a loss function commonly used in GAN training, which aims to deceive the discriminator as much as possible. It is defined as:

[0064] ,

[0065] At the same time, the implicit function encoded by the target generator needs to comply with the Gaussian distribution as much as possible, so the KL divergence of the implicit vector also needs to be considered. Therefore, the final second loss function is represented as:

[0066] ,

[0067] wherein, L G represents the second loss function, D KL represents the KL divergence, is a variable regularization term.

[0068] In one possible implementation, the process of obtaining the first regularization term correction value based on the training method of the image conversion model (also referred to as the Pix2Pix training method) includes: based on the virtual pipeline sample image set and the virtual pipeline prediction image set, using an L1 loss function calculation formula to calculate an L1 loss function; determining the first regularization term correction value based on the L1 loss function.

[0069] In a specific implementation, in this stage, a training method similar to Pix2Pix is used to help the target generator quickly learn the relevant knowledge of the virtual pipeline in the virtual pipeline sample image set, and the loss function between is used as the first regularization term correction value, and the calculation formula of the L1 loss function is:

[0070] ,

[0071] wherein, represents the L1 loss function, h represents a height value of the virtual pipeline sample image or the virtual pipeline prediction image, and w represents a width value of the virtual pipeline sample image or the virtual pipeline prediction image;

[0072] The expression of the first regularization term correction value is

[0073] ,

[0074] wherein,​ represents a first regular term correction value, represents a first regular term correction coefficient.

[0075] Preferably, .

[0076] In one possible implementation, the process of obtaining the second regular term correction value based on the penalty function method includes: generating a set of ideal path curves based on the set of virtual pipe sample images; determining a set of backbone pipes and a set of block pipes based on the set of ideal path curves, and establishing a constraint equation based on the set of backbone pipes, the set of block pipes, and the set of virtual pipe prediction images; determining a penalty function based on the constraint equation; and determining the second regular term correction value based on the penalty function.

[0077] In a specific implementation, in this stage, the penalty function method is used to enable the heuristic target generator to learn the specified geometric topological properties. The penalty function method achieves precise control over the model generation results by adding the required geometric topological constraints as a penalty function to the loss function. This method is particularly suitable for cases where specific connectivity and pipe branching structures need to be maintained. Specifically, the operation process of the penalty function method includes two parts: generation of ideal path curves and establishment of a penalty function.

[0078] Specifically, the ideal path is an expected path generated according to the virtual pipe. It should avoid obstacles, can be continuously reached from the starting point to the end point, and as far as possible conform to the pipe shape or trend.

[0079] First, in order to generate ideal path curves, a binary image thinning technique is used to convert the pipe graph into a binary image and further process it into a skeleton graph. Based on this skeleton graph, the end points of each connected branch and their coordinate positions are identified through a search algorithm, and the distances between these connected branches are calculated. In this process, if there is a path from the starting point to the connected branch where the target point is located, then this path is considered as the ideal path curve. If there is no such ideal path, the end points of the connected branches (which can be considered as leaf nodes) that are closest in distance during the search process are selected to perform path search, in order to generate an ideal path.

[0080] Further, the penalty function is established. After generating the ideal path, the path can be thickened to obtain two pipes by using the dilation algorithm or the pooling method of graphics, including a thinner backbone pipe and a thicker block pipe , as shown in FIG. 7, where the blue part is the backbone pipe B, the dark green part is the block pipe , and the outer area is determined as a branch; and the light green part is the backbone pipe The fractured section. For the generated virtual pipe prediction image. Assuming that the backbone pipeline image, block pipeline image, and generated virtual pipeline prediction image are all located at each pixel in Given a binary image between two points, the connectivity and branchless properties of the generated virtual pipeline can be expressed by the following equation, which represents the ideal constraint equation:

[0081] ,

[0082] Where B represents the backbone pipeline and R represents the block pipeline. Represents a virtual pipeline prediction image;

[0083] This constraint can be directly weighted and penalized onto the second loss function, resulting in the following penalty function;

[0084] ,

[0085] ,

[0086] The final formula for calculating the second regularization term correction value is:

[0087] ,

[0088] in, This represents the adjustment value of the second regularization term. Denotes the coefficients of the first penalty function. This represents the coefficient of the second penalty function.

[0089] Preferred, , .

[0090] As an example, modify the value of the first regularization term. Substitute into the second loss function L G In the middle, the second loss function L is used G The target generator is optimized and trained, and then the value of the second regularization term is adjusted. Substitute into the second loss function L G In this case, the second loss function L is used. G Further optimize and train the target generator.

[0091] S340: Acquire the target environment image and input it into the virtual pipeline generation model to obtain the target virtual pipeline image. The target environment image includes the start and end points of the UAV's flight in the target area, as well as obstacle information in the target area.

[0092] Specifically, the target environment image includes a target obstacle image and a target start and end image, wherein the target start and end image is used to display the start and end points of the UAV flying in the target area, and the target obstacle image is used to display the obstacle information in the target area.

[0093] In a specific implementation, when the virtual pipeline generation model is used for virtual pipeline generation, the target start and end image and the target obstacle image need to be processed into binary images. The target obstacle image needs to define non-collidable obstacles. In practice, it can be a building or a no-fly zone. In the target obstacle image , the specified obstacle is , the feasible region is , and for the target start and end image, the start and end points are , and the other regions are . After being input into the target generator , the encoder receives the specified image and , encodes to generate the expected vector of the hidden space , and then is decoded by the decoder , and finally the obtained image .

[0094] For example, a 256x256 size environment obstacle grayscale image and a 256x256 size start and end grayscale image are received as input of the virtual pipeline generation model, and a 256x256 size virtual pipeline grayscale image is output.

[0095] The technical scheme provided in the application adopts a generative adversarial network and a variational autoencoder to construct a target deep learning model, the target deep learning model comprising a target generator and a target discriminator; based on an obtained environment sample image set and a virtual pipeline sample image set, the target discriminator is trained using a first loss function, and the target generator is trained using a second loss function, to obtain a virtual pipeline generation model, wherein the second loss function comprises a variable regularization term, a first regularization term correction value in the variable regularization term is obtained based on a training method of an image conversion model, and a second regularization term correction value is obtained based on a penalty function method; and then an obtained target environment image is input into the trained virtual pipeline generation model to obtain a target virtual pipeline image. When the target generator in the virtual pipeline generation model is trained, different values of the variable regularization term in the second loss function are obtained through a two-stage training method based on the training method of the image conversion model and the penalty function method, and the obtained different regularization term correction values are used to train the target generator in two stages based on the second loss function, wherein the training method based on the image conversion model is used to help the target generator quickly learn relevant knowledge of the virtual pipeline in the sample data set, and the penalty function method is used to inspire the target generator to learn specified geometric topological properties, so as to realize accurate control of the generation result of the model by adding the required geometric topological constraints as a penalty function into the loss function, effectively improve the training effect of the target generator, improve the accuracy of the virtual pipeline generation model, and improve the generation effect of the virtual pipeline when the virtual pipeline generation model is used to generate the virtual pipeline, and significantly improve the smoothness, connectivity and branch generation of the virtual pipeline.

[0096] The technical scheme provided in the application can significantly improve the smoothness, connectivity and branch generation of the pipeline in the actual virtual pipeline generation task. Figure 8 The technical scheme provided in the application is compared with the existing network generation result, wherein the first column (MAP) represents an environment image, the second column (Ground Truth) represents a virtual pipeline calculated using a traditional method, the third column to the seventh column represent virtual pipelines generated by different generation networks, and the virtual pipelines are respectively denoted as UNet, SAGAN, VAEGAN, C and C+B in the figure. The eighth column is an image of a virtual pipeline generated using the method provided in the application (denoted as C+B+S in the figure). The black part in the figure is an obstacle, and the blue part is the generated virtual pipeline.

[0097] Figure 9 A structural block diagram of a virtual pipeline generation device provided in an embodiment of the application is shown in the figure. For the convenience of description, only parts related to the embodiments of the application are shown. Figure 9 The virtual pipeline generation device 900 can comprise a sample image acquisition module 901, a model construction module 902, a model training module 903 and a virtual pipeline generation module 904.

[0098] The sample image acquisition module 901 is configured to acquire an environment sample image set and a virtual pipeline sample image set, the environment sample image set including a plurality of environment sample images, each environment sample image including an obstacle sample image and a start / endpoint sample image, and the virtual pipeline sample image set including a virtual pipeline sample image corresponding to each environment sample image.

[0099] The model construction module 902 is configured to construct a target deep learning model based on a generative adversarial network and a variational autoencoder, the target deep learning model including a target generator and a target discriminator.

[0100] The model training module 903 is configured to train the target discriminator based on the environment sample image set and the virtual pipeline sample image set using a first loss function, train the target generator using a second loss function, and obtain a virtual pipeline generation model, the second loss function including a variable regularization term, a first regularization term correction value of the variable regularization term being obtained based on a training method of an image conversion model, and a second regularization term correction value being obtained based on a penalty function method.

[0101] The virtual pipeline generation module 904 is configured to acquire a target environment image and input the target environment image into the virtual pipeline generation model to obtain a target virtual pipeline image, the target environment image including a start point and an endpoint of flight of a UAV in a target region and obstacle information in the target region.

[0102] The virtual pipeline generation device provided by the embodiment of the present application has the same beneficial effects as the virtual pipeline generation method.

[0103] It should be noted that the information interaction and execution process between the above-described devices / modules are based on the same concept as the method embodiments of the present application, and the specific functions and technical effects brought by the same can be referred to the method embodiment part, which will not be described here.

[0104] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0105] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 10 As shown, the electronic device 10 of this embodiment includes: at least one processor 100 ( Figure 10 Only one is shown in the diagram), memory 101, and computer program 102 stored in memory 101 and executable on at least one processor 100, wherein the processor 100 executes computer program 102 to implement the above-described... Figure 3 The steps in the method embodiments, or the implementation of the above Figure 9 The functions of each module / unit in the device embodiment.

[0106] Electronic device 10 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. This electronic device 10 may include, but is not limited to, a processor 100 and a memory 101. Those skilled in the art will understand that... Figure 10 This is merely an example of electronic device 10 and does not constitute a limitation on electronic device 10. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0107] The processor 100 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0108] The memory 101 can be an internal storage unit of the electronic device 10, such as a hard disk or a memory of the electronic device 10 in some embodiments. The memory 101 can also be an external storage device of the electronic device 10, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 10 in some other embodiments. Further, the memory 101 can include both the internal storage unit and the external storage device of the electronic device 10. The memory 101 is used to store an operating system, an application program, a boot loader, data, and other programs, such as program codes of computer programs, etc. The memory 101 can also be used to temporarily store data that has been output or will be output.

[0109] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps in the above-mentioned various method embodiments.

[0110] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can be instructed by a computer program to relevant hardware for completion, and the computer program can be stored in a computer readable storage medium. The computer program can be executed by a processor to implement the steps of each method embodiment. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying the computer program code to an electronic device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc.

[0111] The computer readable storage medium provided by the embodiment of the present application has the same beneficial effects as the virtual pipeline generation method.

[0112] The computer program product provided by the embodiment of the present application has the same beneficial effects as the virtual pipeline generation method.

[0113] The computer program product provided by the embodiment of the present application has the same beneficial effects as the virtual pipeline generation method.

[0114] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in a certain embodiment can be referred to the related description of other embodiments.

[0115] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0116] In the embodiments provided by the present application, it should be understood that the disclosed apparatuses / devices and methods can be implemented in other manners. For example, the embodiments of the apparatus / device described above are merely illustrative. For example, the division of the modules or units is merely logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0117] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0118] The above embodiments are merely used to describe the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent ones; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for virtual pipe generation, characterized by, The method comprises: acquiring an environment sample image set and a virtual pipeline sample image set, the environment sample image set comprising a plurality of environment sample images, each of the environment sample images comprising an obstacle sample image and a start and end sample image, and the virtual pipeline sample image set comprising a virtual pipeline sample image corresponding to each of the environment sample images; constructing a target deep learning model based on a generative adversarial network and a variational autoencoder, the target deep learning model comprising a target generator and a target discriminator; training the target discriminator using a first loss function and the target generator using a second loss function based on the environment sample image set and the virtual pipeline sample image set to obtain a virtual pipeline generation model, the second loss function comprising a variable regularization term, a first regularization term correction value of the variable regularization term being obtained based on a training method of an image conversion model and a second regularization term correction value being obtained based on a penalty function method; acquiring a target environment image and inputting the target environment image into the virtual pipeline generation model to obtain a target virtual pipeline image, the target environment image comprising a start point and an end point of a flight of a UAV in a target region and obstacle information in the target region; the training of the target generator using the second loss function comprises: inputting the environment sample image set into the target generator to obtain a virtual pipeline prediction image set; and training the target generator using a second loss function corresponding to a first regularization term correction value of a variable regularization term and a second loss function corresponding to a second regularization term correction value of the variable regularization term in sequence based on the environment sample image set, the virtual pipeline sample image set and the virtual pipeline prediction image set; the first regularization term correction value is determined based on an L1 loss function between the virtual pipeline prediction image set and the virtual pipeline sample image set; the penalty function method adds a required geometric topology constraint as a penalty function into the second loss function.

2. The method of claim 1, wherein, the target generator comprises, in sequence, a first down-sampling convolutional layer, a first double-block residual network layer, a second down-sampling convolutional layer, a second double-block residual network layer, a third down-sampling convolutional layer, a third double-block residual network layer, a fourth down-sampling convolutional layer, a latent vector space, a fourth double-block residual network layer, a first up-sampling convolutional layer, a fifth double-block residual network layer, a second up-sampling convolutional layer, a sixth double-block residual network layer, a third up-sampling convolutional layer, a seventh double-block residual network layer, a fourth up-sampling convolutional layer and a convolutional output layer; the target discriminator comprises, in sequence, a fifth down-sampling convolutional layer, a sixth down-sampling convolutional layer, a seventh down-sampling convolutional layer, an eighth down-sampling convolutional layer, a ninth down-sampling convolutional layer and a tenth down-sampling convolutional layer.

3. The method of claim 2, wherein, After convolution operation based on each down-sampling convolutional layer or up-sampling convolutional layer, batch normalization is used for data processing, and a rectified linear unit is used as an excitation function; a convolution kernel of each of the up-sampling convolutional layers and the down-sampling convolutional layers is 4x4, a step length is 2, a convolution kernel of each of the double-block residual network layers and the convolutional output layer is 3x3, a step length is 1 and padding is 1.

4. The method of claim 1, wherein, The first loss function is used to train the target discriminator, and the second loss function is used to train the target generator based on the environment sample image set and the virtual pipeline sample image set, to obtain a virtual pipeline generation model, including: The virtual pipeline sample image set and the virtual pipeline prediction image set are input into the target discriminator to obtain a probability prediction value set; The target generator is fixed, and the target discriminator is trained based on the environment sample image set, the virtual pipeline sample image set, the virtual pipeline prediction image set and the probability prediction value set by using the first loss function; The virtual pipeline generation model is determined based on the trained target discriminator and the target generator.

5. The method of claim 4, wherein, The process of obtaining the second regular term correction value based on the penalty function method includes: An ideal path curve set is generated based on the virtual pipeline sample image set; A backbone pipeline set and a block pipeline set are determined based on the ideal path curve set, and a constraint equation is established based on the backbone pipeline set, the block pipeline set and the virtual pipeline prediction image set; The penalty function is determined based on the constraint equation; The second regular term correction value is determined based on the penalty function.

6. The method according to any one of claims 1 to 5, characterized in that, The environment sample image set and the virtual pipeline sample image set are obtained, including: A plurality of obstacle sample images and start-end sample images corresponding to each of the obstacle sample images are randomly generated; The environment sample image set is determined based on the plurality of obstacle sample images and the start-end sample images corresponding to each of the obstacle sample images; Based on the environment sample image set, a virtual pipeline sample image corresponding to each of the environment sample images is generated using a traditional heuristic algorithm with strict mathematical constraints, and the virtual pipeline sample image set is constructed.

7. A virtual pipeline generation apparatus characterized by comprising: The device for implementing the method of any one of claims 1 to 6 includes: A sample image acquisition module for acquiring an environment sample image set and a virtual pipeline sample image set, the environment sample image set including a plurality of environment sample images, each of the environment sample images including an obstacle sample image and a start-end sample image, and the virtual pipeline sample image set including a virtual pipeline sample image corresponding to each of the environment sample images; A model construction module for constructing a target deep learning model based on a generative adversarial network and a variational autoencoder, the target deep learning model including a target generator and a target discriminator; A model training module for training the target discriminator by using a first loss function and training the target generator by using a second loss function based on the environment sample image set and the virtual pipeline sample image set, to obtain a virtual pipeline generation model, the second loss function including a variable regular term, a first regular term correction value of the variable regular term being obtained based on an image conversion model training method, and a second regular term correction value being obtained based on a penalty function method. A virtual pipeline generation module is configured to acquire a target environment image and input the target environment image into the virtual pipeline generation model to obtain a target virtual pipeline image, the target environment image including a starting point and an ending point of flight of a UAV in a target region and obstacle information in the target region.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program, when executed by the processor, implements the method of any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program, when executed by the processor, implements the method of any one of claims 1 to 6.

Images