Virtual pipeline generation method and device, electronic equipment and storage medium

Abstract

The invention is suitable for the technical field of artificial intelligence, and provides a virtual pipeline generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an environment sample image set and a virtual pipeline sample image set; building a target deep learning model based on the generative adversarial neural network and the variational automatic coding machine, wherein the target deep learning model comprises a target generator and a target discriminator; based on the environment sample image set and the virtual pipeline sample image set, a first loss function is adopted to train a target discriminator, a second loss function is adopted to train a target generator, a virtual pipeline generation model is obtained, and the second loss function comprises a variable regular term; a first regular term correction value of the variable regular term is obtained based on a training method of an image conversion model, and a second regular term correction value is obtained based on a penalty function method; and obtaining the target environment image and inputting the target environment image into the virtual pipeline generation model to obtain the target virtual pipeline image, so that the generation effect of the virtual pipeline is improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular relates to a virtual pipeline generation method, generation device, electronic device and storage medium. Background Art

[0002] The virtual pipeline is a two-dimensional strip or three-dimensional tubular area similar to a relatively safe flight corridor. It can provide a clear path planning framework for drones and provide a smooth, feasible and safe flight space for drone groups in environments with dense obstacles.

[0003] At present, a machine learning-based image generation network is usually used to generate virtual pipelines. Specifically, the image generation technology is used to conduct supervised learning on the existing virtual pipelines, so that the generation network has the ability to generate virtual pipelines. However, the effect of using this method to generate virtual pipelines depends on the data range of the training set, and its ability to respond to new scenes is poor. After a large amount of training, there may still be a situation where the virtual pipeline generation effect is not good, such as the edges of the generated virtual pipeline are not smooth, the image is discontinuous, and there may be unexpected branches, etc., which may cause accidents when the drone swarm uses the virtual pipeline to fly.

[0004] Therefore, how to improve the generation effect of virtual pipelines has become an urgent problem to be solved. Summary of the invention

[0005] Embodiments of the present application provide a virtual pipeline generation method, a generation device, an electronic device, and a storage medium, aiming to improve the generation effect of the virtual pipeline.

[0006] In a first aspect, an embodiment of the present application provides a virtual pipeline generation method, the method comprising: obtaining an environment sample image set and a virtual pipeline sample image set, the environment sample image set comprising multiple environment sample images, each of the environment sample images comprising an obstacle sample image and a start point and end point sample image, and the virtual pipeline sample image set comprising virtual pipeline sample images corresponding to each of the environment sample images; constructing a target deep learning model based on a generative adversarial neural network and a variational autoencoder, the target deep learning model comprising a target generator and a target discriminator; based on the environment sample image set and the virtual pipeline sample image set, using a first loss function to train the target discriminator, using a second loss function to train the target generator, to obtain a virtual pipeline generation model, the second loss function comprising a variable regularization term, the first regularization term correction value of the variable regularization term being obtained based on a training method of an image conversion model, and the second regularization term correction value being obtained based on a penalty function method; obtaining a target environment image and inputting it into the virtual pipeline generation model to obtain a target virtual pipeline image, the target environment image comprising the start point and end point of a drone flying in a target area, and obstacle information in the target area.

[0007] In one possible implementation, the target generator includes a first downsampling convolution layer, a first dual-block residual network layer, a second downsampling convolution layer, a second dual-block residual network layer, a third downsampling convolution layer, a third dual-block residual network layer, a fourth downsampling convolution layer, a latent vector space, a fourth dual-block residual network layer, a first upsampling convolution layer, a fifth dual-block residual network layer, a second upsampling convolution layer, a sixth dual-block residual network layer, a third upsampling convolution layer, a seventh dual-block residual network layer, a fourth upsampling convolution layer and a convolution output layer connected in sequence; the target discriminator includes a fifth downsampling convolution layer, a sixth downsampling convolution layer, a seventh downsampling convolution layer, an eighth downsampling convolution layer, a ninth downsampling convolution layer and a tenth downsampling convolution layer connected in sequence.

[0008] In a possible implementation, after performing a convolution operation on each downsampling convolution layer or upsampling convolution layer, batch normalization is used for data processing, and a rectified linear unit is used as an excitation function; the convolution kernel of each upsampling convolution layer and each downsampling convolution layer is 4×4, with a step size of 2, and the convolution kernel of each dual-block residual network layer and the convolution output layer is 3×3, with a step size of 1, and a padding of 1.

[0009] In a possible implementation, the target discriminator is trained with a first loss function and the target generator is trained with a second loss function based on the environment sample image set and the virtual pipeline sample image set to obtain a virtual pipeline generation model, including: inputting the environment sample image set into the target generator to obtain a virtual pipeline prediction image set; inputting the virtual pipeline sample image set and the virtual pipeline prediction image set into the target discriminator to obtain a probability prediction value set; fixing the target generator and training the target discriminator with the first loss function based on the environment sample image set, the virtual pipeline sample image set, the virtual pipeline prediction image set and the probability prediction value set; fixing the target discriminator and training the target generator with a second loss function in which a variable regularization term corresponds to a correction value of the first regularization term and a second loss function in which a variable regularization term corresponds to a correction value of the second regularization term in turn based on the environment sample image set, the virtual pipeline sample image set and the virtual pipeline prediction image set; and determining the virtual pipeline generation model based on the trained target discriminator and target generator.

[0010] In a possible implementation, the process of obtaining the first regularization term correction value based on the training method of the image conversion model includes: calculating the L1 loss function based on the virtual pipeline sample image set and the virtual pipeline prediction image set; and determining the first regularization term correction value based on the L1 loss function.

[0011] In a possible implementation, the process of obtaining the second regularization term correction value based on the penalty function method includes: generating an ideal path curve set based on the virtual pipeline sample image set; determining a backbone pipeline set and a block pipeline set based on the ideal path curve set, and establishing a constraint equation based on the backbone pipeline set, the block pipeline set and the virtual pipeline prediction image set; determining a penalty function based on the constraint equation; and determining the second regularization term correction value based on the penalty function.

[0012] In a possible implementation, the obtaining of the environment sample image set and the virtual pipeline sample image set includes: randomly generating a plurality of obstacle sample images and start-end point sample images corresponding to each of the obstacle sample images; determining the environment sample image set based on the plurality of obstacle sample images and the start-end point sample images corresponding to each of the obstacle sample images; and based on the environment sample image set, using a traditional heuristic algorithm with strict mathematical constraints to generate virtual pipeline sample images corresponding to each of the environment sample images, thereby constructing the virtual pipeline sample image set.

[0013] In a second aspect, an embodiment of the present application provides a virtual pipeline generation device, the device comprising: a sample image acquisition module, used to acquire an environment sample image set and a virtual pipeline sample image set, the environment sample image set comprising a plurality of environment sample images, each of the environment sample images comprising an obstacle sample image and a start point and end point sample image, the virtual pipeline sample image set comprising virtual pipeline sample images corresponding to each of the environment sample images; a model construction module, used to construct a target deep learning model based on a generative adversarial neural network and a variational autoencoder, the target deep learning model comprising a target generator and a target discriminator; a model training module, used to construct a target deep learning model based on a generative adversarial neural network and a variational autoencoder, the target deep learning model comprising a target generator and a target discriminator; The target discriminator is trained with a first loss function and the target generator is trained with a second loss function for the environment sample image set and a virtual pipeline generation model is obtained. The second loss function includes a variable regularization term, a first regularization term correction value of the variable regularization term is obtained based on a training method of an image conversion model, and a second regularization term correction value is obtained based on a penalty function method. A virtual pipeline generation module is used to acquire a target environment image and input it into the virtual pipeline generation model to obtain a target virtual pipeline image. The target environment image includes the starting point and end point of the UAV's flight in the target area, and obstacle information in the target area.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in the first aspect or any one of the implementation methods thereof is implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect or any one of the implementation methods thereof is implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the method described in the first aspect or any one of the implementation methods thereof.

[0017] Compared with the prior art, the embodiments of the present application have the following beneficial effects: a target deep learning model is constructed by using a generative adversarial neural network and a variational autoencoder, and the target deep learning model includes a target generator and a target discriminator; based on the acquired set of environmental sample images and the set of virtual pipeline sample images, a first loss function is used to train the target discriminator, and a second loss function is used to train the target generator to obtain a virtual pipeline generation model, wherein the second loss function includes a variable regularization term, a first regularization term correction value in the variable regularization term is obtained based on a training method of an image conversion model, and a second regularization term correction value is obtained based on a penalty function method; and the acquired target environmental image is input into the trained virtual pipeline generation model to obtain a target virtual pipeline image. When training the target generator in the virtual pipeline generation model, different values ​​of the variable regularization term in the second loss function are obtained through a two-stage training method based on an image conversion model and a penalty function method, and the target generator is trained in two stages based on the second loss function using the different regularization term correction values ​​obtained. Among them, the training method based on the image conversion model is used to help the target generator quickly learn the relevant knowledge of the virtual pipeline in the sample data set, and the penalty function method is used to inspire the target generator to learn the specified geometric topological properties. By adding the required geometric topological constraints as penalty functions to the loss function, precise control of the model generation results is achieved, which effectively improves the training effect of the target generator and the accuracy of the virtual pipeline generation model. When the virtual pipeline generation model is used to generate virtual pipelines, the generation effect of the virtual pipeline is improved, and the smoothness, connectivity and branching of the virtual pipeline are significantly improved.

[0018] It can be understood that the virtual pipeline generation device, electronic device, computer-readable storage medium and computer program product provided in the embodiments of the present application have the same beneficial effects as the above-mentioned virtual pipeline generation method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 is a structural schematic diagram of a virtual pipeline; Figure 2 Schematic diagram of a 2D virtual pipeline generated for a machine learning-based image generation network; Figure 3 A flowchart of a method for generating a virtual pipeline provided in one embodiment of the present application; Figure 4 A schematic diagram of the structure of a target generator provided in one embodiment of the present application; Figure 5 A schematic diagram of the structure of a target discriminator provided in one embodiment of the present application; Figure 6 A schematic diagram of the structure of a dual-block residual network layer provided in one embodiment of the present application; Figure 7 A schematic diagram of a backbone pipeline and a block pipeline obtained based on a penalty function heuristic method provided in an embodiment of the present application; Figure 8 A schematic diagram for comparing virtual pipeline generation results provided in an embodiment of the present application; Fig. 9 A structural block diagram of a virtual pipeline generation device provided in one embodiment of the present application; Fig.10 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0021] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0022] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0023] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0024] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0025] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0026] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0027] The virtual pipeline is a two-dimensional strip or three-dimensional tubular area similar to a relatively safe flight corridor. It can provide a clear path planning framework for drones and provide a smooth, feasible and safe flight space for drone groups in environments with dense obstacles.

[0028] Existing virtual pipeline generation technologies can be roughly divided into the following two types: One is the traditional heuristic algorithm with strict mathematical constraints. This method relies on mathematical constraints for numerical solution and can provide an accurate solution. The process is: solve a generating curve through the path planning algorithm, and then solve the pipeline surface that meets the requirements based on the generating curve to obtain a virtual pipeline. Figure 1 shows the basic structure of the virtual pipeline, where the red dotted line part is the generating curve and the blue area is the virtual pipeline.

[0029] Another way is to use a machine learning-based image generation network to generate virtual pipelines. Using image generation technology, supervised learning is performed on existing virtual pipelines, so that the generation network has the ability to generate virtual pipelines.

[0030] The biggest disadvantage of traditional heuristic algorithms with strict mathematical constraints is that they take a long time to compute. Solving such a complex mathematical problem requires a lot of time and computing resources. Table 1 shows the time-consuming comparison of the above two methods, showing that traditional heuristic algorithms require a lot of computing time.

[0031] Table 1 Solution Single scene time Traditional heuristic algorithms with strict mathematical constraints 18.547s Image generation method based on machine learning 0.025s For machine learning-based image generation methods, the generation effect is highly dependent on the data range of the training set and has poor responsiveness to new scenes. After a large amount of training, there may still be problems such as rough edges of the generated virtual pipeline, discontinuous images, and possible unexpected branches, which may cause accidents when the drone swarm uses the virtual pipeline. Figure 2 Schematic diagram of a two-dimensional virtual pipeline generated for a machine learning-based image generation network, where the red part is the generated result and the blue part is the ideal virtual pipeline.

[0032] In order to solve the above technical problems, the present application provides a virtual pipeline generation method, which obtains an environmental sample image set and a virtual pipeline sample image set, the environmental sample image set includes multiple environmental sample images, each environmental sample image includes an obstacle sample image and a start point and end point sample image, and the virtual pipeline sample image set includes virtual pipeline sample images corresponding to each environmental sample image; a target deep learning model is constructed based on an adversarial generative neural network and a variational autoencoder, and the target deep learning model includes a target generator and a target discriminator; based on the environmental sample image set and the virtual pipeline sample image set, a first loss function is used to train the target discriminator, and a second loss function is used to train the target generator to obtain a virtual pipeline generation model, the second loss function includes a variable regularization term, and a first regularization term correction value of the variable regularization term is obtained based on a training method of an image conversion model, and a second regularization term correction value is obtained based on a penalty function method; a target environmental image is obtained and input into the virtual pipeline generation model to obtain a target virtual pipeline image, the target environmental image includes the start point and end point of the drone flying in the target area, and obstacle information in the target area, thereby improving the generation effect of the virtual pipeline.

[0033] A virtual pipeline generation method provided in an embodiment of the present application is applied to generate a virtual pipeline according to an environmental map when performing trajectory planning for a drone cluster, thereby guiding the flight of the drone cluster. The method can be executed by a processor of an electronic device when running a corresponding computer program.

[0034] For ease of understanding, the technical solution of the present application will be described in detail below with reference to the accompanying drawings.

[0035] Figure 3 This is a flow chart of a method for generating a virtual pipeline provided in an embodiment of the present application. For the sake of convenience, only the part related to the present embodiment is shown. The method provided in the present embodiment includes the following steps: S310, obtaining an environment sample image set and a virtual pipeline sample image set, wherein the environment sample image set includes multiple environment sample images, each environment sample image includes an obstacle sample image and a start point and end point sample image, and the virtual pipeline sample image set includes virtual pipeline sample images corresponding to each environment sample image.

[0036] In a specific implementation, environmental sample images are randomly generated based on different environments and scenes, each of which contains corresponding obstacle sample images and start and end point sample images. Then, existing virtual pipeline generation methods such as the traditional heuristic algorithm using strict mathematical constraints or the image generation method based on machine learning are used to generate corresponding virtual pipeline sample images for each environmental sample image, thereby constructing a set of virtual pipeline sample images.

[0037] Preferably, step S310 may optionally include but is not limited to: randomly generating multiple obstacle sample images and start-end point sample images corresponding to each obstacle sample image; determining an environment sample image set based on the multiple obstacle sample images and the start-end point sample images corresponding to each obstacle sample image; based on the environment sample image set, using a traditional heuristic algorithm with strict mathematical constraints to generate virtual pipeline sample images corresponding to each environment sample image, and construct a virtual pipeline sample image set.

[0038] S320, using a generative adversarial neural network and a variational autoencoder to build a target deep learning model, the target deep learning model includes a target generator and a target discriminator.

[0039] Specifically, the Generative Adversarial Network (GAN) includes a generator and a discriminator, and the Variational Auto Encoder (VAE) includes an encoder and a decoder. The generator in VAE is combined with the GAN as the target generator, and the discriminator of GAN is used as the target discriminator of the target deep learning model.

[0040] In one possible implementation, the target generator includes a first downsampling convolution layer, a first dual-block residual network layer, a second downsampling convolution layer, a second dual-block residual network layer, a third downsampling convolution layer, a third dual-block residual network layer, a fourth downsampling convolution layer, a latent vector space, a fourth dual-block residual network layer, a first upsampling convolution layer, a fifth dual-block residual network layer, a second upsampling convolution layer, a sixth dual-block residual network layer, a third upsampling convolution layer, a seventh dual-block residual network layer, a fourth upsampling convolution layer and a convolution output layer connected in sequence; the target discriminator includes a fifth downsampling convolution layer, a sixth downsampling convolution layer, a seventh downsampling convolution layer, an eighth downsampling convolution layer, a ninth downsampling convolution layer and a tenth downsampling convolution layer connected in sequence.

[0041] Furthermore, after performing convolution operations based on each downsampling convolution layer or upsampling convolution layer, batch normalization (BN) is used to normalize the data, and then the rectified linear unit (ReLU) is used as the excitation function. Among them, the upsampling convolution and downsampling convolution use a 4×4 convolution kernel with a step size of 2, and in the residual network block and convolution output layer, a 3×3 convolution kernel with a step size of 1 and a padding of 1 is used.

[0042] As an example, Figure 4 As shown in the figure, the input of the target generator is an obstacle image (Obstacle Map) and a start and end point image (Start & Goal Point), both of which have an image size of 256×256×1. After the first downsampling convolution layer (Conv), BN and ReLU processing, two images of size 128×128×32 are obtained. The two images are connected and processed by the first double-block residual network layer, the second downsampling convolution layer, BN and ReLU to obtain an image of size 64×64×64. After the second double-block residual network layer, the third downsampling convolution layer, BN and ReLU processing, an image of size 32×32×128 is obtained. After the third double-block residual network layer, the fourth downsampling convolution layer, BN and ReLU processing, an image of size 16×16×256 is obtained. After the latent vector space, the fourth double-block residual network layer, the first upsampling convolution layer (Transposed Conv), BN and ReLU to obtain an image of size 32×32×128. After the fifth double-block residual network layer, the second upsampling convolution layer, BN and ReLU, an image of size 64×64×64 is obtained. After the sixth double-block residual network layer, the third upsampling convolution layer, BN and ReLU, an image of size 128×128×32 is obtained. After the seventh double-block residual network layer, the fourth upsampling convolution layer, BN and ReLU, an image of size 256×256×16 is obtained. After the convolution output layer (including the downsampling convolution layer and the hyperbolic tangent function Tanh), a virtual pipeline image (Virtual Tube) of size 256×256×16 is obtained.

[0043] As an example, Figure 5As shown in the figure, the input of the target discriminator includes an obstacle image (ObstacleMap), a start and end point image (Start&Goal Point) and a virtual tube image (Virtual Tube). After the fifth downsampling convolution layer, BN and ReLU processing, three images of size 128×128×64 are obtained. These three images are connected and processed by the sixth downsampling convolution layer, BN and ReLU to obtain an image of size 64×64×128. After the seventh downsampling convolution layer, BN and ReLU processing, an image of size 32×32×256 is obtained. After the eighth downsampling convolution layer, BN and ReLU processing, an image of size 16×16×512 is obtained. After the ninth downsampling convolution layer, BN and ReLU processing, an image of size 8×8×1024 is obtained. After the tenth downsampling convolution layer, BN and ReLU processing, an image of size 1×1×1 is obtained.

[0044] As an example, Figure 6 As shown, the dual-block residual network layer includes two connected residual network layers, and each residual network layer includes a downsampling convolution layer, a BN, and a ReLU connected in sequence.

[0045] S330, based on the environment sample image set and the virtual pipeline sample image set, a first loss function is used to train the target discriminator, and a second loss function is used to train the target generator to obtain a virtual pipeline generation model. The second loss function includes a variable regularization term, and a first regularization term correction value in the variable regularization term is obtained based on the training method of the image conversion model, and a second regularization term correction value is obtained based on the penalty function method.

[0046] Preferably, step S330 may optionally include but is not limited to: inputting the environmental sample image set into the target generator to obtain a virtual pipeline prediction image set; inputting the virtual pipeline sample image set and the virtual pipeline prediction image set into the target discriminator to obtain a probability prediction value set; fixing the target generator, and training the target discriminator with a first loss function based on the environmental sample image set, the virtual pipeline sample image set, the virtual pipeline prediction image set and the probability prediction value set; fixing the target discriminator, and training the target generator with a second loss function in which a variable regularization term corresponds to a correction value of the first regularization term and a second loss function in which a variable regularization term corresponds to a correction value of the second regularization term, respectively, based on the environmental sample image set, the virtual pipeline sample image set and the virtual pipeline prediction image set; and determining the virtual pipeline generation model based on the trained target discriminator and target generator.

[0047] In the specific implementation, for a batch of sample data , the virtual pipeline sample image is , the encoder in the target generator will generate a random latent space vector , and then decoded by the decoder in the target generator to finally obtain the virtual pipeline prediction image , then the target discriminator Receive virtual pipeline sample images respectively And the virtual pipeline prediction image , and try to distinguish the two. During GAN training, we first need to fix the target generator and optimize the target discriminator with the first loss function. The calculation formula of the first loss function is: , Among them, L D represents the first loss function, e represents the obstacle sample image, p represents the start point and end point sample images, r represents the virtual pipeline sample image, represents the virtual pipeline prediction image, D represents the target discriminator, Represents the probability prediction value.

[0048] Then the target discriminator is fixed and the target generator is optimized. In these two stages, the second loss function will be composed of GAN Loss, KL divergence and other loss functions determined by the stage. Among them, GAN Loss is a common loss function for GAN training, which aims to deceive the discriminator as much as possible. It is defined as: , At the same time, the implicit function encoded by the target generator needs to obey the Gaussian distribution as much as possible, so the implicit vector KL divergence. Therefore, the final second loss function is expressed as: , Among them, L G Denotes the second loss function, D KL represents the KL divergence, is a variable regularization term.

[0049] In a possible implementation, a process of obtaining a first regularization term correction value based on an image conversion model training method (also called a Pix2Pix training method) includes: calculating an L1 loss function using an L1 loss function calculation formula based on a virtual pipeline sample image set and a virtual pipeline prediction image set; and determining a first regularization term correction value based on the L1 loss function.

[0050] In the specific implementation, in this stage, a training method similar to Pix2Pix is ​​used to help the target generator quickly learn the relevant knowledge of the virtual pipeline in the virtual pipeline sample image set. and Between The loss function is used as the correction value of the first regular term. The calculation formula of the L1 loss function is: , in, represents the L1 loss function, h represents the height value of the virtual pipeline sample image or the virtual pipeline prediction image, and w represents the width value of the virtual pipeline sample image or the virtual pipeline prediction image; The expression of the first regular term correction value is: , in, represents the correction value of the first regularization term, represents the correction coefficient of the first regularization term.

[0051] Preferably, .

[0052] In one possible implementation, the process of obtaining the second regularization term correction value based on the penalty function method includes: generating an ideal path curve set based on a virtual pipeline sample image set; determining a backbone pipeline set and a block pipeline set based on the ideal path curve set, and establishing a constraint equation based on the backbone pipeline set, the block pipeline set and the virtual pipeline prediction image set; determining a penalty function based on the constraint equation; and determining the second regularization term correction value based on the penalty function.

[0053] In the specific implementation, in this stage, the penalty function method is used to inspire the target generator to learn the specified geometric topological properties. The penalty function method achieves precise control over the model generation results by adding the required geometric topological constraints as penalty functions to the loss function. This method is particularly suitable for situations where specific connectivity and pipeline branching structures need to be maintained. Specifically, the operation process of the penalty function method includes two parts: the generation of the ideal path curve and the establishment of the penalty function.

[0054] Specifically, the ideal path is an expected path generated according to the virtual pipeline, which should avoid obstacles, reach the end point continuously from the starting point, and conform to the shape or trend of the pipeline as much as possible.

[0055] First, in order to generate an ideal path curve, the pipeline graph is converted into a binary image using binary image thinning technology, and then further processed into a skeleton graph. Based on this skeleton graph, the endpoints and coordinate positions of each connected branch are identified through a search algorithm, and the distances between these connected branches are calculated. In this process, if there is a path from the starting point to the connected branch where the target point is located, then the path is considered to be an ideal path curve. If there is no such an ideal path, the connected branch endpoints (which can be considered as leaf nodes) with the closest distance in the search process will be selected for path search to generate an ideal path.

[0056] Furthermore, a penalty function is established. After the ideal path is generated, the path can be "thickened" by using a graphics expansion algorithm or pooling method to obtain two pipelines, including a thinner backbone pipeline. and thicker block pipes , as shown in REF_Ref183615520 \h \* MERGEFORMAT Figure 7, where the blue part is the backbone pipeline B and the dark green part is the block pipeline , the outer area is judged as a branch; the light green part is the backbone pipeline For the generated virtual pipeline prediction image , assuming that the backbone pipeline image, block pipeline image, and generated virtual pipeline prediction image are all located at The connectivity and branchless characteristics of the generated virtual pipeline can be expressed as the following ideal constraint equation: , Among them, B represents the backbone pipeline, R represents the block pipeline, represents the virtual pipeline prediction image; This constraint can be directly weighted to the second loss function, resulting in the following penalty function; , , The final calculation formula for the second regularization term correction value is: , in, represents the correction value of the second regularization term, represents the first penalty function coefficient, Represents the second penalty function coefficient.

[0057] Preferably, , .

[0058] As an example, the first regularization term is corrected to Substitute into the second loss function L G In the example, we use the second loss function L G Optimize the target generator and then correct the value of the second regularization term Substitute into the second loss function L G In this case, the second loss function L is used G Further optimization training is performed on the target generator.

[0059] S340, acquiring a target environment image and inputting it into a virtual pipeline generation model to obtain a target virtual pipeline image, wherein the target environment image includes the starting point and the end point of the UAV's flight in the target area, and obstacle information in the target area.

[0060] Specifically, the target environment image includes a target obstacle image and a target starting point and end point image, wherein the target starting point and end point image is used to display the starting point and end point of the UAV's flight in the target area, and the target obstacle image is used to display the obstacle information in the target area.

[0061] In the specific implementation, when using the virtual pipeline generation model to generate virtual pipelines, it is necessary to process the target start and end point images and the target obstacle images into binary images. The target obstacle image needs to define non-collisionable obstacles. In practice, it can be buildings or no-fly zones. In the example, the obstacle is specified as , the feasible region is For the target start and end images, the start and end points are , other areas are . Input to the target generator After the encoder Receive the specified image and , encoding the expected vector of the latent space Then the decoder Decoding, the final image .

[0062] Exemplarily, a 256×256 grayscale image of an environmental obstacle and a 256×256 grayscale image of a starting point and an end point are received as inputs of a virtual pipeline generation model, and a 256×256 grayscale image of a virtual pipeline is output.

[0063] The technical solution provided by the present application adopts a generative adversarial neural network and a variational autoencoder to construct a target deep learning model, wherein the target deep learning model includes a target generator and a target discriminator; based on the acquired set of environmental sample images and the set of virtual pipeline sample images, a first loss function is used to train the target discriminator, and a second loss function is used to train the target generator to obtain a virtual pipeline generation model, wherein the second loss function includes a variable regularization term, a first regularization term correction value in the variable regularization term is obtained based on a training method of an image conversion model, and a second regularization term correction value is obtained based on a penalty function method; and the acquired target environmental image is input into the trained virtual pipeline generation model to obtain a target virtual pipeline image. When training the target generator in the virtual pipeline generation model, different values ​​of the variable regularization term in the second loss function are obtained through a two-stage training method based on an image conversion model and a penalty function method, and the target generator is trained in two stages based on the second loss function using the different regularization term correction values ​​obtained. Among them, the training method based on the image conversion model is used to help the target generator quickly learn the relevant knowledge of the virtual pipeline in the sample data set, and the penalty function method is used to inspire the target generator to learn the specified geometric topological properties. By adding the required geometric topological constraints as penalty functions to the loss function, precise control of the model generation results is achieved, which effectively improves the training effect of the target generator and the accuracy of the virtual pipeline generation model. When the virtual pipeline generation model is used to generate virtual pipelines, the generation effect of the virtual pipeline is improved, and the smoothness, connectivity and branching of the virtual pipeline are significantly improved.

[0064] The technical solution provided in this application can significantly improve the smoothness, connectivity and branching problems of the pipeline in the actual virtual pipeline generation task. Figure 8 A comparison between the technical solution proposed in this application and the results generated by the existing network is provided, where the first column (MAP) represents the environment image, the second column (Ground Truth) represents the virtual pipeline calculated using the traditional method, the third to seventh columns represent the virtual pipelines generated by different generation networks, which are respectively marked as UNet, SAGAN, VAEGAN, C and C+B in the figure, and the eighth column is the image of the virtual pipeline generated by the method provided by this application (marked as C+B+S in the figure). The black part in the figure is the obstacle, and the blue part is the generated virtual pipeline.

[0065] Fig. 9 This is a structural block diagram of a virtual pipeline generation device provided by an embodiment of the present application. For the sake of convenience, only the part related to the embodiment of the present application is shown. Fig. 9 The virtual pipeline generating device 900 may include a sample image acquiring module 901 , a model building module 902 , a model training module 903 and a virtual pipeline generating module 904 .

[0066] The sample image acquisition module 901 is used to acquire an environmental sample image set and a virtual pipeline sample image set. The environmental sample image set includes multiple environmental sample images, each of which includes an obstacle sample image and a start point and end point sample image. The virtual pipeline sample image set includes virtual pipeline sample images corresponding to each environmental sample image.

[0067] The model building module 902 is used to build a target deep learning model based on the adversarial generative neural network and the variational autoencoder, and the target deep learning model includes a target generator and a target discriminator.

[0068] The model training module 903 is used to train the target discriminator with a first loss function and train the target generator with a second loss function based on the environment sample image set and the virtual pipeline sample image set to obtain a virtual pipeline generation model. The second loss function includes a variable regularization term, and a first regularization term correction value of the variable regularization term is obtained based on the training method of the image conversion model, and a second regularization term correction value is obtained based on a penalty function method.

[0069] The virtual pipeline generation module 904 is used to obtain the target environment image and input it into the virtual pipeline generation model to obtain the target virtual pipeline image. The target environment image includes the starting point and end point of the UAV's flight in the target area, and the obstacle information in the target area.

[0070] A virtual pipeline generation device provided in an embodiment of the present application has the same beneficial effects as the above-mentioned virtual pipeline generation method.

[0071] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0072] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0073] Fig.10 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Fig.10 As shown, the electronic device 10 of this embodiment includes: at least one processor 100 ( Fig.10 Only one is shown in the figure), a memory 101 and a computer program 102 stored in the memory 101 and executable on at least one processor 100, the processor 100 executes the computer program 102 to implement the above Figure 3 The steps in the method embodiment, or the implementation of the above Fig. 9 Functions of each module / unit in the device embodiment.

[0074] The electronic device 10 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device 10 may include but is not limited to a processor 100 and a memory 101. Those skilled in the art will appreciate that Fig.10 This is only an example of the electronic device 10 and does not constitute a limitation on the electronic device 10 . The electronic device 10 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0075] The processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0076] In some embodiments, the memory 101 may be an internal storage unit of the electronic device 10, such as a hard disk or memory of the electronic device 10. In other embodiments, the memory 101 may also be an external storage device of the electronic device 10, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 10. Further, the memory 101 may also include both an internal storage unit of the electronic device 10 and an external storage device. The memory 101 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as program codes of a computer program. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0077] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0078] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to an electronic device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0079] A computer-readable storage medium provided in an embodiment of the present application has the same beneficial effects as the above-mentioned virtual pipeline generation method.

[0080] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0081] A computer program product provided in an embodiment of the present application has the same beneficial effects as the above-mentioned virtual pipeline generation method.

[0082] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0083] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0084] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0085] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0086] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for generating a virtual pipeline, characterized in that: The method comprises: Acquire an environment sample image set and a virtual pipeline sample image set, wherein the environment sample image set includes a plurality of environment sample images, each of the environment sample images includes an obstacle sample image and a start point and end point sample image, and the virtual pipeline sample image set includes virtual pipeline sample images corresponding to each of the environment sample images; Building a target deep learning model based on a generative adversarial neural network and a variational autoencoder, wherein the target deep learning model includes a target generator and a target discriminator; Based on the environment sample image set and the virtual pipeline sample image set, the target discriminator is trained by a first loss function, and the target generator is trained by a second loss function to obtain a virtual pipeline generation model, wherein the second loss function includes a variable regularization term, a first regularization term correction value of the variable regularization term is obtained based on a training method of an image conversion model, and a second regularization term correction value is obtained based on a penalty function method; The target environment image is acquired and input into the virtual pipeline generation model to obtain the target virtual pipeline image, wherein the target environment image includes the starting point and the end point of the UAV flying in the target area, and the obstacle information in the target area.

2. The method according to claim 1, characterized in that The target generator includes a first downsampling convolution layer, a first dual-block residual network layer, a second downsampling convolution layer, a second dual-block residual network layer, a third downsampling convolution layer, a third dual-block residual network layer, a fourth downsampling convolution layer, a latent vector space, a fourth dual-block residual network layer, a first upsampling convolution layer, a fifth dual-block residual network layer, a second upsampling convolution layer, a sixth dual-block residual network layer, a third upsampling convolution layer, a seventh dual-block residual network layer, a fourth upsampling convolution layer and a convolution output layer connected in sequence; The target discriminator includes a fifth down-sampling convolution layer, a sixth down-sampling convolution layer, a seventh down-sampling convolution layer, an eighth down-sampling convolution layer, a ninth down-sampling convolution layer and a tenth down-sampling convolution layer which are connected in sequence.

3. The method according to claim 2, characterized in that After performing convolution operations based on each downsampling convolution layer or upsampling convolution layer, batch normalization is used for data processing, and the rectified linear unit is used as the activation function; The convolution kernel of each upsampling convolution layer and each downsampling convolution layer is 4×4, with a step size of 2, and the convolution kernel of each dual-block residual network layer and the convolution output layer is 3×3, with a step size of 1 and a padding of 1.

4. The method according to claim 1, characterized in that The method of training the target discriminator based on the environment sample image set and the virtual pipeline sample image set by using a first loss function and training the target generator by using a second loss function to obtain a virtual pipeline generation model includes: Inputting the environment sample image set into the target generator to obtain a virtual pipeline prediction image set; Inputting the virtual pipeline sample image set and the virtual pipeline prediction image set into the target discriminator to obtain a probability prediction value set; The target generator is fixed, and based on the environment sample image set, the virtual pipeline sample image set, the virtual pipeline prediction image set and the probability prediction value set, the target discriminator is trained using the first loss function; The target discriminator is fixed, and based on the environment sample image set, the virtual pipeline sample image set and the virtual pipeline prediction image set, the target generator is trained by sequentially using a second loss function in which a variable regularization term is a correction value of the first regularization term and a second loss function in which a variable regularization term is a correction value of the second regularization term; Based on the trained target discriminator and target generator, the virtual pipeline generation model is determined.

5. The method according to claim 4, characterized in that The process of obtaining the first regularization term correction value based on the training method of the image conversion model includes: Calculating an L1 loss function based on the virtual pipeline sample image set and the virtual pipeline prediction image set; The first regularization term correction value is determined based on the L1 loss function.

6. The method according to claim 4, characterized in that The process of obtaining the second regularization term correction value based on the penalty function method includes: generating an ideal path curve set based on the virtual pipeline sample image set; Determine a backbone pipeline set and a block pipeline set based on the ideal path curve set, and establish a constraint equation based on the backbone pipeline set, the block pipeline set and the virtual pipeline prediction image set; determining a penalty function based on the constraint equation; The second regularization term correction value is determined based on the penalty function.

7. The method according to any one of claims 1 to 6, characterized in that: The step of acquiring the environment sample image set and the virtual pipeline sample image set includes: Randomly generate a plurality of obstacle sample images and start point and end point sample images corresponding to each of the obstacle sample images; Determine the environment sample image set based on the plurality of obstacle sample images and the start point and end point sample images respectively corresponding to each of the obstacle sample images; Based on the environmental sample image set, a traditional heuristic algorithm with strict mathematical constraints is used to generate virtual pipeline sample images corresponding to each of the environmental sample images, so as to construct the virtual pipeline sample image set.

8. A virtual pipeline generation device, characterized in that: The device comprises: A sample image acquisition module, used to acquire an environment sample image set and a virtual pipeline sample image set, wherein the environment sample image set includes a plurality of environment sample images, each of which includes an obstacle sample image and a start point and end point sample image, and the virtual pipeline sample image set includes virtual pipeline sample images corresponding to each of the environment sample images; A model building module, used to build a target deep learning model based on a generative adversarial neural network and a variational autoencoder, wherein the target deep learning model includes a target generator and a target discriminator; A model training module, used to train the target discriminator using a first loss function and train the target generator using a second loss function based on the environment sample image set and the virtual pipeline sample image set to obtain a virtual pipeline generation model, wherein the second loss function includes a variable regularization term, a first regularization term correction value of the variable regularization term is obtained based on a training method of an image conversion model, and a second regularization term correction value is obtained based on a penalty function method; The virtual pipeline generation module is used to obtain a target environment image and input it into the virtual pipeline generation model to obtain a target virtual pipeline image. The target environment image includes the starting point and end point of the UAV's flight in the target area, and obstacle information in the target area.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images