Method, device and equipment for generating tree image growth rules based on deep learning

Abstract

The present application discloses a method, device and equipment for generating growth rules of tree images based on deep learning, which relates to the field of plant growth prediction technology and aims to solve the technical problem that the prior art cannot automatically generate L-system sequences. The method comprises: obtaining an image of the tree to be predicted; based on a convolutional neural network module, extracting and encoding features of the image of the tree to be predicted to obtain convolutional coding feature data; inputting the convolutional coding feature data and the first target sequence of L-system into a decoder module so that the decoder module outputs the probability distribution of the next L-system character; based on the probability distribution of the next L-system character, obtaining the second target sequence of L-system; based on the second target sequence of L-system, obtaining the growth rules of the tree to be predicted. Based on the method described in the present application, the growth and morphology of trees can be simulated quickly and accurately, and complex tree structures can be better handled.

Description

Technical Field

[0001] The present application relates to the technical field of plant growth prediction, and in particular to methods, devices and equipment for generating tree image growth rules based on deep learning. Background Art

[0002] At present, the method of generating L-system sequences is mainly based on manual design and optimization. This method requires a lot of manual participation, is time-consuming and labor-intensive, and the effect is not ideal. It is difficult to adapt to the complex and changeable natural environment.

[0003] Therefore, how to automatically generate L-system sequences to better simulate the growth and morphology of trees is an important research issue at present. Summary of the invention

[0004] The present application provides a method, device and equipment for generating tree image growth rules based on deep learning, aiming to solve the technical problem that the prior art cannot automatically generate L-system sequences.

[0005] To solve the above technical problems, the embodiment of the present application provides: a method for generating tree image growth rules based on deep learning, comprising the following steps:

[0006] Get the image of the tree to be predicted;

[0007] Based on the convolutional neural network module, feature extraction and encoding are performed on the image of the tree to be predicted to obtain convolutional coding feature data;

[0008] Inputting the convolutional coding feature data and the L-system first target sequence into a decoder module so that the decoder module outputs a probability distribution of a next L-system character; and obtaining an L-system second target sequence based on the probability distribution of the next L-system character;

[0009] Based on the L-system second target sequence, the growth rule of the tree to be predicted is obtained.

[0010] As some optional implementations of the present application, the convolutional neural network module is obtained by training based on the following training sample set:

[0011] Get multiple tree images;

[0012] After unifying the sizes of the tree images, annotating the L-system sequence corresponding to each type of tree in the tree images to obtain a plurality of first tree images; wherein the L-system sequence refers to sequence information used to describe the growth process and morphological generation of the trees;

[0013] The pixel values ​​of the plurality of first tree images are normalized to obtain a training sample set.

[0014] As some optional implementations of the present application, the convolutional neural network module is trained by the following steps:

[0015] Inputting the training sample set into the Xception structure in the convolutional neural network module, so that multiple channels in the Xception structure respectively perform depth-separable convolution processing on the training sample set to obtain multiple depth-separable convolution features;

[0016] Connect and reshape a plurality of the depth-separable convolutional features so that the convolutional neural network module outputs convolutional encoded feature data.

[0017] As some optional implementations of the present application, the step of connecting and reshaping the multiple depth-separable convolutional features so that the convolutional neural network module outputs convolutional encoding feature data includes:

[0018] Connecting multiple depth-separable convolutional features to merge multiple depth-separable convolutional features; and obtaining a feature map through a convolutional layer, a BatchNormal layer, and a Relu layer; wherein the convolutional layer includes 256 convolution kernels, and the convolution kernel size is 3*3;

[0019] After flattening the last two dimensions of the feature map, convolutional coding feature data is obtained.

[0020] As some optional implementations of the present application, the step of inputting the convolutional coding feature data and the L-system first target sequence into a decoder module so that the decoder module outputs a probability distribution of the next L-system character includes:

[0021] Perform word embedding and position embedding on the first target sequence of the L-system to obtain the feature matrix of the sequence;

[0022] Inputting the feature matrix of the sequence into the Masked Multi-Head Attention layer to obtain a self-attention feature; adding the feature matrix of the sequence and the self-attention feature, performing normalization processing, and obtaining a first output feature;

[0023] Inputting the first output feature and the convolutional coding feature data into the Multi-Head Attention layer to obtain features of the encoder-decoder attention layer; adding the first output feature and the features of the encoder-decoder attention layer, and performing normalization processing to obtain a second output feature;

[0024] Inputting the second output feature into the position-wise Feed Forward layer to obtain a feedforward neural network feature; adding the second output feature and the feedforward neural network feature, and performing normalization processing to obtain a third output feature;

[0025] Based on the third output feature, obtaining a fourth output feature;

[0026] The fourth output feature is input into a preset linear layer and a preset softmax layer to obtain the probability distribution of the next L-system character.

[0027] As some optional implementation manners of the present application, the step of obtaining the fourth output feature based on the third output feature includes:

[0028] The third output feature is used as the initial feature matrix and sequentially input into the Masked Multi-Head Attention layer, the Multi-Head Attention layer, and the position-wise Feed Forward layer, and the input value and the output value of each layer are added and normalized to obtain the first output feature;

[0029] The first output features are then used as the initial feature matrix and sequentially input into the Masked Multi-HeadAttention layer, the Multi-Head Attention layer, and the position-wise Feed Forward layer, and the input value and output value of each layer are added and normalized to obtain the second output features;

[0030] As some optional implementations of the present application, the step of obtaining the L-system second target sequence based on the probability distribution of the next L-system character includes: based on the probability distribution of the next L-system character, selecting the character with the largest probability value as the predicted character, and adding the predicted character to the L-system first target sequence; repeating this step until the predicted character is <eos>, obtain the second target sequence of L-system;

[0031] The step of obtaining the growth rule of the tree to be predicted based on the L-system second target sequence includes: visualizing the L-system second target sequence to simulate the growth rule of the tree to be predicted.

[0032] On the other hand, in order to solve the above technical problems, the embodiment of the present application further provides: a device for generating tree image growth rules based on deep learning, comprising:

[0033] An acquisition module, used for acquiring images of trees to be predicted;

[0034] A feature encoding module, used for extracting and encoding features of the image of the tree to be predicted based on a convolutional neural network module to obtain convolutional encoding feature data;

[0035] A decoding module, configured to input the convolutional coding feature data and the L-system first target sequence into a decoder module, so that the decoder module outputs a probability distribution of a next L-system character; and obtain a L-system second target sequence based on the probability distribution of the next L-system character;

[0036] The prediction module is used to obtain the growth rule of the tree to be predicted based on the L-system second target sequence.

[0037] As some optional implementations of the present application, the decoding module includes an Xception structure, and the Xception structure includes at least four channels with the same number of convolution kernels, a BatchNormal layer and a Relu layer;

[0038] Each of the channels is a depthwise separable convolution branch, and the expansion rate of each of the depthwise separable convolution branches increases exponentially. The size of each convolution kernel in the last three depthwise separable convolution branches is the same and is three times the size of each convolution kernel in the first depthwise separable convolution branch.

[0039] The BatchNormal layer is used to normalize the output value of the depth-separable convolution branch;

[0040] The Relu layer is an activation function.

[0041] On the other hand, to solve the above-mentioned technical problems, an embodiment of the present application further provides: a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned method for generating tree image growth rules based on deep learning.

[0042] Compared with the prior art, the method for generating tree image growth rules based on deep learning described in the present application, after obtaining the image of the tree to be predicted, encodes its features based on the convolutional neural network module, and then decodes the encoded features based on the decoder module to obtain the probability distribution of the next L-system character, and then selects the target sequence of the L-system based on the probability distribution, and further obtains the growth rules of the tree to be predicted based on the target sequence of the L-system. It can be seen that compared with the prior art, the advantage of the present application is that the L-system sequence can be automatically generated, the growth and morphology of trees can be simulated more quickly and accurately, and the workload of manual participation is greatly reduced. At the same time, due to the use of convolutional neural networks and decoders, complex tree structures can be better handled. For example, some tree species may have complex branching patterns, such as pine and oak trees, whose branches may be distributed in all directions to form complex three-dimensional structures and other complex and changeable natural environment conditions, thereby improving the generation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the specific implementation of the present application or the technical solution in the prior art, the following is a brief introduction to the drawings required for the specific implementation or the prior art description. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, each element or part is not necessarily drawn according to the actual scale.

[0044] Figure 1 A flowchart of a method for generating tree image growth rules based on deep learning according to an embodiment of the present application;

[0045] Figure 2 This is a schematic diagram of the first tree image in the embodiment of the present application;

[0046] Figure 3 A schematic diagram of the L-system sequence labeled in the embodiments of the present application;

[0047] Figure 4 A schematic diagram of the training process of the convolutional neural network module described in the embodiment of the present application;

[0048] Figure 5 This is a schematic diagram of the workflow of the decoder module described in the embodiment of the present application;

[0049] Figure 6 This is a schematic diagram of the second target sequence of the L-system generated after the first tree image is processed by the convolutional neural network module and the decoder module in the embodiment of the present application;

[0050] Figure 7 This is a schematic diagram of the growth morphology simulation described in the embodiment of the present application;

[0051] Figure 8 A schematic diagram of a generalized process of generating an L-system sequence of tree images as described in an embodiment of the present application;

[0052] Fig. 9 This is a functional module diagram of the device for generating tree image growth rules based on deep learning as described in an embodiment of the present application.

[0053] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] L-system is a mathematical model used to describe complex patterns in nature. It was originally proposed by mathematician Alan Turing in 1952 to simulate the growth process of plants. Traditional methods for generating L-system sequences mainly rely on manual design and optimization. This method requires designers to have deep knowledge of mathematics and biology in order to be able to understand and apply the complex rules of the L-system. In addition, this method also requires a lot of time and effort, because designers need to repeatedly experiment and adjust parameters to generate L-system sequences that meet specific needs. However, due to the complexity and variability of the natural environment, the L-system sequences generated by this method are often difficult to accurately simulate real plant growth conditions.

[0056] With the development of computer technology, the method of automatically generating L-system sequences has begun to attract attention. This method can automatically learn and optimize the rules of L-system by using machine learning and artificial intelligence technology, thereby generating L-system sequences that better meet actual needs. This method can not only greatly reduce manual participation and improve generation efficiency, but also better adapt to complex and changeable natural environments through learning from large amounts of data.

[0057] However, there are also some challenges in the method of automatically generating L-system sequences. First, how to design an effective learning algorithm to extract useful information from a large amount of data is an important issue. Second, how to deal with the complexity and nonlinearity of the L-system in order to generate high-quality L-system sequences is also a problem that needs to be solved. Finally, how to evaluate the quality of the generated L-system sequence in order to effectively optimize and improve it is also a key issue.

[0058] Based on this, Figure 1 As shown, the embodiment of the present application provides: a method for generating tree image growth rules based on deep learning, comprising the following steps:

[0059] Step S10: Acquire an image of a tree to be predicted.

[0060] In actual application scenarios, the image of the tree to be predicted refers to the tree whose growth rules need to be predicted, such as various types of trees such as pine trees, oak trees, evergreen trees and deciduous trees; the image is in RGB format. In order to improve the efficiency of feature extraction and encoding in the subsequent network module, the image can be uniformly resized to (256, 256) before input. In order to further speed up the efficiency of feature extraction and encoding in the image by the subsequent network module, the pixels of the uniformly resized image can also be normalized to obtain the image of the tree to be predicted.

[0061] Step S20: Based on the convolutional neural network module, feature extraction and encoding are performed on the image of the tree to be predicted to obtain convolutional coding feature data.

[0062] In actual application scenarios, the convolutional neural network module is trained based on the following training sample set:

[0063] Get multiple tree images;

[0064] After unifying the sizes of the tree images, annotating the L-system sequence corresponding to each type of tree in the tree images to obtain a plurality of first tree images; wherein the L-system sequence refers to sequence information used to describe the growth process and morphological generation of the trees;

[0065] The pixel values ​​of the plurality of first tree images are normalized to obtain a training sample set.

[0066] The training sample set is described below with reference to an implementation case. The training sample set may include 10,000 tree images. After the tree images are resized and labeled, a first tree image is obtained. Figure 2 As shown, the L-system sequence marked is as follows Figure 3 As shown; among them, Figure 3 The meanings of the letters are: <pad>Represents padding; <eos>Represents the end mark; <bos>Represents the start mark; F means moving forward a certain distance and drawing a line segment; + means turning left a certain angle; - means turning right a certain angle; [ means starting a branch; ] means ending a branch. The above 8 marks are used as tokens of the decoder.

[0067] It should be noted that the above token: <pad>The function of is to fill the sequence. During the model training process, since each batch of data needs to have the same shape for parallel computing, it is necessary to fill the sequences of different lengths to the same length. This requires the use of <pad>This token is used to fill in.

[0068] It should be noted that the above token refers to the input unit of the decoder. During the decoding process, a starting token is first input: <bos>, and then processed by the self-attention layer and the encoder-decoder attention layer, and then output by the forward layer to get the representation of this token. Next, through a linear layer plus a softmax operation, predict which word in the output dictionary. After completing this step, the word will be put into the decoder input, and the above operation will be repeated until it is decoded to <token> : <eos>So far, the output of the sentence is completed. Therefore, the "8 identifiers as decoder tokens" here refer to the 8 different tags or states that the decoder needs to recognize and process during the processing.

[0069] Specifically, the convolutional neural network module is trained through the following steps: inputting the training sample set into the Xception structure in the convolutional neural network module, so that multiple channels in the Xception structure respectively perform depth-separable convolution processing on the training sample set to obtain multiple depth-separable convolution features; connecting and reshaping multiple depth-separable convolution features so that the convolutional neural network module outputs convolutional encoded feature data.

[0070] In actual application scenarios, the convolutional neural network module includes an Xception structure, which includes at least four channels with the same number of convolution kernels, a BatchNormal layer and a Relu layer; each of the channels is a depth-separable convolution branch, and the expansion rate of each depth-separable convolution branch increases exponentially. The size of each convolution kernel in the last three depth-separable convolution branches is the same and is three times the size of each convolution kernel in the first depth-separable convolution branch; the BatchNormal layer is used to normalize the output value of the depth-separable convolution branch; the Relu layer is an activation function.

[0071] More specifically, the step of connecting and reshaping the multiple depth-separable convolutional features so that the convolutional neural network module outputs convolutional coding feature data includes: connecting the multiple depth-separable convolutional features to merge the multiple depth-separable convolutional features; and obtaining a feature map through a convolutional layer, a BatchNormal layer, and a Relu layer; wherein the convolutional layer includes 256 convolution kernels, and the convolution kernel size is 3*3; after flattening the back two dimensions of the feature map, convolutional coding feature data is obtained. The BatchNormal layer is used to normalize the multiple depth-separable convolutional features, and the Relu layer includes an activation function.

[0072] For example, when the training sample set is input into the Xception structure in the convolutional neural network module to obtain a feature map, Figure 4 As shown, the following steps may be included:

[0073] Step S21, input feature dimension is (N, 3, 256, 256); wherein N is the number of samples, 3 represents the three channels of RGB, and the third 256 and the fourth 256 represent the length and height of the image respectively.

[0074] Step S22: The output features of step S21 are passed through four branches to obtain four feature maps, and the expression ability of the model is improved by capturing the context information of the image at different scales;

[0075] Among them, the first branch is the depth-wise separable convolution branch, which includes 256 convolution kernels, each of which is 1*1 in size, and is normalized using BatchNormal, with Relu as the activation function;

[0076] The second branch is the depthwise separable convolution branch, which includes 256 convolution kernels. Each convolution kernel is 3*3 in size, with a step size of 1 and a dilation rate of 6. BatchNormal is used for normalization, and the activation function is Relu.

[0077] The third branch is the depthwise separable convolution branch, which includes 256 convolution kernels. Each convolution kernel is 3*3 in size, with a step size of 1 and a dilation rate of 12. BatchNormal is used for normalization, and the activation function is Relu.

[0078] The fourth branch is the depthwise separable convolution branch, which includes 256 convolution kernels. Each convolution kernel is 3*3 in size, with a step size of 1 and a dilation rate of 18. BatchNormal is used for normalization, and the activation function is Relu.

[0079] Step S23: perform concat operation on the encoded features of the four branches in step S22, merge the features of the four branches, and pass them through the convolution layer (256 convolution kernels, the convolution kernel size is 3*3), BatchNormal layer, and Relu layer to obtain a feature map with a feature dimension of (N, 256, 16, 16).

[0080] Step S24: Reshape the feature map of step S23 (flatten the last two dimensions of the feature map) to obtain the output features of the convolution module, and the output feature dimension is (N, 256, 256).

[0081] Step S30, inputting the convolutional coding feature data and the L-system first target sequence into a decoder module, so that the decoder module outputs the probability distribution of the next L-system character; based on the probability distribution of the next L-system character, obtaining the L-system second target sequence.

[0082] It should be noted that the input of the decoder module consists of two parts: one is the encoding feature of the convolutional neural network module; the other is an offset of the L-system sequence. For example, the target sequence is " <bos>F[+F] <eos>", when the decoder predicts "[", its input sequence is " <bos>F”.

[0083] Specifically, Figure 5 As shown, the step of inputting the convolutional coding feature data and the L-system first target sequence into the decoder module so that the decoder module outputs the probability distribution of the next L-system character includes:

[0084] Step S31: perform word embedding and position embedding on the first target sequence of the L-system to obtain a feature matrix of the sequence.

[0085] In the Figure 5 In the decoder of the Xception model, "outputs (shifted right)" adds a start symbol to the front of the output sequence and shifts the rest of the output one position to the right. The purpose of this is to facilitate the prediction of the first token and the end of the prediction process. Specifically, suppose we have a normal output sequence whose position relationship is 0-"I", 1-"Love", 2-"China". After the shifted right operation, we added the start symbol to the initial output, which is equivalent to shifting the output right by one position. So the new output sequence becomes 0-start symbol, 1-"I", 2-"Love", 3-"China". In this way, the decoder can refer to the previously generated words when generating the current word, so as to better capture the contextual information in the sequence. Output is the output feature.

[0086] In the Figure 5 In the example, "add&layer norm" is an operation performed on the processed features. Here, "add" refers to the unit addition of two feature maps, while "Layer Normalization" operates on the different features of each sample to achieve scaling and offset adjustments to these features, thereby ensuring that the statistical characteristics of each sample remain stable.

[0087] In the Figure 5 In the code, "linear" stands for Linear Transformation, which is a preset linear layer. Specifically, it is a method of mapping the output of a nonlinear activation function to a new feature space, so as to better adapt to subsequent tasks. This operation is mainly achieved through a fully connected layer, in which each neuron is connected to all neurons in the previous layer, which can capture richer feature information. Therefore, linear transformation plays a vital role in neural networks, which helps to improve the performance and generalization ability of the model.

[0088] Step S32: input the feature matrix of the sequence into the Masked Multi-Head Attention layer to obtain a self-attention feature; add the feature matrix of the sequence and the self-attention feature, perform normalization processing, and obtain a first output feature.

[0089] It should be noted that in the Masked Multi-Head Attention layer, the decoder performs self-attention calculations on the input sequence to obtain a new representation of each position. In order to prevent the current position from obtaining information about future positions, the self-attention layer uses a masking mechanism to prevent the current position from paying attention to the position behind it. The normalization processing involved in steps S32-S35 is based on Layer Normalization.

[0090] Step S33: input the first output feature and the convolutional coding feature data into the Multi-HeadAttention layer to obtain the features of the encoder-decoder attention layer; add the first output feature and the features of the encoder-decoder attention layer, and normalize them to obtain the second output feature.

[0091] It should be noted that when the first output feature and the convolutional coding feature data are input into the Multi-Head Attention layer, the coding features of the convolutional neural network module are used as keys and values, and the first output feature is used as a query condition.

[0092] Step S34, input the second output feature to the position-wise Feed Forward layer to obtain a feedforward neural network feature; add the second output feature and the feedforward neural network feature, and obtain a third output feature after normalization.

[0093] Step S35: Obtain a fourth output feature based on the third output feature.

[0094] It should be noted that the step of obtaining the fourth output feature based on the third output feature includes: inputting the third output feature as the initial feature matrix into the Masked Multi-Head Attention layer, the Multi-Head Attention layer, and the position-wise Feed Forward layer in sequence, and adding the input value of each layer to the output value, and performing normalization processing to obtain the first output feature; then inputting the first output feature as the initial feature matrix into the Masked Multi-Head Attention layer, the Multi-Head Attention layer, and the position-wise Feed Forward layer in sequence, and adding the input value of each layer to the output value, and performing normalization processing to obtain the second output feature; after looping the above steps for a preset number of times, taking the last output feature as the fourth output feature.

[0095] That is, the third output feature is used as input, and the fourth output feature is obtained after repeating the above steps S32 to S34 N times, wherein N is a positive integer for the preset number of cycles, such as 6 times.

[0096] Step S36: input the fourth output feature into a preset linear layer and a preset softmax layer to obtain the probability distribution of the next L-system character.

[0097] When the convolutional neural network module and the decoder module are trained as described above, the Adam gradient descent algorithm can be used to calculate the cross entropy loss function. Through the back propagation of the loss function, the parameters of the preset network model are optimized and adjusted until the network model converges and the trained network model is output.

[0098] After obtaining the probability distribution of the next L-system character based on the above steps, a step of obtaining the second target sequence of the L-system can be performed based on the probability distribution of the next L-system character, that is, based on the probability distribution of the next L-system character, selecting the character with the largest probability value as the predicted character, and adding the predicted character to the first target sequence of the L-system; repeating this step until the predicted character is <eos>, obtain the second target sequence of L-system. In practical application, such as Figure 2 After the tree image shown in the figure is processed by the convolutional neural network module and the decoder module, the generated L-system second target sequence is as follows Figure 6 shown.

[0099] Step S40: Based on the L-system second target sequence, the growth rule of the tree to be predicted is obtained.

[0100] In the actual application scenario, the step of obtaining the growth rule of the tree to be predicted based on the L-system second target sequence includes: visualizing the L-system second target sequence and using it to simulate the growth rule of the tree to be predicted. For example, the description of the L-system second target sequence is visualized, and the growth form of the plant is modeled by an engine such as blender to obtain a growth form simulation diagram such as Figure 7 shown.

[0101] It can be seen that Figure 8 As shown, the present application proposes a method for generating an L-system sequence of tree images based on a convolutional neural network and a Transformer model Decoder. The method can automatically generate an L-system sequence, thereby better simulating the growth and morphology of trees. Compared with the prior art, the advantage of the present invention is that it can automatically generate an L-system sequence, simulate the growth and morphology of trees more quickly and accurately, and greatly reduce the workload of manual participation. At the same time, due to the use of a convolutional neural network and a Transformer Decoder, it can better handle complex tree structures, thereby improving the generation effect.

[0102] On the other hand, Fig. 9 As shown, in order to solve the above technical problems, the embodiment of the present application further provides: a device for generating tree image growth rules based on deep learning, comprising:

[0103] An acquisition module, used for acquiring images of trees to be predicted;

[0104] A feature encoding module, used for extracting and encoding features of the image of the tree to be predicted based on a convolutional neural network module to obtain convolutional encoding feature data;

[0105] A decoding module, configured to input the convolutional coding feature data and the L-system first target sequence into a decoder module, so that the decoder module outputs a probability distribution of a next L-system character; and obtain a L-system second target sequence based on the probability distribution of the next L-system character;

[0106] The prediction module is used to obtain the growth rule of the tree to be predicted based on the L-system second target sequence.

[0107] It should be noted that each module in the device for generating tree image growth rules based on deep learning described in this embodiment corresponds one-to-one to each step in the method for generating tree image growth rules based on deep learning in the aforementioned embodiment. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned method for generating tree image growth rules based on deep learning, which will not be repeated here.

[0108] Based on the same inventive concept as the above embodiment, this embodiment provides a computer-readable storage medium, on which a computer program is stored, and a processor executes the computer program to implement the above method. In some embodiments, the computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it can be various devices including one or any combination of the above memories. The computer can be various computing devices including intelligent terminals and servers.

[0109] It should be noted that all directional indications in the embodiments of the present application (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components in a certain specific posture. If the specific posture changes, the directional indication will also change accordingly.

[0110] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features limited to "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the meaning of "and / or" appearing in the full text includes three parallel solutions. Taking "A and / or B" as an example, it includes solution A, or solution B, or solutions that satisfy both A and B. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0111] The computer device of the hardware operating environment involved in the scheme of this embodiment may include: a processor, such as a central processing unit (CPU), a communication bus, a user interface, a network interface, and a memory. Among them, the communication bus is used to realize the connection and communication between these components. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface may also include a standard wired interface and a wireless interface. The network interface may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory may be a high-speed random access memory (Random Access Memory, RAM) memory, or it may be a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory may also be a storage device independent of the aforementioned processor.

[0112] Those skilled in the art will appreciate that the above structure does not constitute a limitation on the computer device, and may include more or fewer components than those described above, or a combination of certain components, or a different arrangement of components.

[0113] The memory as a storage medium may include an operating system, a network communication module, a user interface module, and an electronic program, and may also include a data storage module. In the above-mentioned computer device, the network interface is mainly used for data communication with a network server; the user interface is mainly used for data interaction with a user; the processor and the memory in the computer device of this embodiment may be set in the computer device, and the computer device calls the device based on deep learning to generate tree image growth rules stored in the memory through the processor, and executes the method based on deep learning to generate tree image growth rules provided in this embodiment.

[0114] It should be noted that the above-mentioned computer device can be an external hardware device that can run independently, or it can be a hardware device that comes with the device that generates tree image growth rules based on deep learning.

[0115] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.< / eos> < / bos> < / eos> < / bos> < / eos> < / token> < / bos> < / pad> < / pad> < / bos> < / eos> < / pad> < / eos>

Claims

1. A method for generating tree image growth rules based on deep learning, characterized in that: The following steps are involved: Get the image of the tree to be predicted; Based on the convolutional neural network module, feature extraction and encoding are performed on the image of the tree to be predicted to obtain convolutional coding feature data; The convolutional coding feature data and the first target sequence of L-system are input into the decoder module so that the decoder module outputs the probability distribution of the next L-system character; based on the probability distribution of the next L-system character, the second target sequence of L-system is obtained; wherein, the step of inputting the convolutional coding feature data and the first target sequence of L-system into the decoder module so that the decoder module outputs the probability distribution of the next L-system character includes: performing word embedding and position embedding on the first target sequence of L-system to obtain a feature matrix of the sequence; inputting the feature matrix of the sequence into the Masked Multi-Head Attention layer to obtain a self-attention feature; adding the feature matrix of the sequence and the self-attention feature, and performing normalization processing to obtain a first output feature; inputting the first output feature and the convolutional coding feature data into the Multi-Head Attention layer to obtain the feature of the encoder-decoder attention layer; adding the first output feature and the feature of the encoder-decoder attention layer, and performing normalization processing to obtain a second output feature; inputting the second output feature into the position-wise Feed Forward layer to obtain feedforward neural network features; add the second output features and the feedforward neural network features, and perform normalization processing to obtain the third output features; input the third output features as the initial feature matrix into the Masked Multi-Head Attention layer, the Multi-Head Attention layer and the position-wise Feed Forward layer in sequence, and add the input value and output value of each layer, and perform normalization processing to obtain the first output features; then input the first output features as the initial feature matrix into the Masked Multi-Head Attention layer, the Multi-Head Attention layer and the position-wise Feed Forward layer in sequence, and add the input value and output value of each layer, and perform normalization processing to obtain the second output features; after the above steps are repeated for a preset number of times, the last output features are used as the fourth output features; the fourth output features are input into the preset linear layer and the preset softmax layer to obtain the probability distribution of the next L-system character; Based on the L-system second target sequence, the growth rule of the tree to be predicted is obtained.

2. The method for generating tree image growth rules based on deep learning according to claim 1, characterized in that: The convolutional neural network module is trained based on the following training sample set: Get multiple tree images; After unifying the sizes of the tree images, annotating the L-system sequence corresponding to each type of tree in the tree images to obtain a plurality of first tree images; wherein the L-system sequence refers to sequence information used to describe the growth process and morphological generation of the trees; The pixel values ​​of the plurality of first tree images are normalized to obtain a training sample set.

3. The method for generating tree image growth rules based on deep learning according to claim 2, characterized in that: The convolutional neural network module is trained by the following steps: Inputting the training sample set into the Xception structure in the convolutional neural network module, so that multiple channels in the Xception structure respectively perform depth-separable convolution processing on the training sample set to obtain multiple depth-separable convolution features; Connect and reshape a plurality of the depth-separable convolutional features so that the convolutional neural network module outputs convolutional encoded feature data.

4. The method for generating tree image growth rules based on deep learning according to claim 3, characterized in that: The step of connecting and reshaping the plurality of the depth-separable convolutional features so that the convolutional neural network module outputs convolutional encoding feature data comprises: Connecting multiple depth-separable convolutional features to merge multiple depth-separable convolutional features; and obtaining a feature map through a convolutional layer, a BatchNormal layer, and a Relu layer; wherein the convolutional layer includes 256 convolution kernels, and the convolution kernel size is 3*3; After flattening the last two dimensions of the feature map, convolutional coding feature data is obtained.

5. The method for generating tree image growth rules based on deep learning according to claim 1, characterized in that: The step of obtaining the second target sequence of the L-system based on the probability distribution of the next L-system character comprises: selecting the character with the largest probability value as the predicted character based on the probability distribution of the next L-system character, and adding the predicted character to the first target sequence of the L-system; repeating this step until the predicted character is <eos> , obtain the second target sequence of L-system;< / eos> The step of obtaining the growth rule of the tree to be predicted based on the L-system second target sequence includes: visualizing the L-system second target sequence to simulate the growth rule of the tree to be predicted.

6. A device for generating tree image growth rules based on deep learning, characterized in that: include: An acquisition module, used for acquiring images of trees to be predicted; A feature encoding module, used for extracting and encoding features of the image of the tree to be predicted based on a convolutional neural network module to obtain convolutional encoding feature data; A decoding module is used to input the convolutional coding feature data and the first target sequence of L-system into a decoder module so that the decoder module outputs the probability distribution of the next L-system character; based on the probability distribution of the next L-system character, obtain the second target sequence of L-system; wherein the step of inputting the convolutional coding feature data and the first target sequence of L-system into the decoder module so that the decoder module outputs the probability distribution of the next L-system character includes: performing word embedding and position embedding on the first target sequence of L-system to obtain a feature matrix of the sequence; inputting the feature matrix of the sequence into a Masked Multi-Head Attention layer to obtain a self-attention feature; adding the feature matrix of the sequence and the self-attention feature, and performing normalization processing to obtain a first output feature; inputting the first output feature and the convolutional coding feature data into a Multi-Head Attention layer to obtain the feature of the encoder-decoder attention layer; adding the first output feature and the feature of the encoder-decoder attention layer, and performing normalization processing to obtain a second output feature; inputting the second output feature into a position-wise Feed Forward layer to obtain feedforward neural network features; add the second output features and the feedforward neural network features, and perform normalization processing to obtain the third output features; input the third output features as the initial feature matrix into the Masked Multi-Head Attention layer, the Multi-Head Attention layer and the position-wise Feed Forward layer in sequence, and add the input value and output value of each layer, and perform normalization processing to obtain the first output features; then input the first output features as the initial feature matrix into the Masked Multi-Head Attention layer, the Multi-Head Attention layer and the position-wise Feed Forward layer in sequence, and add the input value and output value of each layer, and perform normalization processing to obtain the second output features; after the above steps are repeated for a preset number of times, the last output features are used as the fourth output features; the fourth output features are input into the preset linear layer and the preset softmax layer to obtain the probability distribution of the next L-system character; The prediction module is used to obtain the growth rule of the tree to be predicted based on the L-system second target sequence.

7. The device for generating tree image growth rules based on deep learning according to claim 6, characterized in that: The feature encoding module includes an Xception structure, and the Xception structure includes at least four channels with the same number of convolution kernels, a BatchNormal layer and a Relu layer; Each of the channels is a depthwise separable convolution branch, and the expansion rate of each of the depthwise separable convolution branches increases exponentially. The size of each convolution kernel in the last three depthwise separable convolution branches is the same and is three times the size of each convolution kernel in the first depthwise separable convolution branch. The BatchNormal layer is used to normalize the output value of the depth-separable convolution branch; The Relu layer is an activation function.

8. A computer device, characterized in that: The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method for generating tree image growth rules based on deep learning according to any one of claims 1-5.

Images