A citrus color transformation prediction network training method and application method

Abstract

The application provides a citrus color transition prediction network training method and application method, comprising: acquiring an image prediction training set, the image prediction training set comprising a citrus image set and a citrus semantic set; constructing an initial citrus color transition prediction network comprising a semantic segmentation module, an image generation module and a loss optimization module; training the initial semantic segmentation module using the citrus semantic set; training the initial image generation module using the citrus image set, updating the image generation module network parameters with the loss optimization module, and optimizing the image generation module; and obtaining a trained complete citrus color transition prediction network according to the trained complete semantic segmentation module and the image generation module. The application trains the image generation module by inputting images, real images and their time interval mapping to form the citrus image set and introducing an embedding layer in the image generation module, so as to realize the color transition prediction of the citrus according to the input citrus image and the input time interval.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image prediction, in particular to a citrus color transition prediction network training method, an application method and an electronic device. BACKGROUND

[0002] In the process of planting and harvesting of citrus, the timing of picking citrus is an important factor affecting the quality of the fruit. For the growth process of most citrus plants, as the maturity increases, the proportion of pigments on the surface of the citrus will also change, and the peel will correspondingly change from green to other colors such as yellow, orange and red. Therefore, the color of the citrus is a good indicator of the ripening period, and arranging the picking time by predicting the color change of the fruit can effectively improve the quality of the citrus fruit.

[0003] At present, the method for estimating the maturity of citrus according to the color of the citrus is usually based on a deep neural network to calculate a citrus color index (CCI) from the local color of a photographed citrus image or based on a deep neural network to classify the citrus image into unripe, semi-ripe and ripe. However, the current method for judging the color of the citrus generally judges the current color of the citrus based on the photographed image of the citrus, cannot predict the color transition and maturity of the citrus during the growth process, and can only judge the picking time based on experience, lacking accuracy. SUMMARY

[0004] Therefore, it is necessary to provide a citrus color transition network training method, an application method and an electronic device to solve the technical problem that the current technology cannot predict the color transition of the citrus, resulting in inaccurate judgment of the picking time of the citrus.

[0005] To solve the above problems, the present application provides a citrus color transition prediction network training method, comprising:

[0006] An image prediction training set is obtained, which includes a citrus image set and a citrus semantic set, the citrus image set includes an input image, a real image and a time interval mapping between the input image and the real image, and the citrus semantic set includes all citrus images and corresponding semantic labels;

[0007] An initial citrus color transition prediction network is constructed, which includes a semantic segmentation module, an image generation module and a loss optimization module;

[0008] The citrus semantic set is taken as the input of the initial semantic segmentation module, and the semantic segmentation module is iteratively trained to obtain a trained complete semantic segmentation module;

[0009] The input image is taken as input of a semantic segmentation module to obtain a segmentation mask, the input image, time interval mapping and segmentation mask are taken as input of an initial image generation module to obtain a generated image, the input image, generated image and real image are taken as input of the loss optimization module, and a local perception loss is calculated in combination with the segmentation mask of the input image and the semantic label of the real image, and the initial image generation module is iteratively optimized and updated according to the local perception loss until the loss function converges, so as to obtain a trained image generation module;

[0010] A trained citrus color transition prediction network is obtained according to the trained semantic segmentation module and the trained image generation module.

[0011] Further, the image prediction training set is obtained by:

[0012] Citrus image samples in a transition period of the citrus are collected, and serial numbers and collection times are labeled, and semantic labels are obtained by performing semantic labeling on the citrus image samples;

[0013] A time interval mapping is constructed between any two image samples of the same citrus according to the serial numbers and collection times, and an image sample taken before the time interval is taken as an input image, and an image sample taken after the time interval is taken as a real image;

[0014] The input image, real image and corresponding time interval mapping form a citrus image set;

[0015] All the citrus images and corresponding semantic labels form a citrus semantic set.

[0016] Further, the semantic segmentation module is used to obtain a segmentation mask by performing semantic segmentation on the input image to distinguish the positions of the citrus image and the background image;

[0017] The semantic segmentation module comprises a semantic segmentation encoder composed of a semantic convolution layer and a max-pooling layer, and a semantic segmentation decoder composed of a transposed convolution layer and an up-sampling layer.

[0018] The semantic segmentation encoder is used to extract image segmentation features of the input image, and the semantic segmentation decoder is used to decode the segmentation mask according to the image segmentation features.

[0019] Further, the image generation module is used to predict a citrus color transition image.

[0020] The image generation module comprises an image generation encoder composed of a first generation convolution layer and a residual convolution block layer, an image generation decoder composed of a second generation convolution layer, two embedding layers and an output layer.

[0021] The image generation encoder is configured to extract a citrus color transition feature of the input image to obtain a transition feature map; the image generation decoder is configured to realize background re-fusion according to the citrus color transition feature and in combination with the segmentation mask to obtain an output feature map; the two embedding layers are configured to encode the input time interval into a multi-dimensional vector and multiply the transition feature map and the output feature map obtained by the image generation encoder and the image generation decoder respectively; and the output layer takes a scaled hyperbolic tangent function as an activation function and is configured to output a generated image and ensure the range of output pixels by the scaled hyperbolic tangent function.

[0022] Further, the loss optimization module is configured to extract features of the input image, the predicted image and the real image, calculate a local perception loss function, and update network parameters of the image generation module according to the local perception loss;

[0023] The loss optimization module comprises a feature extraction layer composed of five convolution modules and five maximum pooling layers, each convolution module is connected to each maximum pooling layer in a one-to-one correspondence, the first two convolution modules each comprise two convolution layers with the same number of filters, and the last three convolution modules each comprise four convolution layers with the same number of filters; and a model optimization layer composed of an adaptive moment estimation optimizer;

[0024] The feature extraction layer is configured to extract features of the input image, the predicted image and the real image to obtain a local perception loss; and the model optimization layer is configured to update network parameters of the image generation module to reduce the local perception loss.

[0025] Further, the local perception loss comprises a content loss and a local style loss, the content loss is configured to measure the content similarity between the generated image and the input image, and the local style loss is configured to measure the similarity of the color and its distribution of the generated image and the real image.

[0026] Further, the input image, the generated image and the real image are input into the loss optimization module, and the segmentation mask of the input image and the semantic label of the real image are combined to calculate the local perception loss, and the initial image generation module is iteratively optimized and updated according to the local perception loss until the loss function converges, so as to obtain a trained image generation module, comprising:

[0027] The input image and the generated image are input into the loss optimization module for convolution calculation to obtain a content loss;

[0028] The generated image, the real image and the segmentation mask are input into the loss optimization module for convolution calculation to obtain a local style loss;

[0029] The local perception loss is obtained according to the content loss and the local style loss, and the network parameters of the image generation module are updated through the loss optimization module to reduce the local perception loss until the loss function converges, so that the trained image generation module is obtained.

[0030] Further, the trained citrus color transition prediction network is obtained according to the trained semantic segmentation module and the trained image generation module, and comprises:

[0031] The trained semantic segmentation module and the trained image generation module are reserved, and the loss optimization module is deleted to obtain the trained citrus color transition prediction network.

[0032] The application further provides a citrus color transition prediction network application method, comprising:

[0033] Obtaining a to-be-predicted citrus image and a to-be-predicted time interval;

[0034] Inputting the to-be-predicted citrus image and the to-be-predicted time interval into the trained citrus color transition prediction network to predict the citrus color transition, wherein the trained citrus color transition prediction network is determined according to the citrus color transition prediction network training method in any one of the above.

[0035] The trained citrus color transition prediction network outputs a predicted citrus color transition image.

[0036] The application further provides an electronic device, comprising a processor, a memory, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the citrus color transition prediction network training method in any one of the above and / or the citrus color transition prediction network application method is implemented.

[0037] Compared with the prior art, the beneficial effects of the above-mentioned embodiments are: in the citrus color transition prediction network training method provided by the application, first, an image prediction training set is obtained, the image prediction training set includes a citrus image set and a citrus semantic set, the citrus image set includes an input image, a real image, and a time interval mapping between the input image and the real image, and the citrus semantic set includes an image corresponding to the time interval mapping and a semantic label thereof; then, an initial citrus color transition prediction network including a semantic segmentation module, an image generation module, and a loss optimization module is constructed; the citrus semantic set is taken as an input of the initial semantic segmentation module, and the semantic segmentation module is iteratively trained to obtain a training-complete semantic segmentation module; the input image is taken as an input of the semantic segmentation module to obtain a segmentation mask, the input image, the time interval mapping, and the segmentation mask are taken as inputs of the initial image generation module to obtain a generated image, the input image, the generated image, and the real image are taken as inputs of the loss optimization module, and a local perception loss is calculated in combination with the segmentation mask of the input image and the semantic label of the real image, the initial image generation module is iteratively optimized and updated according to the local perception loss until the loss function converges, and a training-complete image generation module is obtained; finally, a training-complete citrus color transition prediction network is obtained according to the training-complete semantic segmentation module and the training-complete image generation module. In summary, the application trains the image generation module by using the citrus image set composed of the input image, the real image, and the time interval mapping between the input image and the real image, and introduces the design of the embedding layer in the image generation module, so as to obtain the citrus color transition prediction network that can obtain the citrus color transition prediction image according to the input citrus image and the input time interval, thereby solving the technical problem that the maturity cannot be predicted according to the citrus color in the prior art, and the picking time of the citrus cannot be accurately determined. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 A flowchart of one embodiment of the citrus color transition prediction network training method provided by the application;

[0039] Figure 2 A schematic diagram of changes in the local perception total loss, the peak signal-to-noise ratio, and the average style loss in the training process of one embodiment of the citrus color transition prediction network application method provided by the application;

[0040] Figure 3 The prediction results of one embodiment of the citrus color transition prediction network application method provided by the application input the input citrus image and the citrus images with different time intervals;

[0041] Figure 4 The prediction results of one embodiment of the citrus color transition prediction network application method provided by the application input the input citrus image and the citrus images with different time intervals;

[0042] Figure 5 Structure diagram of one embodiment of the electronic device provided by the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person skilled in the art without creative work fall within the protection scope of the present application.

[0044] It should be understood that the drawings of the schematic diagram are not drawn according to the actual proportion. The flowchart used in the present application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, a person skilled in the art can add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of the present application.

[0045] Some block diagrams shown in the drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor systems and / or microcontroller systems.

[0046] In this document, the term "embodiment" means that the specific features, structures or characteristics described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment that is not mutually exclusive with other embodiments. A person skilled in the art explicitly and implicitly understands that the embodiments described herein can be combined with other embodiments.

[0047] The embodiment of the present application provides a citrus color change prediction network training method, which is described below.

[0048] Figure 1 The flowchart of one embodiment of the citrus color change prediction network training method provided by the present application is shown in FIG. 1, which includes the following steps. Figure 1

[0049] ​S101, acquire an image prediction training set, the image prediction training set comprising a citrus image set and a citrus semantic set, the citrus image set comprising an input image, a real image and a time interval mapping between the input image and the real image, and the citrus semantic set comprising all citrus images and corresponding semantic labels;

[0050] S102, construct an initial citrus color change prediction network comprising a semantic segmentation module, an image generation module and a loss optimization module;

[0051] S103, input the citrus semantic set into the initial semantic segmentation module to iteratively train the semantic segmentation module to obtain a trained semantic segmentation module;

[0052] S104, input the input image into the semantic segmentation module to obtain a segmentation mask, input the input image, the time interval mapping and the segmentation mask into the initial image generation module to obtain a generated image, input the input image, the generated image and the real image into the loss optimization module, and calculate a local perception loss by combining the segmentation mask of the input image and the semantic label of the real image, and iteratively optimize and update the initial image generation module according to the local perception loss until the loss function converges to obtain a trained image generation module;

[0053] S105, obtain a trained citrus color change prediction network according to the trained semantic segmentation module and the trained image generation module.

[0054] In the citrus color change prediction network training method provided by the application, the initial citrus color change prediction network comprises a semantic segmentation module, an image generation module and a loss optimization module. The semantic segmentation module is trained by a citrus semantic set composed of images corresponding to a time interval mapping and their semantic labels; the image generation module is trained by a citrus image set composed of an input image, a real image and a time interval mapping between the input image and the real image, and the network parameters of the image generation module are updated until the loss function converges by calculating a local perception loss with the loss optimization module; and a trained citrus color change prediction network is obtained according to the trained semantic segmentation module and the image generation module.

[0055] Compared with the prior art, the image generation module is trained by a citrus image set composed of an input image, a real image and a time interval mapping between the input image and the real image, and an embedding layer is introduced in the image generation module to obtain a citrus color change prediction network that can generate a citrus color change prediction image according to an input citrus image and an input time interval. The purpose of predicting the color change of citrus with growth time according to the maturity of citrus color and accurately determining the harvesting time of citrus according to the color change prediction result is achieved.

[0056] In specific embodiments of the present application, an image prediction training set is obtained, comprising:

[0057] Citrus image samples in the transition period of the citrus are collected, and serial numbers and collection times are labeled, and semantic labels are obtained by performing semantic labeling on the citrus image samples;

[0058] A time interval mapping is constructed between any two images of the image samples of the same citrus according to the serial numbers and collection times, and the image sample taken before the time interval is taken as an input image, and the image sample taken after the time interval is taken as a real image;

[0059] The input image, the real image, and the corresponding time interval mapping form a citrus image set;

[0060] The time interval mapping corresponding image and the semantic label form a citrus semantic set.

[0061] Specifically, some citrus are randomly selected on the citrus trees at different positions in the orchard where sampling is performed, and serial number labels are used for differentiation. Photographs are taken from different angles during the color transition of the citrus, and LabelMe is used for semantic labeling of the images. For each sample of the citrus, a time interval mapping is constructed between any two images according to the shooting date, and the citrus image taken earlier in shooting time is taken as an input image, and the input image is taken as a real image according to the corresponding citrus image of the time interval mapping. The input image, the real image, and the corresponding time interval mapping form a citrus image set, and the time interval mapping corresponding image and the semantic label form a citrus semantic set.

[0062] It should be noted that LabelMe is an online image labeling tool, which is used here for semantic labeling of image segmentation of the collected images.

[0063] In specific embodiments of the present application, the semantic segmentation module is used to obtain a segmentation mask by performing semantic segmentation on the input image to distinguish the positions of the citrus image and the background image.

[0064] The semantic segmentation module comprises a semantic segmentation encoder composed of a semantic convolution layer and a max-pooling layer, and a semantic segmentation decoder composed of a transpose convolution layer and an up-sampling layer.

[0065] The semantic segmentation encoder is used to extract image segmentation features of the input image, and the semantic segmentation decoder is used to decode the segmentation mask according to the image segmentation features.

[0066] Specifically, the semantic encoder and the semantic decoder of the semantic segmentation module adopt the structure of U-Net encoder-decoder. The semantic encoder comprises a semantic convolution layer and a max pooling layer, wherein the semantic convolution layer comprises a plurality of 3x3 convolutions, and the max pooling layer comprises a 2x2 max pooling operation; the semantic decoder comprises a transposed convolution layer and an up-sampling layer. For an input image sample, image segmentation features are extracted by the semantic encoder, and the image segmentation features are decoded by the semantic decoder to obtain a segmentation mask.

[0067] For the initial semantic segmentation module, a Dice Loss (Dice similarity coefficient) is used as a loss function, and the module is trained using a citrus semantic set until the loss function converges, thereby obtaining a trained semantic segmentation module.

[0068] In specific embodiments of the present application, the image generation module is used to predict a citrus color transition image.

[0069] The image generation module comprises an image generation encoder composed of a first generation convolution layer and a residual convolution block layer, an image generation decoder composed of a second generation convolution layer, two embedding layers, and an output layer.

[0070] The image generation encoder is configured to extract citrus color transition features of an input image to obtain a transition feature map; the image generation decoder is configured to realize background re-fusion according to the citrus color transition features and in combination with the segmentation mask to obtain an output feature map; the two embedding layers are configured to encode an input time interval into a multi-dimensional vector and multiply the transition feature map and the output feature map obtained by the image generation encoder and the image generation decoder, respectively; and the output layer uses a scaled hyperbolic tangent function as an activation function and is configured to output a generated image and ensure the range of output pixels by the scaled hyperbolic tangent function.

[0071] Specifically, the image generation encoder in the image generation module is composed of a first generation convolution layer and a residual convolution layer, the first generation convolution layer comprises a plurality of 4x4 convolutions with a step size of 2, and the residual convolution layer comprises a plurality of identical 3x3 residual convolution blocks. The decoder is composed of a second convolution layer, which comprises several size-adjusted convolutions and a 3x3 convolution.

[0072] For an input image, after the image generation encoder extracts the citrus color transition features of the input image to obtain a transition feature map, the image generation decoder uses a nearest neighbor interpolation to double the height and width of the feature map through the size-adjusted convolutions in the second convolution layer, and then inputs the result to a 4x4 convolution layer with a step size of 1 to realize background re-fusion according to the citrus color transition features and the segmentation mask to obtain an output feature map.

[0073] In addition, the embedding layer encodes the time interval into two multi-dimensional vectors with a dimension of 64, which are multiplied with the transition feature map and the output feature map obtained by the image generation encoder and the image generation decoder respectively. Except for the output layer, all the convolutional layers are subjected to instance normalization operation and use the ReLu (Linear rectification function) activation function, and the output layer uses the scalable tanh (hyperbolic tangent function) function as the activation function to ensure that the output pixel range is between [0, 255]. Through the design of the embedding layer, the generated image is determined by the input image and the time interval.

[0074] In specific embodiments of the present application, the loss optimization module is used to extract the features of the input image, the predicted image and the real image, calculate the local perception loss function, and update the network parameters of the image generation module with the local perception loss.

[0075] The loss optimization module comprises a feature extraction layer composed of five convolutional modules and five max-pooling layers, each convolutional module being connected in one-to-one correspondence with each max-pooling layer, the first two convolutional modules each comprising two convolutional layers with the same number of filters, and the last three convolutional modules each comprising four convolutional layers with the same number of filters; and a model optimization layer composed of an adaptive moment estimation optimizer.

[0076] The feature extraction layer is used to extract the features of the input image, the predicted image and the real image to obtain the local perception loss, and the model optimization layer is used to update the network parameters of the image generation module to reduce the local perception loss.

[0077] In specific embodiments of the present application, the local perception loss comprises a content loss and a local style loss, the content loss being used to measure the content similarity between the generated image and the input image, and the local style loss being used to measure the similarity of the color and its distribution of the citrus in the generated image and the real image.

[0078] Specifically, VGG19 is a deep neural network model for image classification, and the feature extraction layer of the loss optimization module in the embodiments of the present application is based on the feature extraction part of the VGG19 model and uses the model parameters trained on the ImageNet dataset. Since there are more than one million images on the ImageNet dataset, the trained VGG19 model has strong feature extraction capability. The feature extraction layer comprises a feature extraction layer composed of five convolutional modules and five max-pooling layers, and a model optimization layer composed of an Adam (adaptive moment estimation) optimizer. The first two convolutional modules each comprise two convolutional layers with the same number of filters, and the number of filters is 64 and 128 in turn, and the last two convolutional modules each comprise four convolutional layers with the same number of filters, and the number of filters is 256, 512 and 512 in turn.

[0079] The feature extraction layer is used to extract the citrus image features to obtain a local perception loss. The extracted local perception loss includes two parts of a content loss and a local style loss. The content loss is used to measure the content similarity between the generated image and the input image, so that the predicted image is more realistic. The local style loss is used to measure the similarity of the color of the citrus and its distribution in the generated image and the real image.

[0080] The present application adopts conv4_2 and conv5_2 in VGG19 to calculate the content loss, and adopts conv1_1, conv2_1, conv3_1, conv4_1 and conv5_1 to calculate the local style loss. Let l represent the module convolution layer in the feature extraction layer, and the content loss calculated by the input image through the lth layer is:

[0081]

[0082] Wherein, x is the input image, is the generated image, H l , W l and C l are the height, width and channel number of the feature map obtained after the lth layer convolution, represents the value of the three-dimensional coordinates of the feature map x composed of height, width and channel at (i, j, k).

[0083] After the input image x is convolved through the lth layer, the feature map x l is obtained, and the height-width matrix on the kth channel of the feature map x l is flattened into a vector F l with a length of H l,k E t l,k represents the tth element of F l,k , and the element of the Gram matrix of the input image x after the lth layer convolution is:

[0084]

[0085] The local style loss calculated by the real image through the lth layer is:

[0086]

[0087] Wherein, y is the real image, M l () represents the semantic segmentation mask corresponding to the lth layer of the image y. For different layers, the mask is down-sampled to the same length and width as the output feature. The segmentation mask is obtained according to the segmentation mask to segment the citrus image in the citrus part image and the background part image.

[0088] The local perception loss obtained according to the content loss and the local style loss is:

[0089]

[0090] wherein W content,s and W style,t are layer weight parameters.

[0091] In a specific embodiment of the present application, the input image, the generated image and the real image are taken as inputs of the loss optimization module, and the local perception loss is calculated in combination with the segmentation mask of the input image and the semantic label of the real image, and the initial image generation module is iteratively optimized and updated according to the local perception loss until the loss function converges, so as to obtain a trained image generation module, which comprises:

[0092] The input image and the generated image are input into the loss optimization module for convolution calculation to obtain the content loss;

[0093] The generated image, the real image and the segmentation mask are input into the loss optimization module for convolution calculation to obtain the local style loss;

[0094] The local perception loss is obtained according to the content loss and the local style loss, and the network parameters of the image generation module are updated through the loss optimization module to reduce the local perception loss, until the loss function converges, so as to obtain a trained image generation module.

[0095] Specifically, in the process of training the image generation module, the content loss is calculated according to the input image and the generated image by the feature extraction layer of the loss optimization module, and the local style loss is calculated according to the generated image, the real image and the segmentation mask; then the local perception loss is obtained by adding the content loss and the local style loss; finally, the generated network parameters are updated using the Adam (adaptive moment estimation) optimizer to reduce the local perception loss, and when the loss function converges, a trained image generation module is obtained.

[0096] In a specific embodiment of the present application, a trained citrus color transition prediction network is obtained according to the trained semantic segmentation module and the trained image generation module, which comprises:

[0097] The trained semantic segmentation module and the trained image generation module are retained, and the loss optimization module is deleted to obtain a trained citrus color transition prediction network.

[0098] Specifically, the semantic segmentation module is used to obtain a segmentation mask by performing semantic segmentation on the input image to distinguish the citrus image and the background image position; the image generation module is used to predict a citrus color transition image; and the loss optimization module is used to extract features of the input image, the predicted image and the real image, calculate a local perception loss function, and update network parameters of the image generation module according to the local perception loss. After the semantic segmentation module and the image generation module are trained, the loss optimization module is no longer needed, and only the semantic segmentation module and the image generation module need to be retained to constitute a trained citrus color transition prediction network.

[0099] It needs to be further explained that, in the technical solution of the present application, in addition to being able to output a predicted image of the color transition of the citrus by inputting the citrus image and the time interval, the present application also makes the generated image maintain the semantic content of the input image while being highly similar to the real image in the color distribution of the skin of the citrus through the improved local style loss; and the generated image can be controlled by the input time interval through the embedding layer of the image generation module.

[0100] The technical solution of the present application will be described more clearly in combination with a specific application scenario, and the effectiveness of the present application will be evaluated, and the specific process is as follows:

[0101] I. Preparation of data set:

[0102] The citrus images to be tested are collected, the time interval mapping of the citrus images of the same citrus at different times is established, and semantic annotation is performed to obtain a citrus image data set. The data set is divided into a training set, a validation set and a test set according to a ratio of 3:1:1.

[0103] II. Experimental platform and training details:

[0104] The processor and the graphics card used in the experimental platform are AMD EPYC 7543 32-Core Processor and NVIDIA A40, respectively. The batch size is set to 32, and the epoch is set to 80. In order to verify the training effect, the PSNR (peak signal-to-noise ratio) is used to evaluate the quality of the generated image. The higher the PSNR between the generated image and the original image, the smaller the distortion degree and the higher the image quality. The MLSL (mean style loss) of all prediction results of the data set and the real image is used as an accuracy index, and a low MLSL indicates that the predicted citrus color and distribution are highly similar to the real one. During the training process, the changes of the Total Loss (local perception total loss), the PSNR (peak signal-to-noise ratio) and the MLSL (mean style loss) on the training set and the validation set are as follows: Figure 2As shown. To verify the robustness and generalization of the network, 10 models with the best MLSL performance on the validation set were selected, and then 5 models with the best PSNR performance were selected from these for experiments on the test set. The highest PSNR obtained on the test set was 30.01, indicating that there is a high similarity between the generated image and the real-world situation.

[0105] III. Results Analysis:

[0106] To visualize the comparison between the prediction results of this invention and the actual situation, some images from the test set were randomly selected as input to predict the results when citrus images at different time intervals were input. For example... Figure 3 As shown, the predicted image is generated by the citrus color change prediction network based on the input image, while the real image is an actual photograph of the citrus fruit. As the time interval N increases, the peel color in the predicted image gradually deepens from yellowish-green to orange, showing the same trend as the real image. In the input image and when N=3, the citrus peel color is visibly greenish; when N=27, the citrus image has turned a distinct orange. The color change process is clearly visible in the intermediate stage from N=6 to N=21, but due to the format requirements of the accompanying diagrams, it cannot be represented by a color image and is therefore described in text. Furthermore, the brightness, texture features, and color changes of different peel regions in the generated image also change accordingly based on the features of the input image. Compared to a single peel color, preserving natural lighting conditions and background consistency makes the generated image more realistic.

[0107] To further verify the robustness of the network model, experiments were conducted using citrus images with different viewpoints and colors as input, such as... Figure 4 As shown, Figure 4 The input images are images of citrus fruits at different stages of ripeness, from immature to semi-ripe. The predicted images are generated by a citrus color change prediction network with time intervals N=10 or 20. Compared with the real images, the generated images not only predict color changes well but also preserve the uneven color distribution of the peel, such as yellow stripes and localized green spots. However, due to the format requirements of the accompanying drawings, the uneven color distribution cannot be shown in the drawings, so it is described in words. Compared with the traditional method of using CCI values ​​to measure the average color of citrus fruits, the present invention can show the color distribution of the citrus peel in more detail.

[0108] The present invention also provides an electronic device, combined with Figure 5 Let's take a look. Figure 5The structural schematic diagram of an embodiment of the electronic device provided by the present application is shown in FIG. 5. The electronic device 500 comprises a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the processor 501 executes the program, the training method of the citrus color transition prediction network and / or the application method of the citrus color transition prediction network are implemented.

[0109] As a preferred embodiment, the electronic device further comprises a display 503 for displaying the training method of the citrus color transition prediction network and / or the application method of the citrus color transition prediction network.

[0110] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 502 and executed by the processor 501 to complete the present application. One or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the electronic device 500.

[0111] The electronic device 500 can be a desktop computer, a notebook computer, a palm computer, a smart phone, or the like with a camera module.

[0112] The processor 501 can be an integrated circuit chip with signal processing capability. The processor 501 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), etc. It can implement or execute the disclosed methods, steps, and logic block diagrams in the embodiments of the present application. The general-purpose processor can also be a microprocessor or any conventional processor, etc.

[0113] The memory 502 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a secure digital (SD) card, a flash card, etc. The memory 502 is used to store programs. After receiving an execution instruction, the processor 501 executes the programs. The method defined by the foregoing processes of any of the embodiments of the present application can be applied to the processor 501 or implemented by the processor 501.

[0114] The display 503 can be an LED display screen, a liquid crystal display, or a touch display, etc. The display 503 is configured to display various information of the electronic device 500.

[0115] It can be understood that, Figure 5 The structure shown is only a structural schematic diagram of the electronic device 500, and the electronic device 500 can further include more or less components than those shown. Figure 5 The structure shown is only a structural schematic diagram of the electronic device 500, and the electronic device 500 can further include more or less components than those shown. Figure 5 The components shown in the above description can be implemented in hardware, software, or a combination thereof.

[0116] The above description is only a preferred embodiment of the present application, but the protection scope of the present application is not limited to this. Any changes or replacements within the technical scope disclosed by the present application can be easily thought by those skilled in the art, and should be covered within the protection scope of the present application.

Claims

1. A citrus color transition prediction network training method, characterized in that, The method comprises the following steps: obtaining an image prediction training set, wherein the image prediction training set comprises a citrus image set and a citrus semantic set, the citrus image set comprises an input image, a real image and a time interval mapping between the input image and the real image, and the citrus semantic set comprises all the citrus images and corresponding semantic labels; constructing an initial citrus color transition prediction network comprising a semantic segmentation module, an image generation module and a loss optimization module, wherein the image generation module is used for predicting a citrus color transition image; the image generation module comprises an image generation encoder composed of a first generation convolutional layer and a residual convolutional block layer, an image generation decoder composed of a second generation convolutional layer, two embedding layers and an output layer; wherein the image generation encoder is used for extracting citrus color transition features of the input image to obtain a transition feature map; the image generation decoder is used for realizing background re-fusion according to the citrus color transition features and combining a segmentation mask to obtain an output feature map; the two embedding layers are used for encoding the input time interval into a multi-dimensional vector and multiplying the transition feature map and the output feature map obtained by the image generation encoder and the image generation decoder respectively; and the output layer takes a scaled hyperbolic tangent function as an activation function and is used for outputting a generated image and ensuring the output pixel range through the scaled hyperbolic tangent function; the loss optimization module is used for extracting features of the input image, the predicted image and the real image, calculating a local perception loss function and updating network parameters of the image generation module according to the local perception loss; the loss optimization module comprises a feature extraction layer composed of five convolutional modules and five maximum pooling layers, each convolutional module is connected with each maximum pooling layer in a one-to-one correspondence, the first two convolutional modules each comprise two convolutional layers with the same number of filters, and the last three convolutional modules each comprise four convolutional layers with the same number of filters; and a model optimization layer composed of an adaptive moment estimation optimizer; wherein the feature extraction layer is used for extracting features of the input image, the predicted image and the real image to obtain a local perception loss; and the model optimization layer is used for updating network parameters of the image generation module and reducing the local perception loss; taking the citrus semantic set as an input of the initial semantic segmentation module, iteratively training the semantic segmentation module to obtain a trained semantic segmentation module; taking the input image as an input of the semantic segmentation module to obtain a segmentation mask, taking the input image, the time interval mapping and the segmentation mask as inputs of the initial image generation module to obtain a generated image, taking the input image, the generated image and the real image as inputs of the loss optimization module, and combining the segmentation mask of the input image and the semantic label of the real image to calculate a local perception loss, iteratively optimizing and updating the initial image generation module according to the local perception loss until the loss function converges, and obtaining a trained image generation module; obtaining a trained citrus color transition prediction network according to the trained semantic segmentation module and the trained image generation module.

2. The citrus color break prediction network training method of claim 1, wherein, The method for obtaining the image prediction training set comprises the following steps: Collecting citrus image samples in the transition period of citrus, and labeling serial numbers and collection time, obtaining semantic labels by performing semantic labeling on the citrus image samples; According to the serial numbers and collection time, constructing a time interval mapping between any two image samples of the same citrus, and taking the image sample taken before the time interval as an input image and the image sample taken after the time interval as a real image; The input image, the real image and the corresponding time interval mapping form a citrus image set; The time interval mapping corresponding image and semantic label form a citrus semantic set.

3. The citrus color break prediction network training method of claim 1, wherein, The semantic segmentation module is used to perform semantic segmentation on the input image to obtain a segmentation mask to distinguish the positions of the citrus image and the background image; The semantic segmentation module includes a semantic segmentation encoder composed of a semantic convolution layer and a max pooling layer, and a semantic segmentation decoder composed of a transpose convolution layer and an up-sampling layer; The semantic segmentation encoder is used to extract image segmentation features of the input image, and the semantic segmentation decoder is used to decode the segmentation mask according to the image segmentation features.

4. The citrus color change prediction network training method of claim 1, wherein, The local perception loss includes a content loss and a local style loss, the content loss is used to measure the content similarity between the generated image and the input image, and the local style loss is used to measure the similarity of the color and its distribution of the citrus in the generated image and the real image.

5. The citrus color break prediction network training method of claim 4, wherein, The input image, the generated image and the real image are input into the loss optimization module, and the local perception loss is calculated by combining the segmentation mask of the input image and the semantic label of the real image, and the initial image generation module is iteratively optimized and updated according to the local perception loss until the loss function converges, to obtain a trained image generation module, including: The input image and the generated image are input into the loss optimization module for convolution calculation to obtain a content loss; The generated image, the real image and the segmentation mask are input into the loss optimization module for convolution calculation to obtain a local style loss; According to the content loss and the local style loss, a local perception loss is obtained, and the network parameters of the image generation module are updated through the loss optimization module to reduce the local perception loss, until the loss function converges, to obtain a trained image generation module.

6. The citrus color change prediction network training method of claim 1, wherein, According to the trained semantic segmentation module and the trained image generation module, a trained citrus color transition prediction network is obtained, including: The trained semantic segmentation module and the trained image generation module are retained, and the loss optimization module is deleted to obtain a trained citrus color transition prediction network.

7. A method for citrus color transition prediction network application, characterized in that, Including: Obtaining a to-be-predicted citrus image and a to-be-predicted time interval; The to-be-predicted citrus image and the to-be-predicted time interval are input into the trained citrus color transition prediction network to predict the citrus color transition, wherein the trained citrus color transition prediction network is determined according to the citrus color transition prediction network training method of any one of claims 1 to 6; The citrus color transition prediction image output by the citrus color transition prediction network.

8. An electronic device, comprising: Including: A processor, a memory, and a computer program stored on the memory and capable of running on the processor, wherein the processor implements the citrus color change prediction network training method according to any one of claims 1 to 6 or the citrus color change prediction network application method according to claim 7 when executing the computer program.

Images