Image generation method, storage medium and electronic device

Abstract

The present invention discloses an image generation method, a storage medium and an electronic device, and relates to the field of image processing technology. The image generation method comprises: obtaining a normal grain image without defects; obtaining morphological information representing the outline of broken grain particles, and generating a defect outline based on the morphological information; smearing a defect area on the normal grain image based on the defect outline to obtain an image to be filled; obtaining a filled defective grain image according to a defect type set by a user and the image to be filled, and generating annotation information of the defective grain image based on the defect type. Applying the solution provided by the embodiment of the present invention can reduce the workload required for annotating images.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image generating method, a storage medium and an electronic device. Background Art

[0002] Grain quality evaluation is an important technical support for grain production, procurement, storage and processing. Its evaluation index is of great significance in guiding procurement pricing and guiding the rational use of grain. In related technologies, grain images are often used to evaluate grain quality. Since there are sensory inspection indicators in the observation image, subjective errors are prone to occur, and the accuracy of image inspection is low. Usually, a neural network model is used to perform the recognition task of grain images.

[0003] In order to train these neural network models, a lot of manpower is required to carry out labeling work to obtain labeled images for training. Summary of the invention

[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent. To this end, one object of the present invention is to provide an image generation method, a storage medium and an electronic device to reduce the workload required for annotating an image.

[0005] According to a first aspect of an embodiment of the present invention, there is provided an image generation method, the image generation method comprising:

[0006] Get a normal grain image without defects;

[0007] acquiring morphological information representing the contours of broken cereal particles and generating a defect contour based on the morphological information;

[0008] Based on the defect outline, a defect area is painted on the normal grain image to obtain an image to be filled;

[0009] A filled defective grain image is obtained according to the defect type set by the user and the image to be filled, and labeling information of the defective grain image is generated based on the defect type.

[0010] Optionally, the acquiring morphological information representing the contour of the broken grain particles and generating the defect contour based on the morphological information comprises:

[0011] Sampling random variables that obey normal distribution to obtain sampling information;

[0012] Performing feature fusion on the morphological features representing the morphological information and the sampling features representing the sampling information to obtain fused features;

[0013] A defect contour is obtained based on the fused features.

[0014] Optionally, the acquiring morphological information representing the contour of the broken grain particles and generating the defect contour based on the morphological information comprises:

[0015] Sampling random variables that obey normal distribution to obtain sampling information;

[0016] Inputting the morphological information and the sampling information into a pre-trained defect contour generation model to obtain a defect contour output by the defect contour generation model;

[0017] The defect contour generation model is a model obtained by training a preset first neural network model using a sample grain image containing broken grain grains, and the defect contour generation model is used to obtain an output image according to the grain contour in the sample grain image.

[0018] Optionally, the first neural network model includes an image encoder, a linear encoder, a feature fusion module and a decoder;

[0019] The image encoder is used to extract sample image features of the sample grain image;

[0020] The linear encoder is used to extract sample morphological features of sample morphological information of grain contours in the sample grain image;

[0021] The feature fusion module is used to perform feature fusion on the sample image feature and the sample morphological feature to obtain a sample fusion feature;

[0022] The decoder is used to obtain an output image according to the sample fusion features.

[0023] Optionally, the feature fusion module obtains the sample fusion feature in the following manner:

[0024]

[0025]

[0026]

[0027]

[0028]

[0029] in, represents the first fusion feature, represents the sample morphological characteristics, represents the sample image feature, A first feature matrix and a second feature matrix representing the first fusion feature, represents the activation function, Both represent learnable matrices, is the sampling information, represents the second fusion feature, represents the sample fusion feature, Indicates the sample morphology information; is the feature fusion function, specifically:

[0030]

[0031] in, represents the eigenvector as the independent variable, represents the learnable parameters, Represents the activation function.

[0032] Optionally, the method further includes:

[0033] After the first neural network model obtains the output image, the loss of the first neural network model is calculated in the following manner, and the model parameters of the first neural network model are adjusted based on the loss:

[0034]

[0035] Wherein, Loss represents the loss, n represents the total number of rows of pixel values ​​of the output image, and m represents the total number of columns of pixel values ​​of the output image. represents the pixel value of the i-th row and j-th column in the output image, represents the pixel value of the i-th row and j-th column in the sample grain image, represent the first characteristic matrix and the second characteristic matrix respectively.

[0036] Optionally, obtaining a filled defective grain image according to the defect type set by the user and the image to be filled includes:

[0037] Inputting the defect type and the image to be filled into a pre-trained defect filling model to obtain a defect grain image output by the defect filling model;

[0038] Among them, the defect filling model is: a model obtained by training the second neural network using a sample to-be-filled image provided with label information containing the defect type, and the defect filling model is used to fill in the grain image according to the defect type and output the filled image.

[0039] Optionally, the step of painting out a defective area on the normal grain image based on the defect contour includes:

[0040] Acquire a normal grain contour in the normal grain image;

[0041] Performing image expansion processing on the normal grain outline to obtain a first expanded image including the normal grain outline;

[0042] Performing image corrosion processing on the normal grain contour to obtain a target grain contour;

[0043] determining candidate points from the target grain contour, and determining the position of the defect contour in the second extended image based on the positions of the candidate points;

[0044] generating a second extended image including a defect outline according to the determined position, wherein the second extended image has the same size as the first extended image;

[0045] An intersection image area of ​​an image area included in the normal grain contour in the first extended image and an image area included in the defect contour in the second extended image is determined, and a defect area is smeared out in the normal grain image based on the position of the intersection image area.

[0046] According to a second aspect of an embodiment of the present invention, there is provided a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned image generating method is implemented.

[0047] According to a third aspect of an embodiment of the present invention, there is provided an electronic device comprising: a memory and a processor; a computer program is stored in the memory, and when the computer program is executed by the processor, the above-mentioned image generation method is implemented.

[0048] In the solution provided by the embodiment of the present invention, a defect contour is generated based on the morphological information of the contour of the broken grain particles. Broken grains have natural randomness and irregular shapes, so the defect contour generated by the morphological information in the image to be filled can effectively simulate the defects of the grain, and the defective grain image can be filled in the image to be filled according to the defect type set by the user and the annotation information can be generated. In this process, only the normal grain image needs to be obtained to generate the defective grain image, and when the defect type is set, the defect type to be annotated is known, so that accurate annotation information can be generated, so there is no need to obtain additional images of grains that actually contain defects, and there is no need to pay additional costs to identify defects in such images and then annotate them, which reduces the workload required to annotate images.

[0049] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1is a flow chart of an image generating method provided by an embodiment of the present invention;

[0051] Figure 2 It is a flowchart of the first model training method provided by an embodiment of the present invention;

[0052] Figure 3 is a flowchart of a first model reasoning method provided by an embodiment of the present invention;

[0053] Figure 4 is a structural schematic diagram of a neural network model provided by an embodiment of the present invention;

[0054] Figure 5 is a structural schematic diagram of an image encoder provided by an embodiment of the present invention;

[0055] Figure 6 is a flow chart of a second model training method provided by an embodiment of the present invention;

[0056] Figure 7 is a flow chart of a second model reasoning method provided by an embodiment of the present invention;

[0057] Figure 8 is a flowchart of an image processing method provided by an embodiment of the present invention;

[0058] Fig. 9 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0060] The image generating method, storage medium and electronic device according to the embodiments of the present invention are described below with reference to the accompanying drawings.

[0061] In one embodiment of the present invention, see Figure 1 , provides an image generating method, which includes the following steps S101-S104.

[0062] S101: Acquire a normal grain image without defects.

[0063] The grains in the normal grain image are grains without defects. The normal grain image can be obtained by photographing the normal grains with a photographing device. The photographing device can be a camera, a video camera, etc., which is not limited in the embodiment of the present invention.

[0064] S102: Acquire morphological information representing the contour of broken cereal particles, and generate a defect contour based on the morphological information.

[0065] The morphological information can be obtained by extracting the contour of the grain from the contour image containing the contours of the broken grain particles to obtain the morphological information describing the contour.

[0066] The following examples illustrate three implementation methods of generating morphological information.

[0067] In the first implementation, the area, length, and width information in the contour image can be calculated to obtain a morphological information with a length of 3; in the second implementation, the contour image can be evenly divided into N×N grids, and the ratio of the area of ​​the contour in each grid to the area of ​​the grid is calculated, and finally a morphological information with a length of morphological information, N is a manually preset parameter; in the third implementation method, the average value of the distances from the upper left, lower left, upper right and lower right corners of the contour image to multiple pixel points of the contour can be calculated, and finally a morphological information with a length of 4 is obtained.

[0068] According to the above implementation method, morphological information can also be randomly generated without using a contour image. For example, two random numbers are generated to represent the length and width information respectively, and then the corresponding area is calculated to obtain morphological information with a length of 3, etc. The embodiment of the present invention is not limited to this.

[0069] Grain defects can be of different types, such as worm-eaten grains, moldy grains, diseased grains, etc. These types of defects are random, and the defects in the image appear irregular in shape accordingly. Broken grains also have natural randomness and irregular shapes, so the defect contour generated by the morphological information representing the contour of the broken grain grains can simulate the shape of the defect.

[0070] The defect contour is the contour of the image area occupied by the grain defect in the image. The method of generating the defect contour is shown in the following embodiment.

[0071] S103: Smearing the defect area on the normal grain image based on the defect contour to obtain an image to be filled.

[0072] Specifically, an image region where grains are located in a normal grain image is obtained, and a defect contour is randomly set in the image region. A sub-image region of the image region included in the defect contour is the defect region.

[0073] The specific method of obtaining the defect area can be found in the subsequent embodiments.

[0074] S104: Obtaining a filled defective grain image according to the defect type set by the user and the image to be filled, and generating annotation information of the defective grain image based on the defect type.

[0075] In one embodiment of the present invention, defective grain images can be obtained based on model training, see the subsequent embodiments. In other embodiments, defective grain images corresponding to defect types can also be reconstructed using a super-resolution algorithm to obtain new defective grain images.

[0076] When smearing a defect area, the position of the smeared defect area can be set, so that the filled defect is located in the set position, and the defect positioning box information can also be included in the annotation information. The position of the specified positioning box includes the position of the smeared defect area, thereby providing pixel-level position annotation and defect type annotation, greatly increasing the richness of data volume and data type.

[0077] In the solution provided by the embodiment of the present invention, a defect contour is generated based on the morphological information of the contour of the broken grain particles. Broken grains have natural randomness and irregular shapes, so the defect contour generated by the morphological information in the image to be filled can effectively simulate the defects of the grain, and the defective grain image can be filled in the image to be filled according to the defect type set by the user and the annotation information can be generated. In this process, only the normal grain image needs to be obtained to generate the defective grain image, and when the defect type is set, the defect type to be annotated is known, so that accurate annotation information can be generated, so there is no need to obtain additional images of grains that actually contain defects, and there is no need to pay additional costs to identify defects in such images and then annotate them, which reduces the workload required to annotate images.

[0078] The following examples illustrate the specific implementation methods of two defect contours.

[0079] In the first implementation, random variables that obey a normal distribution are sampled to obtain sampling information; morphological features representing morphological information and sampling features representing sampling information are fused to obtain fused features; and defect contours are obtained based on the fused features.

[0080] Specifically, the above sampling information can be obtained from a normal distribution A set of random numbers are obtained by random sampling, and the corresponding sampling features can be represented by a random number matrix formed by the sampled random numbers.

[0081] In addition, in other implementations, sampling information may be obtained by sampling random variables of other distributions such as Poisson distribution, uniform distribution, sampling from exponential distribution, etc., and the embodiments of the present invention are not limited to this.

[0082] In one embodiment of the present invention, the morphological feature is a feature obtained by encoding the morphological information through an encoder.

[0083] The feature fusion method can be based on the RELU function to perform feature fusion, see the feature fusion function in the subsequent embodiments.

[0084] Alternatively, the fusion may be performed by concatenating or adding features, which is not limited in the embodiment of the present invention.

[0085] In one embodiment of the present invention, the defect contour can be obtained in the following manner: the fused features are upgraded in dimension, and a 2D feature map of the upgraded feature vector is obtained through a Reshape function, and the 2D feature map is then processed through operations such as convolution and bilinear interpolation upsampling to obtain a reconstructed image that is the same size as the contour image and includes the reconstructed contour.

[0086] In the second implementation, random variables that obey the normal distribution can be sampled to obtain sampling information; the morphological information and sampling information are input into a pre-trained defect contour generation model to obtain the defect contour output by the defect contour generation model. Figure 3 As shown, in the inference stage of the defect contour generation model, the normal distribution N (0, 1) is sampled to obtain the sampling information z; in the defect contour generation model, the feature fusion function F is used to fuse the sampling information z and the morphological information s' to obtain the fused feature z', and z' is input into the decoder to obtain the defect contour output by the decoder.

[0087] Among them, the defect contour generation model is: a model obtained by training a preset first neural network model using a sample grain image containing broken grain grains, and the defect contour generation model is used to obtain an output image according to the grain contour in the sample grain image.

[0088] The implementation method for obtaining the sampling information refers to the first implementation method shown in the above embodiment, which will not be described in detail here.

[0089] The first neural network may be a neural network model including CFC-VAE (Cross Fusion Controllable -Variational Autoencoder).

[0090] The autoencoder realizes the function of copying the input to the output through the encoder and decoder included in it. In CFC-VAE, new data can also be generated by random sampling before output to the decoder, so that the output image becomes a reconstructed image containing the image information of the input image and the new data. The specific structure of CFC-VAE is shown below.

[0091] Specifically, the first neural network model may include an image encoder, a linear encoder, a feature fusion module and a decoder;

[0092] The image encoder is used to extract sample image features of the sample grain image; the linear encoder is used to extract sample morphological features of the sample morphological information of the grain contour in the sample grain image; the feature fusion module is used to fuse the sample image features and the sample morphological features to obtain sample fusion features; the decoder is used to obtain the output image according to the sample fusion features.

[0093] The sample image features are image features used to extract features from the sample grain images.

[0094] Specifically, the sample grain image may be input into an image encoder to obtain sample image features extracted by the image encoder.

[0095] The sample morphological features are similar to those in the aforementioned embodiment, the only difference being the outline of the represented grains, which will not be described in detail here.

[0096] The method of obtaining the output image based on the sample fusion features is similar to the method of obtaining the reconstructed image including the reconstructed contour based on the fusion features in the aforementioned embodiment, and will not be described in detail here.

[0097] The following describes the specific implementation method of feature fusion performed by the feature fusion module.

[0098] In one embodiment of the present invention, the feature fusion module obtains the sample fusion feature in the following manner:

[0099]

[0100]

[0101]

[0102]

[0103]

[0104] in, represents the first fusion feature, represents the sample morphological characteristics, represents the sample image feature, A first feature matrix and a second feature matrix representing the first fusion feature, represents the activation function, Both represent learnable matrices, is the sampling information, represents the second fusion feature, represents the sample fusion feature, Indicates the sample morphology information; is the feature fusion function, specifically:

[0105]

[0106] in, represents the eigenvector as the independent variable, represents the learnable parameters, Represents the activation function.

[0107] Learnable matrices and learnable parameters refer to matrices and parameters whose values ​​can be adjusted by the loss function during model training.

[0108] The first fusion feature and the second fusion feature are intermediate features generated based on the above formula in the process of generating the sample fusion feature.

[0109] In one embodiment of the present invention, after the first neural network model obtains the output image, the loss of the first neural network model is calculated in the following manner, and the model parameters of the first neural network model are adjusted based on the loss:

[0110]

[0111] Among them, Loss represents the loss, n represents the total number of rows of pixel values ​​of the output image, and m represents the total number of columns of pixel values ​​of the output image. represents the pixel value of the i-th row and j-th column in the output image, represents the pixel value of the i-th row and j-th column in the sample grain image, denote the first characteristic matrix and the second characteristic matrix respectively.

[0112] The first term on the right side of the equal sign is the MSE function, and the second term is the KL (Kullback-Leibler) divergence.

[0113] The way to adjust the model parameters based on the loss can be to update the learnable matrix, learnable parameters, etc. in the model based on back propagation.

[0114] In one embodiment of the present invention, the training process using the above loss is as follows: Figure 2 As shown, x is a sample grain image, r is the above-mentioned output image, s is the sample morphological information, x and s are input into the first neural network model including CFC-VAE to obtain the output image r, and the loss Loss is calculated based on x and r.

[0115] The rows and columns of the output image and the sample grain image are determined based on the number of pixels of the image. For example, if the resolution of the output image is 400*300, that is, 400 pixels wide and 300 pixels high, the output image has 300 rows and 400 columns of pixel values.

[0116] Figure 4 The schematic diagram of the structure of the first neural network model, where x represents the image of broken grains, x is input into the image encoder, and , that is, the sample image feature, s is the morphological information; input linear encoder to get , that is, the sample morphological characteristics.

[0117] , They are the hidden layer features output by the image encoder and the linear encoder respectively. After the feature fusion of the feature fusion module F, we get , the first fusion feature and the first feature matrix , the second characteristic matrix Multiply and fuse the sampling information of the normal distribution N (0, 1) to obtain the second fusion feature z. The calculation process is shown in the above formula. z and s are fused again through the feature fusion module F to obtain the sample fusion feature , and then decoded by the decoder to obtain the reconstructed image, which is the output image r mentioned above.

[0118] Among them, the structure of the image encoder is as follows Figure 5 As shown in the figure, it consists of multiple convolutional layers (Conv), batch normalization layers (BN), maximum pooling layers (Maxpool), and ResBlock modules and GeM modules containing residual connections.

[0119] The CBR module is a combination of a convolutional layer, a batch normalization layer, and a ReLU activation function layer.

[0120] The Input in the figure is the image input to the image encoder.

[0121] The expression of the GeM module in the figure is:

[0122]

[0123] in, represents the features input by the image encoder, GAP represents the global pooling operation, is a learnable parameter, and f is a feature map of the input GeM module. As described above, the embodiment of the present invention constructs a cross-fusion controllable-variational autoencoder, which can accurately synthesize feature maps based on input image information.

[0124] In one embodiment of the present invention, obtaining a filled defective grain image according to a defect type set by a user and an image to be filled includes:

[0125] Input the defect type and the image to be filled into a pre-trained defect filling model to obtain a defect grain image output by the defect filling model;

[0126] Among them, the defect filling model is: a model obtained by training the second neural network using a sample to-be-filled image provided with label information containing the defect type, and the defect filling model is used to fill in the grain image according to the defect type and output the filled image.

[0127] The second neural network model may be a network model having a similar structure to the first neural network model. For example, the second neural network model may also be a neural network model including CFC-VAE. The difference is that the input images during training are different.

[0128] The second neural network model inputs the sample image to be filled, but when calculating the loss, the loss between the sample grain image before removing the sub-image of part of the area and the output image is still used.

[0129] The sample image to be filled can be obtained by manually removing a sub-image of a part of the area on the sample grain image containing normal grains.

[0130] The training process of the second neural network model is as follows Figure 6 As shown, the worm-eaten S is the label information, and S can specifically include different defect types. is the sample image to be filled, CFC-CAE represents the second neural network model, and the output image is , and calculate the sample grain image and output image The loss between.

[0131] After training is completed, the model inference process is as follows Figure 7 As shown in Figure 2. CFC-VAE represents the defect filling model. The artificially set defect type worm-eaten S is input into the defect filling model, and the image of the defect with the worm-eaten S type after filling output by the defect filling model can be obtained. .

[0132] The following example illustrates a specific implementation of obtaining the defect area in step S103.

[0133] The normal grain contour in the normal grain image is obtained; the normal grain contour is expanded to obtain a first extended image including the normal grain contour; the normal grain contour is eroded to obtain a target grain contour; candidate points are determined from the target grain contour, and the position of the defect contour in the second extended image is determined based on the position of the candidate points; a second extended image including the defect contour is generated according to the determined position, and the second extended image has the same size as the first extended image; an intersection image area of ​​an image area included in the normal grain contour in the first extended image and an image area included in the defect contour in the second extended image is determined, and a defect area is smeared out in the normal grain image based on the position of the intersection image area.

[0134] The intersection image area is the intersection of the image area included in the defect contour and the image area included in the normal grain contour. Since the first extended image and the second extended image have the same size, the position of the intersection image area can be determined based on the position of the pixel points included in the intersection image area in the first extended image or the second extended image.

[0135] like Figure 8 As shown in the figure, the normal grain outline is M. The normal grain outline is expanded and the background is added outside the normal grain outline area to obtain M p . p The normal grain contour in the image is eroded to obtain the image M containing the target grain contour e .

[0136] From M e Get the candidate point Pos from M s Superimposed on M p The second extended image M is obtained on the background image of the same size sp . M s Defect outline.

[0137] Calculate M p and M sp The intersection of the foreground and the background gives M i , according to M i The defect area on the normal grain image I t Apply it in and get I s .

[0138] Image erosion processing reduces the area of ​​the outline to ensure that the generated M sp With M p There is an intersection. In this way, part of the defect contour located based on the candidate points must be located in the area occupied by the normal grain contour, ensuring that the defect is added to the grain.

[0139] In one embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned image generation methods is implemented.

[0140] In one embodiment of the present invention, an electronic device is provided, comprising: a memory and a processor; a computer program is stored in the memory, and when the computer program is executed by the processor, any of the above-mentioned image generation methods is implemented.

[0141] Fig. 9 It is a structural block diagram of an electronic device according to an embodiment of the present invention.

[0142] like Fig. 9 As shown, the electronic device 900 includes: a processor 901 and a memory 903. The processor 901 and the memory 903 are connected, such as through a bus 902. Optionally, the electronic device 900 may also include a transceiver 904. It should be noted that in actual applications, the transceiver 904 is not limited to one, and the structure of the electronic device 900 does not constitute a limitation on the embodiments of the present invention.

[0143] The processor 901 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. The processor 901 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0144] The bus 902 may include a path to transmit information between the above components. The bus 902 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 902 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0145] The memory 903 is used to store a computer program corresponding to the image generation method of the above embodiment of the present invention, and the computer program is controlled and executed by the processor 901. The processor 901 is used to execute the computer program stored in the memory 903 to implement the content shown in the above method embodiment.

[0146] Among them, the electronic device 900 includes but is not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The electronic device 900 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0147] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0148] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0149] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0150] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0151] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0152] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0153] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0154] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An image generation method, characterized in that: The image generation method comprises: Get a normal grain image without defects; Acquiring morphological information representing the contour of the broken grain kernels and generating a defect contour based on the morphological information; the morphological information is obtained by extracting the contour of the grain from a contour image including the contour of the broken grain kernels; Based on the defect outline, a defect area is painted on the normal grain image to obtain an image to be filled; Obtaining a filled defective grain image according to the defect type set by the user and the image to be filled, and generating annotation information of the defective grain image based on the defect type; The step of acquiring morphological information representing the contour of the broken grain particles and generating a defect contour based on the morphological information comprises: Sampling random variables that obey normal distribution to obtain sampling information; Inputting the morphological information and the sampling information into a pre-trained defect contour generation model to obtain a defect contour output by the defect contour generation model; Wherein, the defect contour generation model is a model obtained by training a preset first neural network model using a sample grain image containing broken grain grains, and the defect contour generation model is used to obtain an output image according to the grain contour in the sample grain image; The first neural network model includes an image encoder, a linear encoder, a feature fusion module and a decoder; The image encoder is used to extract sample image features of the sample grain image; The linear encoder is used to extract sample morphological features of sample morphological information of grain contours in the sample grain image; The feature fusion module is used to perform feature fusion on the sample image feature and the sample morphological feature to obtain a sample fusion feature; The decoder is used to obtain an output image according to the sample fusion features; The feature fusion module obtains sample fusion features in the following manner: in, represents the first fusion feature, represents the sample morphological characteristics, represents the sample image feature, A first feature matrix and a second feature matrix representing the first fusion feature, represents the activation function, Both represent learnable matrices, is the sampling information, represents the second fusion feature, represents the sample fusion feature, Indicates the sample morphology information; is the feature fusion function, specifically: in, represents the eigenvector as the independent variable, represents the learnable parameters, Represents the activation function.

2. The method according to claim 1, characterized in that The step of acquiring morphological information representing the contour of the broken grain particles and generating a defect contour based on the morphological information comprises: Sampling random variables that obey normal distribution to obtain sampling information; Performing feature fusion on the morphological features representing the morphological information and the sampling features representing the sampling information to obtain fused features; A defect contour is obtained based on the fused features.

3. The method according to claim 1, characterized in that: The method further comprises: After the first neural network model obtains the output image, the loss of the first neural network model is calculated in the following manner, and the model parameters of the first neural network model are adjusted based on the loss: Wherein, Loss represents the loss, n represents the total number of rows of pixel values ​​of the output image, and m represents the total number of columns of pixel values ​​of the output image. represents the pixel value of the i-th row and j-th column in the output image, represents the pixel value of the i-th row and j-th column in the sample grain image, represent the first characteristic matrix and the second characteristic matrix respectively.

4. The method according to claim 1, characterized in that The step of obtaining a filled defective grain image according to the defect type set by the user and the image to be filled includes: Inputting the defect type and the image to be filled into a pre-trained defect filling model to obtain a defect grain image output by the defect filling model; Among them, the defect filling model is: a model obtained by training the second neural network using a sample to-be-filled image provided with label information containing the defect type, and the defect filling model is used to fill in the grain image according to the defect type and output the filled image.

5. The method according to claim 1, characterized in that: The step of painting out a defective area on the normal grain image based on the defect contour comprises: Acquire a normal grain contour in the normal grain image; Performing image expansion processing on the normal grain outline to obtain a first expanded image including the normal grain outline; Performing image corrosion processing on the normal grain contour to obtain a target grain contour; determining candidate points from the target grain contour, and determining the position of the defect contour in the second extended image based on the positions of the candidate points; generating a second extended image including a defect outline according to the determined position, wherein the second extended image has the same size as the first extended image; An intersection image area of ​​an image area included in the normal grain contour in the first extended image and an image area included in the defect contour in the second extended image is determined, and a defect area is smeared out in the normal grain image based on the position of the intersection image area.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image generating method according to any one of claims 1 to 5 is implemented.

7. An electronic device, characterized in that: include: Memory, processor; The memory stores a computer program, and when the computer program is executed by the processor, the image generation method according to any one of claims 1 to 5 is implemented.

Images