Generating training data in synthetic manner

Abstract

The invention relates to image processing. There is provided a computer-implemented method for generating synthetic training data usable for training a data-driven model for analyzing a surface image of a physical product comprising at least one object, the method comprising: a) providing (210) image data comprising: a set of object image data, the object image data set comprises a plurality of object images of the at least one object, and the at least one object image is associated with a label which can be used for annotating the content of the object image; and a background image representing a background of the surface image of the physical product; b) generating (220) a composite object image dataset from the object image dataset, where the composite object image dataset comprises a plurality of composite object images of the at least one object, at least one composite object image being associated with a tag; and c) generating (230) a plurality of first synthetic training data samples, where each first synthetic training data sample is generated by selecting one or more object images from the synthetic object image dataset and by rendering the selected one or more object images at one or more locations on the background image. The computer-implemented method can be used to improve computer vision technology for application in the agricultural technical field and production environment.

Description

Technical Field

[0001] The present invention generally relates to image processing or computer vision techniques. More specifically, the present invention relates to the following: a computer-implemented method and a synthetic training data generating apparatus for generating synthetic training data that can be used to train a data-driven model for analyzing a surface image of a physical product including at least one object; a computer-implemented method and an image analysis apparatus for analyzing a surface image of a physical product; a method and a system for controlling a production process of a physical product; and a computer program product. Background Art

[0002] In the field of agricultural technology, there is an ongoing effort to make farming or tillage operations more sustainable. Precision farming or precision agriculture is seen as one of the ways to achieve better sustainability and reduce environmental impact. This relies on reliable local detection of plant damage in the field. In a production environment, image-based monitoring and / or control of production processes also relies on reliable detection of defects and their precise localization. Summary of the invention

[0003] Therefore, there may be a need to improve computer vision technology so that the right amount of chemical product can be applied accurately enough.Further, there may be a need to improve computer vision for application in a production environment.

[0004] The objects of the invention are achieved by the subject matter of the independent claims, wherein further embodiments are contained in the dependent claims. It should be noted that the aspects of the invention described below also apply to the following: a computer-implemented method and a synthetic training data generating apparatus for generating synthetic training data that can be used to train a data-driven model for analyzing a surface image of a physical product comprising at least one object; a computer-implemented method and an image analysis apparatus for analyzing a surface image of a physical product; a method and a system for controlling a production process of a physical product; and a computer program product.

[0005] In a first aspect of the present disclosure, there is provided a computer-implemented method for generating synthetic training data that can be used to train a data-driven model for recognizing individual objects in an image of a surface of a physical product that includes at least one object, the method comprising:

[0006] a) providing (210) image data, the image data comprising:

[0007] - an object image dataset, the object image dataset comprising a plurality of object images of the at least one object, the at least one object image being associated with a tag that can be used to annotate the content of the object image, wherein the tag comprises a characteristic value describing a property of the at least one object in the at least one object image and indicating a damage condition of the at least one object in the at least one object image; and

[0008] - a background image, the background image representing the background of the surface image of the physical product;

[0009] b) generating (220) a synthetic object image dataset from the object image dataset, wherein the synthetic object image dataset comprises a plurality of synthetic object images of the at least one object, at least one synthetic object image being associated with a label; and

[0010] c) generating (230) a plurality of first synthetic training data samples, wherein each first synthetic training data sample is generated by selecting one or more object images from the synthetic object image dataset and by drawing the selected one or more object images at one or more locations on the background image.

[0011] For many use cases, certain objects need to be recognized and positioned in front of background objects. Examples include identifying insects on leaves, spores or insects in a petri dish, weeds in a field, and particles on a substrate during production. In order to use deep learning methods to recognize such objects, large training datasets are required. Compared to the development of classical image processing routines, the main effort (and cost) is not the software development cost, but the effort to label the data. Actual deep learning algorithms are adaptive to the training data and require much less development and tuning effort. In some cases, classical algorithms cannot even solve complex image recognition problems. The performance of deep learning methods depends mainly on the availability of large, high-quality training datasets.

[0012] To this end, a computer-implemented method and synthetic training data generation apparatus for synthetically generating training data is provided. The synthetic training data can then be used to train models that work on real data. Generating data in a synthetic manner can potentially address the two mentioned challenges - effort and the possibility of defects in data collection. These algorithms can generate well-distributed datasets with many samples while maintaining real data labels.

[0013] This computer-implemented method includes several steps. First, image data including an object image and one or more background images are provided. For example, a user can provide several background images (e.g., empty petri dishes, leaves, soil, etc.) and object images separated from their backgrounds (e.g., individual spores, insects or eggs, particles). Using rules and parameters, such as the number of objects in the image, whether objects can overlap or touch, how far or narrow the objects that appear should be, etc., the actual distribution of objects in the generated image will be determined. In addition, image enhancement techniques will be used to create differences between background and various object images (about size, shape, orientation, etc.) so that they reflect real variability. In addition, a generation deep learning model (such as a generation adversarial network GAN, Wasserstein GAN, and non-adversarial image synthesis) can be used to further create new and natural appearance variants of the input image. After setting the parameters, actual data set generation can be started. Results, natural-looking artificial combinations, and automatically generated accurate label data can be exported in standard formats that can be used for various image tasks, such as regression, classification, object detection, and object segmentation. The data-driven model can then be evaluated directly, or manual fine-tuning of the data-driven model can be performed based on the generated data set.

[0014] The label data includes a characteristic describing at least one object in at least one object image and a characteristic value indicating the damage condition of the at least one object in the at least one object image. For example, the characteristic may include one or more of the following: an object classification that can be used to identify the at least one object, a coordinate list of the at least one object, a segmentation mask of the at least one object, etc. The characteristic value may include a characteristic value indicating plant damage and / or a characteristic value indicating a deviation from an industrial product standard. The characteristic value may be a numerical value (in the form of a real number), such as a percentage or an absolute value, or the characteristic value may be a classifier (a binary classifier indicating the presence or absence of a specific characteristic, a multi-class classifier). For example, each crop in a field can be evaluated by a disease prediction algorithm and scored from 0% (healthy) to 100% (death due to disease).

[0015] Various types of data-driven models can be trained using synthetic training data. In some examples, the data-driven model can be a classifier, such as for indicating whether a product meets predefined quality standards. In some examples, the data-driven model can be a regression model, such as for determining the number of defects in a product image. In some examples, the data-driven model can be a model for object detection and classification, such as for detecting and classifying defects in a product image. In some examples, the data-driven model can be a model for instance segmentation, such as for determining the class and / or object to which each pixel in an image belongs.

[0016] The data-driven model can be a machine learning algorithm. The machine learning algorithm can be a deep learning algorithm, such as a deep neural network, a convolutional deep neural network, a deep belief network, a recurrent neural network, etc.

[0017] In some examples, data-driven models can be used in the field of agricultural technology. In some examples, data-driven models can be algorithms for identifying pests (e.g., MYZUS, APHIGO, BEMISA adults, BEMISA Stadia, FRANOC stadia) in field trials. In some examples, data-driven models can be algorithms for segmenting the main leaf shape of crops (e.g., tomatoes, peppers, grapes, apple trees). In some examples, data-driven models can be algorithms for identifying weeds.

[0018] In some examples, the data-driven model can be a model for use in a production environment. In one example, an artificial training data set is generated for developing segmentation methods for overlapping objects (particularly cathode active material particles), which in some cases can also include classifying individual particles. Enhancing training data based on the synthesis and controlled placement of a greater number of previously carefully segmented objects can be useful for reducing the number of images that need to be labeled in a time-consuming manner. In some examples, the data-driven model can be an algorithm for object detection and classification of spores. In some examples, the data-driven model can be an algorithm for cell detection. In some examples, the data-driven model can be an algorithm for detecting cell fluorescence.

[0019] Will refer to Figure 3 Exemplary embodiments of computer-implemented methods are described with reference to the examples shown in FIG.

[0020] According to an embodiment of the present invention, the at least one object includes a plurality of objects, and at least two objects of the plurality of objects are associated with tags including different characteristic values.

[0021] For example, the physical product may be a field having a plurality of lettuces therein. At least two of the lettuces in the field may have different damage conditions caused by, for example, a disease. For example, one lettuce may be assigned a score of 0% (healthy) while another lettuce may be assigned a score of 100% (disease). The computer-implemented method is capable of not only identifying individual lettuces in the field, but also determining the damage condition (or health) of the individual lettuces.

[0022] According to an embodiment of the present disclosure, in step b), a generative model is used to generate a synthetic object image dataset.

[0023] Examples of the generative model may include, but are not limited to, GAN, variational autoencoder (VAE), and autoregressive models such as PixelRNN.

[0024] According to an embodiment of the present invention, the generative model includes a conditional generative adversarial network (cGAN).

[0025] Since current datasets of individual object images are labeled and these labels are required in the workflow, the application of GANs must maintain and replicate such labels. It is possible to use conventional GANs to expand the dataset. However, these labels may be lost. Conditional GANs (cGANs) may be a way to overcome this obstacle. This is because cGANs allow the inclusion of constraints, such as feature values ​​that indicate the damage condition of at least one object. In addition to the input images, the labels are fed into the network during training. In this way, the network learns to generate images of corresponding labels. After training the model, the model is able to generate large datasets of object images of each category.

[0026] Additionally, data from field trials often face challenges, especially the small number of samples and unbalanced distribution. Relying solely on field samples may lead to model overfitting, resulting in performance loss during test time. cGANs may be a way to overcome this obstacle, as cGANs can be used to create unbiased and balanced datasets in which undamaged objects (e.g., healthy lettuce) may be roughly the same as damaged objects (unhealthy lettuce). The performance of data-driven models can be improved when the number of synthetic samples is large and the distribution is balanced.

[0027] According to an embodiment of the present disclosure, in step c), the selected one or more object images are drawn on the background image according to a rule derived from one or more surface image samples of the physical product.

[0028] In some examples, the rules can be specific to each object. Examples of rules can include, but are not limited to, the number of objects in an image, whether objects can overlap or touch, how far apart or narrow objects should appear, regular placement of objects (e.g., lettuce) or random placement of objects (e.g., weeds, cathode active material particles in an image of battery material, etc.).

[0029] In some examples, the rules can be defined by the user. For example, for a regular arrangement of objects such as lettuces in a field, the user can define the number of lettuces in the image, the distance between the lettuces, etc.

[0030] In some examples, rules may be derived from one or more sample images, for example using a rule-based machine learning algorithm.

[0031] According to an embodiment of the present disclosure, step c) further includes the step of generating a plurality of second synthetic training data samples from a plurality of first synthetic training data samples using an image-to-image conversion model, wherein the image-to-image conversion model has been trained to generate synthetic surface images that are closer to real surface images of physical products.

[0032] The second synthetic training data sample can also be referred to as optimized synthetic training data. In this way, the large synthetic dataset retains its composition and balance, while the style is modified to a more realistic appearance. Neither the discriminative model nor humans can distinguish this result.

[0033] According to an embodiment of the present disclosure, the image-to-image conversion model includes an image-to-image generative adversarial network.

[0034] According to an embodiment of the present disclosure, the characteristics include one or more of the following:

[0035] - Annotations that can be used to classify plant diseases in images of plants;

[0036] - Annotations that can be used to classify cathode active material particles in images of battery materials;

[0037] - annotations that can be used to classify cells in images of biological material;

[0038] - annotations that can be used to classify insects on leaves; and

[0039] -Annotations that can be used to classify defects in coatings.

[0040] According to an embodiment of the present disclosure, the characteristic value includes one or more of the following:

[0041] - a characteristic value indicative of plant damage; and

[0042] - A characteristic value that indicates a deviation from an industrial product standard.

[0043] According to an embodiment of the present disclosure, the characteristic value is provided in the form of a damage percentage, which can preferably be used to determine the amount of treatment to be applied to the physical product.

[0044] According to an embodiment of the present disclosure, the method further comprises the step of providing a user interface, the user interface allowing a user to provide image data.

[0045] This will be discussed below and with particular reference to Figure 2 This is explained by the example shown in . Figure 4A and Figure 4B An exemplary user interface is shown in FIG.

[0046] According to an embodiment of the present invention, step c) further comprises providing a label for one or more first synthetic training data samples of the plurality of first synthetic training data samples.

[0047] As an example, an annotation file including tags is provided. For example, the annotation file may include a JSON-based file, which is, for example, in COCO (Common Objects in Context) format. The annotation file may be provided by a user or obtained from a database.

[0048] In a second aspect of the present disclosure, a computer-implemented method for analyzing a surface image of a physical product, the method comprising:

[0049] - providing an image of the surface of the physical product; and

[0050] - providing a data-driven model for identifying at least one object on the provided surface image of the physical product and for generating a label that can be used to annotate at least one detected object, wherein the label includes a characteristic value describing the at least one object on the provided surface image of the physical product and indicating a damage condition of the at least one object on the provided surface image of the physical product, wherein the label is preferably usable for monitoring and / or controlling a production process of the physical product,

[0051] Wherein, the data-driven model has been trained on a training dataset, which includes synthetic training data generated according to the first aspect and any associated examples.

[0052] This will be discussed below and with particular reference to Fig.16 The example shown in is explained in detail.

[0053] In a third aspect of the present disclosure, there is provided a method for controlling a production process of a physical product, the method comprising:

[0054] - provide a surface image of the physical product;

[0055] - providing a data-driven model for identifying at least one object on a provided surface image of the physical product and for generating a label that can be used to annotate the at least one object; wherein the label includes a characteristic value describing the at least one object on the provided surface image of the physical product and indicating a damage condition of the at least one object on the provided surface image of the physical product, wherein the data-driven model has been trained on a training data set, the training data set comprising synthetic training data generated according to the first aspect and any associated examples; and

[0056] - generating control data based on the generated tag, the control data comprising instructions for controlling an object modifier to perform an operation acting on at least one detected object.

[0057] In embodiments, an object modifier may include any device configured to perform measures for modifying an object.

[0058] In the case of a farmland, the object modifier may be a plant treatment device configured to apply a crop protection product to the farmland. The plant treatment device may be configured to traverse the farmland. The plant treatment device may be a ground or aerial vehicle, such as a tractor-mounted vehicle, a self-propelled sprayer, a tracked vehicle, a robot, an aircraft, an unmanned aerial vehicle (UAV), a drone, etc.

[0059] In the case of a production environment, the type of object modifier may depend on the application scenario. For example, for particles on a conveyor belt, the object modifier may be a blower that removes defective particles from the conveyor belt.

[0060] This will be discussed below and with particular reference to Fig.16 This is explained by the example shown in .

[0061] In a fourth aspect of the present application, a synthetic training data generating device for generating synthetic training data is provided, wherein the synthetic training data can be used to train a data-driven model for analyzing a surface image of a physical product including at least one object, wherein the synthetic training data generating device comprises one or more processors configured to execute the steps of the method of the first aspect and any associated examples.

[0062] This will be discussed below and with particular reference to Figure 1 and Figure 2 The example shown in is explained in detail.

[0063] In a fifth aspect of the present disclosure, an image analysis device for analyzing a surface image of a physical product, the synthetic training data generation device comprises one or more processors, and the one or more processors are configured to execute the steps of the method of the second aspect.

[0064] This will be discussed below and with particular reference to Figure 2 The example shown in is explained in detail.

[0065] In a sixth aspect of the present disclosure, there is provided a system for controlling a production process of a physical product, the system comprising:

[0066] - a camera configured to capture an image of a surface of the physical product;

[0067] - an apparatus according to the fifth aspect, the apparatus being configured to identify at least one object on the captured surface image of the physical product and for generating a label that can be used to annotate the at least one detected object; and

[0068] - an object modifier configured to perform an operation on the at least one detected object based on the generated label.

[0069] This will be discussed below and with particular reference to Fig.17 and Fig.18 The examples shown in FIG.

[0070] In another aspect of the present disclosure, there is provided a computer program product comprising instructions, which, when executed by a processing unit, cause the processing unit to perform the steps of the method of the first aspect or the method of the second aspect.

[0071] In an embodiment, the image or image data may include any data or electromagnetic radiation imaging that can be obtained or generated by a camera, an image sensor, multiple cameras, or multiple image sensors. The image data is not limited to the visible spectrum range and two-dimensional. Examples of images may include, but are not limited to, grayscale images, near infrared (NIR) images, RGB images, multispectral images, and hyperspectral images. The frame rate of the camera may be in the range of 0.3 Hz to 48 Hz, but is not limited thereto.

[0072] In an embodiment, farmland may include farmland to be processed. Farmland may be any plant or crop growing area, such as a cultivated field, a greenhouse, etc. Plants may be crops, weeds, volunteer plants, crops from a previous growing season, beneficial plants, or any other plants present on the farmland. The farmland may be identified by its geographic location or geo-referenced location data. The farmland may be further specified using reference coordinates, size, and / or shape.

[0073] In an embodiment, damage may include any deviation of a characteristic value from a standard characteristic value.Examples of damage may include plant damage and industrial product damage.

[0074] In embodiments, plant damage may include any deviation from the normal physiological function of a plant that is harmful to the plant, including but not limited to plant diseases (i.e., deviations from the normal physiological function of a plant) caused by:

[0075] a) fungi ("fungal plant diseases"),

[0076] b) Bacteria ("bacterial plant diseases")

[0077] c) viruses ("viral plant diseases"),

[0078] d) insect feeding damage,

[0079] e) Plant nutrient deficiencies,

[0080] f) heat stress, such as temperature conditions above 30°C,

[0081] g) Cold stress, such as temperature conditions below 10°C,

[0082] h) Drought stress,

[0083] i) excessive exposure to the sun, such as exposure to the sun resulting in signs of burning, sunburn or similar signs of exposure,

[0084] j) Acidic or alkaline pH conditions in the soil, pH values ​​below pH 5 and / or pH values ​​above pH 9,

[0085] k) Salt stress, such as soil salinity,

[0086] l) Chemical pollution, such as heavy metal pollution, and / or

[0087] m) Adverse effects of fertilizers or crop protection, such as herbicide damage

[0088] n) Damaging weather conditions, such as hail, frost, and damaging winds.

[0089] In an embodiment, the input unit may include any item or element forming a boundary configured for transmitting information. In particular, the input unit may be configured to transmit information to a computing device, such as a computer, such as to receive information. The input unit is preferably a separate unit configured to receive information or transmit information to a computing device, such as to transmit information to one or more of the following: an interface, in particular a web interface and / or a data interface; a keyboard; a terminal; a touch screen, or any other input device deemed appropriate by the skilled person. More preferably, the input unit includes or is configured to transmit or exchange information as specified below as a data interface.

[0090] In an embodiment, the output unit may include any item or element forming a boundary configured for transmitting information. In particular, the output unit may be configured to transmit information from a computing device (e.g., a computer), such as to send or output the information to another device (e.g., a control unit) that controls and / or monitors the production process of the produced combination. The output unit is preferably a separate unit configured to output or transmit information from a computing device, such as one or more of the following: an interface, in particular a web interface and / or a data interface; a screen, a printer or a touch screen, or any other output device deemed appropriate by the technician. More preferably, the output unit includes or is configured to transmit or exchange a data interface as specified below.

[0091] Preferably, the input unit and the output unit are configured as at least one or at least two separate data interfaces; that is, preferably, a data transmission connection is provided, such as wireless transmission, Internet transmission, Bluetooth, NFC, inductive coupling, etc. As an example, the data transmission connection may be or may include at least one port, the at least one port including one or more of a network or Internet port, a USB port, and a disk drive. The input unit and / or the output unit may also be at least one web interface.

[0092] In an embodiment, the processing unit may include, but is not limited to, any logic circuit system configured to perform the operation of a computer or system, and / or generally includes a device or unit thereof configured to perform a calculation or logical operation. The processing unit may include at least one processor. In particular, the processing unit may be configured to process basic instructions that drive the computer or system. As an example, the processing unit may include at least one arithmetic logic unit (ALU), at least one floating point unit (FPU) (such as a mathematical coprocessor or a digital coprocessor), a plurality of registers and memories (such as cache memories). In particular, the processing unit may be a multi-core processor. The processing unit may include a central processing unit (CPU) and / or one or more graphics processing units (GPUs) and / or one or more application specific integrated circuits (ASICs) and / or one or more tensor processing units (TPUs) and / or one or more field programmable gate arrays (FPGAs), etc. The processing unit may be configured to preprocess the input data. Preprocessing may include performing at least one filtering process on the input data that meets at least one quality standard. For example, the input data may be filtered to remove missing variables.

[0093] It should be understood that all combinations of the aforementioned concepts and the additional concepts discussed in more detail below (provided that these concepts are not mutually contradictory) are considered to be part of the inventive subject matter disclosed herein. In particular, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the inventive subject matter disclosed herein.

[0094] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] In the drawings, like reference numerals generally refer to the same parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention.

[0096] Figure 1 A block diagram of an exemplary synthetic training data generation apparatus is shown.

[0097] Figure 2An exemplary computer network environment for implementing embodiments of the present disclosure is schematically depicted.

[0098] Figure 3 A flowchart describing a computer-implemented method for generating synthetic training data is presented.

[0099] Figure 4A An exemplary user interface for uploading an image of an object is shown.

[0100] Figure 4B An exemplary user interface for uploading a background image is shown.

[0101] Figure 5 Images showing different levels of data.

[0102] Figure 6 The four steps of the pipeline are shown.

[0103] Figure 7 The background subtraction step is shown.

[0104] Figure 8 The resulting image of the cGAN is shown.

[0105] Fig. 9 An exemplary implementation of a pasting algorithm is presented.

[0106] Fig.10 An exemplary implementation of image-to-image conversion using CUT is presented.

[0107] Fig.11 Multi-layer, block-wise contrastive losses are shown.

[0108] Fig.12 Domain A and domain B are shown for image-to-image translation GAN training.

[0109] Fig.13 An example of generating data on a continuous scale from low to high estimates is shown.

[0110] Fig.14 A comparison of the synthesized image and its corresponding optimized image is shown.

[0111] Fig.15 A flowchart describing a computer-implemented method for training a data-driven model is presented.

[0112] Fig.16 A flow chart describing the field management approach is shown.

[0113] Fig.17 An exemplary system for field management is presented.

[0114] Fig.18 An exemplary system for identifying damage conditions of industrial products is shown. DETAILED DESCRIPTION

[0115] In many industrial settings, labeled data is very expensive and difficult to obtain. For example, unmanned aerial vehicles (UAVs) or drones are used to monitor agricultural fields from the air. They are useful tools for collecting images of large parts of the field with good image quality. In order to use algorithms on images from these drones, labeled data captured from the air is needed. However, drone setup and drone flights themselves are expensive, and currently the images are manually annotated by field workers. In addition, there is no guarantee that the captured data has the required variety and distribution, and there is always a risk of human annotation errors. In another example, cameras are used in industrial manufacturing to monitor the production process. The acquired images can be provided to algorithms for detecting defects and accurately locating them. However, image data of defective products can often be difficult to obtain in large quantities, and can be very slow and expensive to collect. This can pose a challenge to the training of complex nonlinear machine learning models that require large amounts of data.

[0116] To this end, a computer-implemented method and a synthetic training data generation device are proposed for synthetically generating training data to expand a training data set of a data-driven model, which can be a model for performing image tasks (such as regression, classification, object detection and / or object segmentation) on surface images of physical objects.

[0117] Figure 1 A block diagram of an exemplary synthetic training data generating apparatus 10 is shown. The synthetic training data generating apparatus 10 includes an input unit 12, a processing unit 14, and an output unit 16.

[0118] In general, the synthetic training data generating apparatus 10 may include various physical and / or logical components for transmitting and manipulating information, which may be implemented as hardware components (e.g., computing devices, processors, logic devices), executable computer program instructions (e.g., firmware, software) to be executed by various hardware components, or any combination thereof, as required by a given set of design parameters or performance constraints. Figure 1 A limited number of components may be shown as an example, but it is understood that a greater or lesser number of components may be employed for a given implementation.

[0119] In some embodiments, the synthetic training data generating apparatus 10 may be embodied as a device or apparatus or in a device or apparatus, such as a server, a workstation, or a mobile device. The synthetic training data generating apparatus 10 may include one or more microprocessors or computer processors that execute appropriate software. The processing unit 14 of the synthetic training data generating apparatus 10 may be embodied by one or more of these processors. The software may have been downloaded and / or stored in a corresponding memory, for example, a volatile memory such as RAM or a non-volatile memory such as flash memory. The software may include instructions that configure one or more processors to perform the functions described herein.

[0120] Note that the synthetic training data generating apparatus 10 may be implemented with or without a processor, and may also be implemented as a combination of dedicated hardware that performs certain functions and a processor (e.g., one or more programmed microprocessors and associated circuitry) that performs other functions. For example, the functional units of the synthetic training data generating apparatus 10 (e.g., input unit 12, one or more processing units 14, and output unit 16) may be implemented in a device or apparatus in the form of programmable logic (e.g., as a field programmable gate array (FPGA)). In general, each functional unit of the synthetic training data generating apparatus may be implemented in the form of a circuit.

[0121] In some embodiments, the synthetic training data generating device 10 can also be implemented in a distributed manner. For example, some or all units of the synthetic training data generating device 10 can be arranged as separate modules in a distributed architecture and connected in a suitable communication network, such as a 3rd Generation Partnership Project (3GPP) network, a Long Term Evolution (LTE) network, the Internet, a LAN (local area network), a wireless LAN (local area network), a WAN (wide area network), etc.

[0122] The processing unit(s) 14 may execute instructions to perform the methods described herein, which will be referred to herein. Figure 3 The examples shown in the are explained in detail.

[0123] Figure 2 An exemplary computer network environment 100 for implementing embodiments of the present disclosure is schematically depicted. As shown, the system 100 includes a synthetic training data generation device 10, a data storage device 20, a decision support system 30, one or more electronic communication devices 50, and a network 60.

[0124] The synthetic training data generating device 10 may have Figure 1 The exemplary training data generating apparatus 10 of the illustrated example may be, for example, a server providing a web service to (a plurality of) electronic communication devices 50. Figure 2As shown in , the exemplary training data generation device 10 may include an input unit 12, one or more processors 14, one or more memory elements 18, an output unit 16, and other components for facilitating or otherwise supporting the generation of synthetic training data. The synthetic training data generation device 10 may provide training data for use during training of a data-driven model.

[0125] The data storage device 20 is configured to store training data including synthetic training data generated by the synthetic training data generating device 10 .

[0126] The decision support system 30 may include a model trainer 32 , an image analysis device 40 , and a control file generation device 34 .

[0127] A model trainer 32 is provided to facilitate or otherwise support the training of a data-driven model. Figure 2 Not shown, but model trainer 32 may include one or more processors, one or more memory elements, and other components for implementing model training logic.

[0128] The image analysis device 40 may include a processor 44 and a memory 48. The processor 44 may execute a machine learning algorithm (i.e., a data-driven model) by executing code 49 that embodies the algorithm. In some examples, the processor 44 may access and execute code for potentially multiple different machine learning algorithms, and may even execute multiple algorithms in parallel in some cases. The processor 44 may include any processor or processing device, such as a microprocessor, an embedded processor, a digital signal processor (DSP), a network processor, a handheld processor, an application processor, a coprocessor, a system on a chip (SOC), or other device for executing code 49. The code 49 embodying the machine learning algorithm may be stored in the memory 48, which may be local or remote from the processor 44. The memory 48 may be implemented as one or more of a shared memory, a system memory, a local processor memory, a cache memory, etc., and may be embodied as software, firmware, or a combination of software and firmware.

[0129] The control file generating device 34 may be configured to generate a control file based on the results generated by the image analyzing device 40. In an embodiment, a control file (also referred to as a configuration file) may include any binary file, data, signal, identifier, code, image, or any other machine-readable or machine-detectable element that can be used to control a machine or device.

[0130] Although in Figure 2A single decision support system 30 is shown as an example, but it should be understood that the functionality of the decision support system 30 (e.g., the model trainer 32, the image analysis device 40, and the control file generation device 34) can be distributed across multiple servers, which can be clustered, geographically distributed across the network 130, or any combination thereof.

[0131] The (multiple) electronic communication devices 50 may act as a terminal, a graphic display client, or other networked client of the synthetic training data generating apparatus 10 and the decision support system 30. The (multiple) electronic communication devices 50 may include an application configured to interface with a web service provided by the synthetic training data generating apparatus 10 and the decision support system 30. For example, a web browser application at the (multiple) electronic communication device 50 may support interfacing with a web server application at the synthetic training data generating apparatus 10 and the decision support system 30. Such a browser may use controls, plug-ins, or applets to support interfacing with the synthetic training data generating apparatus 10 and the decision support system 30. The (multiple) electronic communication devices 50 may use other customized programs, applications, or modules to interface with the synthetic training data generating apparatus 10 and the decision support system 30. The (multiple) electronic communication devices 50 may be a desktop computer, a laptop computer, a handheld device, a mobile device, a mobile phone, a server, a terminal, a thin client, or any other computerized device.

[0132] The network 60 can be any communication network capable of supporting the communication between the synthetic training data generating apparatus 10, the decision support system 30 and (multiple) electronic communication devices 50. The network 60 can be a wired network, a wireless network, an optical network, a radio network, a packet switching network, a circuit switching network, or any combination thereof. The network 60 can use any topology, and the links of the network 60 can support any networking technology, protocol or bandwidth, such as Ethernet, DSL, cable modem, ATM, SONET, MPLS, PSTN, POTS modem, PONS, HFC, satellite, ISDN, WiFi, WiMax, mobile cellular, any combination thereof, or any other data interconnection or networking mechanism. The network 60 can be an intranet, the Internet (or World Wide Web), a LAN, a WAN, a MAN, or any other network for interconnecting computers. In order to support high capacity and load, a distributed computing environment can be implemented by using networking technologies, which can include but are not limited to TCP / IP, RPC, RMI, HTTP, Web services (XML-RPC, JAX-RPC, SOAP, etc.).

[0133] It should be understood that, in addition to Figure 2In addition to the network environment shown in FIG. 1 , the synthetic training data generating apparatus 10 and the decision support system 30 can be combined into a single device. For example, the synthetic training data generating apparatus 10 can be embodied in the decision support system 30, for example, embodied as a module.

[0134] Figure 3 A flow chart describing a computer-implemented method 200 for generating synthetic training data that can be used to train a data-driven model for recognizing individual objects in a surface image of a physical product that includes at least one object. In some examples, the at least one object can include only one object. In some examples, the at least one object can include multiple objects, at least two of the multiple objects being associated with labels that include different characteristic values. Figure 2 The computer-implemented method 200 is described with reference to the system shown in FIG.

[0135] At box 210 (i.e., step a), the following image data is provided: an object image dataset and a background image. The object image dataset includes multiple object images of at least one object. At least one object image is associated with a tag that can be used to annotate the content of the object image. In some examples, each object image is associated with a corresponding tag. In some other examples, it is not necessary to label each object image. For example, only object images that will be identified later must be labeled. Additional unlabeled object images can make the picture more natural. The background image represents the background of the surface image of the physical product.

[0136] For example, the electronic communication device 50 may include a web browser application or other customized program, application or module configured to interface with a web service provided by the synthetic training data generating apparatus 10. Via the web browser application or other customized program, application or module, a user may upload a number of background images (e.g., an empty petri dish, leaves, soil, etc.) and object images separated from their backgrounds (e.g., individual spores, insects or eggs, particles) using, for example, username and password authentication.

[0137] The label includes characteristics that describe at least one object. Depending on the application scenario of the data-driven model, the characteristics may include one or more of the following: annotations that can be used to classify plant diseases in images of plants, annotations that can be used to classify cathode active material particles in images of battery materials, annotations that can be used to classify cells in images of biological materials, annotations that can be used to classify insects on leaves, and defects in coatings. For example, Figure 4A As shown in , an annotation can be a type of plant disease, such as a specific form of fungus.

[0138] The label also includes a characteristic value indicating a damage condition of at least one object. The characteristic value can be a numeric value (real number form), such as a percentage or an absolute value, or the characteristic value can be a classifier (a binary classifier indicating the presence or absence of a specific characteristic, a multi-class classifier). In one example, the characteristic value can include a characteristic value indicating plant damage. In another example, the characteristic value can include a characteristic value indicating a deviation from an industrial product standard. In some examples, the characteristic value is provided in the form of a damage percentage, which is preferably used to determine the amount of treatment to be applied to the physical product. For example, the characteristic value can be a defect value, which can be a score from 0 (healthy) to 100 (disease) or 0% (healthy) to 100% (disease) indicating the plant damage condition.

[0139] Figure 4A and Figure 4B An exemplary user interface of an online web-based app is shown in FIG. Figure 4A and Figure 4B In the example shown in , users can provide image data to generate synthetic training data for training a data-driven model for plant damage detection and classification. Users can Figure 4A The object image name (e.g., 11-m) is entered in the "Object_Image_Name" field in the user interface shown in , and the annotated training value is entered in the "Image Defect Value" field. As an example, the image defect value can be a score from 0 (healthy) to 100 (disease) indicating the plant damage condition. Figure 4A In the example shown in , the user assigns a score of 30 to the object image to be uploaded. The user can optionally enter a defect category in the "Defect Category" field, such as Figure 4B The form of fungal "mildew" shown in . Users can Figure 4B Enter the background image name (e.g., 13-n) in the "Background_Image_Name" field shown in FIG. Figure 4A and Figure 4B The layout, number and order of the facets shown in the figure and the specific names of the facets are presented only for the purpose of illustrating the concept. Of course, other layouts, numbers, orders or names of the facets can be dynamically displayed.

[0140] Return to Figure 3 , at block 220 (i.e., step b)), a synthetic object image dataset is generated from the object image dataset. The synthetic object image dataset includes a plurality of synthetic object images of at least one object. At least one synthetic object image is associated with a label. In some examples, each synthetic object image is associated with a corresponding label. In some other examples, the synthetic object image dataset may include unlabeled synthetic object images to generate more realistic images.

[0141] This step aims to increase the existing datasets of individual object images. Generative deep learning models such as Generative Adversarial Networks (GANs), Wasserstein GANs, and non-adversarial image synthesis can be used to further create new and natural-looking variants of object images.

[0142] Since current datasets of individual object images are labeled and these labels are required in the workflow, the application of GANs must maintain and replicate this labeling. It is possible to use conventional GANs to expand the dataset. However, these labels may be lost. Conditional GANs (cGANs) may be a way to overcome this obstacle. In addition to the input images, the labels are fed into the network during training. In this way, the network learns to generate images with corresponding labels. After training the model, the model is able to generate large datasets of object images for each category.

[0143] At block 230 (i.e., step b), a plurality of first synthetic training data samples are generated. Each first synthetic training data sample is generated by selecting one or more object images from the synthetic object image dataset and by drawing the selected one or more object images at one or more locations on the background image.

[0144] Using rules and parameters, such as the number of objects in the image, whether objects can overlap or touch, how far or narrow the objects should appear, etc., the actual distribution of objects in the generated image will be determined. In addition, image enhancement techniques will be used to create differences in the background and various types of object images (regarding size, shape, orientation, etc.) so that they reflect real variability. After setting the parameters, the actual dataset generation can begin. The results, natural-looking artificial combinations, and automatically generated accurate label data (e.g., coordinate lists, segmentation masks, etc.) can be exported in standard formats that can be used for regression, classification, object detection and / or object segmentation. The data-driven model can then be evaluated directly, or manual fine-tuning of the data-driven model can be performed based on the generated dataset.

[0145] Hereinafter, an exemplary embodiment of a computer-implemented method 200 for generating synthetic training data is described, which can be used to train a data-driven model for analyzing a surface image of a farmland including at least one plant. In this example, the farmland is a lettuce field including a plurality of lettuce heads. Although a lettuce field will be discussed as an example, it will be understood that the computer-implemented method 200 can also be applied to other farmlands.

[0146] At block 210, image data is provided. The image data includes an object image dataset including a plurality of object images of heads of lettuce. At least one object image is associated with a tag that can be used to annotate the content of the object image. The image data further includes a background image representing a background of the surface image of the lettuce field.

[0147] Figure 5 Images of different levels of data are shown. Although their composition is described in detail later, a basic understanding is necessary. A field trial of a field of lettuce consists of multiple plots. Each plot is a separate section, which itself consists of a fixed number of individual lettuce heads. The plot label, which is the target value for the disease prediction algorithm, is the average value of the decay of its individual heads (e.g., between 0 (healthy) and 100 (dead)). The goal of the workflow is to generate images of the plots so that the generated data can be used to train the model.

[0148] The workflow starts at the data level of individual heads and ends at the data level of the entire plot. This hierarchical level allows the use of labels of individual heads, since the label of the entire plot is composed of the labels of its individual heads. This approach further reduces the workload and the amount of labeled data overall, since the available data of labeled individual heads is of better quality and the data acquisition of labeled individual heads is much easier than plot images. This approach also simplifies the control of the composition of the generated plots, as shown in the following workflow.

[0149] In the first step, individual heads are extracted from drone images of the entire plot. They are then cropped and labeled according to a given assessment. Based on this data, a cGAN is trained, which can then be used to generate a much larger dataset of individual heads. The generated individual heads are then selected from the dataset according to their labels and placed on a soil background. In this way, the layout of the actual plot image can be replicated while keeping the true labels and an overall balanced data distribution. The last step is to train and apply an image-to-image GAN. This last step allows the style of the real image to be applied to the synthetically generated plot image while preserving the structure and the given labels.

[0150] Figure 6 The four steps of the pipeline are shown. After outlining the workflow, the individual steps will now be explained in more detail.

[0151] This step of the workflow involves the data source and processing of the data. As mentioned earlier, the case of labeled plot data is insufficient. Therefore, the amount of labeled individual head images used is greater than the amount of labeled plot images. Individual heads are extracted from the drone plot images.

[0152] Due to the altitude and resolution limitations of the drone flight, each head reaches its maximum resolution at 64×64 pixels. Individual heads were cropped and saved as 64×64 pixel PNG files. Since these images will be drawn on a ground background and will be part of the plot formation, the background must be removed from the heads. An ImageJ macro was used to process each head image. In this process, the head was exposed from the background and eroded. Erosion allowed the individual heads to blend realistically with the background.

[0153] Figure 7 The background removal step is shown. Each head in the field is evaluated and scored from 0 (healthy) to 100 (sick). The grades can be divided into 11 categories (0-10). A total of about 40 images can be obtained for each category. This data set of cropped individual heads is the basis for the following steps of the workflow.

[0154] At block 220, a synthetic object image dataset is generated from the object image dataset. The synthetic object image dataset includes a plurality of synthetic object images of at least one object.

[0155] This step aims to increase the existing dataset of individual heads. GANs have been shown to be an effective tool for data augmentation. A larger dataset will help in the future to be able to display more diverse images, not limited to the availability of only 400 individual head images. The description in this subsection aims to provide an understanding of why cGAN is used and its purpose during the workflow. The methodology of cGAN and the results of testing the quality of the output images are described in the following sections.

[0156] Since current datasets for various heads are labeled and these labels are required in the workflow, the application of GANs must maintain and replicate such labels. It is possible to use regular GANs to expand the dataset. However, these labels may be lost. Conditional GANs are a way to overcome this obstacle. In addition to the input images, the labels are fed into the network during training. This way, the network learns to generate images with corresponding labels.

[0157] After training the model, the model is able to generate a large dataset of images of each category. Although there are small visual differences between real images and those generated by the cGAN, this variation may have an impact on the next stage of the workflow. This approach makes it possible to increase the size of those categories that are more difficult to obtain. Since the model learns all categories at the same time, it is easier to achieve good representation of categories that are otherwise underrepresented. For example, the structural diversity of one class of heads with many samples can be combined with the color of another class of heads that may be a feature. As the dataset grows, the data source for each head of lettuce is complete and can be used in the next steps.

[0158] The cGAN approach is discussed in more detail below.

[0159] Data augmentation using cGAN

[0160] Since the database of individual heads is limited, the GAN method was introduced to supplement and expand the existing data. GAN has been shown to be a useful tool for data expansion. In this case, it is necessary to be able to control the output of the GAN because different head labels are required during the pasting process. This leads to the use of conditional GAN. cGAN is able to generate images of different categories based on specific inputs. For the current case, the manually labeled heads can be divided into 11 categories and used for training with cGAN.

[0161] The open Github repository StudioGAN1 is used as a reference. Several popular GAN architectures have been implemented in it and benchmarks have been performed with the following three datasets: Cifar10 (3×32×32), TinyImageNet (3×64×64), and ImageNet (3×128×128).

[0162] Three factors were critical to the choice of architecture. Performance on TinyImageNet (which corresponds to the input size of the Raisin data approach), computational cost, and whether the GAN uses labels. Based on these criteria, SAGAN was chosen. After initial training, SAGAN was able to converge quickly and capture the characteristics of the data in visual matching.

[0163] Conditional GAN ​​extends the generator and discriminator by introducing additional inputs. In addition to the noise or input image, auxiliary information Y is included, which for this disclosure is a numeric class label. SAGAN builds on this principle but provides a more advanced and higher performance architecture. SAGAN adapts to non-local models and introduces self-attention. This allows both the generator and the discriminator to recognize relationships between even spatially distant parts of the image.

[0164] train

[0165] The model calculates self-attention between hidden layers. For this purpose, the image features in each previous hidden layer are transformed into two feature spaces f and g to calculate the attention.

[0166] Where s(i, j) = f(x i ) T g(x j )

[0167] Here, β indicates how much the model's attention is on the i-th position when considering the j-th position. The output of the attention layer is o=(o1,o2,o3,...,on), where C is the number of channels and N is the number of feature classes from the previous hidden layer. The final output of the attention layer is obtained by multiplying by a scalar and adding the input feature map. The scalar causes spatially closer details to be considered first. Thereafter, the model should gradually solve more complex tasks. The generator and discriminator can be trained with a hinged version of the adversarial loss.

[0168]

[0169] To stabilize the training, SAGAN uses two additional techniques. First, spectral normalization is applied to both the generator and the discriminator. This approach requires only slightly more computational effort, but has been shown to stabilize the training behavior. Second, unbalanced learning rates are used for the generator and discriminator updates, which has been shown to produce better results.

[0170] Resulting image of cGAN

[0171] Figure 8 The resulting image of the cGAN is shown. There are minor differences in appearance between the real data and the synthesized data. The model is able to replicate colors and shapes well, but the synthesized heads have a pixelated texture. However, this is consistent with the goal of cGAN to learn the underlying distribution of the data. Comparison with the real heads shows that there are outliers in the field data whose color scheme matches another category better. The cGAN is unable to replicate this phenomenon. The samples of each category appear homogeneous and different from those of the other categories. There is a visually distinct continuum from healthy to dead. Although this was not planned, it is a side effect that proves useful. The generated categories now have a reliable and unique appearance. In addition, each of the eleven categories has significantly more samples than the real dataset. Although there are some heads in these categories that look very similar, there is a wider variety of shapes and textures overall.

[0172] At block 230, a plurality of first synthetic training data samples are generated. Each first synthetic training data sample is generated by selecting one or more object images from the synthetic object image dataset and by drawing the selected one or more object images at one or more locations on the background image. In this step, labels may be provided separately in an annotation file (such as a JSON-based file, e.g., COCO format).

[0173] The last two steps create and refine the database of individual lettuce heads. These heads now form the plots. This subsection explains the relevance and details of this step.

[0174] Data from field trials often present challenges, particularly small sample sizes and unbalanced distribution. Relying solely on field samples can lead to model overfitting, resulting in performance loss during test time. The presence of diseased heads and their arrangement in the plots is arbitrary. Since deep learning models are best trained on balanced training sets, there are many opportunities for improvement. Some target values ​​for regression will not even appear in data and possible compositions that are not included in the dataset. The purpose of this step is to lay the foundation for overcoming these challenges. To increase variance and balance the training data, this part of the workflow creates a large, well-distributed dataset of plot images. By algorithmically arranging these plots, both the composition and label of each image can be controlled.

[0175] During algorithm compilation, real data labels are available. The target value of the field plot is defined as the average of all internal heads. Outsiders are not affected by the processing and are therefore not considered. Since the plot consists of a fixed pattern, the algorithm simulates the pattern to obtain an appearance similar to the real image. This step focuses on the optimal distribution of realistic patterns and data sets, while the realistic style is applied later. This approach allows the data to be balanced across multiple dimensions. Each regression value can have examples. When examining combinations that may result in, for example, a mean of 50, it is obvious that only samples with a value of 50, half of each sample with a value of zero and a value of 100, or any combination between these extreme values ​​are possible. Although not every combination occurs naturally, the deep learning model is preferably trained to the highest possible degree of balance.

[0176] Hereinafter, the pasting algorithm will be described in detail.

[0177] The pasting algorithm composes a dataset of 1000 plot images (SynPlot). Again, while this number can be arbitrarily high, early experiments suggest that doubling, quintupleing, or 10x the number of images will not yield significant improvements. The purpose of the pasting algorithm is to synthesize a balanced dataset.

[0178] Therefore, the first task of this step is to draw a sample with a numerical distribution of a certain mean and standard deviation. These two parameters can be used to control both the resulting mean and the degree of distribution to achieve the mean. Next, a random soil image is selected, and the outer row of each lettuce head is drawn with a healthy head. The inner row is then filled according to the drawn sampling distribution, and each head is randomly selected from its category folder. In order to further increase the difference, the size of the head and the distance between them are changed to smaller random values. This is done to simulate the irregularity of nature. The modularity of the algorithm is taken into account in the design of this step. Therefore, this workflow can also be applied to other data sources. In addition, the random parameters such as the control interval can be changed. There is also a parameter for the number of rows and columns, which will make it possible to apply the workflow to experiments with different layouts.

[0179] The pasting algorithm can be implemented in Python 3.7. The PIL1 library is used to process the images, and the functions of the scikit-learn2 Python library are used to generate the random distribution. The parameters defining the layout, image size, and the randomness of the spacing and header size are set externally. Fig. 9 An exemplary implementation of the pasting algorithm is shown in .

[0180] Optionally, at block 330, a plurality of second synthetic training data samples may be generated from the plurality of first synthetic training data samples using an image-to-image conversion model. The image-to-image conversion model has been trained to generate synthetic surface images that more closely resemble realistic surface images of physical products.

[0181] So far, the workflow has processed and generated individual heads of lettuce and produced a balanced dataset of images of plots. However, these images should have an artificial appearance and be significantly different from the plot data. This work hypothesizes that visual properties, such as primitive-to-background fusion and overall realism, can be improved by image-to-image translation GANs. This hypothesis is supported by the fact that image-to-image translation GANs have previously been shown to increase the realism of synthetic images. This section describes the role of image-to-image translation GANs in the workflow, while the functionality and implementation will be described later. The CUT model, which is a variant of the image-to-image translation GAN, receives two image domains as input. Domain A consists of previously generated images, and domain B consists of field data. CUT learns to apply the style of domain B to images of domain A. In this way, the large synthetic dataset retains its composition and balance, while the style is modified to a more realistic appearance. Neither the discriminant model nor humans can distinguish this result. Because ideally the composition of the plots and the degree of decay of each head are maintained, the labels remain the same during the application of the trained model. Fig.10 An exemplary implementation of image-to-image conversion using CUT is shown in FIG.

[0182] In the following, methods and selections of GANs for image-to-image translation are described.

[0183] The principle of image-to-image GAN is comparable to the human ability to transfer style from one scene to another. For example, it is easy to imagine a landscape in different seasons of the year, even if that particular landscape has never been seen at the time. Therefore, humans are able to transfer a familiar style to a particular object. It is this ability that image-to-image GAN simulates. The model is trained on two domains and learns to capture the style and characteristics of one region and transfer it to another. Of course, this assumes that the two domains have basic similarities. For example, the original CycleGAN material mentions seasonal transfer of landscapes or style transfer between horses and zebras. Since the two domains in this work are similar and differ only in style, it is assumed that the application of this approach is possible. The rationale behind using GAN is the underlying goal of the training. The goal is to make the output indistinguishable from the original image of the domain. When this is achieved, the model has learned the weights to transfer an image from domain A to domain B.

[0184] Contrastive Learning

[0185] When choosing an image-to-image GAN architecture to fit, the first and most prominent option is cycleGAN. However, there is a recent architecture called contrastive unpaired transformation (CUT) that provides better results. This architecture is also more suitable for the challenges of this work, as described below. CUT introduces a method built on the cycleGAN architecture. The cycle consistency of cycleGAN requires the relationship between the two domains to be bijective. This is too strict, so the authors propose to maximize the mutual information between the input patch and the output patch instead. To do this, a contrastive loss function is used that aims to change the appearance of the object while preserving the shape and geometry. The method learns to map in one direction and performs image-to-image transformation with a more streamlined architecture and therefore more efficiently. The lack of cycle consistency also allows unidirectional image transformation using only one image. This offers great potential, especially for data-scarce situations, such as in this work.

[0186] train

[0187] This work consists of two domains. Domain A consists of synthetically generated plot images, and domain B consists of actual plot images. Due to the unpaired image-to-image transfer, the shape and basic colors of domain A are preserved. Therefore, the labeling of domain A is also preserved. The style, appearance, and in some cases the differences of the individual lettuce heads of the plot are transferred from domain B. This does not require labels, because the model learns to identify areas with high similarity and transfer the style of these areas accordingly. By matching patches (such as patches with healthy lettuce heads), the corresponding patches of equally healthy lettuce heads with the style of domain B are found and used as reference. This applies similarly to other parts of the image. In order to transfer this to the entire image, and for different regions down to the pixel level, a multi-layer patch-based learning objective is established. Genc and its feature stack can be used to calculate it. Each layer and region of this feature stack represents a region of the input image. Here, the deeper layers correspond to larger patches respectively. L layers of interest are selected and fed into a multilayer perceptron H. The number of spatial locations is defined as SL. The corresponding features are defined as zls, and the other features are defined as zlS / s. Finally, yls is the encoded output image. In order to match the corresponding input-output patches, PatchNCE loss is introduced:

[0188]

[0189] The PatchNCE loss together with the GAN loss helps achieve the ultimate goal, i.e., the generator should produce realistic images with corresponding input-output patches while preventing too much change through the loss. Fig.11 Multi-layer, block-wise contrastive losses are shown.

[0190] Workflow Image

[0191] Fig.12 Domain A and Domain B are shown for which an image-to-image translation GAN was trained. Domain A is a synthetic image from an earlier step in the workflow, and Domain B consists of real field images. There are appearance differences between the two domains, making it impossible to train a model on Domain A data to predict Domain B images. Even if the individual heads are very similar, as shown in previous experiments, the overall visual appearance has large differences. One of these differences is the color composition, since, for example, the sun incidence is different between different drone images. Another large difference is that the interpolation algorithm cannot realistically blend the image with the background. Again, this is not the focus of this work and is therefore not a shortcoming of this step or the entire workflow. On the other hand, it can also be seen that the appearance of the optimized plots is very similar to the field data. In terms of realism, the optimized images are almost indistinguishable from the Domain B images. However, in terms of color, the similarity depends on the sample of Domain B. Some images are closer than others, depending on the lighting and growth state.

[0192] Fig.14 Comparison of the synthesized images and their corresponding optimized images is shown. The optimized curves in the right column are the result of applying the GAN on the corresponding images in the left column. These examples prove that most of the unhealthy heads are also converted to unhealthy heads. Although some heads are not converted, the overall composition is maintained and the labels remain similar. The style of the real plots is successfully transferred. Even the slight differences in the head placement are maintained to give a more realistic appearance.

[0193] The workflow is not only able to replicate images it has seen before, but it is also able to interpolate images. Specifically, although the evaluation values ​​range from 0 to 100, the field data does not contain a sample for every value. Using the workflow, multiple samples of each value can be generated. Therefore, the workflow is able to generate combinations that are not included in the field data and therefore expand the distribution of the data at hand. Fig.13 As presented in , the workflow is able to generate data on a continuous scale from low to high estimates.

[0194] In summary, each head is processed and multiplied by the conditional GAN ​​model.

[0195] Then, a balanced dataset of field images was generated through an interpolation algorithm. To make these images useful for training, the final step was to apply the style of real field images to the data.

[0196] The generated synthetic training data can be stored in a training database, such as Figure 2 The training database in the synthetic training data generating apparatus 10 shown in FIG. The training database may be any organized data collection that can be stored and accessed electronically from a computer system and from which data may be input or transferred to a computer system. Figure 2 The decision support system 30 and the electronic communication device(s) 50 are shown in FIG.

[0197] Fig.15 A flowchart depicting a computer-implemented method 300 for training a data-driven model is shown. Figure 2 1 and 2. The system 100 shown in FIG.

[0198] At block 310, a training data set is provided. For example, the model trainer 32 of the decision support system 30 may be provided from Figure 2 The training data is obtained from the training database 20 shown in FIG.

[0199] The training data set includes synthetic training data generated by the synthetic training data generating device 10. In addition to the synthetic training data, the training data set may also include real images, i.e., images acquired by a camera. The real images do not need to be labeled. In the field of agricultural technology, the real images may be real field data, which may also come from different regions or have different crop / weed plant arrangements. In this way, fewer images are required.

[0200] At block 320, a data-driven model is trained on a training data set. For example, Figure 2 The model trainer 32 shown in FIG. 3 may include one or more processors, one or more memory elements, and other components for implementing a model to support training of a data-driven model. The data-driven model may be a machine learning algorithm (e.g., Figure 2 The machine learning algorithm may be a deep learning algorithm, such as an algorithm based on a deep neural network, a convolutional deep neural network, a deep belief network, a recurrent neural network, etc.

[0201] The performance of a data-driven model can be characterized by training a model or network (e.g., of artificial "neurons") to adapt the model to operate and return results in response to a set of one or more inputs. Training can be fully unsupervised, partially supervised, or supervised.

[0202] After training, the resulting data-driven model can be used to perform image tasks such as object detection (e.g., plant detection, insect detection, etc.), image segmentation, and data augmentation.

[0203] In one example, a data-driven model can be trained to identify damage conditions on agricultural fields, which can be used for field management. Fig.16 and Fig.17 The examples shown in are discussed.

[0204] Fig.16 A flow chart describing a field management method 400 is shown, which combines Fig.17 1 and 2. The system 100 shown in FIG. Fig.17 The system 100 shown in FIG. Figure 2 The difference is that Fig.16 The system 100 in includes a trained data-driven model.

[0205] In some examples, Fig.17In the exemplary system shown in , the trained data-driven model can be deployed in the network 60 and no further training is required in the application of the farmland. In some other examples, the training can be continuous, so that the training continues based on the newly generated synthetic training data received from the synthetic training data generating device. Such continuous training may include error correction training or other incremental training (which may be algorithm specific).

[0206] Starting at block 410, a surface image is provided. For example, an image of the farmland 100 may be acquired by a camera, which may be mounted on a Fig.16 The UAV 70, aircraft, etc. shown in FIG. 8 can automatically capture each image without the user controlling the UAV.

[0207] The acquired images are then uploaded to the decision support system 30. If multiple images are acquired, these images may be provided to the decision support system 30 to stitch the captured images together. It is worth noting that the transmission of each image may be performed immediately after it is captured, or after all images are captured as a group. In this regard, it is preferred that the UAV 70 includes a corresponding communication interface configured to directly or indirectly transmit the collected images to the decision support system 30, which may be, for example, a cloud computing solution, a centralized or decentralized computer system, a computer center, etc. Preferably, during the collection, the images are automatically transmitted from the UAV 70 to the decision support system 30 using a suitable wireless communication interface (e.g., a mobile interface, a long-range WLAN, etc.), for example, via an upload center or a cloud connection. Although it is preferred that the collected images are transmitted via a wireless communication interface, the UAV 70 may also include a field data transmission interface, such as a USB interface, from which the collected images may be received via manual transmission and then transmitted to a corresponding computer device for further processing.

[0208] At block 420, a data-driven model is provided. For example, using the trained data-driven model, the image analysis device 40 in the decision support system 30 can be configured to identify and locate defects in the image, such as based on a score from 0 (healthy) to 100 (disease) for each head of lettuce as described above. For example, the image analysis device 40 can detect damaged plants, such as plants damaged by fungi at a point at location (X, Y).

[0209] At box 430, the control file generation device 34 of the decision support system 30 can generate a control file based on the identified damage location. The control file may include instructions to move to the identified location and apply the treatment. The identified location can be provided as location data, which can be geographical location data, such as GPS coordinates. The control file can be provided, for example, as a control command for an object modifier, for example, the control command can be read into a data memory of the object modifier before the field is processed, for example, through a wireless communication interface, a USB interface, etc. In this example, the object modifier can also be referred to as a processing device. In this context, it is preferred that the control file allows more or less automated processing of the field, that is, for example, the sprayer automatically distributes the required herbicides and / or pesticides at the corresponding coordinates without manual intervention by the user. It is particularly preferred that the control file also includes a control command for driving away from the field. It should be understood that the present disclosure is not limited to the specific content of the control data, but may include any data required to operate the processing device.

[0210] In another example, a data-driven model can be trained to identify damage to industrial products. Fig.18 The examples shown in are discussed.

[0211] Fig.18 An exemplary system 500 for identifying damage conditions of industrial products is shown. System 500 may include a data storage device 20, a decision support device 30, an electronic communication device 50, an object modifier 80 having a processing device 90, and a camera 95.

[0212] The decision support device 30 may be embodied as a workstation or a server, or may be embodied in a workstation or a server. The decision support device 30 may provide web services to, for example, an electronic communication device 50. The decision support device 30 may have Figure 2 For example, the decision support device 30 may include an image analysis device having a trained data-driven model configured to identify damage to industrial products. The data-driven model may have been trained on a training data set obtained from the data storage device 20. The training database includes synthetic training data that has been generated according to the methods described herein. Figure 3 The example shown in describes an exemplary synthetic training data generation method.

[0213] During deployment of the trained data-driven model, the camera 95 may capture images of particles on the conveyor belt. The images are provided to, for example, the image analysis device 40 ( Fig.18Using the trained data-driven model, the image analysis device 40 is configured to detect damaged locations where its surface may show deviations from normal (or from a standard). The object modifier 80 may receive position information of the damaged location from the controller and trigger the processing device 90 to act on the damaged location of the surface. The operation of the processing device 90 is not limited to a single specific point, and its operator may apply measures to substantially all points of the object, where point-specific strengths are derived from the position information. For example, Fig.18 As shown in , the system can be used to detect defective particles on a conveyor belt. If one or more defective particles are detected at one or more points, a processing device 90 (e.g., a blower) controlled by an object modifier 80 can remove the defective particles from the conveyor belt.

[0214] In another exemplary embodiment of the present invention, a computer program or a computer program element is provided, characterized in that it is suitable for executing the method steps of the method according to one of the aforementioned embodiments on an appropriate system. Therefore, the computer program element can be stored on a computer unit, which can also be part of an embodiment of the present invention. The computing unit can be suitable for executing or causing the execution of the steps of the above-mentioned method. In addition, the computing unit can be suitable for operating the components of the above-mentioned device. The computing unit can be suitable for automatic operation and / or executing the commands of the user. The computer program can be loaded into the working memory of a data processor. Therefore, the data processor can be equipped to perform the method of the present invention.

[0215] This exemplary embodiment of the invention covers both a computer program which right from the beginning uses the invention and a computer program which by means of an update turns an existing program into a program which uses the invention.

[0216] Furthermore, the computer program element may be capable of providing all necessary steps to complete the procedure of the exemplary embodiment of the method described above.

[0217] According to another exemplary embodiment of the invention, a computer readable medium, such as a CD-ROM, is presented, wherein the computer readable medium has stored thereon a computer program element, which has been described in the preceding section.

[0218] The computer program may be stored and / or distributed on a suitable medium, such as an optical storage medium or solid-state medium provided with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0219] However, the computer program may also be presented via a network such as the World Wide Web and may be downloaded from such a network into a working memory of a data processor. According to another exemplary embodiment of the present invention, a medium for making a computer program element available for download is provided, the computer program element being arranged for performing a method according to one of the aforementioned embodiments of the present invention.

Claims

1. A computer-implemented method (200) for generating synthetic training data that can be used to train a data-driven model for identifying individual objects in an image of a surface of a physical product that includes at least one object, the method comprising: a) providing (210) image data, the image data comprising: - an object image dataset, the object image dataset comprising a plurality of object images of the at least one object, the at least one object image being associated with a tag that can be used to annotate the content of the object image, wherein the tag comprises a characteristic value describing a property of the at least one object in the at least one object image and indicating a damage condition of the at least one object in the at least one object image; and - a background image, the background image representing the background of the surface image of the physical product; b) generating (220) a synthetic object image dataset from the object image dataset, wherein the synthetic object image dataset comprises a plurality of synthetic object images of the at least one object, at least one synthetic object image being associated with a label; and c) generating (230) a plurality of first synthetic training data samples, wherein each first synthetic training data sample is generated by selecting one or more object images from the synthetic object image dataset and by drawing the selected one or more object images at one or more locations on the background image.

2. The computer-implemented method of claim 1, in, The at least one object includes a plurality of objects, at least two objects of the plurality of objects are associated with tags including different characteristic values.

3. The computer-implemented method according to claim 1 or 2, in, In step b), a generative model is used to generate the synthetic object image dataset.

4. The computer-implemented method of claim 3, in, The generative model includes a conditional generative adversarial network (cGAN).

5. A computer-implemented method according to any one of the preceding claims, in, In step c), the selected one or more object images are drawn on the background image according to a rule derived from one or more surface image samples of the physical product.

6. A computer-implemented method according to any one of the preceding claims, in, Step c) further comprises the step of generating a plurality of second synthetic training data samples from the plurality of first synthetic training data samples using an image-to-image conversion model, wherein the image-to-image conversion model has been trained to generate synthetic surface images that are closer to real surface images of the physical product.

7. The computer-implemented method of claim 6, in, The image-to-image translation model includes an image-to-image generative adversarial network.

8. A computer-implemented method according to any one of the preceding claims, in, This characteristic includes one or more of the following: - Annotations that can be used to classify plant diseases in images of plants; - Annotations that can be used to classify cathode active material particles in images of battery materials; - annotations that can be used to classify cells in images of biological material; - annotations that can be used to classify insects on leaves; and -Annotations that can be used to classify defects in the coating.

9. A computer-implemented method according to any one of the preceding claims, in, The property value includes one or more of the following: - a characteristic value indicative of plant damage; and - A characteristic value that indicates a deviation from an industrial product standard.

10. The computer-implemented method of claim 9, in, The characteristic value is provided in the form of a damage percentage which can preferably be used to determine the amount of treatment to be applied to the physical product.

11. A computer implemented method according to any one of the preceding claims, further comprising the step of providing a user interface allowing a user to provide the image data.

12. A computer-implemented method according to any one of the preceding claims, in, Step c) further includes providing the label for one or more first synthetic training data samples of the plurality of first synthetic training data samples.

13. A computer-implemented method for analyzing a surface image of a physical product, the method comprising: - providing (410) a surface image of the physical product; as well as - providing (420) a data-driven model for identifying at least one object on the provided surface image of the physical product and for generating a label that can be used to annotate at least one detected object, wherein the label includes a characteristic value describing the at least one object on the provided surface image of the physical product and indicating a damage condition of the at least one object on the provided surface image of the physical product, wherein the label is preferably capable of being used to monitor and / or control a production process of the physical product, Wherein, the data-driven model has been trained on a training dataset, which training dataset includes synthetic training data generated according to any of the preceding claims.

14. A method for controlling a production process of a physical product, the method comprising: - providing (410) a surface image of the physical product; - providing (420) a data-driven model for identifying at least one object on the provided surface image of the physical product and for generating a label that can be used to annotate the at least one object; wherein the label includes a characteristic value describing the at least one object on the provided surface image of the physical product and an indication of a damage condition of the at least one object on the provided surface image of the physical product, wherein the data-driven model has been trained on a training data set comprising synthetic training data generated according to any one of claims 1 to 12; and - generating (430) control data based on the generated tag, the control data comprising instructions for controlling an object modifier to perform an operation acting on at least one detected object.

15. A synthetic training data generating device (10) for generating synthetic training data, wherein the synthetic training data can be used to train a data-driven model for analyzing a surface image of a physical product including at least one object, wherein the synthetic training data generating device comprises one or more processors configured to perform the steps of the method of any one of claims 1 to 12.

16. An image analysis device (40) for analyzing a surface image of a physical product, the synthetic training data generation device comprising one or more processors configured to perform the steps of the method of claim 13.

17. A system (100) for controlling a production process of a physical product, the system comprising: - a camera (95) configured to capture an image of a surface of the physical product; - an image analysis device (40) according to claim 16, configured to identify at least one object on the captured surface image of the physical product and to generate a label that can be used to annotate the at least one detected object; as well as - an object modifier (80) configured to perform an operation on the at least one detected object based on the generated tag.

18. A computer program product comprising instructions which, when executed by a processing unit, cause the processing unit to perform the method of claim 1 or the steps of the method of claim 13.

Images