TRAINING METHOD FOR A NEURAL NETWORK INTENDED FOR CONTROLLING AN OBJECT SORTING SYSTEM

Abstract

Method (100) of processing and characterizing objects in by means of a neural network, including the generation (150) of digital training data (T.DAT) of the neural network, representative of simulated objects to be processed (S.IT), generated by a digital twin of the object processing system, and associated with annotations (ANN) indicating at least one characteristic of the simulated objects (S.IT); the training (160) of the neural network by means of the digital training data (T.DAT); the acquisition (170) of digital images of a real scene including at least one real object to be processed; the provision (180) of data (R.DAT) generated from the digital images of the real scene to the neural network, which in turn provides data representative of a characteristic (PGO) of the real object; and the sending (190) of a control signal to an object processing device. Figure to be published with the abbreviation: Fig. 2

Description

Title of the invention: TRAINING METHOD FOR A NEURAL NETWORK INTENDED FOR CONTROLLING AN OBJECT SORTING SYSTEM TECHNICAL FIELD OF THE INVENTION

[0001] The invention relates to a method for processing objects, such as a sorting method, the method including training a neural network intended for controlling an object processing system, said neural network being trained by means of images comprising objects to be processed each associated with a characteristic used as a processing criterion. TECHNOLOGICAL BACKGROUND

[0002] Object processing systems such as baggage, parcel, postal or other object sorting systems use conveyors and / or gripping robots as handling means. These handling means are controlled by means of various types of sensors such as photoelectric cells, scales, force sensors, barcode readers or digital cameras connected to electronic and / or computer systems.

[0003] Among the methods for controlling these sorting systems, the use of artificial neural networks is considered a very effective solution that can have high reliability, with a very low error rate. One example is the control of gripping robots using images captured by digital cameras. In this case, the function of the neural network is to characterize, possibly by identification, objects to be sorted found in the images captured by the cameras in order to control in return gripping robots whose function is to move these objects. Thus, so-called "deep learning" methods are developed to process complex stacks of items to be sorted, as described in the article "Visual Sorting of Express Parcels Based on Multi-Task Deep Learning" by Song Han et al., Sensors 2020, 20, 6785; doi: 10.3390 / s20236785.

[0004] Neural networks are trained on sets of manually labeled images to adjust their internal parameters, such as the synaptic weights of each of the neurons constituting the network. Labeling images includes clipping the objects of interest by making a mask excluding parts of the image other than these objects, then annotating each of the objects. For quality training, the training data should have as many images as possible, with 100,000 images being considered a minimum. However, processing and manually labeling these images is time-consuming and prone to human error. It is therefore difficult and expensive to produce large labeled datasets for training neural networks.

[0005] In view of these difficulties, patent document WO 2018 / 170512 A1 discloses a solution based on a sequence of images, in which an object is manually labeled in a first image of the sequence, then this object is automatically tracked and labeled in subsequent images of the sequence, thus providing a large number of training images for limited human effort. It is also proposed to carry out automatic annotations of images, as for example in patent document EP3 885 998A1.

[0006] However, a satisfactory solution for sorting applications still remains to be implemented. Statement of the invention

[0007] An object of the invention is a method of object processing including the training of a neural network based on the use of automatically produced annotated images.

[0008] In order to achieve this aim, a first aspect of the invention is a method for processing objects in an object processing installation comprising a computer configured to characterize, by means of a neural network, objects to be processed visible in digital images of objects to be processed located in the object processing installation, the method comprising: a step of generating digital training data representative of objects to be processed simulated within the object processing installation, these data being intended for training the neural network, generated by means of a digital twin of the object processing system, and associated with annotations indicating at least one characteristic of the simulated objects to be processed;a step of training the neural network comprising providing the digital training data as input to the neural network and during which the parameters of the neural network are adjusted on the basis of the digital training data; a step of acquiring digital images of a real scene of the object processing facility comprising at least one real object to be processed; a step comprising providing data generated from the digital images of the real scene as input to the neural network, the neural network providing in return data representative of a characteristic of the real object to be processed; and a step comprising sending a control signal to an object processing device on the basis of the data representative of the characteristic of the at least one real object to be processed.

[0009] This method benefits from the automatic, simple and economical production of a large number of annotated images adapted to the purpose of the method, which is to realize the mode detailed below consisting of sorting items arriving on an item sorting line. Thus, the training of the neural network, based on a large number of annotated images without errors and generated in ways that correspond to the situation for which the neural network is used, is of high quality, and the use of this neural network after its training implies a high level of reliability of the sorting process to which it is applied.

[0010] According to additional non-limiting characteristics of the first aspect of the invention, considered individually or in any technically feasible combination:

[0011] - the at least one characteristic of the simulated objects to be processed can be chosen from the position, geometry and orientation of the simulated objects to be processed;

[0012] - the digital training data may comprise at least one image digital of a simulated scene representing a working area of ​​the object processing facility;

[0013] - the at least one digital image can be generated so as to be substantially as if obtained from digital images acquired by a digital camera installed above the working area of ​​the actual processing facility;

[0014] - the digital training data may further comprise a map of depth of the simulated scene;

[0015] - the depth map can be generated so as to be substantially such that if it was obtained from digital images acquired by a digital camera installed above the working area of ​​the actual processing facility;

[0016] - the digital twin integrates a dynamic interaction simulation engine physical between simulated elements of the treatment facility, so as to reproduce movements and shocks between simulated objects to be treated and simulated elements of the treatment facility;

[0017] - the method may further comprise a step of projecting textures from digital images of real objects on surfaces of simulated objects to be treated;

[0018] - the digital twin of the object processing system can be parameterized in so as to generate digital images of homogeneous batches of objects to be processed comprising more than 40%, preferably more than 50%, preferably more than 80%, preferably more than 90% or 100% of objects belonging to the same type of objects to be processed; and

[0019] - the type of objects can be defined by a texture mapped onto object surfaces simulated.

[0020] The invention extends to a treatment installation comprising a computer configured to implement the method according to the invention. BRIEF DESCRIPTION OF THE FIGURES

[0021] Other characteristics and advantages of the invention will emerge from the detailed description of the invention which follows with reference to the appended figures in which:

[0022] [Fig.l] [Fig.l] represents an overall view of an installation in which the system according to the invention is integrated;

[0023] [Fig.2] [Fig.2] illustrates a mode of operation of the system according to the invention, using a top view; and

[0024] [Fig.3] [Fig.3] illustrates a method according to the invention, which can be implemented by the system of figures 1 and 2. DETAILED DESCRIPTION OF THE INVENTION

[0025] Figures 1 to 3 illustrate an embodiment of a method for training a neural network, specialized in the analysis of digital images in order to identify articles to be sorted within an article sorting line.

[0026] A context in which the invention can be applied is illustrated using the example of [Fig. 1] and 2, which represent a schematic portion of a FAC installation for processing articles comprising a sorting line SL equipped with a system SYS for controlling the sorting line. The straight arrows indicate the paths followed by articles to be sorted and being sorted within the installation.

[0027] [Fig.l] illustrates a sorting installation FAC, comprising a work area WZ into which IT articles to be sorted brought by a truck Trck are poured in bulk, a robotic manipulator arm MA as an individual article handling device configured to unpack the IT articles and place them in an orderly manner on a conveyor belt CB leading to an inserter INS, a circular conveyor belt CCB receiving the articles from the inserter INS and sorting outlets Out (three outlets shown here) to which articles are directed according to their final destination. This may, for example, involve sorting postal parcels and dividing them into distribution rounds, each corresponding to one of the sorting outlets.

[0028] [Fig. 2] illustrates the work zone WZ. The manipulator arm MA is equipped with a gripping head GH, and the conveyor belt CB moves the articles It unloaded from the truck in bulk in the direction Dir represented by an arrow, to the inserter INS. The gripping head GH can be equipped with suction cups SC connected to a pumping system, each suction cup can be controlled individually so as to apply a vacuum to the surface of an article to be gripped. The articles arrive in bulk from the right of the figure, the manipulator arm being responsible for unstacking them, that is to say, gripping them one by one to place them in order on the conveyor belt CB towards the left of the figure. A system SYS for controlling the sorting line comprises a digital camera CAM and a digital computer CALC comprising a computer memory operatively connected to the digital camera CAM and the manipulator arm MA. The manipulator arm MA has the function of individually manipulating IT articles located in the work area to move them from this area to the conveyor belt CB.

[0029] We will focus on the process of identifying items in the work area with a view to their being picked by the manipulator arm.

[0030] At this level, the articles are handled one by one so as to order them for subsequent sorting operations, here the insertion of the articles at regular intervals on the circular conveyor belt. This individualized handling of the articles must be done as quickly as possible, while ensuring good quality of capture of the articles by the robotic arm. To this end, the system must provide the control system of the robotic arm with the characteristics, positions, orientations, and geometries of each of the articles. These characteristics are obtained from digital images of the work area, processed by the CALC computer using a NN neural network trained using annotated images.

[0031] Conventionally, these images are annotated manually, or at best, with semi-automated annotation systems, as mentioned in the Technological Background section. The solution proposed here is based on the use of a digital twin of the sorting facility to computationally generate digital images and depth maps representative of scenes encompassing the work area.

[0032] The use of a digital twin is particularly relevant in a situation where it is difficult to carry out manual annotation of training images. When it comes to associating a binary content annotation with images, such as "presence or absence of a yes / no address block", then a human operator can be efficient in terms of time, reliability and cost to carry out the annotation. On the other hand, when the required annotation consists for example of determining and indicating a position, geometry or dimensions of items present in an image, then the time, reliability and cost of a human operator are very unfavorable. In such situations, it is particularly interesting to implement an automatically annotated training image generation, which will be faster, more reliable and / or more economical than if it were carried out by a human operator.

[0033] [Fig. 3] represents a diagram of the method 100 for handling articles according to the invention, including the generation of annotated images for training a neural network, the training of this network, then its use in the control of a robotic arm for unpicking articles to be sorted.

[0034] In a step 110A, the DAT data from the FAC processing facility are collected. of articles, which includes the geometric and functional characteristics of the equipment used (sorters, conveyors, etc.) and the installation of this equipment, the interactions between the different elements.

[0035] In a step 120, on the basis of the collected DAT data, a digital twin NUM.TWIN of the sorting installation is developed, stored in a computer memory of the CALC computer. This digital twin integrates a dynamic engine configured to reproduce the physical interactions between the simulated elements (objects such as articles to be processed within the installation, conveyors, sorting systems, etc.), so that the path, the movements, the impacts between each simulated article and the elements constituting its environment are faithfully reproduced by the digital twin. The dynamic engine may consist of a software grouping configured to reproduce the laws of dynamic mechanics.In combination with an automation system that reproduces the control system of the installation elements, the digital twin makes it possible to generate realistic configurations corresponding to actual processing of articles by the installation, in particular in the WZ working area. The dynamic engine allows for high realism and a wide variety of configurations, ensuring effective training of the neural network subsequently.

[0036] In a step 130, the digital twin generates simulated scenes of the sorting installation comprising in particular configurations of simulated articles to be sorted S.IT in the work zone WZ. These configurations can be characterized by one or more characteristics such as positions, orientations, and geometries of each of the articles of a given configuration, as well as by textures applied to volumes representing each of the objects, textures can include logos, patterns, address blocks, or other visual elements distinguishing one article or one type of article from another article or another type of article.Of course, the generation of articles from the simulated scenes can be controlled by conventional means such as simulation parameters, so as to reproduce the situations envisaged by the practitioner, for example with regard to the shapes, dimensions or even textures of the articles, the density, the number or the distribution of other different types of these.

[0037] In a step 110B, digital images R.IMG of real articles representative of the articles intended to be sorted by the FAC installation are collected and stored on a computer storage medium. This operation may, for example, consist of taking digital photos of each of the surfaces of postal articles or parcel packaging such as those which are to be sorted, by sampling in the sorting installation of already sorted articles or at suppliers of article packaging.

[0038] At a step 140, the CALC calculator projects textures from the naked images R.IMG merics on the surfaces of the simulated articles S.IT, thus subsequently making it possible to obtain photorealistic viewpoints of the simulated scene. At a step 150, the CALC computer generates a digital image and a depth map of the scene. This digital image and this depth map constitute annotated T.DAT training data, the annotations indicating the respective characteristics of the articles in the scene: position, geometry, orientation, for example. The digital image can be generated in such a way as to include masks taking into account the occlusion of each of the articles.

[0039] The digital image and the depth map of the scene are preferably generated so as to be substantially as if they were obtained from digital images acquired by the digital camera CAM in the real sorting installation. This means that the viewing angle of the simulated scene chosen to generate the digital image and the depth map corresponds to the viewing angle from which the digital camera CAM observes the real scene. The objective is to train the neural network with data as close as possible to the data obtained from real shots that it will have to process following its training period. The training can be done without the depth map, which is only optional, but can improve the quality of the training.

[0040] An important advantage of this training data is that, being entirely generated automatically by calculation, the characteristics of the items in the scene, generated during the simulation of the scene, are perfectly known and the annotations, also automatic, are immediately available and necessarily accurate, which gives this method a decisive advantage over conventional methods for producing annotated images.

[0041] An additional advantage of the method described herein is that the operator can choose to simulate expected scenes, in order to anticipate a potentially problematic situation.

[0042] An example is the arrival of a new type of packaging for the items to be sorted, for example by the arrival of a new economic actor producing items to be sorted having a new design, such as a set of particular patterns on the packaging. In this case, this known design can be integrated in advance into the training data in order to prepare the neural network for sorting this new type of packaging.

[0043] Another example is that of the massive arrival of a type of articles to be sorted, thus forming homogeneous batches of articles to be sorted. A type of article can be defined by a shape (cylindrical, parallelepiped, hexagonal), a texture (pattern which will be applied to the volume of a simulated article), a logo, a specific color etc... or a combination of any number of these characteristics. A type of article can include subtypes: in the case of a type defined by a parallelepiped shape, specific relationships between a length, a width and a depth can form a subtype. Articles of the same type can, for example, come from the same economic actor and present the same visual identity, characterized by the type to which the articles belong, such as a texture of the packaging used by this economic entity.

[0044] Outside of this context of the massive arrival of articles of the same type, a usual situation could be that of heterogeneous batches of articles to be sorted, that is to say batches made up of articles of different types. In this usual situation, the articles can then be distinguished relatively easily from one another. On the other hand, in the case of the arrival of a homogeneous batch of articles, that is to say a batch of articles resembling each other, in particular because they belong to the same type of article, these articles can be difficult to distinguish from one another in the images used for sorting. Such a situation can be anticipated by generating training images comprising homogeneous batches of articles in order to train the neural network to manage this specific situation.“Images comprising homogeneous batches of items” are images generated by the digital twin configured to generate images with items of the same type. The operator can thus choose to generate training digital images of homogeneous batches of items comprising more than 40%, more than 50%, more than 80%, more than 90% or 100% of items belonging to the same type, according to conventional methods of adjusting the simulation parameters. Trained on the basis of these images, the neural network will be better prepared to process real images.

[0045] Steps 130 to 150 are repeated so as to obtain a number of images considered sufficient by the system operator for training the neural network.

[0046] At a step 160 of training the neural network, the annotated training data are provided by the calculator CALC as input to the neural network NN during its training phase. During this training phase, the internal parameters P; of the network, which characterize its architecture and therefore its response to data provided as input, converge towards values ​​allowing a reliability deemed sufficient for the processing of this data, reliability estimated for example by a success rate of the network in processing the data and which must exceed a given threshold to be considered sufficient. This rate reflects the level of adequacy of the responses of the network to the data provided in relation to the expected responses, corresponding to the annotations.

[0047] Once the training phase is completed, the internal parameters of the neural network are adjusted so as to provide appropriate responses to the data. provided as input to the network, and it can be used during the sorting process of real items.

[0048] In a step 170, the digital camera CAM acquires digital images of a real scene encompassing the working area WZ in the form of a color digital image and two images intended to form a depth map of the scene. The computer CALC generates the depth map.

[0049] At a step 180, the resulting data R.DAT (unannotated) from step 170 comprising the color image and the depth map of a real scene are provided to the neural network NN. The neural network, trained for this task, then identifies the positions, geometries and PGO orientations of the items in the scene. Generally speaking, the neural network could be designed and trained to identify any other characteristic of the item: a destination or origin address or an identification in a textual or digital code form such as a barcode or a QR code, a logo, a category such as a type of packaging (polybag, cardboard, envelope, etc.). Furthermore, this principle can be applied to any kind of object such as an item, a parcel, a box, a letter, a package or natural product or a manufactured product.

[0050] At a step 190, based on the positions, geometries and orientations PGO of the articles identified by the neural network, the computer CALC sends a control signal to the robotic manipulator arm MA so as to unpick the articles by manipulating them individually according to at least one of their respective positions, geometries and orientations, according to conventional methods.

[0051] Steps 170 to 190 are repeated so as to process all of the items appearing in the work zone WZ.

[0052] In the implementation described above, a CALC calculator is used for all the phases of generating the training data, training the neural network, and during the article sorting process. It is not necessary for the same calculator to be used for these different tasks: each could be carried out by a dedicated calculator, and the neural network could be transmitted from one to the other of these calculators.

[0053] In this document, the figures are not necessarily to scale. Some features and components may be shown exaggerated relative to other components or in a somewhat schematic form, and some details of conventional elements may not be shown in the interest of clarity and conciseness.

[0054] Of course, the invention is not limited to the embodiment described and variant embodiments can be made without departing from the scope of the invention as defined by the claims. 10

Claims

Claims

1. Method (100) for processing objects in an object processing installation (FAC) comprising a computer configured to characterize, by means of a neural network (NN), objects to be processed visible in digital images of objects to be processed located in the object processing installation (FAC), the method comprising: - a step (150) of generating digital training data (T.DAT) representative of simulated objects to be processed (S.IT) within the object processing installation (FAC), these data being intended for training the neural network, generated by means of a digital twin of the object processing system, and associated with annotations (ANN) indicating at least one characteristic of the simulated objects to be processed (S.IT); - a step (160) of training the neural network comprising providing the digital training data (T.DAT) as input to the neural network (NN) and during which the parameters of the neural network are adjusted on the basis of the digital training data; - a step (170) of acquiring digital images of a real scene of the object processing installation (FAC) comprising at least one real object (IT) to be processed; - a step (180) comprising the supply of data (R.DAT) generated from the digital images of the real scene as input to the neural network (NN), the neural network providing in return data representative of a characteristic (PGO) of the real object (IT) to be processed; and - a step (190) comprising sending a control signal to an object processing device (MA) on the basis of the data representative of the characteristic of the at least one real object (IT) to be processed.

2. The method (100) of claim 1, wherein the at least one characteristic of the simulated objects to be processed is selected from the position, geometry and orientation (PGO) of the simulated objects to be processed.

3. Method (100) according to claim 1 or 2, wherein the digital training data (T.DAT) comprises at least one digital image of a simulated scene representing a working area (WZ) of the object processing facility (FAC).

4. The method (100) of claim 3, wherein the at least one digital image is generated in such a way as to be as if it were obtained from digital images acquired by a digital camera (CAM) installed above the working area (WZ) of the actual processing facility.

5. The method (100) of claim 3 or 4, wherein the digital training data further comprises a depth map of the simulated scene.

6. A method (100) according to claim 5, wherein the depth map is generated so as to be as if it were obtained from digital images acquired by a digital camera (CAM) installed above the working area (WZ) of the actual processing facility.

7. Method (100) according to any one of the preceding claims 1 to 6, in which the digital twin integrates a dynamic engine for simulating physical interactions between simulated elements of the treatment installation, so as to reproduce movements and shocks between simulated objects to be treated and simulated elements of the treatment installation.

8. Method (100) according to any one of the preceding claims 1 to 7, further comprising a step (140) of projecting textures from digital images (R.IMG) of real objects onto surfaces of the simulated objects to be processed (S.IT).

9. Method (100) according to any one of the preceding claims 1 to 8, in which the digital twin of the object processing system is configured so as to generate digital images of homogeneous batches of objects to be processed comprising more than 40%, preferably more than 50%, preferably more than 80%, preferably more than 90% or 100% of objects belonging to the same type of objects to be processed.

10. The method (100) of claim 9, wherein the type of objects is defined by a texture mapped onto surfaces of simulated objects.

11. Treatment installation (FAC) comprising a calculator (CALC) configured to implement the method according to any one of the preceding claims 1 to 10.

Images