How to process at least one image

Abstract

The present invention relates to a method for processing at least one image (1), comprising the steps of receiving image data, inputting the unmasked image data into an artificial neural network for detecting one or more pre-trained target regions (3), detecting one or more regions of interest (2) in the image (1) and creating an image mask dependent on the one or more regions of interest (2), and applying the created image mask to the image data to remove image regions (4) that do not contain the one or more regions of interest (2).

Description

[Technical Field]

[0001] The present invention relates to a method for processing at least one image. Furthermore, the present invention relates to a data processing device and an image acquisition device including such a data processing device. Furthermore, the present invention relates to a computer program product, a computer readable medium and a data transmission signal. [Background technology]

[0002] Image processing using artificial neural networks has been known for some time. A neural network can be used to detect a pre-trained target region within an image. To do this, a mask is applied to the image to remove portions of the image that are not of interest to the application. The masked image data is then input to the neural network. However, neural networks provide accurate results when the input data is normally distributed. For example, neural networks work well when color values ​​are normally distributed. When a mask is applied to remove one or more uninteresting image portions, the remaining image data input to the neural network no longer has a normally distributed color value. As a result, the pre-trained target region is not accurately recognized by the neural network. A similar problem applies to the training process when training images are applied to the neural network. As a result, the neural network cannot be trained to provide accurate results. Summary of the Invention [Problem to be solved by the invention]

[0003] It is an object of the present invention to provide a method that allows for more accurate recognition of pre-trained target regions. [Means for solving the problem]

[0004] The object of the present invention is solved by a method for processing at least one image, said method comprising the steps of: receiving image data; inputting the unmasked image data into an artificial neural network to detect one or more pre-trained target regions; detecting one or more regions of interest in the image and creating an image mask dependent on the one or more regions of interest; applying the created image mask to the image data to remove image regions that do not include the one or more regions of interest.

[0005] According to the present invention, unmasked image data is input to a neural network. This ensures that the input data, particularly color values, are normally distributed during normal operation or training. This allows the neural network to provide accurate results. In particular, the neural network accurately detects one or more pre-trained target regions. It has been recognized that target region detection and region of interest detection must be performed separately from each other. In particular, the same image data is used to detect target regions and regions of interest. This provides better results than masking the received image data before it is input to the neural network.

[0006] As will be described in more detail below, by removing image regions that do not contain one or more regions of interest, target regions of interest for a user and / or application can be easily identified. This has the advantage that training is simplified because training can be focused on identifying regions of interest and target regions. That is, training does not need to be performed on all other uninteresting objects, and training is independent of uninteresting regions, especially uninteresting objects. Training can be performed separately for target regions and regions of interest, which also simplifies training. Another advantage of the present invention is that by separately detecting one or more target regions and one or more regions of interest, hardware performance during the training and operational phases is improved. Another advantage of the present invention is that by separately detecting multiple regions of interest, whether overlapping or non-overlapping, different detections of one or more target regions for each region of interest can be applied and combined in parallel.

[0007] "Pre-trained" means that the neural network is trained based on training images to detect target regions in a training process.

[0008] The pre-trained target region can be selected flexibly. It can be one or more parts of one or more objects. Alternatively, the pre-trained target region can be one or more objects. The pre-trained target region can depend on the application in which the method is used. If the method is used for quality inspection, the pre-trained target region can be one or more defects, such as scratches. In the method of the present invention, the one or more defects are of interest located in a region of interest, particularly an object. Alternatively, the pre-trained target region can be the color of an object or object part, and the method can check whether the object or object part has a predetermined color. As another example, the pre-trained target region can be multiple parts of an object, and the method checks whether the image data includes the multiple object parts. If not, a mistake may have occurred during the manufacture of the object.

[0009] The phrase "detecting one or more pre-trained target regions" means that the neural network determines all pre-trained target regions in the input image. Thus, the output of the neural network is whether one or more pre-trained target regions exist in the image, particularly in the image data. This includes cases where the image does not contain a target region. In this case, the output of the neural network is that the image does not contain a target region.

[0010] The phrase "detecting one or more regions of interest" means that all regions of interest are determined in the input image. This includes the case where the input image is determined to contain no regions of interest. In this case, the image mask removes the entire image, since the entire image is deemed uninteresting. The regions of interest may be a portion of the image or may cover the entire image.

[0011] "Unmasked image data" means that the image data corresponds to the original image data acquired by the image acquisition device. By applying an image mask to the received image data, the data is altered, i.e., masked, so that it does not correspond to the original image data. If the region of interest covers the entire image, applying the image mask does not remove any image portions.

[0012] When multiple images are acquired, each image is processed in the manner described above. That is, for each image, one or more pre-trained target regions are detected, and one or more regions of interest are detected. Furthermore, for each image, an image mask is created, and each image mask is applied to each image. It is clear that the image data received in the method of the present invention corresponds to an image, particularly an acquired image.

[0013] The method can be performed by a data processing device. The data processing device can include or be one or more processors. Alternatively, the data processing device can be a computer. Image data acquired by the image acquisition device is sent to the data processing device, whereby the data processing device receives the image data from the image acquisition device and processes the image data.

[0014] According to one embodiment, the data processing device can process the image data to detect one or more pre-trained target regions, particularly those located in at least one pre-trained region of interest. This allows the location of the pre-trained target region and the location of the pre-trained region of interest to be determined. This allows determining whether the detected pre-trained target region is located in the pre-trained region of interest. The one or more pre-trained target regions, particularly those located in the pre-trained region of interest, can be output by a neural network. This is only possible if the image data includes one or more pre-trained target regions. The one or more pre-trained target regions, particularly those located in the at least one pre-trained region of interest, are target regions relevant to the user. The pre-trained target regions can be detected separately from the pre-trained region of interest, allowing accurate detection and taking into account all pre-trained regions of interest. As described below, other pre-trained target regions not located in the region of interest can be ignored for further processing of the image data or the pre-trained target regions.

[0015] One or more pre-trained regions of interest can be detected before the image mask is applied to the acquired image data. As described below, the one or more pre-trained regions of interest can be detected independently of the detection of the pre-trained target regions, or vice versa. That is, the one or more pre-trained regions of interest can be detected before, in parallel with, or after the detection of the pre-trained target regions. This also applies to the training process described above. The training process is also independent of whether the regions of interest or target regions are detected first.

[0016] The pre-trained region of interest may be at least one part of at least one object or at least one object. The object may be a discrete object. A discrete object is an object that has a clearly defined boundary and spatial extension, thereby spatially invariant properties. Thus, the object may be anything that is visible and / or tangible and can be touched, such as a car, a chair, etc. Additionally or alternatively, the object may be small enough that its image can be captured by an image capture device, in particular a camera and / or a mobile phone and / or a tablet and / or a microscope, etc.

[0017] The pre-trained target region may have different optical properties than the region of interest. For example, the pre-trained target region may have a different color, brightness, etc. Additionally or alternatively, the pre-trained target region may correspond to the region of interest or may be smaller than the region of interest. In other words, the pre-trained target region may have a particular shape, and the cross-section of the shape may be smaller than the cross-section of the region of interest. The pre-trained target region may correspond to a region of interest, particularly a defect in an object. Additionally or alternatively, the pre-trained target region may have different optical properties, e.g., color, than the optical properties of the pre-trained region of interest.

[0018] The image mask can be configured to remove one or more pre-trained target regions located outside the pre-trained region of interest. Additionally or alternatively, the image mask can be applied to the image data after one or more trained target regions have been detected. That is, the image is configured not to remove one or more pre-trained target regions located in at least one region of interest. If the image data does not contain a region of interest, the image mask removes all pre-trained target regions. Masking the image data is used to simplify and / or accelerate processing of the pre-trained target regions because all image regions that are not interesting and / or relevant are removed from the image data. In this way, image processing after image masking is faster because there is less image data to process than if the entire image data were processed. The image mask can be applied to the entire image data, not just a portion of the image data.

[0019] The image mask creation process can be performed as follows: The received image is represented by a matrix. The matrix can include a value for each pixel in the image. The image mask can also be represented by a matrix. In a first step, the values ​​of the image mask can have a value of 0. After one or more regions of interest are detected, in a second step, the values ​​of the image mask can be modified. The pixel values ​​of the image mask are modified by applying the image mask with the modified values ​​to the received image so that image regions that do not contain regions of interest are removed. This can be achieved when pixel values ​​corresponding to regions that are not regions of interest are 0. Alternatively, the pixel values ​​can have a different value, which can determine that the pixel does not belong to the region of interest.

[0020] The image mask, represented by the matrix determined as described above, is applied to the received image as follows: As described above, the received image can also be represented by a matrix. By applying the image mask to the received image, the output is a matrix whose values ​​are the values ​​of the received image or modified by the image mask. The values ​​can be modified by the image mask to be 0, but different values ​​are also possible. Finally, pixels belonging to the region of interest can be identified by their values ​​after the image mask has been applied to the image.

[0021] The one or more target regions located in the pre-trained region of interest may be visualized, i.e., the output of the method and / or data processing device may be visualized, for example, on a display. Additionally or alternatively, the pre-trained region of interest and / or the removed image region are not visualized.

[0022] In one embodiment, the neural network may be a convolutional neural network. The neural network may include at least two layers. If the neural network includes only two layers, the neural network includes an input layer and an output layer. The neural network may include more than two layers.

[0023] The neural network can be configured to create an image mask and / or detect one or more regions of interest. In this case, the neural network performs all tasks, i.e., detecting pre-trained targets, creating an image mask, and detecting regions of interest. The tasks can be performed in parallel or subsequent to each other. The outputs of the neural network are the detected pre-trained target regions, the image mask, and the detected regions of interest. In this case, only one neural network is executed in the data processing device.

[0024] Alternatively, a further neural network can be implemented in the data processing device. The further neural network can be configured to create an image mask and / or detect regions of interest. The further neural network can output the created image mask and / or the detected regions of interest. The neural network and the further neural network can be implemented in parallel with each other or subsequent to each other. The same image data is input to both the neural network and the further neural network.

[0025] The further neural network may be a convolutional neural network. The further neural network may include at least two layers. If the further neural network includes only two layers, the further neural network includes an input layer and an output layer. The further neural network may include more than two layers. In the above embodiment, the neural network outputs the detected pre-trained target regions, and the further neural network outputs the detected regions of interest and the created image mask.

[0026] The one or more pre-trained target regions located in the at least one region of interest are determined by an intersection between the detected one or more pre-trained target regions and the detected one or more regions of interest. The intersection processing can be performed by the data processing device based on the output of the neural network and / or the further neural network. Alternatively, the intersection processing can be performed by the neural network. This is possible when only one neural network is provided.

[0027] Intersection processing is performed on the detected two-dimensional image and operates as follows: One or more detected target regions can be assigned one or more polygons, and one or more detected regions of interest can be assigned one or more other polygons. That is, a polygon is assigned to each target region, and another polygon is assigned to each region of interest. The polygons are configured to contain the target region. Similarly, the other polygons are configured to contain the region of interest. If a polygon in the target region and another polygon in the region of interest overlap or share a common surface, the target region and the region of interest intersect with each other. In this case, depending on the overlap, either a partial target region or the complete target region is considered. If a polygon and another polygon do not overlap or share a common surface, the target region and the region of interest do not intersect, and the target region is not considered.

[0028] According to one embodiment, the data processing device can be configured to detect a region of interest based on a tentative region of interest. The tentative region of interest is detected by a neural network or a further neural network based on input image data. This tentative region of interest often does not coincide with the region of interest shown in the image. This allows the data processing device to determine whether the periphery of the region of interest shown in the image is located in a predetermined region that includes at least a part of the periphery of the tentative region of interest. The predetermined region may be an area having a predetermined number of pixels in height and width.

[0029] The data processing device can set the region of interest, particularly to the periphery of the region of interest shown in the image, when the provisional region of interest in the predetermined region is offset from the periphery of the region of interest shown in the image. If the predetermined region does not include the periphery of the region of interest that is offset from the provisional region of interest, the data processing device sets the provisional region of interest as the region of interest. The above approach improves the detection of the region of interest. In particular, it is guaranteed that the detected region of interest, particularly the object, corresponds to the region of interest, particularly the object, shown in the image.

[0030] The data processing device can determine the number of detected pre-trained target regions. Furthermore, the data processing device can determine whether the number of pre-trained target regions corresponds to a predetermined number of target regions. The number of pre-trained target regions can be used for quality control. The data processing device can determine the quality of the region of interest based on the one or more detected pre-trained target regions. For example, the data processing device can determine that the quality of the region of interest, particularly the object, is poor if the number of detected pre-trained target regions does not correspond to the predetermined number of target regions.

[0031] According to one embodiment, a training process is performed. In the training process, the neural network and / or the further neural network are trained for the operational phase. The goal of the training is for the neural network to detect one or more target regions in the image data. If only one neural network is provided, the training goal is for the neural network to detect one or more further regions of interest and create an image mask. If the further neural network is executed on the data processing device, the goal of the further neural network is for the further neural network to detect one or more regions of interest and create an image mask.

[0032] The training process of the neural network may correspond to the training process of the further neural network. That is, both neural networks may be trained in the same way. That is, the same training images may be input to both neural networks. The neural network may be trained with unmasked or masked training images, and / or the further neural network may be trained with unmasked or masked training images. "Unmasked images" means that no mask is applied to the training images, and the training images input to the neural network and / or the further neural network correspond to the original training images. Therefore, no objects are removed in the unmasked training images. The training images may show multiple objects that do not correspond to the region of interest.

[0033] The training images input to the neural network and / or further neural network in the training phase may contain context information. Training images with context information means that they contain the target region and / or region of interest to be trained on. In contrast, training images with no context information means that the training images may or may not contain the target region, but may contain multiple different objects, i.e. regions of no interest.

[0034] The training process for training the neural network and / or the further neural network may include two training phases: The training may be performed to train the detection of target regions and / or regions of interest.

[0035] In the first training phase, training images containing non-contextual information are input to the neural network and / or the further neural network. The first training phase can be performed only once and can be considered a general training in which the neural network is trained with multiple different objects. In the first training phase, the neural network and / or the further neural network are trained with labeled images. By "labeled," we mean that the image data includes information about image height, image width, and other image information, such as color. Furthermore, the image signal includes information about what objects, such as screws, chairs, stairs, etc., are represented in the image signal. In this case, the training images input in the first training phase can include target regions and / or regions of interest. The designated or classified objects are at least partially surrounded by bounding boxes, allowing the neural network to recognize where a given transport material is located in the image data. The bounding boxes can be polygonal.

[0036] In a first training process, the neural network being trained is input with a large number of images, particularly millions of images containing information about objects and their locations, as discussed above. As discussed above, the images may show a variety of different objects, thereby allowing for target regions and / or regions of interest to be included in the images, but need not.

[0037] After the first training phase is completed, the neural network and / or the further neural network are trained in a second training phase. In the second training phase, training images containing the target region and / or region of interest are input to the neural network and / or the further neural network. The second training phase is used to train the neural network and / or the further neural network for the application in which the neural network will be used. Typically, the neural network is retrained if it is to be used in a different application.

[0038] During training, the neural network and / or further neural network to be trained can receive training images containing target regions and / or regions of interest, and possibly other objects, or training images that do not contain target regions and / or regions of interest. However, it is advantageous if 20-100%, particularly 80-95%, and preferably 90-95% of the training images provided show target regions and / or regions of interest. In the second training phase, the same training images as in the first training phase can be provided. Alternatively, different training images can be provided. At least one training image, particularly multiple training images, can be provided to the neural network and / or further neural network in the second training phase. This allows for labeling some of the training images while leaving other parts unlabeled. Alternatively, not all images can be labeled. In this case, all training images containing target regions and / or regions of interest are labeled. Training images that do not contain target regions and / or regions of interest are not labeled.

[0039] According to another aspect, a data processing device is provided. The data processing device includes means for performing the method of the present invention. Further, an image capture device for capturing an image is provided. The image capture device can be at least one of a camera, a mobile phone, a microscope, and a tablet. The data processing device can be part of the image capture device. Alternatively, the data processing device can be electrically connected to the image capture device. The capture device can be configured to capture visible light. By "visible light" it is meant that the captured light has a wavelength in the range of 380 nanometers to 750 nanometers.

[0040] According to a further aspect of the present invention, there is provided a computer program product comprising instructions which, when executed by a data processing apparatus, in particular a computer, cause the data processing apparatus, in particular a computer, to perform the steps of the method of the present invention. Further, there is provided a computer-readable data carrier having stored thereon the computer program product. Also provided is a data transmission signal carrying the computer program product.

[0041] The method of the present invention can be used in conveyor systems, in particular for determining the quality of objects, in particular goods, transported by the conveyor system, where the objects, in particular goods, correspond to or form part of the area of ​​interest, or a part of the object, in particular goods, can be the area of ​​interest. [Brief explanation of the drawings]

[0042] In the figures, the subject matter of the invention is shown diagrammatically, with identical or similar active elements being provided with mostly identical reference numerals. [Figure 1] 1. Apparatus comprising an image acquisition device and a data processing device for carrying out the inventive method according to a first embodiment. [Figure 2] 2. Apparatus comprising an image acquisition device and a data processing device for carrying out the method of the invention according to a second embodiment. [Figure 3A] 5A to 5C are images illustrating the different processing steps of the method of the invention. [Figure 3B] 5A to 5C are images illustrating the different processing steps of the method of the invention. [Figure 3C] 5A to 5C are images illustrating the different processing steps of the method of the invention. [Figure 3D] 5A to 5C are images illustrating the different processing steps of the method of the invention. [Figure 4A] 1 illustrates a training phase for training a neural network to detect target regions. [Figure 4B] 1 illustrates a training phase for detecting regions of interest. [Figure 5] Processing to determine regions of interest. [Figure 6] A conveyor system including an apparatus according to Figure 1 or Figure 2. DETAILED DESCRIPTION OF THE INVENTION

[0043] 1 shows an apparatus 8 including an image acquisition device 6 and a data processing device 5, where the data processing device 5 is electronically connected to the image acquisition device 6 such that data transfer between the two devices is possible. The image acquisition device 6 may include or be a camera. Alternatively, the image acquisition device 5 may be a camera, a mobile phone, a tablet, or a microscope. The data processing device 5 may include or be at least one processor. Alternatively, the data processing device 5 may be a computer.

[0044] The image is acquired by optical means such as a lens of the image acquisition device 6. The acquired image data is output via an output section 60 of the image acquisition device 6. The output section 60 is connected to the input section 50 of the data processing device 5 so as to enable data exchange between the image acquisition device 6 and the data processing device 5.

[0045] The data processing unit 5 includes a processing section 51. The processing section 51 includes a first processing section 510 and a second processing section 511. The first processing section 510 includes a neural network, and the second processing section 511 includes a further neural network. Both sections receive the image data received from the input section 50. The data processing unit 5 further includes an intersection section 52. The intersection section 52 receives the outputs of the neural network and the further neural network. The intersection section 52 also serves as an output of the data processing unit 5 and is connected to a display input section 90 for data exchange between the data processing unit 5 and a display 9 of the device 8.

[0046] Next, a method for processing the images acquired by the image acquisition device 6 will be described.

[0047] In a first step, an image 1 is acquired by the image acquisition device 6. This image 1 is sent to the data processing device 5, particularly to the input section 50 of the data processing device 5. The received image data is processed in the processing section 50. In particular, in a second step, this image data is sent to a neural network of the first processing unit 510. The sent image data is unmasked. In the second step, the neural network detects one or more pre-trained regions 3. The output of the neural network is whether the image data includes one or more pre-trained target regions 3. In particular, the output of the neural network is the number and positions of one or more pre-trained target regions 3 in the image 1. The output also includes the case where the image data does not include a pre-trained region 3. In this case, the output of the neural network is that the image data does not include a pre-trained region 3. The neural network is processed in the first processing section (not shown) of the data processing device 5.

[0048] The neural network is pre-trained to detect one or more pre-trained target regions 3. Training of the neural network is performed in a training process T, which is described in more detail in FIG. 4A. The neural network may be a convolutional neural network. Additionally or alternatively, the neural network may include at least two layers.

[0049] In a third step, one or more regions of interest 3 in the image 1 are detected in the processing section 51. The detection can be performed by the same neural network as described above in the second step S2.

[0050] Alternatively, the detection can be performed by a further neural network. This case is shown in FIG. 1. For this, the image data received at the input section 50 is transmitted in a third step to a further neural network of the second processing unit 511. The transmitted image data is not masked. The further neural network detects in a third step whether the image data contains one or more regions of interest 2. Furthermore, in a third step, the further neural network creates an image mask, where the image mask depends on the one or more regions of interest 2. The image mask is created after the one or more regions of interest 2 have been detected. Furthermore, in a third step, the image mask is applied to the image data. This removes all image data that does not contain one or more regions of interest 2.

[0051] The output of the further neural network is whether the image data contains one or more regions of interest. In particular, the further neural network determines the number of regions of interest and / or their positions and the shape of the regions of interest. By shape, it is meant that the perimeter of the regions of interest is determined. Image regions 4 that do not contain one or more regions of interest 2 are not output.

[0052] The further neural network is pre-trained to detect one or more regions of interest 2. Training of the further neural network is performed in a training process T, which is described in more detail in FIG. 4B. The further neural network may be a convolutional neural network. Additionally or alternatively, the further neural network may include at least two layers.

[0053] The received image data is input to the neural network and the further neural network, which can process the input data in parallel with each other.

[0054] In the fourth step, the output of the neural network is crossed with the output of a further neural network in the intersection section 52. That is, by crossing the outputs of the two neural networks, a target region 3 located in the region of interest 2 is detected. The output of the fourth step is information that the target region 3 is located in the region of interest 2. In the fifth step, this output is sent to the display input section 90 and displayed.

[0055] It is possible to display a region of interest 2 having one or more target regions 3 disposed therein. Alternatively, the region of interest 2 may not be displayed, and only one or more target regions 3 may be displayed.

[0056] The above method steps are performed in a data processing unit 5 and define a computer-implemented method for processing an image 1 according to the present invention.

[0057] 2 shows an apparatus 8 having an image acquisition device 6 and a data processing unit 5 for performing the method of the present invention according to a second embodiment. The second embodiment differs from the first embodiment shown in FIG. 1 in the processing order. In the second embodiment, the image data received from the image acquisition device 6 is input only to the neural network of the first processing unit 510 in a second step. The input image data is not masked.

[0058] In contrast, the image output received by the neural network is sent in a third step to a further neural network of the second processing unit 511. The further neural network processes the output of the neural network. In particular, the further neural network processes the image data that is part of the output of the neural network, as shown in Figure 1. Furthermore, the further neural network does not process the target region 3 detected by the neural network of the first processing unit 510.

[0059] The data processing device 5 is configured to process the output of the further neural network in a fourth step. The output of the further processing device corresponds to the outputs of the neural network and the further neural network shown in Figure 1. That is to say, the output of the further neural network shown in Figure 2 is information about one or more target regions and information about one or more regions of interest 2. This information is processed in a fourth step so that an intersection processing takes place in the intersection section 52. In the intersection processing, one or more target regions 3 located within the region of interest 2 are identified. Other method steps correspond to the method steps described in Figure 1.

[0060] A further difference from the method described in Figure 1 is that the data processing device 5 must include a neural network and a further neural network to process the second and third steps, whereas, as mentioned above, in the embodiment shown in Figure 1 the second and third steps can be performed by the same neural network.

[0061] 3A-3D show images 1 illustrating the different processing steps of the method of the present invention. FIG. 3A shows image 1 acquired by image acquisition device 6. Image data of the acquired image 1 is sent to data processing device 5. In both embodiments, the image data is input to a neural network. The neural network in processing section 51 detects target regions 3 and does not determine regions of interest 2, as shown in FIG. 3B. That is, the output of the neural network is the location and number of target regions 3. This information is processed in a fourth step by data processing device 5. The image data input to the neural network is unmasked. That is, no image parts of the image acquired by acquisition device 6 are removed, and the input image data corresponds to the original image data acquired by image acquisition device 6.

[0062] Figure 3C shows the output of a further neural network of processing section 50. Image data of image 1 shown in Figure 3A is input to the further neural network. The further neural network generates an image mask that is dependent on region of interest 2. The further neural network then applies the image mask to the input image data. This removes image region 4 that does not contain region of interest 2. The image region 4 and region of interest 2 are shown in Figure 3C. The image data of the image shown in Figure 3C is the output of the further neural network.

[0063] Figure 3D shows the result after an intersection process has been performed by the data processing device 5. In the intersection process, the image data of the image shown in Figure 3B and the image data of the image shown in Figure 3C are intersected. In particular, one or more target regions 3 located within the region of interest 2 are determined. Other target regions 3 located outside the region of interest 2 are removed. Figure 3D shows a target region detected by a neural network located within a region of interest 2 detected by a further neural network.

[0064] 4A shows a training process for training a neural network to detect a target region 3. The training process T includes two training phases T1 and T2.

[0065] The first training phase T1 corresponds to a general training in which multiple training images are input to the neural network. The training images may, but do not necessarily, include the target region 3. The training images, in particular all training images, input to the neural network in the first training phase T1 are labeled. Furthermore, the training images show multiple different objects and / or information about the object locations. Thus, after the first training phase T1, the neural network knows to identify multiple different types of objects. The first training phase T1 can be performed only once to train the neural network.

[0066] After the first training phase is finished, a second training phase T2 is started. The second training phase T2 is used to train the neural network for the application in which the device 8 is used. Thus, the training images input to the neural network include the target region 3. Additionally, training images that do not include the target region 3 are input to the neural network. Contrary to the first training phase, at least some or all of the training images are unlabeled.

[0067] After the neural network has performed the second training phase T2, the neural network is able to recognize whether the input image data contains one or more target regions 3.

[0068] 4B shows the training process for detecting regions of interest. Depending on whether the data processing device 5 includes only a neural network or a neural network and a further neural network, the training process is performed for the neural network or the further neural network. The following statements apply to both cases. That is, the training is the same regardless of whether a further neural network is present or not.

[0069] The first training phase T1a corresponds to a general training in which multiple training images are input to the neural network or further neural network. The training images may, but do not necessarily, include a region of interest 2. The training images, in particular all training images, input to the neural network in the first training phase T1a are labeled. Furthermore, the training images show multiple different objects and / or information about the object locations. Thus, after the first training phase T1a, the neural network or further neural network knows to identify multiple different types of objects. The first training phase T1a can be performed only once to train the neural network.

[0070] After the first training phase T1a is finished, a second training phase T2a is started. The second training phase T2a is used to train a neural network for the application in which the device 8 is used. Thus, the training images input to the neural network or further neural network include the target region 3. Furthermore, training images that do not include the target region 3 are input to the neural network or further neural network. Contrary to the first training phase, at least some or all of the training images are unlabeled.

[0071] After the neural network has performed the second training phase T2a, the neural network is able to recognize whether the input image data contains one or more regions of interest 2.

[0072] Fig. 5 shows a process for determining a region of interest 2. As described above, the neural network or further neural network detects one or more regions of interest 2, on which an object detection process is performed. Fig. 5 shows a preliminary region of interest 2a determined by the neural network or further neural network. Furthermore, Fig. 5 shows an original region of interest 2b in an image 1 acquired by an image acquisition device 6.

[0073] The data processing unit 5 checks whether the periphery of the preliminary region of interest 2a is offset from the periphery of the original region of interest 2b in the predetermined region 10, as shown in the image 1. If the periphery of the preliminary region of interest 2a is offset, the data processing unit 5 sets a portion of the region of interest 2 to correspond to the periphery of the original region of interest. This process is performed along the circumferential direction of the original region of interest 2b. Therefore, after the process, the region of interest 2 to be used in the intersection process is determined. This region of interest 2 is shown on the right side of Figure 5. Figure 5 is used in the above-mentioned intersection process.

[0074] 6 shows a conveyor system 11 including an apparatus 8 according to FIG. 1 or 2. The conveyor system 11 includes a conveyor belt 8 on which objects are transported. The objects correspond to the region of interest 2 described above. The conveyor belt 12 is arranged to pass through a monitoring region 13 of the image acquisition device 6. That is, all objects placed on the conveyor belt 12 pass through the monitoring region 13. The objects may correspond to the region of interest 2 or may be part of the region of interest 2.

[0075] An image capture device 6 captures images 1 of a surveillance area 13 containing objects. The method described above is then performed by a data processing device 5 to detect one or more target areas 3 within each area of ​​interest 2. [Explanation of symbols]

[0076] 1 image 2. Areas of Interest 2a Preliminary Areas of Interest 2b Original region of interest 3. Target Object 4. Areas of no interest 5 Data Processing Unit 6. Image acquisition device 8 equipment 9. Display 10 Predetermined area 11 Conveyor System 12 Conveyor Belt 13 Monitoring area 60 Output Section 50 Input Section 51 Processing Section 52 Intersection 510 First Processing Section 511 Second Processing Section 90 Display Input Section T Training process T1 First training phase T2 Second training phase

Claims

1. 1. A method of processing at least one image, the method comprising: receiving image data; inputting the unmasked image data into an artificial neural network to detect one or more pre-trained target regions; detecting one or more regions of interest in the image and creating an image mask dependent on the one or more regions of interest; applying the created image mask to the image data to remove image regions that do not include the one or more regions of interest; wherein one or more pre-trained target regions located in at least one pre-trained region of interest are detected.

2. The method of claim 1 , wherein the one or more pre-trained regions of interest are detected before the image mask is applied to the received image data.

3. The method of claim 1 , wherein the pre-trained region of interest is at least one portion of at least one object or at least one object.

4. a) the pre-trained target region has different optical properties than the region of interest; and / or b) the pre-trained target region corresponds to or is smaller than the region of interest; The method of claim 1.

5. a) the image mask is configured to remove one or more pre-trained target regions that are located outside the pre-trained region of interest; and / or b) the created image mask is applied to the image data after the one or more pre-trained target regions are detected; The method of claim 1.

6. a) the one or more target regions located in the pre-trained region of interest are visualized; and / or b) the pre-trained region of interest and / or the removed image region are not visible; The method of claim 1.

7. a) the neural network is a convolutional neural network, and / or b) the neural network comprises at least two layers; The method of claim 1.

8. The method of claim 1 , wherein the neural network generates the image mask and / or determines the region of interest.

9. The method of claim 1 , wherein a further neural network generates the image mask and / or determines the region of interest.

10. a) the further neural network is a convolutional neural network, and / or b) the further neural network comprises at least two layers; 10. The method of claim 9.

11. 10. The method of claim 1, wherein the one or more pre-trained target regions located in the at least one region of interest are determined by an intersection of the detected one or more pre-trained target regions and the detected one or more regions of interest.

12. The method of claim 11, wherein to determine the region of interest, a perimeter of a tentative region of interest is determined, and it is determined whether the perimeter of the region of interest shown in the image is located in a predetermined area including at least a portion of the perimeter of the tentative region of interest.

13. The method according to claim 12 , wherein when the tentative region of interest is offset from the periphery of the region of interest within the predetermined region, the region of interest is set to the periphery of the region of interest.

14. a) determining whether the number of pre-trained target regions corresponds to a predetermined number of target regions; and / or b) determining a quality of the region of interest based on the detected one or more pre-trained target regions; The method of claim 1.

15. The method of claim 9 , wherein the training of the neural network corresponds to the training of the further neural network.

16. the neural network is trained using unmasked training images; and / or the further neural network is trained using unmasked training images; 10. The method of claim 9.

17. The method of claim 1 , wherein the neural network and / or the further neural network are trained using training images, the training images including context information.

18. The method of claim 1 , wherein the training of the neural network and / or the further neural network comprises two training phases.

19. 19. The method of claim 18, wherein in a first training phase, training images containing non-contextual information are input to the neural network and / or the further neural network.

20. 19. The method of claim 18, wherein in a second training phase, training images containing the target region and / or the region of interest are input to the neural network or the further neural network.

21. Data processing apparatus including means for carrying out the method according to any one of claims 1 to 20.

22. 22. An image acquisition device for acquiring an image of an object, comprising a data processing device according to claim 21.

23. A computer program which, when executed by a data processing apparatus, causes the data processing apparatus to carry out a method according to any one of claims 1 to 20.

24. 24. A computer readable medium storing the computer program of claim 23.

25. 22. A conveyor system for determining the quality of transported goods, the conveyor system including a data processing device according to claim 21.

Images