Method, device and mechanism for positioning a bone portion present in a poultry leg

Abstract

The invention relates to a method and an arrangement for training at least one neural network for locating a bone portion present in a poultry leg (10). The invention also relates to a method and an apparatus for locating a bone portion present in a poultry leg (10), wherein the method comprises the steps of conveying a plurality of poultry legs (10) along a conveying direction (16) by a conveying device (23), acquiring a plurality of digital images (25) of the front or rear side of each poultry leg (10) conveyed past the imaging system (24) by a first optical imaging system (24), sequentially providing the plurality of digital images (25) as input data to a first neural network for locating a bone portion, wherein the first neural network for locating a bone portion has been trained by the method according to any one of claims 1 to 6, and determining location data (31) of the bone portion by the first neural network, and providing the displayed and / or transmitted location data (31) to a downstream machine (26) for processing the plurality of poultry legs (10) based on the determined location data.

Description

Technical Field

[0001] This invention relates to a method for training at least one neural network for locating bone portions present in poultry legs, and a non-volatile computer-readable storage medium comprising a program including instructions for causing a computer to perform the method. The invention also relates to a method for locating bone portions present in poultry legs, a mechanism for training a method for locating bone portions present in poultry legs, and an apparatus for locating bone portions present in poultry legs. Background Technology

[0002] This method, mechanism, and equipment are used for automated processing of poultry. To separate the meat from the bone or bony portion of the poultry leg, the first step is to determine the exact location of the bone portion so that the necessary incisions for deboning can be optimally placed. In particular, the location and / or placement of the thigh bone, tibia, and knee are especially important.

[0003] Document EP 2 532 246 B1 discloses a method for deboning bone-in meat using an X-ray system. The poultry leg to be processed is passed between an X-ray source and an X-ray detector, and the location and position of the bone are determined by analyzing the obtained X-ray data.

[0004] The drawback is that these known methods always require X-ray imaging during operation. This presents particular challenges in terms of workplace safety, and is also costly and requires extensive maintenance. Furthermore, handling X-rays necessitates highly trained and skilled personnel. Summary of the Invention

[0005] Therefore, the object of the present invention is to provide corresponding methods, apparatus and mechanisms that allow the bone portions in poultry legs to be located with high precision and at the lowest possible cost in terms of equipment.

[0006] The objective is achieved through the aforementioned method for training at least one neural network for locating bony portions present in poultry legs, the method comprising the steps of: providing a plurality of poultry legs; recording images of the front or rear sides of the plurality of poultry legs in the optically visible wavelength range using an optical camera to generate optical image data for each of the plurality of poultry legs; irradiating the rear or front side of the plurality of poultry legs with X-rays from an X-ray source, and recording X-ray images of the side of the plurality of poultry legs away from the X-ray source using an X-ray imaging system to generate X-ray images for each of the plurality of poultry legs. Linear image data; defining reference points for marking the location of the bone portion based on the X-ray image data; overlaying the optical image data with the location of the bone portion and / or the X-ray image data to generate mixed image data for each of the plurality of poultry legs; defining reference points for marking the location of the bone portion based on the mixed image data; inputting the image data from the optical camera as input data and the reference points as target data as training data for the neural network; repeatedly adjusting the weights of the neural network based on the difference between the target data and the output data generated by the neural network.

[0007] The method according to the invention has the advantage of using X-ray imaging only during the training or learning phase. Once the neural network has been trained using the method according to the invention, it is possible to reliably determine the location of bony portions in a poultry leg based solely on optical images of the leg. For this purpose, optical image data and X-ray image data are superimposed on each other. Thus, the mixed image data represents a superimposed image containing both optical and X-ray images of the complete poultry leg. Therefore, it is possible to identify the exact location and position of bony portions in the poultry leg based on the mixed image and to correlate them with an external view of the complete poultry leg. In this way, a correlation is established between the location of the bony portions and the external morphology of the poultry leg.

[0008] An advantageous embodiment of the invention is characterized in that the reference points include a thigh reference point, a calf reference point, and a knee reference point. These reference points provide sufficient accuracy for determining the location and position of the thigh, calf, and knee. Simultaneously, the algorithmic complexity for determining these reference points is reduced to a necessary minimum.

[0009] According to another preferred embodiment of the invention, the thigh reference point and the lower leg reference point are point pairs, each representing the location of a bone end region. Based on these point pairs, the location and orientation of the bones in the poultry leg can be defined sufficiently accurately. Particularly preferably, in each case, the point pair is located at the head of the bone, preferably at the midpoint of the relevant bony portion relative to the longitudinal direction of the bone.

[0010] A preferred further improvement of the invention is that the knee reference point is formed as a point cloud comprising at least one point, wherein the point of the point cloud references the edge position of the knee. Generally, a point cloud with one point is sufficient to indicate the location of the knee. Alternatively, this one point is chosen such that it is located in the middle of the knee.

[0011] Another advantageous embodiment of the invention features that the point cloud includes at least an upper knee reference point and a lower knee reference point, wherein the upper and lower knee reference points are located in the knee edge region. By specifying the above two reference points, it is possible not only to determine the position of the knee but also to estimate its size, particularly its length and width. Particularly preferably, in addition to the above upper and lower knee reference points, the point cloud also includes a third reference point, such that the three reference points form a triangle whose area covers the knee as much as possible. As previously mentioned, the above three reference points are then preferably located at the edge of the knee.

[0012] According to another preferred embodiment, before generating the mixed image data, object-related image regions of the optical image data and the X-ray image data are extracted from the image background. Through extraction, image regions that do not display the poultry leg or its bony parts are masked. This prevents structures located in the background from being considered during the training of the neural network. Overall, the reliability and accuracy of bone localization are thus improved.

[0013] The objective can also be achieved by the aforementioned non-volatile computer-readable storage medium, the medium comprising a program including instructions for causing a computer to execute a method for training at least one neural network for locating bone portions present in poultry legs.

[0014] The objective is also achieved by the method described above for locating bone portions present in poultry legs, the method comprising the following steps: conveying a plurality of poultry legs along a conveying direction via a conveying device; acquiring a plurality of digital images of the front or rear sides of each of the plurality of poultry legs conveyed through the imaging system via a first optical imaging system; sequentially providing the plurality of digital images as input data to a first neural network configured for locating the bone portions, wherein the first neural network for locating the bone portions has been trained by the method described above for training at least one neural network; determining positional data of the bone portions via the first neural network; and providing the displayed and / or transmitted positional data to a downstream machine for processing the plurality of poultry legs based on the determined positional data.

[0015] The method according to the invention has the following advantages: it only requires acquiring images of the poultry leg within the optically visible range to locate the bony parts present in the leg. Therefore, the method according to the invention eliminates the need for X-ray imaging during operation. Consequently, the equipment costs are significantly reduced compared to methods known in the prior art. In particular, by eliminating X-ray imaging, the risks associated with X-ray imaging are eliminated, and the use of highly skilled personnel is eliminated. Therefore, the method according to the invention is much cheaper than methods known in the prior art. However, in terms of the accuracy of bone location, it is in no way inferior to X-ray-based methods.

[0016] An advantageous embodiment of the present invention is characterized in that the reference points include a thigh reference point, a lower leg reference point, and a knee reference point. These reference points provide sufficient accuracy for determining the location and position of the femur, lower leg, and knee. Simultaneously, the algorithmic complexity for determining these reference points is reduced to a necessary minimum.

[0017] According to another preferred embodiment of the invention, the thigh reference point and the lower leg reference point are point pairs, each representing the location of the bone end region. Based on these point pairs, the location and orientation of the bone in the poultry leg can be defined with sufficient accuracy. Particularly preferred is that the point pairs are located at the head of the bone in each case.

[0018] A preferred further improvement of the invention differs in that the knee reference point is formed as a point cloud comprising at least one point, wherein the point of the point cloud references the edge position of the knee. Generally, a point cloud with one point is sufficient to indicate the location of the knee. Alternatively, this one point is chosen such that it is located in the middle of the knee.

[0019] Another advantageous embodiment of the invention features that the point cloud includes at least an upper knee reference point and a lower knee reference point, wherein the upper knee reference point and the lower knee reference point are located in the knee edge region. By specifying the above two reference points, it is possible to determine not only the position of the knee but also its size. Particularly preferably, in addition to the above upper knee reference point and lower knee reference point, the point cloud also includes a third reference point, such that the three reference points form a triangle whose area covers as much of the knee as possible. As mentioned above, the above three reference points are then preferably located at the edge of the knee.

[0020] Another advantageous embodiment of the invention features a downstream machine control unit that, based on provided location data, determines the cutting path of the positioned bone portion and moves the downstream machine's blade along this cutting path to debone the poultry leg; the blade is adapted for controllable movement. In this way, the poultry leg can be optimally deboned. Knowing the location of the bone portion allows for the determination of an optimal cutting path that separates the meat from the bone, while avoiding contact between the downstream machine's blade and the bone portion of the poultry leg. Simultaneously, knowing the location of the bone portion allows the cutting line to be as close to the bone as possible, leaving as little residual meat as possible in the bone area. The location of the incision, typically in the knee area, can therefore also be accurately determined. Thus, the invention allows for the automatic and thorough deboning of the poultry leg.

[0021] A preferred embodiment of the invention is characterized in that, before providing the acquired digital images of the plurality of poultry legs as input data to the first neural network, they are fed to a leg-side detection device adapted to perform leg-side detection and determine whether each specific digital image is of a right or left poultry leg. If the digital image does not match the specified leg side, the image data of the relevant digital image is mirrored along a virtual axis to convert the digital image of the right poultry leg into a virtual digital image of the left poultry leg, and vice versa. Preferably, the virtual axis is a vertical axis. Advantageously, the first neural network is thus designed such that it is sufficient to detect only one leg type, i.e., left or right poultry legs. In this way, the complexity of the neural network is reduced, and the cost required for training the neural network is also greatly reduced since only one poultry leg type needs to be trained. For example, if the neural network is designed to process left poultry legs, then the digital image of the right poultry leg is first mirrored as described. As a result of the mirroring, a portion of the poultry leg image looks like an image of the left poultry leg. In this way, image data of both the right and left poultry legs can be used to locate the bony parts of the poultry leg through a neural network designed to process the left leg. If the neural network is designed to process the right leg, the procedure is reversed accordingly, and the partial image of the left leg is mirrored as described above before being processed by the neural network.

[0022] Another advantageous embodiment of the invention features that the leg-side detection is performed via a second neural network trained with images of poultry legs on the specified leg side. This offers the advantage of very high detection accuracy. Preferably, the second neural network is trained using only images of either the left or right poultry leg in each case. For example, if the second neural network is designed to detect the left poultry leg, then when an image of a right poultry leg is input, it indicates that the left leg was not detected. The absence of a detected left poultry leg means that the poultry leg in question is the right leg.

[0023] A preferred further improvement of the invention is that, before providing the plurality of digital images of the plurality of poultry legs as input data to the first neural network and / or before the leg-side detection, the plurality of digital images of the plurality of poultry legs are fed to a front and rear detection device adapted to perform front and rear detection and determine whether each particular digital image displays the front or rear side of the poultry leg. If the digital image does not match the specified front / rear side, the suspension bracket of the conveying device is rotated 180°, and a digital image of the side of the poultry leg facing the first optical imaging system is acquired by a second optical imaging system arranged downstream of the first optical imaging system relative to the conveying direction. The suspension bracket holds the poultry leg and is controllably pivoted about its vertical axis. This has the advantage that the poultry legs do not need to be pre-classified according to front / rear orientation. Therefore, front refers to the outer or skin side of the poultry leg, while rear refers to the inner or meat side of the poultry leg. Thus, regardless of the orientation of the poultry leg, the bone portion can be automatically and adequately located. Preferably, the front side of the poultry leg is always selected as the designated side because the greater amount of meat and the resulting curved surface on the front side provide a better reference point in the digital image for locating the bone portion.

[0024] Another advantageous embodiment of the invention features that the front and rear detections are performed using a third neural network trained with images of the specified front / rear poultry legs. This offers the advantage of very high detection accuracy. Preferably, in each case, the third neural network is trained using only images of the front or rear of the poultry leg. For example, if the third neural network is designed to detect the front of the poultry leg, then when an image of the rear of the poultry leg is input, it will indicate that the front was not detected. The absence of a detected front means that the relevant side is the rear.

[0025] The objective is further achieved by a mechanism for training at least one neural network for locating bony portions present in poultry legs, the mechanism comprising: an optical camera adapted to record images of the front or rear sides of the plurality of poultry legs in the optically visible wavelength range, and configured to generate optical image data for each of the plurality of poultry legs; an X-ray source and an X-ray imaging system, the X-ray source adapted to irradiate the rear or front side of the plurality of poultry legs with X-rays, the X-ray imaging system adapted to record X-ray images of the side of the plurality of poultry legs away from the X-ray source, and configured to generate X-ray image data for each of the plurality of poultry legs; and a display and input device configured to display the X-ray image data. The system includes: X-ray image data and / or displaying mixed image data, and inputting reference points to be defined, the reference points being used to mark the location of the bone portion; an overlay unit configured to overlay the optical image data with the X-ray image data and / or the reference points to generate the mixed image data for each of the plurality of poultry legs; at least one neural network; and a learning cycle control unit configured and adapted to input the image data as input data and the reference points as target data as training data for the neural network, wherein the learning cycle control unit is adapted to repeatedly adjust the weights of the neural network based on the difference between the target data and the output data generated by the neural network.

[0026] The mechanism according to the invention has the advantage of using X-ray imaging only during the training or learning phase. Once the neural network has been trained using the method according to the invention, it is possible to reliably determine the location of bony parts in a poultry leg based solely on optical images of the leg. During the training phase, optical image data and X-ray image data are superimposed. Thus, the mixed image data represents a superimposed image containing both optical and X-ray images of the complete poultry leg. Therefore, it is possible to identify the exact location and position of bony parts in the poultry leg based on the mixed image and associate them with an external view of the complete poultry leg. In this way, the correlation between the location of bony parts and the external morphology of the poultry leg is established and learned through the neural network.

[0027] A preferred further improvement of the present invention is that the reference points include a thigh reference point, a calf reference point, and a knee reference point. These reference points provide sufficient accuracy for determining the location and position of the femur, calf, and knee. Simultaneously, the algorithmic complexity for determining these reference points is reduced to a necessary minimum.

[0028] According to another preferred embodiment of the invention, the thigh reference point and the lower leg reference point are point pairs, each representing the location of the bone end region. Based on these point pairs, the location and orientation of the bone in the poultry leg can be defined accurately. Particularly preferably, the point pairs are located at the head of the bone in each case.

[0029] Another advantageous embodiment of the invention features that the knee reference point is formed as a point cloud comprising at least one point, wherein the point in the point cloud references the edge position of the knee. Generally, a point cloud with one point is sufficient to indicate the location of the knee. Alternatively, this one point is chosen such that it is located in the middle of the knee.

[0030] According to another preferred embodiment of the invention, the point cloud includes at least an upper knee reference point and a lower knee reference point, wherein the upper and lower knee reference points are located in the edge region of the knee. By specifying the above two reference points, it is possible to determine not only the position of the knee but also its size. Particularly preferably, in addition to the above upper and lower knee reference points, the point cloud also includes a third reference point, such that the three reference points form a triangle whose area covers as much of the knee as possible. Preferably, as mentioned above, the above three reference points are located at the edge of the knee. Alternatively, as mentioned above, the point cloud may include only one point, preferably only the upper knee reference point or a point located in the middle of the knee.

[0031] Another advantageous embodiment of the invention features a mechanism configured to extract object-related image regions of the optical image data and the X-ray image data from the image background before generating the mixed image data. Through extraction, image regions that do not display the poultry leg or its bony parts are thus masked. This prevents structures located in the background from being considered during the training of the neural network. Overall, the reliability and accuracy of the bone location are thus improved.

[0032] The objective is further achieved by a device for locating bone portions present in poultry legs, the device comprising: a conveying device adapted to convey a plurality of poultry legs in a conveying direction; a first optical imaging system configured to acquire a plurality of digital images of the front or rear sides of the plurality of poultry legs; a first neural network configured to locate the bone portions and trained by the method according to any one of claims 1 to 6; and an input unit adapted to sequentially provide the plurality of digital images as input data to the first neural network, wherein the first neural network is adapted to determine position data of the bone portions and provide the displayed and / or transmitted position data to a downstream machine for processing the plurality of poultry legs based on the determined position data.

[0033] Therefore, the device according to the invention does not require X-ray imaging. Consequently, the cost of the equipment is significantly reduced compared to devices known in the prior art. In particular, by eliminating X-ray imaging, the risks associated with X-ray imaging are eliminated, and highly skilled personnel are no longer required. Furthermore, expensive X-ray imaging components are no longer needed during operation, and maintenance costs are significantly reduced. Therefore, the device according to the invention is much cheaper than devices known in the prior art. However, in terms of the accuracy of bone segment localization, it is by no means inferior to X-ray-based devices.

[0034] An advantageous embodiment of the present invention is characterized in that the reference points include a thigh reference point, a lower leg reference point, and a knee reference point. These reference points provide sufficient accuracy for determining the location and position of the femur, lower leg, and knee. Simultaneously, the algorithmic complexity for determining these reference points is reduced to a necessary minimum.

[0035] An advantageous embodiment of the invention is characterized in that the thigh reference point and the lower leg reference point are point pairs, each representing the location of the bone end region in each case. Based on these point pairs, the location and orientation of the bone in the poultry leg are defined sufficiently accurately. Particularly preferably, the point pair is located at the head of the bone in each case. Generally, a point cloud with a single point is sufficient to indicate the location of the knee. Alternatively, this single point is chosen such that it is located in the middle of the knee.

[0036] According to another preferred embodiment, the knee reference point is formed as a point cloud comprising at least one point, wherein the point in the point cloud references the edge position of the knee. Generally, a point cloud with one point is sufficient to indicate the location of the knee. Alternatively, this single point is chosen such that it is located in the middle of the knee.

[0037] Another advantageous embodiment of the invention features that the point cloud includes at least an upper knee reference point and a lower knee reference point, wherein the upper and lower knee reference points are located in the knee edge region. By specifying the above two reference points, it is possible to determine not only the position of the knee but also its size. Particularly preferably, in addition to the above upper and lower knee reference points, the point cloud also includes a third reference point, such that the three reference points form a triangle whose area covers as much of the knee as possible. As mentioned above, preferably, the above three reference points are located at the edge of the knee.

[0038] Another advantageous embodiment of the invention features a control unit in the downstream machine adapted to determine a cutting path along the located bone portion based on provided location data. This control unit is further configured to move the blade of the downstream machine along this cutting path to debone the poultry legs in order to process the plurality of poultry legs; the blade is adapted to move controllably. In this way, the poultry legs can be deboned optimally. Knowing the location of the bone portion allows for the determination of an optimal cutting path that separates the meat from the bone, while avoiding contact between the downstream machine's blade and the bone portion of the poultry leg in any way. At the same time, knowing the location of the bone portion allows the cutting line to be as close to the bone as possible, leaving as little residual meat as possible on the bone. The location of the incision, typically in the knee area, can therefore also be accurately determined. Thus, the present invention allows for the complete and automated deboning of poultry legs.

[0039] According to another preferred embodiment of the invention, the device further includes a leg-side detection device configured to perform leg-side detection based on the acquired digital images of the plurality of poultry legs before providing them as input data to the first neural network, and to determine whether each specific digital image is a right or left poultry leg. If the digital image does not match a specified leg side, the image data of the relevant digital image is mirrored along a virtual axis to convert the digital image of the right poultry leg into a virtual digital image of the left poultry leg, and vice versa. Advantageously, the first neural network is designed to detect only one leg type, i.e., left or right poultry leg, which is therefore sufficient. Thus, the complexity of the neural network is reduced, and the cost of training the neural network is also greatly reduced since it only needs to be trained with one poultry leg type. For example, if the neural network is designed to process left poultry legs, then, as described, the digital image of the right poultry leg is first mirrored. As a result of the mirroring, a portion of the poultry leg image then appears to be an image of the left poultry leg. In this way, image data of both the right and left poultry legs can be used to locate the bony parts of the poultry leg through a neural network designed to process the left leg. If the neural network is designed to process the right leg, the procedure is reversed accordingly, and the partial image of the left leg is mirrored as described above before being processed by the neural network.

[0040] According to another preferred embodiment of the invention, the leg-side detection includes a second neural network that has been trained using images of poultry legs on the specified leg side. This offers the advantage of very high detection accuracy. Preferably, the second neural network is trained using only images of either the left or right poultry leg in each case. For example, if the second neural network is designed to detect the left poultry leg, then when an image of a right poultry leg is input, it will indicate that the left leg was not detected. The absence of a detected left poultry leg means that the leg in question is the right leg.

[0041] Another advantageous embodiment of the invention features that the device further includes front and rear detection devices adapted to perform front and rear detection before the plurality of digital images of the plurality of poultry legs are provided as input data to the first neural network and / or before the leg side detection, and to determine whether each particular digital image displays the front or rear side of the poultry leg, and if the digital image does not match the designated front / rear side, to cause the suspension bracket of the conveying device to rotate 180° and acquire a digital image of the side of the poultry leg facing the first optical imaging system by a second optical imaging system arranged downstream of the first optical imaging system relative to the conveying direction, the suspension bracket holding the poultry leg and pivoting controllably about its vertical axis (44). This has the advantage that the poultry legs do not need to be pre-classified according to their front / rear orientation. Therefore, the bone portion can be located completely automatically regardless of the orientation of the poultry leg. Preferably, the front side of the poultry leg is always selected as the designated side because it provides a better reference point for the location of the bone portion in the digital image due to the larger proportion of meat and the resulting curved surface.

[0042] A preferred further improvement of the invention is that the front and rear detection devices include a third neural network trained using images of the specified front / rear poultry legs. This offers the advantage of very high detection accuracy. Preferably, in each case, the third neural network is trained using only images of the front or rear of the poultry leg. For example, if the third neural network is designed to detect the front of the poultry leg, then when an image of the rear of the poultry leg is input, it will indicate that the front side was not detected. The absence of a detected front side implies that the relevant side is the rear side.

[0043] The objective is also achieved by a non-volatile computer-readable storage medium comprising a program including instructions for causing a computer to execute the method described above for locating bone portions present in poultry legs. Attached Figure Description

[0044] Other preferred and / or advantageous features and embodiments of the invention will become apparent from the dependent claims and description. Particularly preferred embodiments will be explained in more detail with reference to the accompanying drawings, in which:

[0045] Figure 1 This is a plan view of a mechanism according to the invention for training at least one neural network for locating bone portions present in the legs of poultry.

[0046] Figure 2 This is a schematic diagram of the first optical imaging system and a poultry leg located in front of the first optical imaging system.

[0047] Figure 3 This is a schematic diagram of a hybrid image based on hybrid image data.

[0048] Figure 4 This is a schematic diagram of a device for positioning bone portions according to the present invention.

[0049] Figure 5 It is a block diagram.

[0050] Figure 6 This is a flowchart of the preprocessing.

[0051] Figure 7 yes Figure 4 The side view of the device shown in the image, and

[0052] Figure 8 It is a graphical representation of the front and rear detection devices. Detailed Implementation

[0053] The method according to the invention, the storage medium according to the invention, and the device according to the invention will be explained in more detail below.

[0054] Figure 1 This is a plan view of a mechanism according to the invention for training at least one neural network for locating bone portions present in a poultry leg 10. Figure 3 As shown, these bony parts are particularly the femur 11, tibia 12, and knee. The following description is intended both to explain the aforementioned mechanisms and to describe in more detail the method for training the aforementioned neural network according to the present invention.

[0055] To train the neural network, a plurality of poultry legs 10 are first required. The arrangement according to the invention includes an optical camera 14 adapted to record images of the front or rear sides of the poultry legs 10 in the optically visible wavelength range. The optical camera 14 is thus configured to generate optical image data for each poultry leg 10. Preferably, each poultry leg 10 is oriented with its front side facing the optical camera 14, such that only images of the front side of the poultry leg are recorded. However, it is also possible for each poultry leg 10 to be oriented with its rear side facing the optical camera 14. In this case, only images of the rear side of the poultry leg are recorded.

[0056] Each poultry leg 10 can be conveyed along the conveying direction 16 by a conveying device, for example, not shown in the figure. However, it is also possible to manually position the poultry leg 10 in front of the optical camera 14.

[0057] The mechanism according to the invention further includes: an X-ray source 18 adapted to irradiate the rear or front side of the poultry leg 10 with X-rays 17; and an X-ray imaging system 19 or an X-ray imaging sensor adapted to record X-ray images. The X-ray imaging system 19 is arranged on the side of the poultry leg 10 away from the X-ray source 18 and configured to generate X-ray image data. In this way, X-ray image data for each poultry leg 10 is generated.

[0058] The acquired optical and X-ray image data form the basis for training the first neural network. The optical and X-ray image data are fed into a superposition unit (not shown), which is adapted to superimpose the optical image data of one of the plurality of poultry legs 10 with the X-ray image data of the same poultry leg 10 to generate mixed image data for each poultry leg 10. Thus, the mixed image 15 of each poultry leg 10 represents a superimposed image obtained through superposition, in which the locations of the bony portions, particularly the femur 11, tibia 12, and knee, as well as the external shape of the poultry leg 10, are visible. Preferably, the images are recorded by the optical camera 14 and the X-ray imaging system 19 in such a way that the recorded image portions of each poultry leg are matched to each other as closely as possible. Preferably, the superposition is further adapted to determine this matching of image portions.

[0059] According to another advantageous embodiment of the invention, reference points 20 for marking the positions of bone portions are first defined based on X-ray image data. These reference points can be defined by an inspector or semi-automatically. The positions of the bone portions thus determined are then superimposed with the optical image data to generate mixed image data for each poultry leg 10.

[0060] The method and apparatus for training the neural network according to the invention further include displaying mixed image data via a display and an input device (not shown). Based on the displayed mixed data, reference points 20 for marking the location of bone portions are then defined, for example by an inspector or semi-automatically. Reference points 20 are input via the input device.

[0061] Figure 3 A schematic diagram of such a hybrid image 15 based on hybrid image data is shown as an example.

[0062] The invention also includes a neural network (not shown) and a learning cycle control unit. The learning cycle control unit is configured and adapted to receive optical image data and a plurality of reference points 20, preferably, reference points 20a, 20b, 20c, 20d, 20e, 20f, and 20g as training data for the neural network. Thus, the optical image data forms the input to the neural network, while the reference points 20 in each case correspond to the neural network's expected output data in response to the relevant optical image data, thereby forming the target data.

[0063] To train the neural network, the learning cycle control unit is adapted to repeatedly adjust the weights of the neural network based on the difference between the target data and the output data generated by the neural network. Preferably, the neural network is a multi-layer neural network with a corresponding number of hidden layers. Preferably, the weights are adjusted during training using stochastic gradient descent. For example, the mean squared error of the difference between the target data and the output data generated by the neural network is used as the loss function.

[0064] The structure of this neural network and how it adjusts the weights based on the error between the expected output data and the target data are well-known, so they will not be elaborated upon here.

[0065] Preferably, the reference points 20 in each case include two points for marking the thigh 11 and the lower leg 12, and three points for marking the knee. Thus, thigh reference points 20a and 20b mark the location of the femur 11, lower leg reference points 20c and 20d mark the location of the tibia 12, and knee reference points 20e, 20f, and 20g mark the location of the knee. Of course, the invention is not limited to the number of reference points 20 described above. On the contrary, it is possible to specify more reference points 20.

[0066] More preferably, thigh reference points 20a, 20b and lower leg reference points 20c, 20d form point pairs in each case. Preferably, these point pairs mark the location of the bone end region 21. The bone end region 21 in each case refers to the region of the bone where the articular head is located. Preferably, knee reference points 20e, 20f, 20g form a point cloud 22, the points of which reference the edge location of the knee. The point cloud 22 includes at least one of the knee reference points 20e, 20f, 20g. However, preferably, the point cloud 22 includes... Figure 3 The three knee reference points 20e, 20f, and 20g shown reference the edge of the knee.

[0067] According to another preferred embodiment of the present invention, the point cloud 22 includes at least two reference points 20e and 20g, namely the lower knee reference point and the upper knee reference point.

[0068] Preferably, before generating the mixed image data, the overlay unit is adapted to extract object-related image regions from the image background of both the optical image data and the X-ray image data. In other words, image regions that represent only the background are masked in the related data.

[0069] The present invention also relates to a non-volatile computer-readable storage medium having a program comprising instructions for causing a computer to execute the methods described above for training a neural network. All common storage types are suitable as storage media, such as CD-ROMs, DVDs, Memory Sticks, fixed disks, or cloud storage services.

[0070] The present invention also includes an apparatus and method for locating bone portions present in a poultry leg 10. The following will first refer to... Figure 4 The apparatus and method according to the invention will be explained in more detail below. The apparatus includes a conveying device 23 configured to convey poultry legs 10 along a conveying direction 16. A first optical imaging system 24 (which in...) Figure 2 (Illustrated schematically) Multiple digital images 25 of the front or rear sides of the plurality of poultry legs 10 are acquired. It has been found particularly advantageous to always acquire the front side of the poultry legs 10. However, it is theoretically possible to always acquire the rear side of the poultry legs 10. The device also includes a first neural network (not shown) configured to locate the bony portions in the poultry legs 10. The first neural network has been trained by a method according to the invention for training a neural network for locating the bony portions present in the poultry legs.

[0071] The device according to the invention further includes an input unit (not shown) adapted to sequentially provide the plurality of digital images as input data to a first neural network. In other words, preferably, the first neural network receives a digital image of the front side of the poultry leg 10 as input data. The correspondingly trained first neural network is adapted to determine position data 31 of the bone portion based on this input data and provide this determined display and / or transmission position data 31 to a downstream machine 26 for processing the poultry leg 10 based on the determined position data 31.

[0072] according to Figure 5 The steps of this method will become clearer with the block diagram. The plurality of poultry legs 10, conveyed along the conveying direction 16 by the conveying device 23, pass through the first optical imaging system 24, which acquires multiple digital images 25 of the front or rear sides of the plurality of poultry legs 10 at a given time point. The plurality of digital images 25 are optionally preprocessed 28, and then their orientation is checked. As previously stated, the preprocessing 28 of the plurality of digital images 25 and the execution of steps 29 and 30 are entirely optional. Further details regarding the above steps will be discussed below. Therefore, it is possible, in principle, to sequentially provide the plurality of digital images 25 as input data to the first neural network in step 27 without further preprocessing. As described above, position data 31 is then determined by the first neural network in step 27. More preferably, at least when recording optical image data, the poultry legs 10 are preferably illuminated by a flashlight.

[0073] Neural networks can take many forms. In principle, all multi-layer networks can be considered. A network structure with 29 layers has been found to offer particular advantages in detection accuracy while maintaining acceptable algorithmic complexity. Of these 29 layers, preferably, sixteen are two-dimensional convolutional layers. Furthermore, preferably, the convolutional layers are divided into four blocks, each followed by a max-pooling layer and a culling layer. Advantageously, all layers except the last one, which uses the sigmoid function, are activated by a rectified function.

[0074] Preferably, the input layer of the first neural network is adapted to process digital images with a resolution preferably of 300 x 300 pixels. Preferably, the output layer of the first neural network comprises fourteen nodes, each node representing the x and y coordinates of seven reference points 20. Furthermore, preferably, the first neural network is adapted to perform all calculations using floating-point arithmetic. Specifically, calculations are performed using floating-point numbers of the preferred "floating-point" type, with a resolution of 16 or 32 bits. Furthermore, preferably, the first neural network is configured with multiple processors for parallel computation.

[0075] Preferably, the reference points 20 in each case include two points for marking the thigh 11 and the lower leg 12, and three points for marking the knee. Thus, thigh reference points 20a and 20b mark the location of the femur 11, lower leg reference points 20c and 20d mark the location of the tibia 12, and knee reference points 20e, 20f, and 20g mark the location of the knee. Of course, the invention is not limited to the number of reference points 20 described above. On the contrary, it is possible to specify more reference points 20.

[0076] More preferably, thigh reference points 20a, 20b and lower leg reference points 20c, 20d form point pairs in each case. These point pairs preferably mark the location of the bone end region 21. Preferably, knee reference points 20e, 20f, 20g form a point cloud 22, with each point in the point cloud referencing the edge location of the knee. Point cloud 22 includes at least one of the knee reference points 20e, 20f, 20g. However, more preferably, point cloud 22 includes... Figure 3 The three knee reference points 20e, 20f, and 20g shown in the figure reference the edge of the knee. Alternatively, as previously mentioned, point cloud 22 may include only one point, preferably only the upper knee reference point 20g or a point located in the middle of the knee (not shown in the figure).

[0077] According to another preferred embodiment of the present invention, the point cloud 22 includes at least two reference points 20e and 20g, namely the lower knee reference point and the upper knee reference point.

[0078] The method according to the invention preferably further includes determining a cutting line path based on provided position data via a control unit (not shown) of the downstream machine 26. The control unit of the downstream machine 26 moves a controllable blade along this cutting line path to debone the poultry leg 10. Knowing the location of the bone portion in the poultry leg 10 allows for the determination of an optimal cutting line path to leave as little residual meat as possible on the relevant bone, while preventing the blade from cutting into the bone portion itself.

[0079] Figure 6 A flowchart of the aforementioned preprocessing 28 is shown. Step 28 preferably involves first performing lens correction 38 on the digital image 25. Then, advantageously, background masking 39 is performed to extract the image region in which the poultry leg 10 is visible. Furthermore, preferably, in the next downsampling step 40, the resolution of the digital image is reduced, for example, to 300x300 pixels, or preferably 100x100 or 128x128 pixels. However, the invention is not limited to the above resolutions. Rather, any other resolution reduction can be applied. Then, optionally, color conversion 41 is performed, for example, to a BGR or grayscale image. To adjust the digital image input to the first neural network, a conversion to floating-point numbers is then preferably performed via a floating-point conversion step 42.

[0080] More preferably, before the acquired digital image 25 of the poultry leg 10 is provided as input data to the first neural network, the acquired digital image 25 of the poultry leg 10 is fed to... Figure 5 The leg-side detection device 32 is shown. This device is adapted to perform leg-side detection and determine whether each specific digital image 25 is of a right or left poultry leg. To keep the training cost and algorithm complexity of the first neural network as low as possible, it is preferably trained only on one type of poultry leg 10, i.e., either the left or right leg 10. Therefore, to ensure reliable detection of the bone portion by the first neural network, it is necessary to prepare the "correct" digital images 25 of the poultry leg 10; that is, for example, if the neural network has previously been trained with the left poultry leg 10, it will always be a digital image of the left leg.

[0081] Advantageously, the method and apparatus according to the invention are adapted to automatically determine whether the recorded digital image 25 is of the right or left poultry leg 10. If the digital image 25 does not match the specified leg side, the leg side detection device 32 is adapted to mirror the image data of the digital image 25 with a virtual axis in order to convert the digital image 25 of the right poultry leg 10 into a virtual digital image 25 of the left poultry leg 10, and vice versa.

[0082] For example, if the first neural network has been trained with the left poultry leg 10, then if the leg-side detection device 32 detects that the relevant poultry leg 10 is a left poultry leg, the digital image 25 will not be altered by the leg-side detection device 32. Then, the digital image 25 is transmitted via the signal flow arrow 34, as... Figure 5 As shown. Otherwise, digital image 25 is guided along signal flow arrow 35 for mirroring in step 33 above. Thus, digital image 25 is either directly used as input to the first neural network in step 27 or transmitted along signal flow arrows 34 or 36 after image mirroring.

[0083] Leg side detection is preferably performed using a second neural network that has been trained with images of a poultry leg 10 (i.e., the left leg) of a specified leg side. Therefore, the leg side detection device 32 preferably includes a second neural network. If a digital image 25 corresponding to the specified leg side is detected, the digital image 25 remains unchanged as described above. If the second neural network does not detect the specified leg side, the digital image 25 is mirrored.

[0084] More preferably, the digital image 25 of the poultry leg 10 is fed to the front and rear detection device 37 before and / or before the leg-side detection is provided as input data to the first neural network. The front and rear detection device 37 is configured to determine whether each particular digital image 25 shows the front or rear side of the poultry leg 10. If the digital image 25 does not match the specified front / rear side, the front and rear detection device is adapted to rotate the poultry leg 10 so that it is oriented with the corresponding opposite side facing the first optical imaging system 24. Figure 8 The text provides a brief overview of this operation.

[0085] For this purpose, the conveying device 16 includes a plurality of suspension brackets 43, each configured and adapted to receive one of a plurality of poultry legs 10. Each suspension bracket 43 is configured to be controllably pivot about its vertical axis 44. Figure 8 The side view of the conveyor 16 shows the suspension bracket 43 in detail. If the front and rear detection devices 37 determine that the poultry leg 10 is not oriented to the desired side toward the first optical imaging system 24, the front and rear detection devices will cause the corresponding suspension receiver 43 of the conveyor 16 to perform a 180° rotation in step 46. Then, the side of the poultry leg facing the second optical imaging system 45 is obtained by the second optical imaging system 45, which is arranged downstream of the first optical imaging system 24 relative to the conveying direction 16.

[0086] Preferably, the front and rear detection are performed using a third neural network trained with images of designated front / rear poultry legs. Therefore, the front and rear detection device preferably includes a third neural network.

[0087] The objective is achieved by a non-volatile computer-readable storage medium comprising a program including instructions for causing a computer to execute a method for locating a bone portion present in a poultry leg 10.

[0088] According to an advantageous embodiment of the invention, the lenses of the optical camera 14 and the first and second optical imaging systems 24, 25 include polarizing filters. These filters are configured to reduce possible reflections, for example, caused by the wet surface of the poultry leg 10.

Claims

1. A method for training at least one neural network for locating bone portions present in a poultry leg (10), comprising the steps of: - Provide multiple poultry legs (10); - An image of the front or rear side of the plurality of poultry legs (10) is recorded by an optical camera (14) in the optical visible wavelength range, so as to generate optical image data of each of the plurality of poultry legs (10), wherein the front side refers to the outer side or skin side of the plurality of poultry legs (10), and the rear side refers to the inner side or meat side of the plurality of poultry legs (10). - Irradiate the rear or front side of the plurality of poultry legs (10) with X-rays (17) from an X-ray source (18), and record X-ray images of the side of the plurality of poultry legs (10) away from the X-ray source (18) using an X-ray imaging system (19) to generate X-ray image data for each of the plurality of poultry legs (10); - Based on the X-ray image data, a reference point (20) is defined for marking the location of the bone portion; - The positions of the optical image data and the X-ray image data are superimposed to generate mixed image data of each of the plurality of poultry legs (10); - The optical image data of the optical camera, which is input as input data, and the reference point (20), which is input as target data, are used as training data for the neural network; - Based on the difference between the target data and the output data generated by the neural network, the weights of the neural network are repeatedly adjusted.

2. The method according to claim 1, characterized in that, The reference points (20) include thigh reference points (20a, 20b), calf reference points (20c, 20d) and knee reference points (20e, 20f, 20g).

3. The method according to claim 2, characterized in that, The thigh reference points (20a, 20b) and the lower leg reference points (20c, 20d) are point pairs that represent the location of the bone end region (21).

4. The method according to claim 2 or 3, characterized in that, The knee reference points (20e, 20f, 20g) form a point cloud (22) including at least one point, wherein the point in the point cloud (22) refers to the edge position of the knee.

5. The method according to claim 4, characterized in that, The point cloud (22) includes at least an upper knee reference point (20g) and a lower knee reference point (20e), wherein the upper and lower knee reference points (20g, 20e) are located in the knee edge region.

6. The method according to claim 1, characterized in that, Before generating the hybrid image data, object-related image regions of the optical image data and the X-ray image data are extracted from the image background.

7. A non-volatile computer-readable storage medium comprising a program, the program including instructions for causing a computer to perform the method according to any one of claims 1 to 6.

8. A method for locating a bony portion present in a poultry leg (10), comprising the following steps: - Multiple poultry legs (10) are conveyed along the conveying direction (16) by a conveying device (23); - A first optical imaging system (24) acquires multiple digital images (25) of the front or rear side of each of the plurality of poultry legs (10) being transported through the first optical imaging system (24), wherein the front side refers to the outer side or skin side of the plurality of poultry legs (10) and the rear side refers to the inner side or meat side of the plurality of poultry legs (10); - The plurality of digital images (25) are sequentially provided as input data to a first neural network configured for locating the bone portion, wherein the first neural network for locating the bone portion has been trained by the method according to any one of claims 1 to 6; - Determine the location data (31) of the bone portion using the first neural network; and - The location data (31) to be displayed and / or transmitted is provided to a downstream machine (26) for processing the plurality of poultry legs (10) based on the determined location data (31).

9. The method according to claim 8, characterized in that, The reference points include thigh reference points (20a, 20b), calf reference points (20c, 20d), and knee reference points (20e, 20f, 20g).

10. The method according to claim 9, characterized in that, The thigh reference points (20a, 20b) and the lower leg reference points (20c, 20d) are point pairs that represent the location of the bone end region (21).

11. The method according to claim 9 or 10, characterized in that, The knee reference points (20e, 20f, 20g) form a point cloud (22) including at least one point, wherein the point in the point cloud (22) refers to the edge position of the knee.

12. The method according to claim 11, characterized in that, The point cloud includes at least an upper knee reference point (20g) and a lower knee reference point (20e), wherein the upper and lower knee reference points (20g, 20e) are located in the knee edge region.

13. The method according to claim 8, characterized in that, The control unit of the downstream machine (26) determines the cutting line path based on the provided position data (31) and moves the cutter of the downstream machine (26) along the cutting line path to debone the poultry leg (10), the cutter being adapted to move controllably.

14. The method according to claim 8, characterized in that, Before providing the acquired digital images (25) of the plurality of poultry legs (10) as input data to the first neural network, the acquired digital images (25) of the plurality of poultry legs (10) are fed to a leg side detection device (32), which is adapted to perform leg side detection and determine whether each particular digital image (25) is of the right poultry leg (10) or the left poultry leg (10), and if the digital image (25) does not match the specified leg side, the image data of the relevant digital image (25) is mirrored with a virtual axis so as to convert the digital image (25) of the right poultry leg (10) into a virtual digital image (25) of the left poultry leg (10), and vice versa.

15. The method according to claim 14, characterized in that, The leg side detection is performed by a second neural network that has been trained with images of poultry legs (10) on the specified leg side.

16. The method according to claim 14, characterized in that, Before providing the plurality of digital images (25) of the plurality of poultry legs (10) as input data to the first neural network and / or before the leg-side detection, the plurality of digital images (25) of the plurality of poultry legs (10) are fed to a front and rear detection device (37), which is adapted to perform front and rear detection and determine whether each particular digital image (25) shows the front or rear side of the poultry leg (10), and if the digital image (25) does not match the specified front / rear side, the suspension bracket (43) of the conveying device (23) is rotated 180° and a digital image (25) of the side of the poultry leg (10) facing the first optical imaging system (24) is acquired by a second optical imaging system (45) arranged downstream of the first optical imaging system (24) relative to the conveying direction (16), the suspension bracket (43) holding the poultry leg (10) and pivoting controllably about its vertical axis (44).

17. The method according to claim 16, characterized in that, The front and rear detections are performed by a third neural network that has been trained with images of the specified front / rear poultry legs (10).

18. A mechanism for training at least one neural network for locating bone portions present in a poultry leg (10), comprising: Multiple poultry legs (10); An optical camera (14) is adapted to record images of the front or rear sides of the plurality of poultry legs (10) in the optical visible wavelength range and is configured to generate optical image data for each of the plurality of poultry legs (10), wherein the front side refers to the outer or skin side of the plurality of poultry legs (10) and the rear side refers to the inner or meat side of the plurality of poultry legs (10). An X-ray source (18) and an X-ray imaging system (19) wherein the X-ray source (18) is adapted to irradiate the rear or front side of the plurality of poultry legs (10) with X-rays (17), and the X-ray imaging system (19) is adapted to record X-ray images of the side of the plurality of poultry legs (10) away from the X-ray source (18) and is configured to generate X-ray image data for each of the plurality of poultry legs (10); A display and input device configured to display the X-ray image data and / or display mixed image data, and input a reference point to be defined, the reference point being used to mark the position of the bone portion; The overlay unit is configured to overlay the optical image data with the X-ray image data and / or the reference point to generate the mixed image data of each of the plurality of poultry legs (10); At least one neural network; and A learning cycle control unit is configured and adapted to input the optical image data as input data and the reference points as target data as training data for the neural network, wherein the learning cycle control unit is adapted to repeatedly adjust the weights of the neural network based on the difference between the target data and the output data generated by the neural network.

19. The mechanism according to claim 18, characterized in that, The reference points (20) include thigh reference points (20a, 20b), calf reference points (20c, 20d) and knee reference points (20e, 20f, 20g).

20. The mechanism according to claim 19, characterized in that, The thigh reference points (20a, 20b) and the lower leg reference points (20c, 20d) are point pairs that represent the location of the bone end region (21).

21. The mechanism according to claim 19 or 20, characterized in that, The knee reference points (20e, 20f, 20g) form a point cloud (22) including at least one point, wherein the points in the point cloud (22) reference the edge position of the knee.

22. The mechanism according to claim 21, characterized in that, The point cloud (22) includes at least an upper knee reference point (20g) and a lower knee reference point (20e), wherein the upper and lower knee reference points (20g, 20e) are located in the knee edge region.

23. The mechanism according to claim 18, characterized in that, The overlay unit is adapted to extract object-related image regions of the optical image data and the X-ray image data from the image background before generating the mixed image data.

24. A device for locating bone portions present in a poultry leg (10), comprising A conveying device (23) is adapted to convey multiple poultry legs (10) along the conveying direction (16). A first optical imaging system (24) is configured to acquire multiple digital images (25) of the front or rear sides of the plurality of poultry legs (10), wherein the front side refers to the outer or skin side of the plurality of poultry legs (10), and the rear side refers to the inner or meat side of the plurality of poultry legs (10). A first neural network, configured to locate the bone portion, and trained by the method according to any one of claims 1 to 6, and An input unit is adapted to sequentially provide the plurality of digital images (25) as input data to the first neural network, wherein the first neural network is adapted to determine the position data (31) of the bone portion and provide the displayed and / or transmitted position data (31) to a downstream machine (26) for processing the plurality of poultry legs (10) based on the determined position data (31).

25. The device according to claim 24, characterized in that, Reference points (20) include thigh reference points (20a, 20b), calf reference points (20c, 20d) and knee reference points (20e, 20f, 20g).

26. The device according to claim 25, characterized in that, The thigh reference points (20a, 20b) and the lower leg reference points (20c, 20d) are point pairs that represent the location of the bone end region (21).

27. The device according to claim 25 or 26, characterized in that, The knee reference points (20e, 20f, 20g) form a point cloud (22) including at least one point, wherein the points in the point cloud (22) reference the edge position of the knee.

28. The device according to claim 27, characterized in that, The point cloud (22) includes at least an upper knee reference point (20g) and a lower knee reference point (20e), wherein the upper and lower knee reference points (20g, 20e) are located in the knee end region.

29. The device according to claim 24, characterized in that... The control unit of the downstream machine (26) is adapted to determine the cutting line path of the located bone portion based on the provided position data (31), wherein the control unit is further configured to move the cutter of the downstream machine (26) along the cutting line path to debone the poultry legs (10) in order to process the plurality of poultry legs (10), the cutter being adapted to move controllably.

30. The apparatus of claim 24 further includes a leg-side detection device (32) configured to perform leg-side detection based on the acquired digital images (25) of the plurality of poultry legs (10) before providing the acquired digital images (25) of the plurality of poultry legs (10) as input data to the first neural network, and to determine whether each particular digital image (25) is of the right poultry leg (10) or the left poultry leg (10), and if the digital image (25) does not match the specified leg side, to mirror the image data of the relevant digital image (25) with a virtual axis so as to convert the digital image (25) of the right poultry leg (10) into a virtual digital image (25) of the left poultry leg (10), and vice versa.

31. The device according to claim 30, characterized in that, The leg-side detection includes a second neural network that has been trained with images of poultry legs (10) on the specified leg side.

32. The device of claim 30 further includes a front and rear detection device (37) adapted to perform front and rear detection before and / or before the leg side detection, providing the plurality of digital images (25) of the plurality of poultry legs (10) as input data to the first neural network, and to determine whether each particular digital image (25) shows the front or rear side of the poultry leg (10), and if the digital image (25) does not match the specified front / rear side, to rotate the suspension bracket (43) of the conveying device (23) 180° and to acquire a digital image (25) of the side of the poultry leg facing the first optical imaging system (24) by a second optical imaging system (45) arranged downstream of the first optical imaging system (24) relative to the conveying direction (16), the suspension bracket (43) holding the poultry leg (10) and pivoting controllably about its vertical axis (44).

33. The device according to claim 32, characterized in that, The front and rear detection device (37) includes a third neural network that has been trained with images of the designated front / rear poultry legs (10).

34. A non-volatile computer-readable storage medium comprising a program, the program including instructions for causing a computer to perform the method according to any one of claims 8 to 17.

Images