Estimation of animal body parameters using image processing

Abstract

A computer-implemented image processing method for estimating the physical parameters of a specific species or breed of animal. To capture images of the animal, an image acquisition device acquires an image dataset containing depth map images. A plane estimation algorithm analyzes the depth map images to determine the background plane behind the animal. A segmentation algorithm identifies the animal's mask. Depth values ​​outside the animal mask are reset to the distance between the viewpoint of the image acquisition device and the estimated background plane. The depth map images are normalized to restrict the data to a fixed numerical range, thereby generating a depth map image of processed distance-tagged triple depth data (T3D). The processed depth map images, along with the reset depth values ​​of their background plane, are fed into a pre-trained neural network, which outputs estimates of the animal's physical parameters.

Description

Technical Field

[0001] The present invention relates to determining values of animal body parameters by computer-implemented image processing using a neural network.

Background Art

[0002] In livestock production, it is often important to regularly measure one or more body parameters of animals, particularly weight and size, so as to monitor the growth of the animals.

[0003] In a standard approach, animals are individually moved from an enclosure where they are confined, through a narrow passage, into a gated shed where a weighing device is installed. Then, using the weighing device, the weight of each animal passing through the passage and entering the shed is measured. This process not only requires a great deal of labor but also risks stressing the animals.

[0004] To avoid the need for standard weighing approaches, various computer-implemented image processing methods have been developed over the past few decades for the purpose of estimating the weight of animals (particularly pigs). In recent years, many of them use deep learning neural networks (DLNNs).

[0005] In their introduction, Zhang, J., Zhuang, Y., Ji, H., and Teng, G., "Estimating Pig Weight and Size Using a Multi-Output Regressive Convolutional Neural Network: A Fast and Fully Automated Method," Sensors 2021, 21, 3218, https: / / doi.org / 10.3390 / s21093218[1], they summarize known machine vision approaches for estimating pig weight and classify them into four approaches: stereo projection of a pig's shadow onto the floor, image processing of a two-dimensional (2D) image of the pig's back, processing of a depth map image of the pig's back, and elliptic fitting based on the correlation between pig weight and geometric parameters that can be identified from the pig's image. The subject of Zhang et al.'s paper is the use of a deep learning neural network model that takes both 2D and depth map images as input and processes these images to determine the pig's weight.

[0006] Suwannakhun, S., Daungmala, P., Estimation of pig weight by digital image processing using deep learning, in the proceedings of the 14th International Conference on Signal-Image Technologies and Internet-Based Systems (SITIS), Beijing, China, November 26–29, 2018 [2], which describes an example of the elliptic fitting method described above using a neural network.

[0007] U.S. Patent Application No. 2022 / 221325(A1) (Viking Genetics FMBA)[3] discloses a computer-implemented image processing method for estimating the weight of an animal (particularly a cattle). A depth map image is acquired from above to capture an image of the animal's back. The depth map image is then associated with a reference model containing information on the dorsal topology and weight of the cattle breed being evaluated, thereby calculating the animal's weight.

[0008] Chinese Patent No. 113240574(A) (Shenzhen Xiwei Intelligent Technology Co Ltd)[4] discloses a computer-implemented image processing method for estimating the weight of an animal. A fisheye camera is used to acquire a 2D image of the animal's back from above, and the animal's height is estimated from this 2D image. Optionally, a distance sensor is also provided to obtain a more accurate value for the animal's height than with the fisheye lens. In this case, the first height obtained from the 2D image captured by the fisheye lens is scaled by a second height obtained by the distance sensor. The region of the 2D image related to the animal is segmented using a segmentation algorithm based on conventional image processing techniques (thresholding, morphological transformation, blob analysis, etc.) or, alternatively, a neural network (i.e., artificial intelligence). The height of the animal is then measured in sub-regions around several specific target points on the animal's back. These height values ​​are either a first height value obtained solely from the 2D image, or a combination of first and second height values ​​obtained from the 2D image and distance sensor data, respectively. Then, weight values ​​are determined for sub-regions around each target point, and these weight values ​​are combined to obtain the total weight of the animal. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] [3] U.S. Patent Application No. 2022 / 221325 (A1) (Viking Genetics FMBA) [Patent Document 2] [4] Chinese Patent No. 113240574(A) (Shenzhen Xiwei Intelligent Technology Co Ltd) [Non-patent literature]

[0010] [Non-Patent Document 1] [1] Zhang, J., Zhuang, Y., Ji, H., Teng, G., Estimation of pig weight and size using a multi-output regression convolutional neural network: A fast and fully automated method, Sensors 2021, 21, 3218, https: / / doi.org / 10.3390 / s21093218 [Non-Patent Document 2] [2] Suwannakhun, S., Daungmala, P., Estimation of pig weight using digital image processing with deep learning, Proceedings of the 14th International Conference on Signal-Image Technologies and Internet-Based Systems (SITIS), Beijing, China, November 26-29, 2018. [Non-Patent Document 3] [5] Jouppi, Young, Patil, et al., "Performance analysis of tensor processing units in a data center," 44th International Symposium on Computer Architecture (ISCA), Toronto, Canada, June 24-28, 2017 (submitted April 16, 2017), arXiv:1704.04760 [cs.AR] [Overview of the Initiative]

[0011] The above references are incorporated herein by reference in their entirety.

[0012] According to one aspect of this disclosure, a computer-implemented image processing method is provided for determining the values ​​of physical parameters of a particular species or breed of animal, the method being: The present invention provides an image dataset of an image region containing a depth map image captured by a camera from a certain viewpoint, and the present invention provides a step in which the image region includes an animal that is imaged against a background plane. The depth map image is analyzed using a plane estimation algorithm to identify the background plane, and then the distance value between the viewpoint and the background plane is determined. The process involves applying a segmentation algorithm to an image dataset to identify animals within an image region and define corresponding animal masks. The steps include: correcting the depth map image to reset depth values ​​outside the animal mask to the distance from the viewpoint to the background plane; The steps include: feeding a depth map image along with its reset depth values ​​to a neural network to determine the animal's body parameters; The neural network receives estimates of animal body parameters as output, and the neural network is pre-trained through the analysis of similar image datasets of animals of a specific species and / or breed for which body parameter values ​​are known. It is equipped with.

[0013] The inventors conducted performance comparison tests to compare specific implementations of their method described above with other approaches using the same entire dataset, namely, neural networks trained on the same training dataset and using the same input image dataset (or a subset thereof). The performance tests revealed that the method according to the present invention performs significantly better than the following: -When using only RGB images (i.e., without depth map images) -When using a depth map image (i.e., without an RGB image) -When one channel of an RGB image is replaced with a depth map image - The same as the method of the present invention using both RGB images and depth map images, except that no background plane estimation is performed and the values ​​of the depth map image outside the animal mask are not reset to background values ​​but retained as acquired.

[0014] A major advantage of the embodiments of the present invention is that the distance of the background plane obtained through the analysis of depth map images provides absolute scaling of segmented animals. The absolute scaling provided by identifying the background plane replaces the inclusion of reference objects such as checkerboards in the image for absolute scaling, as is known from the prior art, particularly in imaging dead pigs. The absolute scaling provided by identifying the background plane also means that the viewpoint can be changed from image to image without affecting the method, thereby enabling image acquisition using a handheld imaging device. For example, in the inventors' specific implementation, a neural network is used to predict parameters, and this neural network was trained with a training set of image data acquired with a vertical distance from the camera viewpoint to the ground in the range of 1.4 to 1.9 meters and a constant camera tilt angle. Thus, the trained neural network can account for a person holding a handheld imaging device at a constant height and a constant tilt angle. Even if the tilt angle changes from image to image, once the background plane is identified from the depth map image, the tilt angle is also automatically identified, so there is no impediment to plane estimation. This method offers the freedom to use handheld imaging devices, which is in contrast to some prior art methods that rely on the fixed position of the imaging device (i.e., camera) rather than using a reference object for absolute scaling. The fixed position ensures that the camera is positioned at a known, fixed distance (and at a fixed tilt angle) from the ground, and this fixed distance is used to provide absolute scaling. In such prior art methods, the known fixed position of the camera allows for the distinction between small animals close to the viewpoint and large animals farther away. Similarly, this method is suitable not only for processing image datasets acquired by handheld imaging devices but can also be used for processing image datasets acquired by fixed-position image acquisition devices where positioning is not critical.For example, since the training dataset used to train a parameter-predictive neural network is known in advance, the specifications regarding camera mounting can be defined accordingly. In the inventors' embodiment, the specification is to mount the camera at a height of 1.4 to 1.9 meters from the ground at an arbitrary tilt angle, but this tilt angle is generally downward, taking into account the field of view of the image acquisition device, so that a sufficiently large area of ​​the ground is included in the image area.

[0015] A further significant advantage of the present invention is the ability to realize embodiments that can process multiple animals contained within a single image region. This is possible because the segmentation algorithm can be configured to segment regardless of how many animals are in the image region, and the plane estimation algorithm can function regardless of the number of animals present in the image region. Furthermore, there is no need to physically restrain the animals in a pen to control their posture, for example, to ensure that they are standing rather than lying down when acquiring the image dataset. Specifically, a neural network can be used as the segmentation algorithm and trained to distinguish between desirable postures, such as standing (for which the parameter-predictive neural network is trained), and other postures (for which the parameter-predictive neural network is not trained). Therefore, it is not necessary to keep only one animal in the pen at a time to ensure that the animals are standing or that there is only one animal in the image region. Rather, the animals can roam freely together in a normal enclosure, and the image dataset can be acquired over a period of time so that parameter prediction is performed for all animals, for example, animals that are lying down for part of the time or partially hidden by contact with other animals are not excluded.

[0016] In the following, the present invention will be further described, using the attached drawings as an example. [Brief explanation of the drawing]

[0017] [Figure 1] Perspective view of a handheld device equipped with an imaging device for acquiring a color (RGB) image and a depth map image above the image area A. FIG. 1A is a view of the imaging device seen from below. [Figure 2] Example of an RGB image acquired by the handheld imaging device of FIG. 1. The image includes at least one animal. [Figure 3] Example of a depth map image acquired by the handheld imaging device of FIG. 1 simultaneously with acquiring the RGB image of FIG. 2. [Figure 4] Flow diagram of an image processing method according to an embodiment of the present invention. [Figure 5] Processed version of the RGB image of FIG. 2 in which the pixels of the animal are segmented. [Figure 6] Intermediate processed version of the depth map image of FIG. 3. Pixels of non-animals identified by segmenting the RGB image are set to NULL values. [Figure 7] Further processed version of the processed depth map image of FIG. 6. Pixels of non-animals are set to values representing the estimated distance to the ground in the depth map image determined by processing the pixels of non-animals in the depth map image of FIG. 3 with a plane estimation algorithm. [Figure 8] Block diagram of a tensor processing unit (TPU) that can be used to perform calculations associated with the implementation of a neural network architecture in a training or production environment. [Figure 9] Block diagram of a computing device that can be used as a host computer for the TPU of FIG. 8, for example.

Mode for Carrying Out the Invention

[0018] In the following detailed description, specific details are shown for the purpose of explanation and not limitation, to better understand the present disclosure. It will be apparent to those skilled in the art that the present disclosure can be implemented in other embodiments that depart from these specific details.

[0019] The following describes one embodiment of a method for predicting animal body parameters using image processing of an image dataset. Here, the body parameter is body weight, and the animal species is a pig. The image dataset supplied as input to this method consists of 2D images, specifically RGB (color) images and depth maps. A depth map is sometimes called a point cloud, and each point is similar to a pixel in a monochrome 2D image, but has a distance value instead of a grayscale value. For this reason, the depth map will be referred to as a depth map image, and its points as pixels. The image dataset is acquired from a viewpoint located above an animal standing on the ground. In this way, the RGB image and the depth map image form a pair of images taken from substantially the same viewpoint, so the two images have a close spatial correspondence and can be compared, for example, pixel by pixel. The 2D image is a 3-channel image, and the depth map image is a 1-channel image.

[0020] The format for training deep learning-based neural networks is as follows: Task (T): Predict the weight of an animal. • Experience (E): A set of RGB (R) images and corresponding depth (D) images [X -> (R+D)], along with the target weight (Y) of the animal. • Performance (P): Mean absolute error between target weight (Y) and predicted weight (y).

[0021] In this method, the axioms are that the projected surface area (A) of a standing animal viewed from above is proportional to the animal's volume (V), and furthermore, the animal's volume (V) is proportional to the animal's weight (Y). · A ∝ V ∝ Y This method considers not only the animal's local context but also its global context, therefore a pipeline is required to determine the animal's weight.

[0022] Local context is necessary to identify animals in an image without noise from nearby objects and / or to exclude other instances of the same animal. Specifically, the inventors' method uses segmentation to isolate and select a single animal for further processing. This can be achieved by using neural network-based instance segmentation models such as "MaskRCNN," "MaskFormer," and "Yolact." An RGB image is used as the basis for segmentation to extract a mask of a single animal within the RGB image.

[0023] In segmentation processing to separate animals, the global context of the animals is lost. Specifically, in the segmented portion of an RGB image, the distance between the viewpoint and the animal is unknown. Therefore, the absolute area of ​​the animal is also unknown, and thus the axiom A ∝ Y cannot be applied based solely on the segmented portion of the RGB image containing the animal. For example, the closer an animal is to the viewpoint, the larger its projected area will be on the handheld imaging device and, consequently, within the RGB image. A small animal close to the viewpoint will appear the same size as a large animal farther away from the viewpoint in the segmented portion of the RGB image.

[0024] To address this issue and provide a global context, a plane estimation algorithm is applied to pixels in the depth map image outside the animal mask to determine the background plane. In this embodiment, the background plane is the plane of the ground on which the animal is standing, and a distance value d is generated for the distance between the camera viewpoint and the background plane.

[0025] Next, the corrected depth map images are synthesized. In this process, depth map values ​​within the animal's sub-region (i.e., within the mask) are retained to provide a local context, and depth map values ​​outside the animal's sub-region (i.e., outside the mask) are populated with unidirectional (e.g., vertical) distance values ​​to the estimated background plane to provide a global context.

[0026] In reality, most open-source neural network models available for image processing require a 3-channel data input, such as for RGB images with red, green, and blue frames. Therefore, to use such a model for processing depth-mapped images, the modified depth-mapped image is duplicated three times, providing three arrays as input to the neural network. In the inventors' specific example, the model used is based on the model "Xception: Deep Learning with Depth-Separable Convolution" 2017 published by Francois Chollet (https: / / arxiv.org / abs / 1610.02357 DOI:https: / / doi.org / 10.48550 / arXiv.1610.02357), with the classifier stage replaced with a regressor to suit the purpose.

[0027] Figure 1 is a perspective view of a handheld imaging device 10 for acquiring image datasets. The handheld imaging device 10 comprises an imaging device 12. As shown in the view from below in Figure 1A, the imaging device 12 comprises a first camera 16 (e.g., a conventional RGB camera) for acquiring 2D images and a second camera 15 (e.g., a stereo camera with two adjacent lenses) for acquiring depth map images. The 2D camera 16 can operate to acquire red, green, and blue image frames that make up an RGB image. The depth map camera 15 can operate to acquire a point cloud indicating depth or distance from the camera viewpoint O. In the settings of the main embodiments of the present invention described below, the handheld imaging device 10 is held by a grip 14, and the imaging device 12 is supported in a downward position at a high position by an elongated frame member 13. The frame member 13 also includes a mount for a smartphone 18, which can be used to store image datasets acquired by the imaging device 12 and to wirelessly transfer those image datasets to the cloud or a local network for further processing. The imaging device 12 can operate to simultaneously acquire pairs of RGB images and depth map images. Preferably, cameras 15 and 16 have the same or similar fields of view so that the RGB images and depth map images they each capture cover the same or nearly the same image region A. Also preferably, cameras 15 and 16 capture their respective RGB images and depth map images from the same viewpoint O, or from viewpoints that are closely enough to be considered identical for the purpose of integrating the simultaneously acquired pairs of RGB images and depth map images. Image region A is illustrated as a plane at a vertical distance d from viewpoint O.

[0028] Figure 2 is an example of an RGB image acquired by the handheld imaging device shown in Figure 1, which includes a whole, standing pig.

[0029] Figure 3 shows an example of a depth map image acquired simultaneously with the acquisition of the RGB image in Figure 2 using the handheld imaging device shown in Figure 1.

[0030] Figure 4 is a flowchart of a body parameter estimation method according to one embodiment of the present invention, which comprises the steps of data acquisition (step S1), instance segmentation to segment standing, complete animals from the image (step S2), background plane estimation to estimate the distance to the background of the image (step S3), generation of a modified depth map image (step S4), and body parameter prediction by a neural network using classification by a convolutional neural network (CNN) architecture (step S5).

[0031] Step S1 is a data acquisition stage in which a pair of corresponding images is provided using the handheld imaging device shown in Figure 1, from a camera viewpoint positioned above an animal standing on the ground. Each pair of corresponding images consists of a 2D image and a corresponding depth map image, with each image being composed of an array of pixels.

[0032] Step S2 is the segmentation stage. In this stage, the 2D image is processed by applying a segmentation algorithm to identify one or more consecutive pixel subregions that represent a complete, standing animal, each defined by its respective binary mask. In the inventors' implementation, a neural network-based segmentation algorithm (e.g., a publicly available vision transformer (ViT) such as Maskformer or SegViT) is used to segment different instances of animals in the RGB image. A classification model then determines whether an animal is standing and complete (i.e., appears complete). It has been shown that imposing the condition of being standing and complete significantly improves the accuracy of the prediction. For example, pigs that are lying down but appear complete, and pigs that are standing but partially hidden by other objects or partially outside the image region are excluded. Once pigs that satisfy these conditions are identified, the pixels associated with those pigs are filtered from the RGB image using a segmentation mask (segMask) generated by the ViT model. Figure 5 shows the segmentation mask (segMask) of the RGB image determined by processing the RGB image in Figure 2. A major advantage of the inventors' segmentation algorithm is that if an image region contains multiple animals, the algorithm will segment all of them, provided that they are standing and fully visible. This feature is schematically illustrated in the superimposed boxes in step S2 of Figure 4.

[0033] Step S3 is the background plane estimation stage. In this stage, the depth map image is processed with a plane estimation algorithm to estimate the ground plane on which the animal is standing. The vertical distance from the camera viewpoint to the ground, i.e., the height, provides a global context, which in turn provides absolute scaling to the depth image data in segmented animal sub-regions that have only a local context.

[0034] As schematically illustrated in Figure 4, steps S2 and S3 are performed independently of each other. In this embodiment, the independence is clear because step S2 is performed on the RGB image and step S3 is performed on the depth map image. However, in other embodiments, segmentation, i.e., step S2, can also be performed on the depth map image. Even in this case, steps S2 and S3 are independent of each other because a segmentation mask is not used in background plane determination, and knowledge of the background plane is not required for segmentation.

[0035] In step S4, the depth map image is modified based on the results of steps S2 and S3, namely the segmentation mask and the determined background plane. Specifically, the depth values ​​of pixels outside the animal sub-regions in the depth map image, i.e., outside the mask, are reset to values ​​corresponding to the distance between the viewpoint of the corresponding image pair and the estimated background plane, providing a uniform background without clutter. For example, referring to Figures 2 and 3, it can be seen that the depth information of the enclosure fence and gate does not convey any information useful for weight estimation. Therefore, excluding them from the input of the dataset to the neural network improves the accuracy of weight estimation. Figure 6 shows how the pig is segmented from the depth map image using the segmentation mask segMask generated in step S2. For this to work, the RGB image and the depth map image must be placed side by side, i.e., have a one-to-one spatial correspondence. After step S4, the points in the depth map image will be measured distances from the viewpoint to the animal within the mask, and NULL values ​​outside the mask.

[0036] In the inventors' implementation, it is assumed that there is a one-to-one spatial correspondence between pixels in the RGB image and points in the depth map image, based on the fact that the lenses and sensors of the handheld imaging device are close to each other and the pair of images they capture can be considered to have been taken from the same viewpoint. Alternatively, if this assumption is not made, a warp transform can be applied to one of the two images to spatially map it to the other. The distance from the background plane (in this case, the ground), determined according to the estimated background plane, to the camera viewpoint is then embedded in the background of the depth data values. In other words, all distance values ​​in the point cloud are set to the vertical distance between the viewpoint and the estimated background plane, as shown in Figure 7. That is, Figure 7 shows an example of a modified depth map image where the points in the depth map image are measured distances from the viewpoint to the animal within the mask, and are single values ​​outside the mask that represent the vertical distance from the viewpoint to the background plane determined by the plane estimation algorithm.

[0037] In step S5, the modified depth map image is fed to the neural network. Since the modified depth map image is single-channel, it needs to be duplicated into three channels to match the three-channel input required by standard open-source CNN architectures. Before input to the CNN, the three-channel depth image is normalized so that the data fits within a fixed range (e.g., [0, 255]). The resulting depth map image processed as input to the CNN is called "Triplicated Depth Data with Distance (T3D)". The CNN then outputs a predicted value for the animal's weight. The CNN is pre-trained through the analysis of similar data of animals with known weights. A CNN classification model, such as the modified version of Francois Chollet's XceptionNet model mentioned above, i.e., the Chollet Xception model with the classifier stage replaced by a regressor, can be used as the weight estimator.

[0038] If the segmentation algorithm identifies n valid animal instances in step S2, steps S4 and S5 are modified as follows: Step S4 is performed n times, once for each identified animal instance, and each execution of step S4 is labeled as step S4-n. The mask regions of the unselected animal instances are treated as non-animal subregions of the image dataset. In other words, within the masks of the selected animal instances, the points in the depth map image are measurements of the distance from the viewpoint to the animal, and outside the selected masks, the points are NULL values. Step S5 is also performed n times, once for each identified animal instance, using the results of each step S4-n. The result of step S5-n is a depth map image where the points have measurements of the distance from the viewpoint to the animal within the selected mask n, and the same single value outside the selected mask n (including within the unselected masks <> n), which is the distance from the viewpoint to the background plane determined by the plane estimation algorithm. In this way, step S5 runs the neural network n times to determine the n weights of the n animal instances identified within the image region. The correspondence to multiple instances is schematically shown in the overlapping boxes of S2, S4, and S5 in Figure 4.

[0039] To summarize the main embodiments described above, what has been described is a computer-implemented image processing method for estimating the weight of a certain species or breed of animal, and this method is The process involves providing a pair of corresponding images captured by a camera from a viewpoint positioned above an animal standing on the ground, where the pair of corresponding images consists of a 2D image and a depth map image, and The process involves analyzing a depth map image using a ground plane estimation algorithm to determine the estimated ground plane on which the animal is standing, while referring to the depth map image. The process involves processing a 2D image by applying a segmentation algorithm to identify a 2D image representing an animal and a contiguous sub-region within the corresponding depth map image. A step of resetting the depth value outside the animal's sub-region to a value corresponding to the distance between the viewpoint and the estimated ground plane of the corresponding image pair, The steps include: normalizing the depth map image to restrict the data to a fixed numerical range and generating a depth map image of the processed distance-attached triple depth data (T3D); The processed depth map image is fed to a neural network along with reset depth values ​​outside the animal's sub-regions, and the neural network is pre-trained through the analysis of similar depth map images of animals with known weights. The step of receiving the predicted weight of the animal as output from the neural network. It is equipped with.

[0040] [Differentiation] While the main embodiments described above use examples of images being taken from above, it is also possible to take side or frontal images of an animal from the side, for example, using a vertical wall behind the animal as the background plane. Please understand that different training will be required when using images from different directions, such as above, the side, or the front.

[0041] In the embodiments described above, the animal species is a pig as an example, but it should be understood that this method can be applied to other species, including, but not limited to, livestock such as cattle, sheep, horses, ostriches, turkeys, chickens, and ducks. Furthermore, in the case of species with multiple breeds, this method can be applied to one or more specific breeds of a species, such as Berkshire, Chester White, Duroc, and Hampshire pigs. Thus, the animal species can be bipedal or quadrupedal.

[0042] The animal species may be humans. For example, this method can be applied to an image taken from the side of a corridor with one wall of the corridor as the background plane, showing people walking down the corridor. This can be useful, for example, at an airport gate to determine the total weight of all passengers boarding a commercial flight. The total weight of the passengers and the amount of cargo can then be added together to estimate the total pay load for the entire flight. This can be used as an additional check to ensure that the pay load does not exceed certain limits. The estimated pay load can also be used as a cross-check to ensure that the aircraft has an appropriate amount of fuel for the scheduled flight. This amount should not be too much, as carrying excess fuel will result in excess fuel consumption, nor too little, for obvious safety reasons.

[0043] As mentioned in the description of the main embodiments above, the present invention also allows for embodiments that rely solely on depth map images, i.e., do not require 2D images. In such alternative embodiments, step S2 of instance segmentation is performed on depth map images. The practical advantage of the main embodiments over the alternative embodiments is that most open-source neural network models have already been extensively trained on RGB images, and the amount of additional training required to adapt such neural network models specifically to the present invention is relatively small; in other words, very good results can be achieved with little effort. On the other hand, when performing segmentation on depth map images, it is known that pre-trained open-source neural network models are not available; in this case, the neural network model must be trained from scratch, which would likely involve considerable work with large, custom-built training datasets. Nevertheless, it is still possible to use depth map images for segmentation.

[0044] In the primary embodiments described above, the segmentation algorithm is based on a neural network. In alternative embodiments, the segmentation algorithm is based on one or more classical image processing techniques, such as the following: • Distributed-based analysis for seed region identification Adaptive thresholding • Morphological operations (blob analysis, etc.) • Contour identification • Contour joining based on proximity heuristic rules • Calculation of invariant image moments • Edge detection (such as Sobel edge detection) • Curvature flow filtering • Histogram matching to remove intensity variations between consecutive intersections • Multi-resolution rigid body / affine image registration (gradient descent optimizer) • Non-rigid deformation / transformation • Super Pixel Clustering In addition, in further embodiments, it is also possible to provide a segmentation algorithm that first preprocesses image data by applying some classical image processing, and then supplies the preprocessed image data to a neural network.

[0045] Figure 8 shows the TPU of Jouppi et al. 2017[5], a simplified reproduction of Jouppi's Figure 1. The TPU 100 includes a systolic matrix multiplication unit (MMU) 102 with a 256 × 256 MAC capable of performing 8-bit multiplication and addition on signed and unsigned integers. The weights of the MMU are supplied through a weight FIFO buffer 104, which reads the weights from memory 106 in the form of off-chip 8GB DRAM via a suitable memory interface 108. A unified buffer (UB) 110 is provided for storing intermediate results. The MMU 102 receives input from the weight FIFO interface 104 and the UB 110 (via a systolic data setup unit 112) and is connected to output the 16-bit product of the MMU processing to an accumulator unit 114. An activation unit 116 executes a nonlinear function on the data held in the accumulator unit 114. After further processing by the normalization unit 118 and the pooling unit 120, the intermediate results are sent to the UB 110 and resupplied to the MMU 102 via the data setup unit 112. The pooling unit 120 can perform maximum value pooling (i.e., Maxpooling) or average value pooling as desired. The programmable DMA controller 122 transfers data between the TPU and the host computer, and between the UB 110 and the TPU. TPU instructions are sent from the host computer to the controller 122 via the host interface 124 and the instruction buffer 126.

[0046] It should be understood that the computing power used to run neural networks, whether based on CPUs, GPUs, or TPUs, may be hosted locally on a computer network (e.g., as described below) or remotely hosted in data centers that comprise the network nodes. Furthermore, an AI processing node may correspond to a single physical processing unit (e.g., located in one or more racks of servers) or it may be distributed across two or more physical processing units.

[0047] Figure 9 is a block diagram showing an example of a computing device 500 that can be used in connection with the various embodiments described herein. For example, the computing device 500 can be used as a host computer that performs CNN processing in conjunction with a computer node in the computer network system described above, for example, a suitable GPU or TPU as shown in Figure 8.

[0048] The computing device 500 may be a server, a conventional personal computer, or another processor-equipped device capable of wired or wireless data communication. Furthermore, as will be apparent to those skilled in the art, other computing devices, systems, and / or architectures, including devices not capable of wired or wireless data communication, may also be used.

[0049] The computing device 500 preferably includes one or more processors, for example, a processor 510. The processor 510 may be, for example, a CPU, a GPU, a TPU, or an array or combination thereof (e.g., a combination of a CPU and a TPU, a combination of a CPU and a GPU). Additional processors may also be provided, for example, an auxiliary processor for managing input / output, an auxiliary processor for performing floating-point arithmetic (e.g., a TPU), a dedicated microprocessor with an architecture suitable for high-speed execution of signal processing algorithms (e.g., a digital signal processor, an image processor), a slave processor subordinate to the main processing system (e.g., a backend processor), an additional microprocessor or controller for a dual or multiprocessor system, a coprocessor, etc. Such auxiliary processors may be separate processors or may be integrated with the processor 510. Examples of CPUs that can be used in the computing device 500 include Pentium processors, Core i7 processors, and Xeon processors, all of which are available from Intel Corporation in Santa Clara, California. An example of a GPU usable with Computing Unit 500 is the Tesla K80 GPU from Nvidia Corporation in Santa Clara, California.

[0050] The processor 510 is connected to the communication bus 505. The communication bus 505 may include data channels to facilitate the transfer of information between the storage and other peripheral components of the computing device 500. The communication bus 505 may further provide a set of signals used for communication with the processor 510, including a data bus, an address bus, and a control bus (not shown). The communication bus 505 may have a standard or non-standard bus architecture, such as the Industry Standard Architecture (ISA), Extended Industry Standard Architecture (EISA), Microchannel Architecture (MCA), Peripheral Interconnection (PCI) local bus, or a bus architecture compliant with standards published by the Institute of Electrical and Electronics Engineers (IEEE) (e.g., IEEE 488 General Purpose Interface Bus (GPIB), IEEE 696 / S-100).

[0051] The computing device 500 preferably includes main memory 515 and may also include secondary memory 520. Main memory 515 provides storage for instructions and data for a program executed by the processor 510 (e.g., one or more of the functions and / or modules described above). It should be understood that computer-readable program instructions stored in memory and executed by the processor 510 can include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, and source code or object code written and / or compiled from a combination of one or more programming languages ​​(including, but not limited to, Smalltalk, C / C++, Java, JavaScript, Perl, Visual Basic, .NET, etc.). Main memory 515 is typically semiconductor-based memory such as dynamic random access memory (DRAM) or static random access memory (SRAM). Other types of semiconductor-based memory include, for example, synchronous dynamic random access memory (SDRAM), Rambus dynamic random access memory (RDRAM), and ferroelectric random access memory (FRAM), as well as read-only memory (ROM).

[0052] Computer-readable program instructions can run entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenarios, the remote computer can connect to the user's computer through any type of network, including local area networks (LANs) and wide area networks (WLANs), or it can connect to an external computer (for example, via the Internet using an Internet service provider).

[0053] The secondary memory 520 may optionally include internal memory 525 and / or removable media 530. Reading from and / or writing to the removable media 530 can be done in any known manner. Examples of removable storage media 530 include magnetic tape drives, compact disc (CD) drives, digital multipurpose disc (DVD) drives, other optical drives, and flash memory drives.

[0054] The removable storage medium 530 is a non-temporary computer-readable medium on which computer executable code (i.e., software) and / or data is stored. The computer software or data stored in the removable storage medium 530 is read into the computing device 500 and executed by the processor 510.

[0055] The secondary memory 520 may include other similar elements that enable computer programs or other data or instructions to be loaded into the computing device 500. Such means may include, for example, an external storage medium 545 and a communication interface 540 that enables the transfer of software and data from the external storage medium 545 to the computing device 500. Examples of the external storage medium 545 include an external hard disk drive, an external optical drive, and an external magneto-optical drive. Other examples of secondary memory 520 include semiconductor-based memories such as programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (block-oriented memory similar to EEPROM).

[0056] As described above, the computing device 500 may include a communication interface 540. The communication interface 540 enables the transfer of software and data between the computing device 500 and external devices (e.g., printers), networks, or other information sources. For example, computer software or executable code can be transferred from a network server to the computing device 500 via the communication interface 540. Examples of the communication interface 540 include an internal network adapter, a network interface card (NIC), a Personal Computer Memory Card International Association (PCMCIA) network card, a CardBus network adapter, a wireless network adapter, a Universal Serial Bus (USB) network adapter, a modem, a network interface card (NIC), a wireless data card, a communication port, an infrared interface, an IEEE 1394 FireWire, and other devices that can connect the system 550 to a network or another computing device. The communication interface 540 preferably implements industry-published protocol standards such as Ethernet IEEE 802 standard, Fibre Channel, Digital Subscriber Line (DSL), Asynchronous Digital Subscriber Line (ADSL), Frame Relay, Asynchronous Transfer Mode (ATM), Integrated Digital Services Network (ISDN), Personal Communication Services (PCS), Transmission Control Protocol / Internet Protocol (TCP / IP), and Serial Line Internet Protocol / Point-to-Point Protocol (SLIP / PPP), but may also implement customized or non-standard interface protocols.

[0057] Software and data transferred via the communication interface 540 generally take the form of telecommunication signals 555. These signals 555 can be provided to the communication interface 540 via a communication channel 550. In one embodiment, the communication channel 550 can be a wired or wireless network or various other communication links.

[0058] Communication channel 550 transmits signal 555 and can be implemented using various wired or wireless communication means. Such means include, to name a few, wires, cables, optical fibers, conventional telephone lines, mobile phone links, wireless data communication links, radio frequency ("RF") links, and infrared links.

[0059] Computer executable code (i.e., computer programs or software) is stored in main memory 515 and / or secondary memory 520. Computer programs can also be received via the communication interface 540 and stored in main memory 515 and / or secondary memory 520. When such computer programs are executed, the computing device 500 can perform various functions of the disclosed embodiments as described elsewhere in this specification.

[0060] In this specification, the term “computer-readable medium” is used to refer to any non-temporary computer-readable storage medium used to provide computer executable code (e.g., software and computer programs) to the computing device 500. Examples of such media include main memory 515, secondary memory 520 (including internal memory 525, removable media 530 and external storage medium 545), and any peripheral devices (network information servers or other network devices) that are communicably connected to the communication interface 540. These non-temporary computer-readable media are means for providing executable code, programming instructions and software to the computing device 500. In embodiments implemented using software, the software can be stored in a computer-readable medium and loaded into the computing device 500 via the removable media 530, input / output interface 535, or communication interface 540. In such embodiments, the software is loaded into the computing device 500 in the form of telecommunication signals 555. Once the software is executed by the processor 510, the processor 510 preferably performs the functions described elsewhere in this specification.

[0061] The input / output interface 535 provides an interface between one or more components of the computing device 500 and one or more input and / or output devices. Examples of input devices include, but are not limited to, keyboards, touchscreens or other touch-sensitive devices, biometric authentication devices, computer mice, trackballs, and pen-based pointing devices. Examples of output devices include, but are not limited to, cathode ray tubes (CRTs), plasma displays, light-emitting diode (LED) displays, liquid crystal displays (LCDs), printers, vacuum fluorescent displays (VFDs), surface electron emission displays (SEDs), and field emission displays (FEDs).

[0062] The computing device 500 also includes optional wireless communication components to facilitate wireless communication over voice and / or data networks. The wireless communication components comprise an antenna system 570, a radio system 565, and a baseband system 560. In the computing device 500, radio frequency (RF) signals are transmitted and received wirelessly by the antenna system 570 under the control of the radio system 565.

[0063] The antenna system 570 may comprise one or more antennas and one or more multiplexers (not shown) that perform switching functions to provide transmit and receive signal paths to the antenna system 570. In the receive path, the received RF signal is coupled from the multiplexer to a low-noise amplifier (not shown), which amplifies the received RF signal and transmits the amplified signal to the radio system 565.

[0064] The wireless system 565 may comprise one or more radios configured to communicate at various frequencies. In one embodiment, the wireless system 565 may have a demodulator (not shown) and a modulator (not shown) combined in a single integrated circuit (IC). The demodulator and modulator may be separate components. In the receiving path, the demodulator removes the RF carrier signal, leaving the baseband received voice signal, which is transmitted from the wireless system 565 to the baseband system 560.

[0065] If the received signal contains voice information, the baseband system 560 decodes the signal and converts it to an analog signal. The signal is then amplified and sent to the speaker. The baseband system 560 also receives analog voice signals from the microphone. These analog voice signals are converted to digital signals and encoded by the baseband system 560. The baseband system 560 also codes the digital signals for transmission to generate a baseband transmit voice signal, which is sent to the modulator section of the radio system 565. The modulator mixes the baseband transmit voice signal and the RF carrier signal to generate an RF transmit signal, which is sent to the antenna system 570, and may also pass through a power amplifier (not shown). The power amplifier amplifies the RF transmit signal and sends it to the antenna system 570, where the signal is switched to the antenna port for transmission.

[0066] The baseband system 560 is also communicatively connected to a processor 510, which may be a central processing unit (CPU). The processor 510 has access to data storage areas 515 and 520. The processor 510 is preferably configured to execute instructions (i.e., computer programs or software) that can be stored in main memory 515 or secondary memory 520. The computer program can be received from the baseband processor 560 and stored in main memory 510 or secondary memory 520, or it can be executed immediately upon receipt. Once such a computer program is executed, the computing device 500 can perform various functions of the disclosed embodiments. For example, the data storage areas 515 and 520 may contain various software modules.

[0067] The computing device further includes a display 575 directly connected to the communication bus 505, which may be provided in place of, or in addition to, any display connected to the input / output interface 535 described above.

[0068] Various embodiments can be implemented primarily using hardware components, such as application-specific integrated circuits (ASICs), programmable logic arrays (PLAs), and field-programmable gate arrays (FPGAs). Those skilled in the art will also see implementations of hardware state machines capable of performing the functions described herein. Various embodiments can also be implemented using a combination of both hardware and software.

[0069] Furthermore, those skilled in the art will understand that the various exemplary logic blocks, modules, circuits, and method steps described in relation to the above figures and embodiments disclosed herein can often be implemented as electronic hardware, computer software, or a combination of both. Above, various exemplary components, blocks, modules, circuits, and steps are described in general terms of their function in order to clearly demonstrate this hardware-software interoperability. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art will understand that the described functions can be implemented in various ways for specific applications, but such implementation decisions should not be interpreted as causing a departure from the scope of the invention. Also, the grouping of functions within modules, blocks, circuits, or steps is for the sake of clarity. It is possible to move specific functions or steps from one module, block, or circuit to another without departing from the invention.

[0070] Furthermore, various exemplary logic blocks, modules, functions, and methods described in relation to the embodiments disclosed herein can be implemented or executed using general-purpose processors, digital signal processors (DSPs), ASICs, FPGAs or other programmable logic devices, discrete gates, transistor logic, discrete hardware components, or combinations thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but instead, it may be any processor, controller, microcontroller or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or other such configurations.

[0071] Furthermore, steps of methods or algorithms described in relation to embodiments disclosed herein can be embodied directly in hardware, in software modules executed by a processor, or in a combination of both. Software modules can be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium (including network storage media). Exemplary storage media can be connected to a processor so that the processor can read information from and write information to the storage medium. Alternatively, the storage medium can be integrated with the processor. The processor and storage medium can also be located within an ASIC.

[0072] The computer-readable storage medium as used herein should not be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0073] Any software component described herein can take various forms. For example, a component may be a standalone software package or a software package incorporated as a “tool” into a larger software product. It may also be available for download from a network, such as a website, as a standalone product or as an add-in package for installation into an existing software application. It may also be provided as a client-server software application, a web-enabled software application, and / or a mobile application.

[0074] Embodiments of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present invention. It should be understood that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0075] Computer-readable program instructions can be provided to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device to generate a machine, and instructions executed via the processor of the computer or other programmable data processing device can create means for performing functions / operations defined in the blocks of a flowchart and / or block diagram. These computer-readable program instructions can be stored in a computer-readable storage medium that can instruct a computer, a programmable data processing device, and / or other device to function in a particular way, and the computer-readable storage medium storing the instructions can become a product containing instructions that perform aspects of functions / operations defined in the blocks of a flowchart and / or block diagram.

[0076] Computer-readable program instructions can be loaded into a computer, other programmable data processing device, or other device to cause the computer, other programmable device, or other device to execute a series of operational steps, generate a computer implementation process, and cause the instructions executed by the computer, other programmable device, or other device to perform functions / operations defined by blocks in a flowchart and / or block diagram.

[0077] The flowcharts and block diagrams shown illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or part of an instruction containing one or more executable instructions for performing a defined logical function. In some alternative implementations, the functions described in a block may be executed in a different order than that shown in the diagram. For example, two blocks shown consecutively may actually be executed substantially simultaneously or in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs a defined function / operation or a combination of dedicated hardware and computer instructions.

[0078] Apparatus and methods embodying the present invention can be hosted and provided by a cloud computing environment. Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0079] In summary, the detailed description above describes a computer-implemented image processing method and apparatus for estimating the physical parameters of a specific species or breed of animal. To capture images of the animal, an image acquisition device acquires an image dataset containing depth map images. A plane estimation algorithm analyzes the depth map images to determine the background plane behind the animal. A segmentation algorithm identifies the animal's mask. Depth values ​​outside the animal mask are reset to the distance between the viewpoint of the image acquisition device and the estimated background plane. The depth map images are normalized to restrict the data to a fixed numerical range, thereby generating a depth map image of the processed distance-tagged triple depth data (T3D). The processed depth map images, along with the reset depth values ​​of their background plane, are fed into a pre-trained neural network, which outputs estimates of the animal's physical parameters.

[0080] Those skilled in the art will see that many improvements and modifications can be made to the exemplary embodiments described above without departing from the scope of this disclosure.

Claims

1. A computer-implemented image processing method for determining the values ​​of physical parameters of a specific species or breed of animal, The image dataset is provided, which includes an image region containing a depth map image captured by a camera from a certain viewpoint, and the image region includes an animal that is imaged against a background plane. The steps include: analyzing the depth map image using a plane estimation algorithm to identify the background plane, and then determining the distance value between the viewpoint and the background plane; The steps include applying a segmentation algorithm to the image dataset to identify animals within the image region and defining corresponding animal masks, The steps include modifying the depth map image to reset depth values ​​outside the animal mask to the distance values ​​from the viewpoint to the background plane, The steps include: normalizing the depth map image to restrict the data to a fixed numerical range and generating a depth map image of the processed distance-attached triple depth data (T3D); The depth map image is supplied to a neural network along with its reset depth value to determine the animal's body parameters, and the neural network is pre-trained through the analysis of similar image datasets of the particular species or breed of animal. The steps include receiving an estimate of the animal's physical parameters as output from the neural network, and A method for providing this.

2. The image dataset of the image region further includes a two-dimensional image taken from the viewpoint, and the segmentation algorithm is applied to the two-dimensional image to identify the animal in the image region and define the corresponding animal mask. The method according to claim 1.

3. The segmentation algorithm is applied to the depth map image to identify the animals within the image region and define the corresponding animal masks. The method according to claim 1.

4. The viewpoint is located above the animal, and the background plane is the plane of the ground on which the animal is located. The method according to any one of claims 1 to 3.

5. The viewpoint is located on one side of the animal, and the background plane is the plane of the wall to the side of where the animal is located. The method according to any one of claims 1 to 3.

6. The physical parameter of the animal is one of body weight and distance dimension. The method according to any one of claims 1 to 5.

7. The segmentation algorithm identifies animals on the condition that the animal mask matches a complete animal of the species or breed to be processed by the method. The method according to any one of claims 1 to 6.

8. The segmentation algorithm identifies an animal, further provided that the animal mask matches the animal in a specific posture. The method according to claim 7.

9. The aforementioned posture is a standing posture, and the aforementioned animal species is a bipedal or quadrupedal animal. The method according to claim 8.

10. The segmentation algorithm is based on image processing using a further neural network, and the further neural network is pre-trained through the analysis of similar image datasets of the particular species or breed of animal. The method according to any one of claims 1 to 9.

11. The segmentation algorithm is based on classical image processing. The method according to any one of claims 1 to 9.

12. The image region includes multiple animal instances captured relative to the background plane, and when the segmentation algorithm identifies multiple animal instances within the image region, one of these animal instances is selected for further processing, while the other animal instances are processed as background by resetting their depth values ​​to the distance from the viewpoint to the background plane. The method according to any one of claims 1 to 11.

13. The aforementioned further processing is performed separately for each animal instance to determine the body parameter values ​​of each animal instance identified within the image region. The method according to claim 12.

14. A computing device loaded with machine-readable instructions for performing the method according to any one of claims 1 to 13.

Images