Information processing device, information processing method, and program

Abstract

To provide an information processing device with which it is possible to estimate the imaging position of a captured image from a background image included in the captured image, and with which an inconvenience that the imaging position becomes unknown is suppressed.SOLUTION: The information processing device comprises: an acquisition unit for acquiring a captured image in which an animal in a breeding region is three-dimensionally imaged together with a background; a storage unit for storing a map image which is a three-dimensional image representing the breeding region; an identification unit for identifying an animal image representing the animal from the captured image; a generation unit for generating weight information pertaining to the animal on the basis of the animal image; and an estimation unit for estimating the imaging position of the captured image by using a background image in the captured image that represents the background and the map image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there has been known a technique for generating weight information regarding an animal by using a photographed image of the animal (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2023-33103 Summary of the Invention [Problem to be solved by the invention]

[0004] Some animals are kept in vast breeding areas. In such cases, in order to photograph all the animals, it is necessary to take multiple images while changing the shooting position. However, for example, when there are many shooting positions, an inconvenience may occur in which shooting position each image was taken. In consideration of the above circumstances, the present invention aims to suppress the above-mentioned inconvenience. [Means for solving the problem]

[0005] In order to solve the above problems, the information processing device of the present invention includes an acquisition unit that acquires a 3D image of an animal in a breeding area together with its background, a memory unit that stores a map image which is a 3D image representing the breeding area, an identification unit that identifies an animal image representing the animal from the captured image, a generation unit that generates weight information relating to the animal based on the animal image, and an estimation unit that estimates the shooting location of the captured image using a background image representing the background and the map image from the captured image. Effect of the Invention

[0006] According to the present invention, the shooting position of a captured image is estimated from a background image contained in the captured image, thereby preventing the inconvenience of the shooting position becoming unclear. [Brief description of the drawings]

[0007] [Figure 1] FIG. 2 is a diagram for explaining each component of the information processing system 1. [Diagram 2] FIG. 2 is a functional block diagram of an information processing device. [Diagram 3] FIG. 11 is a diagram for explaining a specific example of a captured image. [Figure 4] FIG. 11 is a diagram for explaining an estimation process. [Diagram 5] FIG. 11 is a diagram for explaining a calculation process. [Figure 6] FIG. 13 is a diagram for explaining the creation process. [Figure 7] 4 is a flowchart of each process executed by the information processing device. [Figure 8] FIG. 13 is a diagram for explaining a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] 1 is a diagram for explaining each component of an information processing system 1 according to this embodiment. The information processing system 1 according to this embodiment is configured to be capable of generating weight information Dw of each livestock A by photographing the livestock A raised in a breeding area R. The weight information Dw is, for example, information indicating the weight (live weight) of the livestock A. However, the system may be configured to generate weight information Dw indicating the weight of the carcass of the livestock A.

[0009] In this embodiment, it is assumed that pigs are used as a specific example of livestock A. It is also assumed that a pig farm is used as the breeding area R. The breeding area R is configured to include a plurality of breeding sections Rp (1, 2, 3, 4, 5, ...). A plurality of livestock A are raised in each of the breeding sections Rp. However, livestock A other than pigs may also be used. The breeding area R is not limited to the example of FIG. 1.

[0010] 1, the information processing system 1 includes a conveying device E, a camera C, and a computer 100. The computer 100 includes a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), and a hard disk drive (HDD).

[0011] The HDD of the computer 100 stores various types of data. For example, programs that enable the execution of a measurement process (see FIG. 7(a)) and a map image creation process (see FIG. 7(b)) described below are stored in the HDD of the computer 100. The CPU of the computer 100 executes the above programs to realize various functions (such as the acquisition unit 11) described below. The RAM of the computer 100 temporarily stores various types of information that the CPU refers to when executing a program, for example. Furthermore, the ROM of the computer 100 stores various types of information in a non-volatile manner.

[0012] As shown in Fig. 1, the conveying device E can have a computer 100 attached to its lower end. The conveying device E is also configured to be capable of being suspended from a wire rope W. The conveying device E is provided with rollers that are rotated by a motor, and when the conveying device E is suspended from the wire rope W, the rollers come into contact with the wire rope W. In the above configuration, the conveying device E moves along the wire rope W by rotating the rollers. Specifically, the conveying device E can move in the direction of arrow Q in Fig. 1.

[0013] As shown in FIG. 1, the camera C is attached to the lower end side of the computer 100 and is configured to be able to photograph the livestock A. FIG. 1 shows the shooting direction S of the camera C. The camera C includes a color (RGB) camera and a depth camera. The color camera captures two-dimensional color images. On the other hand, the depth camera captures three-dimensional images (distance images) including depth information indicating the distance to the subject. For example, the three-dimensional images captured by the camera C are assumed to be point cloud images captured using LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) technology.

[0014] In this embodiment, livestock A is photographed at approximately the same time (from the same position) by both the color camera and the depth camera. When livestock A is photographed, image information is generated by associating a two-dimensional image and a three-dimensional image showing the livestock A. After generating the image information, camera C transmits the image information to computer 100.

[0015] Although details will be described later, in this embodiment, weight information Dw is generated using a three-dimensional image captured by camera C (hereinafter, "captured image Gx"). Specifically, the conveying device E can be operated from a remote location by a remote control. For example, when a user appropriately operates the remote control, the conveying device E can be moved along the wire rope W. Also, the camera C can be operated from a remote location by a remote control. For example, when a user appropriately operates the remote control, a captured image Gx is captured.

[0016] In this embodiment, each livestock A is photographed for each of the above-mentioned breeding sections Rp (1, 2, 3, ...). In the above configuration, the user first moves the transport device E (including camera C) to the upper side of approximately the center of the breeding section Rp to be photographed, and photographs the breeding section Rp with the camera C. Thereafter, the transport device E is moved, for example, to the upper side of the adjacent breeding section Rp. By repeating the above moving and photographing operations, it is possible to photograph all of the breeding sections Rp (all of the livestock A). However, the transport unit 200 may be configured to automatically stop and move repeatedly. A marker may be provided at the position on the wire rope W where the transport device E should be stopped.

[0017] Each captured image Gx of each livestock A is acquired by a computer 100. The computer 100 generates weight information Dw for each livestock A displayed in the captured image Gx. According to the present embodiment described above, each livestock A is captured by a camera C, and weight information Dw indicating the weight of each livestock A is generated. Therefore, compared to a configuration in which the weight of each livestock A is measured by a weighing scale, for example, there is an advantage that the weight of each livestock A can be easily managed.

[0018] Moreover, the computer 100 (camera C) of this embodiment is configured to be movable using a wire rope W. Let us assume a configuration in which the computer 100 moves on rails. The above configuration has the disadvantage that the rails must be permanently installed above the breeding area R. This embodiment has the advantage that when photographing of the breeding area R is completed, the wire rope W can be removed and installed in another breeding area R. Note that the present invention does not exclude a configuration in which the computer 100 moves on rails.

[0019] However, in a configuration in which the wire rope W is installed every time the livestock A is photographed, it is assumed that there is a disadvantage that the photographing position of the photographed image Gx is easily unknown. In addition, in a configuration in which the camera C is moved using the wire rope W (for example, in this embodiment), the camera C is more likely to vibrate when photographing the photographed image Gx than in a configuration in which the camera C moves using a rail, for example. As will be described in detail later, if the camera C vibrates when photographing the photographed image Gx, there is a disadvantage that the weight information Dw cannot be generated with high accuracy from the photographed image Gx. In consideration of the above circumstances, in this embodiment, a configuration is adopted in which the above-mentioned inconveniences that are easily manifested when the wire rope W is used are suppressed. The above configuration will be described in detail below.

[0020] Fig. 2 is a functional block diagram of the information processing device 10. For example, the above-mentioned CPU executes a program, causing the computer 100 to function as the information processing device 10. Note that Fig. 2 shows the above-mentioned camera C.

[0021] As shown in FIG. 2, the information processing device 10 includes an acquisition unit 11, a recognition unit 12, a generation unit 13, a calculation unit 14, an estimation unit 15, a creation unit 16, and a storage unit 17. The acquisition unit 11 acquires a captured image Gx captured by a camera C. FIG. 2 shows some of the images contained in the captured image Gx (see also FIG. 3 described below). As shown in FIG. 2, the captured image Gx usually includes an animal image Ga and a background image Gb. The animal image Ga is an image representing livestock A. The background image Gb is an image other than livestock A. The above background image Gb includes, for example, a floor image Gf representing the floor area F of the breeding area R.

[0022] The identification unit 12 identifies an animal image Ga representing a livestock A from each image in the photographed image Gx. The identification unit 12 also identifies a floor image Gf representing a floor area F of the breeding area R. Specifically, the information processing device 10 segments each object included in the photographed image Gx. AI (Artificial Intelligence) technology may be adopted for the above segmentation.

[0023] For example, each object in the captured image Gx is segmented using a trained FCN (Fully Convolutional Networks) (for example, the technology described in JP 2022-29169 A may be adopted). In addition, each object (livestock A, floor area F, partition, etc.) in the captured image Gx is classified. However, the configuration for identifying each object in the captured image Gx is not limited to the above example.

[0024] The generation unit 13 generates weight information Dw regarding the livestock A based on the animal image Ga. Specifically, the generation unit 13 generates the weight information Dw from the animal image Ga using the technology described in Japanese Patent No. 7210862. As described above, the weight information Dw in this embodiment indicates the weight of the livestock A. However, the weight information Dw may be generated to indicate the weight of the carcass of the livestock A. Furthermore, the weight information Dw may be generated using a technology other than that described in Japanese Patent No. 7210862.

[0025] Incidentally, in a configuration in which weight information Dw is generated from an animal image Ga, it is necessary to correct the inclination of the animal image Ga according to the inclination of the camera C when the photographed image Gx is captured. The inclination of the camera C is detected by a tilt sensor provided in the camera C. However, as described above, in a configuration in which the camera C is suspended from a wire rope W, the camera C may vibrate when the photographed image Gx is captured. In addition, the tilt sensor of the camera C has a circumstance in which it becomes difficult to detect the accurate inclination when it vibrates. As can be understood from the above explanation, if the inclination of the animal image Ga is corrected based only on the inclination detected by the tilt sensor of the camera C, a problem may occur in which the weight information Dw is not generated with high accuracy.

[0026] In order to suppress the above inconveniences, this embodiment includes a calculation unit 14. The calculation unit 14 calculates the horizontal direction in the captured image Gx from the floor image Gf in the captured image Gx. Specifically, the floor area F represented by the floor image Gf is approximately parallel to the horizontal direction. The calculation unit 14 calculates a plane parallel to the floor image Gf, and identifies a normal vector V of the plane (see FIG. 5 described later). The direction of the normal vector V can be assumed to correspond to the direction of gravity.

[0027] The generation unit 13 uses the calculation result (normal vector V) of the calculation unit 14 to generate weight information Dw from the animal image Ga. Specifically, assuming that the direction of the normal vector V is the direction of gravity, the generation unit 13 adjusts the direction of the animal image Ga and generates weight information Dw from the animal image Ga. The above configuration will be described in detail later with reference to FIG. 5.

[0028] The estimation unit 15 estimates the shooting position Ps of the captured image Gx by using the background image Gb and the map image Gy of the captured image Gx. Specifically, the estimation unit 15 estimates which of the breeding sections Rp (1, 2, 3, ...) the captured image Gx was captured in. The map image Gy is a point cloud image (map data) representing the entire breeding area R (see FIG. 4 described later). The above map image Gy is stored in the storage unit 17. The estimation unit 15 identifies an area that matches the background image Gb of the captured image Gx from the map image Gy. Furthermore, the breeding section Rp (1, 2, 3, ...) corresponding to the identified area of ​​the map image Gy is estimated as the shooting position Ps of the captured image Gx.

[0029] However, the livestock A moves freely in the breeding section Rp. Therefore, when estimating the shooting position Ps, it is difficult to estimate the shooting position with high accuracy from the captured image Gx including the animal image Ga. In consideration of the above circumstances, the estimation unit 15 of this embodiment deletes the animal image Ga from the captured image Gx, and estimates the shooting position Ps of the captured image Gx using the captured image Gx after the animal image Ga has been deleted. The above configuration will be described in detail later with reference to FIG. 4.

[0030] The information processing device 10 stores the position information Dp indicating the photographing position Ps estimated by the estimation unit 15 in association with the weight information Dw generated by the generation unit 13. Note that a plurality of livestock A are raised in one breeding section Rp. Therefore, one photographed image Gx usually contains a plurality of animal images Ga. In the above configuration, a plurality of pieces of weight information Dw are generated from one photographed image Gx. Each piece of weight information Dw generated from a common photographed image Gx is associated with a common position information Dp.

[0031] The creation unit 16 creates the map image Gy (map data) described above. Specifically, the creation unit 16 creates the map image Gy by linking the captured images Gx of each breeding area R (1, 2, 3, ...). However, there is a situation in which the captured images Gx including the animal image Ga cannot be linked with high precision (cannot be positioned accurately). In consideration of the above situation, the creation unit 16 creates the map image Gy using the captured images Gx from which the animal image Ga has been deleted. The above configuration will be described in detail later with reference to FIG. 6.

[0032] Fig. 3 is a diagram for explaining a specific example of the captured image Gx. Fig. 3 shows the X-axis direction, the Y-axis direction, and the Z-axis direction in the image space. Fig. 3 also shows the position of the information processing device 10 in the XY plane at the time of capturing the captured image Gx with a dashed line. The specific example of Fig. 3 is the captured image Gx captured when the information processing device 10 is located on the upper side near the center of the breeding section Rp.

[0033] As shown in FIG. 3, the captured image Gx includes an animal image Ga and a background image Gb. As the background image Gb, a partition image Gp representing a partition forming the outer edge of the breeding section Rp and a floor image Gf representing the floor area F are assumed. If the captured image Gx is captured in a state where an obstruction (e.g., livestock A) is located in the floor area F, a part of the floor area F is not captured. Therefore, the floor image Gf in the captured image Gx represents the floor area F with a part missing. For example, the floor image Gf in FIG. 3 has a hole H in the part obstructed by the livestock A. However, in this embodiment, even when the above floor image Gf is used, the capturing position Ps can be estimated with high accuracy.

[0034] Fig. 4 is a diagram for explaining a configuration that enables estimation of the shooting position Ps. In the upper left part of Fig. 4, a specific example of a captured image Gx is shown, similar to Fig. 3 described above. As described above, the captured image Gx includes an animal image Ga and a background image Gb. Of the captured images Gx, the animal image Ga is used to generate weight information Dw of the livestock A. However, since the livestock A moves, there are cases where the shooting position Ps cannot be estimated with high accuracy from the captured image Gx that includes the animal image Ga.

[0035] Therefore, in this embodiment, a configuration is adopted in which the photographing position Ps is estimated using a photographed image Gx from which the animal image Ga has been deleted. Specifically, a deletion process (Sa in FIG. 4) is executed on the photographed image Gx. In the deletion process, each image identified as an animal image Ga from the photographed image Gx is deleted. In other words, the deletion process generates a background image Gb from the photographed image Gx. The upper right part of FIG. 4 shows a specific example of the background image Gb from which the animal image Ga has been deleted.

[0036] After generating the background image Gb, the information processing device 10 executes an estimation process (Sb in FIG. 4). In the estimation process, a portion of the map image Gy (map data) that matches the currently captured background image Gb is identified. In this embodiment, an ICP (Iterative Closest Point) algorithm is used to identify a portion of the map image Gy that matches the background image Gb.

[0037] A conceptual diagram of a specific example of the map image Gy is shown in the lower part of Fig. 4. The map image Gy in Fig. 4 includes a portion P1 representing the breeding area Rp1, a portion P2 representing the breeding area Rp2, a portion P3 representing the breeding area Rp3, and a portion P4 representing the breeding area Rp4. In the specific example of Fig. 4, it is assumed that the background image Gb is determined to substantially match the portion P4 in the estimation process. In the above cases, the current shooting position Ps is estimated to be the breeding area Rp4.

[0038] Note that the position information Dp (shooting position Ps) is not limited to the above example. For example, a configuration may be adopted in which information indicating a combination of latitude, longitude, and altitude of the camera C at the time when the photographed image Gx was photographed is generated as the position information Dp. For example, a configuration is assumed in which a combination of latitude, longitude, and altitude of the camera C is estimated from the range of the map image Gy that matches the background image Gb. In the above configuration, a configuration may be adopted in which information indicating the estimation result is generated as the position information Dp.

[0039] 5 is a diagram for correcting the orientation of the animal image Ga. For example, in the technology described in the above-mentioned Japanese Patent No. 7210862, in order to generate the weight information Dw with high accuracy, it is necessary to correct (rotate) the orientation of the animal image Ga so that it stands in the Z-axis direction corresponding to the gravity direction.

[0040] In the prior art, the orientation of the animal image Ga was corrected using a tilt sensor provided in the camera C. The tilt sensor detects the tilt of the shooting direction S (see FIG. 1). Specifically, the tilt sensor detects the pitch angle θp and low angle θr of the shooting direction S with respect to the direction of gravity (Z axis). The pitch angle θp is the tilt of the shooting direction S around the X axis with respect to the direction of gravity. The low angle θr is the tilt of the shooting direction S around the Y axis with respect to the direction of gravity. When the camera C captures the captured image Gx, it detects the pitch angle θp and low angle θr, and generates tilt information that identifies the detected pitch angle θp and low angle θr.

[0041] In the above conventional technology, the orientation of the animal image Ga is adjusted based on the tilt information. For example, the coordinates of each point constituting the animal image Ga are rotated using a rotation matrix. With the above tilt correction, the animal image Ga is corrected to a three-dimensional image captured from the Z-axis direction (directly above). That is, the orientation of the animal image Ga is corrected to stand in the Z-axis direction corresponding to the direction of gravity.

[0042] However, as described above, in a configuration in which the camera C is suspended from the wire rope W, the camera C may vibrate during shooting. If the camera C vibrates during shooting, it is expected that accurate tilt information will not be generated by the tilt sensor. Therefore, in the above-mentioned conventional technology, there is a problem that the orientation of the animal image Ga cannot be corrected with high accuracy. In consideration of the above-mentioned circumstances, the information processing device 10 of this embodiment adopts a configuration that can adjust the orientation of the animal image Ga with high accuracy even if the camera C vibrates. The above-mentioned configuration will be described in detail below.

[0043] The information processing device 10 acquires a floor image Gf from the captured image Gx. A simulated diagram of the floor image Gf is shown in the left part of FIG. 5. The floor area F (floor image Gf) is usually positioned approximately parallel to the horizontal direction. Therefore, the information processing device 10 calculates the plane through which the floor image Gf passes as the horizontal plane Hs (Sc in FIG. 5). Specifically, the plane through which the floor image Gf passes is expressed by the following formula (1) in mathematical expression 1. Furthermore, "a", "b", and "c" in formula (1) can be obtained from formula (2) in mathematical expression 1.

[0044]

number

[0045] In formula (2), (x1, y1, z1), (x2, y2, z2), ... (xn, yn, zn) are the coordinates of each point constituting the floor image Gf. In this embodiment, the coordinates of 1000 points (n = 1000) of the points constituting the floor image Gf are substituted into formula (2) to obtain the equation of the horizontal plane Hs. However, the number of coordinates substituted into formula (2) is not limited to 1000 and may be changed as appropriate.

[0046] The right side of Fig. 5 shows a conceptual diagram of a horizontal plane Hs calculated from the floor image Gf. When the information processing device 10 calculates the horizontal plane Hs, it specifies a normal vector V of the horizontal plane Hs. If the shooting direction S coincides with the direction of gravity when the captured image Gx is captured, the normal vector V is parallel to the Z axis in the image space. On the other hand, if the shooting direction S is inclined with respect to the direction of gravity, the normal vector V is not parallel to the Z axis.

[0047] When the information processing device 10 identifies the normal vector V, it executes a correction process (Sd in FIG. 5). In the correction process, the captured image Gx (including the animal image Ga) is rotated in a direction such that the normal vector V is parallel to the Z-axis direction. The information processing device 10 also acquires an animal image Ga from the captured image Gx on which the correction process has been executed. The orientation of the above animal image Ga has been corrected to a direction standing in the Z-axis direction corresponding to the direction of gravity. Therefore, weight information Dw can be generated from the animal image Ga with high accuracy. Note that the orientation of the animal image Ga may be adjusted after the animal image Ga is separated from the captured image Gx.

[0048] In the above embodiment, the shooting direction S (tilt of the camera C) can be appropriately changed according to the shape of the shooting area R (pig farm), etc. For example, the camera C may be installed so that the shooting direction S is approximately parallel to the direction of gravity. Also, the camera C may be installed tilted so that the shooting direction S intersects with the direction of gravity. Note that the present invention does not exclude a technique for adjusting the orientation of the captured image Gx (animal image Ga) based on tilt information generated by a tilt sensor. For example, after adjusting the orientation of the captured image Gx based on the tilt information generated by a tilt sensor, the orientation of the captured image Gx may be adjusted by a correction process. Also, the tilt (shooting posture) of the camera C may be calculated (estimated) using the normal vector V and stored in correspondence with the position information Dp, or the above-mentioned tilt information and position information Dp may be stored in correspondence with each other.

[0049] FIG. 6 is a diagram for explaining a configuration for creating a map image Gy (map data). As described above, in this embodiment, the map image Gy is created by concatenating the images Gx of each breeding section Rp. In the specific example of FIG. 6, for ease of explanation, it is assumed that the map image Gy is created from two images Gx including an image Gx of breeding section Rp1 and an image Gx of breeding section Rp2. However, it is also possible to configure the map image Gy to be created from three or more images Gx by separately photographing three or more breeding sections Rp.

[0050] 6, it is assumed that, of each breeding section Rp, breeding section Rp1 is photographed at time t1, and breeding section Rp2 is photographed at a later time t2. Hereinafter, for the sake of explanation, the captured image Gx of breeding section Rp1 of each breeding section Rp may be referred to as "captured image Gx1," and the captured image Gx of breeding section Rp2 may be referred to as "captured image Gx2."

[0051] Fig. 6 shows the photographing area Rx1 displayed in the photographed image Gx1 of the breeding area R. As can be seen from Fig. 6, the photographing area Rx1 includes not only the breeding section Rp1, which is the original subject of the photographing, but also a part of the adjacent breeding section Rp2. Specifically, the photographing area Rx1 includes the area r2 of the breeding section Rp2. For the sake of explanation, the area r2 is shown shaded in Fig. 6.

[0052] Fig. 6 also shows an image capturing area Rx2 displayed in the captured image Gx2 of the breeding area R. As can be seen from Fig. 6, the image capturing area Rx2 includes not only the breeding section Rp2, which is the original subject of the image capturing, but also a portion of the adjacent breeding section Rp1. Specifically, the image capturing area Rx2 includes an area r1 of the breeding section Rp1. For the sake of explanation, the area r1 is shown shaded in Fig. 6.

[0053] As can be understood from the above explanation, the area r1 of the breeding area Rp1 and the area r2 of the breeding area Rp2 are included in both the photographed image Gx1 and the photographed image Gx2. Hereinafter, for the sake of explanation, an area included in both the photographed area Rx1 and the photographed area Rx2 (for example, the area r1 and the area r2) may be referred to as an "overlapping area r". As will be described in detail later, the information processing device 10 identifies the overlapping area r from each photographed image Gx(1, 2) and arranges each photographed image Gx at a position where each overlapping area r of each photographed image Gx(1, 2) overlaps. According to the above configuration, each photographed image Gx is synthesized to create a map image Gy.

[0054] However, during the period between capturing the photographed image Gx1 and capturing the photographed image Gx2 (the period from t1 to t2), the livestock A may move. For example, in the specific example of FIG. 6, at the photographed time t1 of the photographed image Gx1, the head of the livestock A is located in the overlapping region r. On the other hand, at the photographed time t2 of the photographed image Gx2, the buttocks of the livestock A are located in the overlapping region r. In the above cases, the image of the overlapping region r changes between the photographed images Gx1 and Gx2. This can easily cause a problem in that it becomes difficult to identify the overlapping region r as a common region in the photographed images Gx1 and Gx2.

[0055] In consideration of the above circumstances, in this embodiment, a configuration is adopted in which a deletion process is performed to delete the animal image Ga from each photographed image Gx (1, 2) before identifying the overlapping region r. The middle left part of FIG. 6 shows a simulated image of the photographed image Gx1 (background image Gb1) after the deletion process (Se in FIG. 6) is executed. The middle right part of FIG. 6 shows a simulated image of the photographed image Gx2 (background image Gb2) after the deletion process (Se in FIG. 6) is executed. Hereinafter, for the sake of explanation, the part of the background image Gb1 representing the overlapping region r may be referred to as the "overlapping portion Gp1". The part of the background image Gb2 representing the overlapping region r may be referred to as the "overlapping portion Gp2".

[0056] The information processing device 10 executes the creation process (in FIG. 6) and creates a map image Gy using the background image Gb1 and the background image Gb2. Specifically, the information processing device 10 combines the images so that the overlapping portion Gp1 of the background image Gb1 and the overlapping portion Gp2 of the background image Gb2 overlap (registration). For example, an ICP algorithm is preferably adopted for the registration of the point cloud images.

[0057] The information processing device 10 stores the map image Gy created in the creation process in the storage unit 17. The timing for executing the creation process can be changed as appropriate. For example, the latest map image Gy may be generated from the captured image Gx every time the captured image Gx is captured to generate the weight information Dw of the livestock A. The information processing device 10 may also be configured to be able to switch to a map image creation mode. In the above configuration, in the map image creation mode, the map image Gy is created from the captured image, and the weight information Dw is not generated. For example, when creating the map image Gy, the livestock A is evacuated from the breeding area R, and then the breeding area R is photographed in the map image creation mode. In the above configuration, the deletion process (Se, Sf) can be omitted in the map image creation mode.

[0058] Fig. 7(a) is a flowchart of the measurement process. In the measurement process, weight information Dw of each livestock A is generated (the weight of the livestock A is measured). The information processing device 10 executes the measurement process, for example, every time the camera C captures an image Gx. However, the timing at which the measurement process is executed can be changed as appropriate. For example, the measurement process may be executed after the images Gx of all the breeding sections Rp have been captured.

[0059] When the measurement process is started, the information processing device 10 executes an acquisition process (S101). In the acquisition process, a photographed image Gx is acquired from the camera C. After executing the acquisition process, the information processing device 10 executes a classification process (S102). In the classification process, each object (livestock A, floor area F, etc.) in the photographed image Gx acquired in the immediately preceding acquisition process is classified. After executing the classification process, the information processing device 10 executes a calculation process (S103). In the calculation process, a normal vector V is calculated from the floor image Gf of the photographed image Gx acquired in the immediately preceding acquisition process (see FIG. 5 above).

[0060] After executing the calculation process, the information processing device 10 executes a correction process (S104). In the correction process, the orientation of each animal image Ga in the captured image Gx is corrected using the normal vector V calculated in the immediately preceding calculation process. After executing the correction process, the information processing device 10 executes a generation process (S105). In the generation process, weight information Dw is generated using the animal image Ga of the currently acquired captured image Gx. In the above generation process, when the captured image Gx includes multiple animal images Ga, multiple pieces of weight information Dw are generated.

[0061] The photographed image Gx may include livestock A in a breeding area Rp other than the breeding area Rp that is the original subject of the photograph (for example, an adjacent breeding area Rp). In this embodiment, the photographed image Gx is configured to be able to distinguish between the livestock A in the breeding area Rp that is the original subject of the photograph and the livestock A in the other breeding areas Rp. For example, the livestock A in the breeding area Rp that is the subject of the photograph is distinguished from the other livestock A based on the positional relationship with the partition image Gp in the photographed image Gx. In the generation process, weight information Dw is generated from the animal image Ga representing the livestock A in the breeding area Rp that is the subject of the photograph, and weight information Dw is not generated from the other animal images Ga. However, the photographed image Gx may be configured to generate weight information Dw from all animal images Ga, regardless of whether they are the subject of the photograph or not.

[0062] After executing the generation process, the information processing device 10 executes a deletion process (S106). In the deletion process, the animal image Ga is deleted from the currently acquired photographed image Gx. After executing the deletion process, the information processing device 10 executes an estimation process (S107). In the estimation process, the current photographing position Ps (position information Dp) is estimated using the photographed image Gx (background image Gb) from which the animal image Ga has been deleted and the map image Gy (map data) (see FIG. 4 above).

[0063] After executing the estimation process, the information processing device 10 executes a storage process (S108). In the storage process, each piece of weight information Dw generated in the immediately preceding generation process is stored in association with the position information Dp generated in the immediately preceding estimation process. The storage medium for storing the weight information Dw and the position information Dp may be included in the information processing device 10 itself, or may be included in an external device. In a configuration in which each piece of information (Dw, Dp) generated in the measurement process is stored in an external device, the information is transmitted to the external device every time the measurement process is executed. After executing the storage process, the information processing device 10 ends the measurement process.

[0064] 7B is a flowchart of the map image creation process. The information processing device 10 of this embodiment can transition to a map image creation mode. The information processing device 10 executes the map image creation process in the map image creation mode.

[0065] When the map image creation process is started, the information processing device 10 executes an acquisition process (S201) to acquire a photographed image Gx from the camera C. Furthermore, when the photographed image Gx is acquired by the acquisition process, the information processing device 10 executes a classification process (S202) to identify each object (livestock A, background) in the photographed image Gx. After executing the classification process, the information processing device 10 executes a deletion process (S203) to delete the animal image Ga from the photographed image Gx acquired in the immediately preceding acquisition process.

[0066] After executing the deletion process, the information processing device 10 judges whether or not all of the breeding areas Rp have been photographed (S204). For example, when the operator of the information processing device 10 has photographed all of the breeding areas Rp, the operator executes an end operation to end the map image creation mode. The above-mentioned end operation is assumed to be an operation of a remote control for remotely operating the information processing device 10. If the end operation is not executed, the information processing device 10 judges that there are remaining breeding areas Rp that have not been photographed (S204: No) and repeatedly executes the above-mentioned steps S201 to S203.

[0067] On the other hand, if an end operation is executed, the information processing device 10 determines that all of the breeding sections Rp have been photographed (S204: Yes), and executes a creation process (S205). In the creation process, each background image Gb (point cloud image) acquired in each acquisition process is combined using a registration technique to create a map image Gy (see FIG. 6 above). The map image Gy created in the creation process is stored in the storage unit 17 in a storage process (S206). After executing the storage process, the information processing device 10 ends the map image creation process.

[0068] FIG. 7(c) is a diagram for explaining a specific example of the position information Dp and weight information Dw stored in the measurement process. As described above, each weight information Dw of each livestock A is stored in correspondence with the position information Dp indicating the photographing position Ps where the livestock A was photographed. In addition, the photographing date on which each weight information Dw was generated is stored in correspondence with the weight information Dw. Furthermore, the average value Da of the weight information Dw of each livestock A (weight of livestock A) is automatically calculated and stored for each photographing position. Each of the above information can be displayed, for example, by a computer capable of communicating with the information processing device 10.

[0069] <Modification> The above embodiments can be modified in various ways. Specific modified embodiments are exemplified below. Two or more embodiments selected from the following examples can be appropriately combined.

[0070] (1) In each embodiment, a configuration is assumed in which breeding areas R1 and R2 are provided as breeding areas R. The breeding areas R1 and R2 are provided adjacent to each other. For example, breeding areas R1 and R2 are provided in a common pigpen. However, the position of the wire rope W capable of photographing each livestock A in breeding area R1 is different from the position of the wire rope W capable of photographing each livestock A in breeding area R1.

[0071] In the above case, after photographing the photographed area R1, the installation position of the wire rope W is changed and photographing the photographed area R2, thereby generating weight information Dw for all livestock A. Also, a configuration may be adopted in which a wire rope W1 for photographing the photographed area R1 and a wire rope W2 for photographing the photographed area R2 are separately provided. In the above configuration, the information processing device 10 is installed on the wire rope W1, photographing the photographed area R1, and then the information processing device 10 is installed on the wire W2 and photographing the photographed area R2, thereby generating weight information Dw for all livestock A.

[0072] Furthermore, the information processing device 10A may be installed on the wire rope W1, and the information processing device 10B may be installed on the wire rope W2. In the above configuration, the information processing device 10A generates weight information Dw1 for each livestock A in the breeding area R1, and the information processing device 10B generates weight information Dw2 for each livestock A in the breeding area R2. In the above configuration, a server device may be provided that receives the weight information Dw1 from the information processing device 10A and receives the weight information Dw2 from the information processing device 10B. Also, a configuration may be adopted in which a list of each piece of weight information Dw received by the server device can be displayed on a monitor.

[0073] In each of the above modified examples, a map image Gy representing the breeding area R1 and a map image Gy representing the breeding area R2 may be provided separately. Alternatively, a single map image Gy representing both the breeding area R1 and the breeding area R2 may be provided. In the above modified examples, after deleting the animal images Ga from each photographed image Gx of the breeding area R1 and each photographed image Gx of the breeding area R2, the photographed images Gx (background images Gb) are combined by a registration technique to create a map image Gy representing the entire breeding area R(1, 2). In the above modified examples, the photographing position is estimated by a common map image Gy in both cases where the breeding area R1 and the breeding area R2 are photographed.

[0074] In each of the above modified examples, the method of creating the map image Gy is not limited to the above examples. For example, in each embodiment, the installation position of the wire rope W can be easily changed. Therefore, the position of the wire rope W may be changed two or more times, and the breeding area R may be photographed from various positions, and the map image Gy may be created from the photographed images. The above configuration has the advantage that the map image Gy can be created with high accuracy, compared to a configuration in which the map image Gy is created only from photographed images without changing the position of the wire rope W.

[0075] (2) In each embodiment, the map image Gy may be updated by the photographed image Gx every time the photographed image Gx is photographed. For example, when the photographed image Gx is acquired in the above-mentioned measurement process, the map image Gy created in advance by the map image creation process may be automatically updated by the photographed image Gx. Specifically, the background image Gb of the newly acquired photographed image Gx may be superimposed on the map image Gy using an ICP algorithm. In the above configuration, every time the photographed image Gx is acquired, a missing part in the map image Gy is complemented. Therefore, the accuracy of the map image Gy can be improved.

[0076] (3) In each embodiment, the method for identifying the image capture position may be changed as appropriate. Below, a modified example of the configuration for identifying the image capture position will be described with reference to Fig. 8(a) and Fig. 8(b). Fig. 8(a) is a diagram for explaining the breeding area R in the modified example. Fig. 8(a) assumes that the breeding area R is viewed from above.

[0077] As shown in FIG. 8(a), a plurality of breeding sections Rp are provided in the breeding area R, similar to the first embodiment. FIG. 8(a) shows breeding sections Rp1 and Rp2 selected from among the breeding sections Rp. In this modification, similar to the first embodiment, the breeding section Rp is photographed in three dimensions to obtain a photographed image Gx (see FIG. 3 above). An animal image Ga is identified from the photographed image Gx, and weight information Dw is generated from the animal image Ga. Note that in this modification, similar to the first embodiment, the orientation of the animal image Ga is adjusted by the normal vector V (see FIG. 5 above).

[0078] As shown in FIG. 8(a), each rearing section Rp (1, 2...) is provided with a respective feeding device M (1, 2...). The feeding device M stores feed for the livestock A internally and discharges the feed to the outside at a predetermined timing. Also, each rearing section Rp (1, 2...) is provided with a respective sticker St (1, 2...). The above stickers St show a code Co. As the code Co, for example, a QR code (registered trademark) can be suitably adopted.

[0079] From the code C of the breeding section Rp, specific information Dc (for example, a section number described later) regarding the breeding section Rp is generated. Specifically, the camera C of this modified example performs two-dimensional shooting substantially simultaneously with three-dimensional shooting. The shooting area for three-dimensional shooting and the shooting area for two-dimensional shooting are substantially the same. Furthermore, the information processing device 10 acquires a shot image Gw generated by two-dimensionally shooting the breeding section Rp in addition to the above-mentioned shot image Gx generated by three-dimensionally shooting the breeding section Rp. Hereinafter, for the sake of explanation, the shot image Gx, which is a three-dimensional image, may be referred to as a "first shot image Gx," and the shot image Gw, which is a two-dimensional image, may be referred to as a "second shot image Gw."

[0080] In this modified example, similarly to the first embodiment, each livestock A is photographed (2D photographed, 3D photographed) by a camera C for each breeding section Rp. Each sticker St in each breeding section Rp is provided at a position where the code C of that sticker St is photographed when that breeding section Rp is photographed. For example, in the specific example of FIG. 8(a), each sticker St is affixed to the top surface of each feeding apparatus M. However, it is sufficient that the code C is photographed when each livestock A in the breeding section Rp is photographed, and the position where the sticker St is provided is not limited to the above example.

[0081] According to the above configuration, the code image Gc representing the code Co of the breeding section Rp that is the subject of the image capture is included in the second captured image Gw. The information processing device 10 identifies the code image Gc from the second captured image Gw. The information processing device 10 also generates the identification information Dc from the code Co represented by the identified code image Gc.

[0082] For example, in the specific example of FIG. 8, it is assumed that a sticker St1 indicating a code Co1 is provided in the breeding section Rp1 of each breeding section Rp. In the above case, a code image Gc indicating the code Cо1 is identified from the second captured image Gw obtained by two-dimensionally capturing the breeding section Rp1. From the code Cо1, specific information Dc1 regarding the breeding section Rp1 is generated. Also, in the specific example of FIG. 8, it is assumed that a sticker St2 indicating a code Co2 is provided in the breeding section Rp2 of each breeding section Rp. In the above case, a code image Gc indicating the code Cо2 is identified from the second captured image Gw obtained by two-dimensionally capturing the breeding section Rp2. From the code Cо2, specific information Dc2 regarding the breeding section Rp2 is generated.

[0083] Examples of the specific information Dc relating to the breeding section Rp include the pigpen number corresponding to the pigpen in which the breeding section Rp is located, the section number corresponding to the breeding section Rp, the number of livestock A in the breeding section Rp, and the date on which the livestock A was brought into the breeding section Rp (hereinafter, the "date of delivery"). However, the specific information Dc is not limited to the above examples and may be changed as appropriate. Among the various pieces of the specific information Dc, the section number corresponds to the position information Dp (photographing position) in the first embodiment described above.

[0084] 8(b) is a diagram for explaining a specific example of each piece of information stored in the modified example. Each piece of information described above is stored in the information processing device 10 or an external device. In this modified example, each piece of weight information Dw generated from the first captured image Gx (each animal image Ga) is stored in association with specific information Dc generated from the second captured image Gw (code image Gc) captured substantially simultaneously with the first captured image Gx.

[0085] For example, in the specific example of FIG. 8(b), it is assumed that the code Cо1 is photographed in a breeding section Rp1 with section number "Rp1" in a pigpen with pigpen number "R1". From the above code Cо1, specific information Dc1 is generated. In addition to the pigpen number "R1" and section number "Rp1", the specific information Dc1 includes the number of livestock A in the breeding section Rp1, "n", and the delivery date "y / m / d1" when the livestock A was delivered to the breeding section Rp1. In the specific example of FIG. 8(b), weight information Dw (Dwa1...Dwan) for each livestock A in the breeding section Rp1 is stored in correspondence with the above specific information Dc1.

[0086] As described above, according to this modification, the photographing position (breeding section Rp) is identified from the second photographed image Gw, and thus the inconvenience of the photographing position becoming unclear is suppressed, as in the above-mentioned first embodiment. Also, according to this modification, each piece of information (Dc, Dw) for managing each livestock A is generated using both the three-dimensional image (first photographed image Gx) and the two-dimensional image (second photographed image Gw). Therefore, compared to a configuration in which each piece of information for managing each livestock A is generated from only one of the three-dimensional image and the two-dimensional image, for example, there is an advantage that the number of types of information that can be generated is increased.

[0087] In addition, when a breeding section Rp (for example, Rp1) is set as a shooting target, a code C0 in another breeding section Rp (for example, Rp2) may be shot. In the above case, in addition to the code image Gc representing the code C0 of the breeding section Rp that is the shooting target, the code image Gc representing the code C0 of the other breeding section Rp is included in the second shooting image Gw. In consideration of the above, when a plurality of code images Gc are included in the second shooting image Gw, the information processing device 10 is preferably configured to be able to identify the code image Gc representing the code C0 that is the shooting target from the plurality of code images Gc. For example, the information processing device 10 identifies the largest code image Gc among the plurality of code images Gc as the code image Gc representing the code C0 that is the shooting target. In addition, in this modification, the section number of the specific information Dc may be estimated using the estimation process (see FIG. 4) of the first embodiment described above.

[0088] <Summary of the functions and effects of the embodiment> <First aspect> The information processing device (10) of this embodiment includes an acquisition unit (11) that acquires a captured image (Gx) of an animal (livestock A) in a breeding area (R) captured in 3D together with a background, a storage unit (17) that stores a map image (Gy) that is a 3D image representing the breeding area, an identification unit (12) that identifies an animal image (Ga) representing the animal from the captured image, a generation unit (13) that generates weight information (Dw) related to the animal based on the animal image, and an estimation unit (15) that estimates a capture position (Ps) of the captured image using a background image (Gb) representing the background of the captured image and the map image. According to this embodiment, the capture position of the captured image can be estimated from a background image included in the captured image, and the inconvenience of the capture position becoming unknown is suppressed.

[0089] <Second aspect> The information processing device (10) of this embodiment includes an acquisition unit (11) that acquires a captured image (Gx) of an animal (livestock A) in a breeding area (R) captured in 3D together with a floor area (F), an identification unit (12) that identifies an animal image (Ga) representing the animal from the captured image and identifies a floor image (Gf) representing the floor area from the captured image, a calculation unit (14) that calculates the horizontal direction in the captured image from the floor image, and a generation unit (13) that generates weight information (Dw) related to the animal from the animal image using the calculation result (normal vector V) of the calculation unit. According to this embodiment, even if the camera vibrates when capturing the captured image, there is an advantage that the orientation of the animal image can be adjusted with high precision, making it easier to generate accurate weight information.

[0090] <Third aspect> The information processing device (10) of this aspect includes a storage unit (17) that stores a map image (Gy) that is a three-dimensional image showing the breeding area, and an estimation unit (15) that estimates the position where the captured image was captured by using a background image (Gb) that shows the background including the floor area in the captured image and the map image. According to this aspect, the effects of both the first and second aspects can be achieved.

[0091] <Fourth aspect> In the information processing device (10) of this aspect, the estimation unit (15) deletes the animal image from the captured image, and estimates the position at which the captured image was captured using the captured image after the animal image has been deleted. This aspect has the advantage that the capturing position can be estimated with high accuracy, as compared to a configuration in which the capturing position is estimated using a captured image including an animal image, for example.

[0092] <Fifth aspect> The information processing device (10) of this aspect includes a creation unit (16) that creates a map image using a captured image from which animal images have been deleted. This aspect has the advantage that the map image can be created with high accuracy, as compared to a configuration in which a map image is created using a captured image including animal images, for example.

[0093] <Sixth aspect> The information processing device (10) of this aspect is an information processing device capable of generating weight information (Dw) regarding animals in a breeding area, and includes a display unit (sticker St) indicating a predetermined code (Co) in a specific area (Rp) in the breeding area (R), a first acquisition unit that acquires a first captured image (Gx) obtained by capturing the specific area in three dimensions, a second acquisition unit that acquires a second captured image (Gz) obtained by capturing the specific area in two dimensions, a first identification unit that identifies an animal image (Ga) representing an animal from the first captured image, a second identification unit that identifies a code image representing a code from the second captured image, a first generation unit that generates weight information based on the animal image, and a second generation unit that generates specific information regarding the specific area based on the code image. According to this aspect, there is an advantage that weight information can be generated from an animal image included in the first captured image, and specific information regarding the specific area (shooting position) can be generated from a code image included in the second captured image.

[0094] <Seventh aspect> The information processing method of this embodiment includes the steps of storing a map image (Gy) which is a three-dimensional image showing the breeding area (R), acquiring a photographed image (Gx) in which an animal (livestock A) in the breeding area is photographed in three dimensions together with the background (S101 in FIG. 7(a)), identifying an animal image (Ga) showing the animal from the photographed image (S102 in FIG. 7(a)), generating weight information (Dw) on the animal based on the animal image (S105 in FIG. 7(a)), and estimating the photographing position of the photographed image using a background image (Gb) showing the background of the photographed image and the map image (S107 in FIG. 7(a)). According to this embodiment, the same effects as those of the first embodiment described above can be achieved.

[0095] <Eighth aspect> The program of this aspect causes a computer to execute each step of the sixth aspect. According to this aspect, the same effects as those of the first aspect described above can be achieved. [Explanation of symbols]

[0096] 10: information processing device, 11: acquisition unit, 12: identification unit, 13: generation unit, 14: calculation unit, 15: estimation unit, 16: creation unit, 17: memory unit.

Claims

1. an acquisition unit that acquires a three-dimensional image of the animals in the rearing area together with the background; A storage unit for storing a map image, which is a three-dimensional image representing the breeding area; an identification unit that identifies an animal image representing the animal from the captured image; a generating unit that generates weight information regarding the animal based on the animal image; an estimation unit that estimates a photographing position of the photographed image by using a background image representing the background of the photographed image and the map image; An information processing device comprising:

2. an acquisition unit that acquires a three-dimensional image of the animals in the rearing area together with the floor area; an identification unit that identifies an animal image representing the animal from the captured image and identifies a floor image representing the floor area from the captured image; A calculation unit that calculates a horizontal direction in the captured image from the floor image; a generating unit that generates weight information about the animal from the animal image using the calculation result of the calculating unit; An information processing device comprising:

3. A storage unit for storing a map image, which is a three-dimensional image representing the breeding area; an estimation unit that estimates a position where the photographed image was taken by using a background image that represents a background including the floor area in the photographed image and the map image; The information processing apparatus according to claim 2 , further comprising:

4. The estimation unit deletes the animal image from the photographed image, and estimates a position where the photographed image was photographed using the photographed image after the animal image has been deleted. The information processing device according to claim 1 .

5. A creating unit creates the map image by using the photographed image from which the animal image has been deleted. The information processing device according to claim 1 .

6. An information processing device capable of generating weight information regarding animals in a breeding area, A display unit showing a predetermined code is provided in the specific area of ​​the breeding area, A first acquisition unit that acquires a first captured image obtained by three-dimensionally capturing the specific region; A second acquisition unit that acquires a second captured image obtained by two-dimensionally capturing the specific region; a first identification unit that identifies an animal image representing the animal from the first captured image; a second identification unit that identifies a code image representing the code from the second captured image; A first generator that generates the weight information based on the animal image; and a second generating unit configured to generate specific information regarding the specific region based on the code image.

7. storing a map image, which is a three-dimensional image representing a breeding area; acquiring a three-dimensional image of the animals in the rearing area together with a background; identifying an animal image representative of the animal from the captured images; generating weight information about the animal based on the animal image; estimating a photographing position of the photographed image by using a background image representing the background of the photographed image and the map image; An information processing method comprising:

8. A program causing a computer to execute each step according to claim 6.

Images