Determination device, determination method, and program

Abstract

This enhances the accuracy of identification methods using machine learning and image processing. [Solution] The system includes an imaging unit 200 that acquires imaging data including image data and point cloud data, an image processing unit 110 that identifies an object from the image data captured by the imaging unit 200, a reference position calculation unit 120 that calculates a reference position on the RGB image of the identified object OB, an object position information calculation unit 130 that calculates the actual dimensional coordinates in the X-axis direction and the Z-axis direction based on the calculated reference position, depth information corresponding to the reference position, and parameters of the imaging unit, an object-to-object angle calculation unit 140 that calculates the angle of the line between the object OBs with respect to the line in the X-axis direction based on the calculated actual dimensional coordinates of each object OB, and a determination unit 150 that determines whether the identification of the OB object is correct or incorrect based on whether the calculated angle is within a predetermined threshold range.

Description

Technical Field

[0001] The present invention relates to a determination device, a determination method, and a program.

Background Art

[0002] In recent years, against the backdrop of the growing awareness of health and the desire for a longer healthy life, for example, the demand for leafy vegetables has been expanding. However, the cultivation and growth of vegetables are greatly affected by weather conditions. Although imported vegetables are inexpensive, many consumers have concerns about their quality when it comes to maintaining and enhancing health.

[0003] On the other hand, although domestically produced vegetables are more expensive than imported vegetables, they are safe and reliable agricultural products that meet the growing awareness of health and the desire for a longer healthy life. However, in Japan, the aging of agricultural workers and the shortage of labor are becoming serious problems in the entire agricultural sector, and there is a concern about a large-scale abandonment of farming in about 10 years. In addition, since the entire agricultural industry depends on a method of inheritance that is close to "tradition" based on empirical rules, it is a major problem to promote the entry of young people into farming and to inherit and establish technologies for new entrants.

[0004] In response to these situations, the movement to apply digital technology and robot technology to the agricultural field has been active.

[0005] As an example of such movements, a mobile platform has been disclosed that has a simple configuration and travels along the ridges or rows of a field where crop plants are planted, while sweeping weeds between rows and between plants, while avoiding the crop plants, and comprises a steering means 140, a sweeping means 150 that moves back and forth in the width direction of the mobile platform while making contact with the soil between multiple ridges or rows that are in contact with the soil, an imaging means 110 that images the soil surface in front of the direction of travel, an image processing means that recognizes a pre-set object from the image acquired by the imaging means and stores the position of the object on the ridge or row, and a control means 130 that sets the direction of travel and instructs the steering means to avoid the position of the object, and sets the speed of the sweeping means's back and forth movement in the width direction according to the travel speed of the mobile platform (see, for example, Patent Document 1). [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2022-52948 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, the technology described in Patent Document 1 above does not always correctly recognize crops depending on the training data and the environment being identified. There is a problem that if crops and weeds have similar shapes and colors, weeds may be misidentified as crops, or crops as weeds.

[0008] Therefore, the present invention has been made in view of the above-mentioned problems, and aims to provide a determination device, determination method, and program that complement the certainty of identification methods using machine learning and image processing by adding a determination element based on the tendency for crops to be planted in a straight line when planted. [Means for solving the problem]

[0009] Embodiment 1; One or more embodiments of the present invention propose a determination device comprising: an imaging unit that acquires imaging data including image data and point cloud data of depth information; an image processing unit that identifies an object from the image data captured by the imaging unit; a reference position calculation unit that calculates a reference position on the RGB image of the identified object; an object position information calculation unit that calculates the actual dimensional coordinates in the X and Z directions from the center of the RGB image based on the calculated reference position, depth information corresponding to the reference position, and parameters of the imaging unit; an object-to-object angle calculation unit that calculates the angle of the line between objects with respect to the line in the X direction based on the calculated actual dimensional coordinates of each object; and a determination unit that determines whether the identification of the object is correct or incorrect based on whether the calculated angle is within a predetermined threshold range.

[0010] Embodiment 2; One or more embodiments of the present invention propose a determination device in which the object is an agricultural crop, and the determination unit determines whether the identification of the object is correct or incorrect based on a predetermined threshold range based on the characteristics of the positional relationship of the agricultural crop at the time of planting.

[0011] Embodiment 3; One or more embodiments of the present invention propose a determination device in which the imaging unit is an RGBD camera.

[0012] Embodiment 4; One or more embodiments of the present invention propose a determination device in which the object-to-object angle calculation unit calculates the object-to-object angle based on the depth difference between at least two objects and the lateral distance difference in the RGBD camera's field of view, in a plane extending in the lateral and depth directions of the RGBD camera's field of view.

[0013] Embodiment 5; One or more embodiments of the present invention propose a determination device in which the object position information calculation unit calculates the actual horizontal dimensional coordinates of each object in the field of view of the RGBD camera based on the size of the image sensor of the RGBD camera or the total number of pixels.

[0014] Embodiment 6; One or more embodiments of the present invention propose a determination device in which the object position information calculation unit calculates the actual dimensional coordinates in the depth direction within the field of view of the RGBD camera for each object, based on pixel information corresponding to the center of the object and depth information.

[0015] Embodiment 7; One or more embodiments of the present invention are a determination method in a determination device including an imaging unit, an image processing unit, a reference position calculation unit, an object position information calculation unit, an object-to-object angle calculation unit, and a determination unit, wherein the imaging unit acquires imaging data including image data and point cloud data of depth information; the image processing unit identifies an object from the image data acquired in the first step; the reference position calculation unit calculates a reference reference position on the RGB image of the object identified in the first step; and the object position information calculation unit calculates the A determination method is proposed, which includes: a fourth step of calculating the actual dimensional coordinates in the X and Z directions from the center of the RGB image based on the reference position calculated in the third step, depth information corresponding to the reference position, and parameters of the imaging unit; a fifth step of the object-to-object angle calculation unit calculating the angle of the line between objects with respect to the line in the X direction based on the actual dimensional coordinates of each object calculated in the fourth step; and a sixth step of the determination unit determining whether the identification of the object is correct or incorrect based on whether the angle calculated in the fourth step is within a predetermined threshold range.

[0016] Embodiment 8; One or more embodiments of the present invention are programs for causing a computer to execute a determination method in a determination device including an imaging unit, an image processing unit, a reference position calculation unit, an object position information calculation unit, an object angle calculation unit, and a determination unit, wherein the imaging unit performs a first step of acquiring imaging data including image data and point cloud data of depth information; the image processing unit performs a second step of identifying an object from the image data captured in the first step; the reference position calculation unit performs a third step of calculating a reference position on the RGB image of the object identified in the first step; and the object position information calculation unit performs The present invention proposes a program for causing a computer to execute the following steps: a fourth step of calculating the actual dimensional coordinates in the X and Z directions from the center of the RGB image based on the reference position calculated in the third step, depth information corresponding to the reference position, and parameters of the imaging unit; a fifth step of the object-to-object angle calculation unit calculating the angle of the line between objects with respect to the line in the X direction based on the actual dimensional coordinates of each object calculated in the fourth step; and a sixth step of the determination unit determining whether the identification of the objects is correct or incorrect based on whether the angle calculated in the fourth step is within a predetermined threshold range. [Effects of the Invention]

[0017] According to one or more embodiments of the present invention, by adding a determination element based on the tendency for crops to be planted in a straight line during planting, it is possible to enhance the accuracy of identification methods using machine learning or image processing. [Brief explanation of the drawing]

[0018] [Figure 1] This figure shows the configuration of a determination device according to an embodiment of the present invention. [Figure 2] This figure illustrates the angles formed by straight lines between multiple objects in a determination device according to an embodiment of the present invention. [Figure 3]In the determination device according to an embodiment of the present invention, it is a diagram schematically showing images of a plurality of objects as seen from an imaging unit. [Figure 4] In the determination device according to an embodiment of the present invention, it is a diagram showing the relationship between a subject and an image sensor via a lens and an example of calculating the actual dimensions of the subject. [Figure 5] In the determination device according to an embodiment of the present invention, it is an image exemplifying the positional relationship between a crop and a weed. [Figure 6] In the determination device according to an embodiment of the present invention, it is an image exemplifying the positional relationship between a crop and a weed. [Figure 7] In the determination device according to an embodiment of the present invention, it is an image exemplifying the positional relationship between a crop and a weed. [Figure 8] It is a flowchart of the processing of the determination device according to an embodiment of the present invention. <000​​​​​​​​​​​​​​​​​​​​​​​​To avoid such situations, the determination device 1 according to this embodiment focuses on the fact that crops are planted in relatively straight lines and aims to accurately distinguish between crops and weeds.

[0021] <Configuration of the determination device 1> As shown in Figure 1, the determination device 1 according to this embodiment is configured to include an RGBD camera 200 and a processor 100 that performs determination processing.

[0022] The data captured by the RGBD camera 200 includes image data and point cloud data. More specifically, the data captured by the RGBD camera 200 includes RGB image data and point cloud data containing depth information. In the above, depth information corresponding to the reference position is used as point cloud data in order to calculate the actual dimensions, but depth information obtained using a stereo method, such as the parallax of the reference position, may also be used.

[0023] <Configuration of Processor 100> Furthermore, as shown in Figure 1, the processor 100 is configured to include an image processing unit 110, a reference position calculation unit 120, an object position information calculation unit 130, an object-to-object angle calculation unit 140, a determination unit 150, a storage unit 160, and a control unit 170.

[0024] The image processing unit 110 performs the process of identifying the target object OB from the RGB image data acquired by the RGBD camera (imaging unit) 200. The processing results from the image processing unit 110 are sent to the control unit 170, which will be described later, via the bus line BL.

[0025] The reference position calculation unit 120 calculates a reference position on the RGB image for the object OB identified by the image processing unit 110. More specifically, the reference position calculation unit 120 calculates the reference position as the reference pixel position on the RGB image for the object OB (in this embodiment, agricultural crops are used as an example) identified by the image processing unit 110.

[0026] In this embodiment, the center position is calculated using a bounding box as an example, but the centroid of the silhouette of the object OB identified by the image processing unit 110 may be calculated and used as the reference position. The calculation result of the reference position calculation unit 120 is sent to the control unit 170, which will be described later, via the bus line BL.

[0027] The object position information calculation unit 130 calculates the actual dimensional coordinates in the X-axis and Z-axis directions based on the reference position calculated by the reference position calculation unit 120, the depth information corresponding to the reference position, and the parameters of the RGBD camera 200. More specifically, the object position information calculation unit calculates the actual dimensional coordinates in the X-axis and Z-axis directions based on the reference position of each object OB calculated by the reference position calculation unit 120, the depth information corresponding to that reference position (which may be the average of the surrounding area), and parameters of the RGBD camera 200 such as the sensor size and focal length. The calculation result from the object position information calculation unit 130 is sent to the control unit 170, which will be described later, via the bus line BL.

[0028] Here, Figure 4 illustrates the image-capturing principle of a camera. By applying this principle to an image sensor, the actual dimensions of the object OB can be obtained. In other words, regarding the horizontal direction, as shown in Figure 4, if the size h' of the image sensor and the focal length d' are known, then if the object distance d is known, then h':d' = h:d, and the actual dimensions of the image from the RGBD camera 200 can be calculated using the following equation 1. Therefore, it becomes possible to set extraction conditions as described above.

[0029]

number

[0030] As shown in Figure 3, more specifically, the method for calculating the actual dimensions in the X-axis direction from the RGBD camera 200 to the object OB1 is, for example, if the image sensor size ISS of the RGBD camera 200 is ISS = 2.7288w × 1.5498h (mm), the focal length f is f = 1.88 (mm), the image size PS is PS = 1280 × 720 (pixels), the size of one pixel of the image sensor is 1PS = 2.7288 / 1280w × 1.5498 / 720h (mm), the RGB image reference coordinates (x, y) of object 1 are (960, 0), and the depth D of object 1 is D = 500 mm, then, The actual AD in the X-axis direction from the RGBD camera 200 is: AD = (960 pixels - 640 pixels) × 2.7288 mm / 1280 pixels × 1.88 mm / 500 mm = 181 mm This is the result.

[0031] As shown in Figure 3, when the captured image is viewed in plan view, the horizontal direction of the paper is the X-axis direction, and the vertical direction of the paper, perpendicular to the X-axis direction, is the Y-axis direction. Furthermore, the direction of the paper's depth, which intersects the XY plane perpendicularly, becomes the Z-axis direction. Furthermore, when viewing the RGBD camera 200 from above, the depth direction corresponds to the Z-axis direction, which represents the depth dimension.

[0032] In the diagram, the rectangle surrounding object OB is the bounding box, the vertical line in the center indicates the center of the camera image, and the number of pixels at the center position in the X-axis direction of the camera image is, for example, 640 pixels. From this, for example, the center of object 2 (the center of its bounding box) is at a distance of W2 from the center of the camera image, and the center of object 1 (the center of its bounding box) is at a distance of W1 from the center of the camera image, and the distance between object 1 and object 2 in the X-axis direction is W12. The depth information corresponding to the reference position may be depth information directly corresponding to the reference position, or it may be the average of depth information for the area surrounding the reference position. The calculation result from the object position information calculation unit 130 is sent to the control unit 170, which will be described later, via the bus line BL.

[0033] The object-to-object angle calculation unit 140 calculates the angle of the line between objects relative to the line in the X-axis direction, based on the actual dimensional coordinates of each object OB calculated by the object position information calculation unit 130. More specifically, the object-to-object angle calculation unit 140 calculates the angle between the straight line between the objects OB and the straight line in the X-axis direction from the actual dimensional coordinates of each object OB calculated by the object position information calculation unit 130. The calculation result from the object-to-object angle calculation unit 140 is sent via the bus line BL to the control unit 170, which will be described later.

[0034] The determination unit 150 determines whether the identification of object OB is correct or incorrect based on whether the angle calculated by the object-to-object angle calculation unit 140 is within a predetermined threshold range. More specifically, the determination unit 150 determines that objects 1, 2, and 3 can be correctly identified if the difference between the angle formed by the line L1 between object 1 and object 2 shown in Figure 2 and the line LX1 in the X-axis direction, and the angle formed by the line L2 between object 2 and object 3 and the line in the X-axis direction, is within a predetermined threshold. The determination result from the determination unit 150 is sent to the control unit 170, which will be described later, via the bus line BL.

[0035] In other words, as shown in Figure 6, the determination unit 150 determines that crops (1), (2), (3), and (4) are planted in a straight line on the furrows of the field, and when detecting crops using machine learning or image processing, if the straight line formed by crops (1)-(2) is called line A, the straight line formed by crops (2)-(3) is called line B, and the straight line formed by crops (3)-(4) is called line C, then if crops planted in a straight line can be identified, lines A, B, and C will also be close to being the same straight line, and the determination is made by utilizing this fact.

[0036] On the other hand, as shown in Figure 7, if the determination unit 150 mistakenly identifies that the crop (4) is located in a position different from where the crop should actually be planted, the angle formed by line I and line U changes significantly compared to the angle formed by line A and line I. Therefore, the determination unit 150 utilizes this property to accurately distinguish between crops and weeds planted in a straight line.

[0037] In this embodiment, the determination is made based on the difference in the angle between the two lines and the line in the X-axis direction, but this is not the only method. The determination result from the determination unit 150 is sent to the control unit 170, which will be described later, via the bus line BL.

[0038] The memory unit 160 consists of ROM (Read Only Memory) or RAM (Random Access Memory), and stores programs, data, etc. In this embodiment, the storage unit 160 stores RGB image data and depth information acquired by the RGBD camera 200.

[0039] The control unit 170 controls the operation of the entire determination device 1 based on a control program stored in ROM or the like. More specifically, the control unit 170 controls the image processing in the image processing unit 110, the reference position calculation process in the reference position calculation unit 120, the object position information calculation process in the object position information calculation unit 130, the object angle calculation process in the object angle calculation unit 140, the determination process in the determination unit 150, and so on.

[0040] <Processing by the determination device 1> The processing of the determination device 1 according to this embodiment will be explained using Figure 8.

[0041] The image processing unit 110 performs the process of identifying the target object OB from the RGB image data acquired by the RGBD camera 200 (step S110). Furthermore, methods such as rule-based image processing detection or machine learning-based identification may be used for identification. The processing results from the image processing unit 110 are sent to the control unit 170 via the bus line BL.

[0042] The reference position calculation unit 120 calculates a reference position on the RGB image for the object OB identified by the image processing unit 110 (step S120). More specifically, the reference position calculation unit 120 calculates the reference position as the reference pixel position on the RGB image for the object OB (in this embodiment, agricultural crops are used as an example) identified by the image processing unit 110. In this embodiment, the center position is calculated using a bounding box as an example, but the centroid of the silhouette of the object OB identified by the image processing unit 110 may be calculated and used as the reference position. The calculation result from the reference position calculation unit 120 is sent to the control unit 170 via the bus line BL.

[0043] The object position information calculation unit 130 calculates the actual dimensional coordinates in the X-axis and Z-axis directions based on the reference position calculated by the reference position calculation unit 120, the depth information corresponding to the reference position, and the parameters of the RGBD camera 200 (step S130). More specifically, the object position information calculation unit 130 calculates the actual dimensional coordinates in the X-axis and Z-axis directions based on the reference position of each object OB calculated by the reference position calculation unit 120, the depth information corresponding to that reference position (which may be the average of the surrounding area), and parameters of the RGBD camera 200 such as the sensor size and focal length. The calculation result from the object position information calculation unit 130 is sent to the control unit 170 via the bus line BL.

[0044] The object-to-object angle calculation unit 140 calculates the angle of the line between objects relative to the line in the X-axis direction based on the actual dimensional coordinates of each object OB calculated by the object position information calculation unit 130 (step S140). More specifically, the object-to-object angle calculation unit 140 calculates the angle between the objects and the line in the X-axis direction from the actual dimensional coordinates of each object OB calculated by the object position information calculation unit 130. The calculation result from the object-to-object angle calculation unit 140 is sent via the bus line BL to the control unit 170, which will be described later.

[0045] The determination unit 150 determines whether the identification of object OB is correct or incorrect based on whether the angle calculated by the object-to-object angle calculation unit 140 is within a predetermined threshold range (step S150). The determination unit 150 determines, based on whether the angle calculated by the object-to-object angle calculation unit 140 is within a predetermined threshold range, that the identification of object OB is correct (YES in step S150), and sends a message to the control unit 170 to that effect. Specifically, the determination unit 150 determines, for example, that objects 1, 2, and 3 can be correctly identified if the difference between the angle formed by the line L1 between object 1 and object 2 shown in Figure 2 and the line LX1 in the X-axis direction, and the angle formed by the line L2 between object 2 and object 3 and the line in the X-axis direction, is within a certain threshold.

[0046] On the other hand, if the determination unit 150 determines that the identification of object OB is incorrect (NO in step S150) based on whether the angle calculated by the object-to-object angle calculation unit 140 is within a predetermined threshold range, it sends a message to the control unit 170 to that effect. Specifically, the determination unit 150 determines, for example, that objects 1, 2, and 3 cannot be correctly identified if the difference between the angle formed by the line L1 between object 1 and object 2 shown in Figure 2 and the line LX1 in the X-axis direction, and the angle formed by the line L2 between object 2 and object 3 and the line in the X-axis direction, is greater than or equal to a certain threshold.

[0047] <Operation Verification> The verification results of the operation of the determination device 1 according to this embodiment, as described above, will be explained using Figures 9 to 11. For this operational verification, an orange was placed on a tatami mat to serve as the target object OB.

[0048] Figure 9 shows the results of calculating the angle between objects OB from the image and depth information of the RGBD camera 200 in a top view. The image on the left of Figure 9 shows the relative positions of oranges 1 before they moved, and the image on the right of Figure 9 shows the relative positions of oranges 1 after they moved. Specifically, the determination is made by calculating the angles (θ1, θ2) formed by the horizontal parallel lines of the RGBD camera 200 and the lines connecting the oranges. If the angle difference between θ1 and θ2 is less than 10°, it is determined that there is a reasonable degree of agreement. For example, in the image on the right in Figure 9, the angle difference between θ1 and θ2 is 3°, so it is determined that they are in agreement to some extent. In the image on the left in Figure 9, the angle difference between θ1 and θ2 is 38°, so it is determined that they are not in agreement.

[0049] In Figure 10(A), the oranges are placed on the floor, and the size of the oranges, the depth to the oranges, and the angle of the straight line connecting the oranges are shown. Furthermore, Figure 10(B) shows how the oranges are arranged in a straight line. In Figure 10, th represents the angle formed by the line between the oranges and the line in the horizontal direction of the camera, h represents the vertical size of the oranges, w represents the horizontal size of the oranges, d represents the depth from the camera to the oranges, the numbers indicate the order of proximity to the camera, and the mesh represents the length in the depth and horizontal directions.

[0050] In other words, Figure 10 shows the process of a device that, based on the depth information from the RGBD camera 200, measures the distance from the RGBD camera 200 to the oranges, the size of each orange, and the angle of the straight line connecting the oranges. When the angle of the straight line connecting the first and second oranges matches the angle of the straight line connecting the second and third oranges to a certain extent, the device draws a rectangle on the floor.

[0051] Figure 11 shows the orange placed on top of the water bottle in Figure 11(A). Figure 11(B) also shows the orange being returned to the floor from the top of the water bottle. Note that each symbol indicates the same information as in Figure 10.

[0052] In other words, Figure 11 shows the process when oranges are lined up and one of them is placed on top of a water bottle (approximately 20 cm high). The upper right image in Figure 11 is an image from above, and when converted to an angle viewed from above, the angle at which all the oranges are viewed from above is roughly the same as the angle at which all the oranges are viewed from approximately ground level. In the system used for operational verification, for example, a point is drawn on the orange when the angles of each line match to a certain extent.

[0053] <Effects and Actions> As described above, the determination device 1 according to this embodiment includes an imaging unit 200 that acquires imaging data including image data and point cloud data; an image processing unit 110 that identifies an object OB from the image data captured by the imaging unit 200; a reference position calculation unit 120 that calculates a reference position on the RGB image of the identified object OB; an object position information calculation unit 130 that calculates the actual dimensional coordinates in the X and Z directions based on the calculated reference position, depth information corresponding to the reference position, and parameters of the imaging unit 200; an object-to-object angle calculation unit 140 that calculates the angle of the line between objects with respect to the line in the X direction based on the calculated actual dimensional coordinates of each object OB; and a determination unit 150 that determines whether the identification of the object OB is correct or incorrect based on whether the calculated angle is within a predetermined threshold range. In other words, the determination device 1 according to this embodiment is based on the fact that crops tend to be planted in a straight line when planted. The image processing unit 110 identifies the object OB from the image data captured by the imaging unit 200, the reference position calculation unit 120 calculates a reference position on the RGB image of the identified object OB, the object position information calculation unit 130 calculates the actual dimensional coordinates in the X and Z directions based on the calculated reference position, depth information corresponding to the reference position, and parameters of the imaging unit 200, the object-to-object angle calculation unit 140 calculates the angle of the line between objects with respect to the line in the X direction based on the calculated actual dimensional coordinates of each object OB, and the determination unit 150 determines whether the identification of the object OB is correct or incorrect based on whether the calculated angle is within a predetermined threshold range. Therefore, it can complement the accuracy of identification methods using machine learning and image processing.

[0054] In the determination device 1 according to this embodiment, the object OB is a crop, and the determination unit 150 determines whether the identification of the object OB is correct or incorrect based on a predetermined threshold range that is based on the characteristics of the positional relationship of the crop at the time of planting. In other words, the determination unit 150 determines whether the identification of object OB is correct or incorrect based on the characteristics of the positional relationship of the crops at the time of planting, and whether the deviation from those characteristics is within a predetermined threshold range. More specifically, the determination unit 150 focuses on the fact that crops are planted in a relatively straight line, and determines whether the identification of object OB is correct or incorrect based on whether the deviation from that straight line is within a predetermined threshold range. Therefore, it can complement the accuracy of identification methods using machine learning and image processing.

[0055] In the determination device 1 according to this embodiment, the imaging unit 200 is an RGBD camera. In other words, the imaging unit 200 acquires imaging data that includes image data and point cloud data. Furthermore, the imaging information acquired by the imaging unit 200 includes RGB image data and depth information. Therefore, using imaging information from a single imaging unit 200, the object position information calculation unit 130 can accurately and quickly calculate the actual dimensional coordinates in the X-axis and Z-axis directions. Therefore, it is possible to enhance the accuracy of identification methods using machine learning and image processing.

[0056] In the determination device 1 according to this embodiment, the object-to-object angle calculation unit 140 calculates the object-to-object angle based on the depth difference between at least two objects OB and the lateral distance difference in the field of view of the RGBD camera 200, in a plane that extends in the lateral and depth directions within the field of view of the RGBD camera 200. In other words, the object position information calculation unit 130 calculates the angles between objects and the lines in the X-axis direction from the actual dimensional coordinates of each object OB calculated by the unit. Therefore, it can complement the accuracy of identification methods using machine learning and image processing.

[0057] In the determination device 1 according to this embodiment, the object position information calculation unit 130 calculates the actual horizontal dimension coordinates of each object OB within the field of view of the RGBD camera 200 based on the size of the image sensor of the RGBD camera 200 or the total number of pixels. In other words, the object position information calculation unit 130 calculates the actual dimensional coordinates in the X-axis direction based on the reference position of each object OB calculated by the reference position calculation unit 120, the depth information corresponding to that reference position, and parameters of the RGBD camera 200 such as sensor size and focal length. Therefore, it can complement the accuracy of identification methods using machine learning and image processing.

[0058] In the determination device 1 according to this embodiment, the object position information calculation unit 130 calculates the actual dimensional coordinates in the depth direction within the field of view of the RGBD camera 200 for each object OB, based on the pixel information corresponding to the center of the object OB and the depth information. In other words, the object position information calculation unit 130 calculates the actual dimensional coordinates in the Z-axis direction based on the reference position of each object OB calculated by the reference position calculation unit 120, the depth information corresponding to that reference position, and parameters of the RGBD camera 200 such as sensor size and focal length. Therefore, it can complement the accuracy of identification methods using machine learning and image processing.

[0059] The determination method in the determination device 1 according to this embodiment includes: a first step in which the imaging unit 200 acquires imaging data including image data and point cloud data; a second step in which the image processing unit 110 identifies an object from the image data captured in the first step; a third step in which the reference position calculation unit 120 calculates a reference position on the RGB image of the object OB identified in the first step; a fourth step in which the object position information calculation unit 130 calculates the actual dimensional coordinates in the X and Z directions based on the reference position calculated in the third step, depth information corresponding to the reference position, and parameters of the imaging unit 200; a fifth step in which the object-to-object angle calculation unit 140 calculates the angle of the line between the object OBs with respect to the line in the X direction based on the actual dimensional coordinates of each object OB calculated in the fourth step; and a sixth step in which the determination unit 150 determines whether the identification of the object OBs is correct or incorrect based on whether the angle calculated in the fourth step is within a predetermined threshold range. In other words, the determination method in the determination device 1 according to this embodiment is based on the fact that crops tend to be planted in a straight line when planted. The image processing unit 110 identifies the object OB from the image data captured by the imaging unit 200, the reference position calculation unit 120 calculates a reference position on the RGB image of the identified object OB, the object position information calculation unit 130 calculates the actual dimensional coordinates in the X-axis direction and the Z-axis direction based on the calculated reference position, depth information corresponding to the reference position, and parameters of the imaging unit 200, the object-to-object angle calculation unit 140 calculates the angle of the line between objects with respect to the line in the X-axis direction based on the calculated actual dimensional coordinates of each object OB, and the determination unit 150 determines whether the identification of the object OB is correct or incorrect based on whether the calculated angle is within a predetermined threshold range. Therefore, it can complement the accuracy of identification methods using machine learning and image processing.

[0060] A program for causing a computer to execute the determination method in the determination device 1 according to this embodiment includes: a first step in which the imaging unit 200 acquires imaging data including image data and point cloud data; a second step in which the image processing unit 110 identifies an object from the image data captured in the first step; a third step in which the reference position calculation unit 120 calculates a reference position on the RGB image of the object OB identified in the first step; a fourth step in which the object position information calculation unit 130 calculates the actual dimensional coordinates in the X and Z directions based on the reference position calculated in the third step, depth information corresponding to the reference position, and parameters of the imaging unit 200; a fifth step in which the object-to-object angle calculation unit 140 calculates the angle of the line between the object OBs with respect to the line in the X direction based on the actual dimensional coordinates of each object OB calculated in the fourth step; and a sixth step in which the determination unit 150 determines whether the identification of the object OBs is correct or incorrect based on whether the angle calculated in the fourth step is within a predetermined threshold range. In other words, the program for causing a computer to execute the determination method in the determination device 1 according to this embodiment is based on the fact that crops tend to be planted in a straight line when planted, and causes the image processing unit 110 to identify the object OB from the image data captured by the imaging unit 200, causes the reference position calculation unit 120 to calculate a reference position on the RGB image of the identified object OB, causes the object position information calculation unit 130 to calculate the actual dimensional coordinates in the X-axis direction and the Z-axis direction based on the calculated reference position, depth information corresponding to the reference position, and parameters of the imaging unit 200, causes the object-to-object angle calculation unit 140 to calculate the angle of the line between objects with respect to the line in the X-axis direction based on the actual dimensional coordinates of each object OB calculated, and causes the determination unit 150 to determine whether the identification of the object OB is correct or incorrect based on whether the calculated angle is within a predetermined threshold range. Therefore, it can complement the accuracy of identification methods using machine learning and image processing.

[0061] Furthermore, the determination device 1 of the present invention can be realized by recording the processing of the processor 100 on a recording medium readable by a computer system, and then having the processor 100 read and execute the program recorded on this recording medium. The computer system referred to here includes hardware such as an operating system and peripheral devices.

[0062] Furthermore, "computer system" shall also include the homepage provisioning environment (or display environment) if the WWW (World Wide Web) system is being used. Furthermore, the above program may be transmitted from a computer system that stores the program in a memory device or the like to another computer system via a transmission medium or by transmission waves within the transmission medium. Here, the "transmission medium" used to transmit a program refers to a medium that has the function of transmitting information, such as a network (communication network) like the Internet or a communication line (communication line) like a telephone line.

[0063] Furthermore, the above program may be intended to implement some of the functions described above. Furthermore, the aforementioned functions may be realized in combination with programs already recorded in the computer system, such as so-called differential files (differential programs).

[0064] While embodiments of this invention have been described in detail above with reference to the drawings, all determination devices that a person skilled in the art can implement by appropriately modifying the design based on the determination device 1 described above as an embodiment of the present invention also fall within the technical scope of the present invention, insofar as they encompass the gist of the present invention. Within the scope of the concept of this invention, a person skilled in the art can conceive of various modifications and alterations, and it is understood that these modifications and alterations also fall within the technical scope of this invention. For example, any modifications made by a person skilled in the art to the above-described embodiments, such as adding, deleting, or changing the design of components, or adding, omitting, or changing the conditions of processes, are also included within the technical scope of the present invention, as long as they retain the essence of the present invention.

[0065] Furthermore, any other effects and advantages brought about by the embodiments described herein that are obvious from this specification or that can be appropriately conceived by those skilled in the art are naturally considered to be brought about by the present invention. Various inventions can be formed by appropriately combining the multiple components disclosed in the above embodiments. For example, some components may be removed from all the components shown in the embodiment. Furthermore, components from different embodiments may be combined as appropriate. [Explanation of Symbols]

[0066] 1; Judgment device 100; processor 110; Image Processing Unit 120;Reference position calculation unit 130; Object position information calculation unit 140; Angle calculation unit between objects 150; Judgment section 160;Memory part 170; Control Unit 200; RGBD camera OB; object

Claims

1. An imaging unit that acquires imaging data including image data and point cloud data of depth information, An image processing unit that identifies an object from image data captured by the aforementioned imaging unit, A reference position calculation unit calculates a reference position on the RGB image of the identified object, An object position information calculation unit calculates the actual dimensional coordinates in the X-axis and Z-axis directions from the center of the RGB image based on the calculated reference position, depth information corresponding to the reference position, and the parameters of the imaging unit. An object-to-object angle calculation unit calculates the angle between the lines of the objects relative to the line in the X-axis direction based on the actual dimensional coordinates of each object, A determination unit that determines whether the identification of the object is correct or incorrect based on whether the calculated angle is within a predetermined threshold range, A determination device, including a determination device.

2. The aforementioned object is an agricultural product, The determination device according to claim 1, wherein the determination unit determines whether the identification of the object is correct or incorrect based on a predetermined threshold range based on the characteristics of the positional relationship of the crops at the time of planting.

3. The determination device according to claim 1, wherein the imaging unit is an RGBD camera.

4. The determination device according to claim 3, wherein the object-to-object angle calculation unit calculates the object-to-object angle based on the difference in depth between at least two objects and the difference in distance in the lateral direction of the RGB camera's field of view, in a plane extending in the lateral and depth directions of the RGB camera's field of view.

5. The determination device according to claim 3, wherein the object position information calculation unit calculates the actual horizontal coordinates of each object in the field of view of the RGB camera based on the size of the image sensor of the RGB camera or the total number of pixels.

6. The determination device according to claim 5, wherein the object position information calculation unit calculates the actual dimensional coordinates in the depth direction in the field of view of the RGBD camera for each object based on the pixel information corresponding to the center of the object and the depth information.

7. A determination method in a determination device including an imaging unit, an image processing unit, a reference position calculation unit, an object position information calculation unit, an object angle calculation unit, and a determination unit, The imaging unit performs a first step of acquiring imaging data including image data and point cloud data of depth information, The image processing unit performs a second step of identifying an object from the image data captured in the first step, The reference position calculation unit performs a third step of calculating a reference position on the RGB image of the object identified in the first step, The object position information calculation unit performs a fourth step in which it calculates the actual dimensional coordinates in the X-axis and Z-axis directions from the center of the RGB image based on the reference position calculated in the third step, the depth information corresponding to the reference position, and the parameters of the imaging unit. The object-to-object angle calculation unit performs a fifth step in which it calculates the angle of the line between the objects with respect to the line in the X-axis direction based on the actual dimensional coordinates of each object calculated in the fourth step, The determination unit performs a sixth step in which it determines whether the identification of the object is correct or incorrect based on whether the angle calculated in the fourth step is within a predetermined threshold range, A determination method that includes this.

8. A program for causing a computer to execute a determination method in a determination device including an imaging unit, an image processing unit, a reference position calculation unit, an object position information calculation unit, an object angle calculation unit, and a determination unit, The imaging unit performs a first step of acquiring imaging data including image data and point cloud data of depth information, The image processing unit performs a second step of identifying an object from the image data captured in the first step, The reference position calculation unit performs a third step of calculating a reference position on the RGB image of the object identified in the first step, The object position information calculation unit performs a fourth step in which it calculates the actual dimensional coordinates in the X-axis and Z-axis directions from the center of the RGB image based on the reference position calculated in the third step, the depth information corresponding to the reference position, and the parameters of the imaging unit. The object-to-object angle calculation unit performs a fifth step in which it calculates the angle of the line between the objects with respect to the line in the X-axis direction based on the actual dimensional coordinates of each object calculated in the fourth step, The determination unit performs a sixth step in which it determines whether the identification of the object is correct or incorrect based on whether the angle calculated in the fourth step is within a predetermined threshold range, A program that causes a computer to perform a process that includes [a specific step].

Images