Shape acquisition system, processing device, shape acquisition method, and computer program

Abstract

This technology provides a way to understand the changes in branch shape after cutting. [Solution] The shape acquisition system disclosed herein is a shape acquisition system that acquires the shape of branches after cutting a branch included in a crop. The system comprises a first imaging device that images the crop, and a processing device that performs processing based on first image data obtained by the first imaging device. The processing device performs a process of acquiring shape information of the crop over time, with the base of the main stem of the crop as the reference point, based on the first image data, and an acquisition process of acquiring coordinate information indicating the shape of the branch after cutting, based on the shape information.

Description

Technical Field

[0001] The present disclosure relates to a shape acquisition system, a processing device, a shape acquisition method, and a computer program.

Background Art

[0002] Patent Document 1 discloses a technique for providing a reference point on a specific part of a plant and measuring the growth amount and growth rate of the plant based on the change in the length from the reference point to the shoot apex.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, the shape of agricultural crops may change not only due to growth but also due to cutting of branches by workers such as pruning, leaf removal, and harvesting. If the change in the shape of the branch after such cutting can be grasped, the shape of the branch can be grasped, and the numerical quantification of the branch cutting work by the worker can be achieved. Therefore, a technique capable of grasping the change in the shape of the branch after pruning is desired.

Means for Solving the Problems

[0005] The shape acquisition system according to the present disclosure is a shape acquisition system that acquires the shape of a branch after cutting the branch included in a crop. This system includes a first imaging device that images the crop, and a processing device that performs processing based on first image data obtained by the first imaging device. The processing device executes processing for acquiring shape information of the crop with a reference point at the root of the main stem of the crop over time based on the first image data, and acquisition processing for acquiring coordinate information indicating the shape of the branch after cutting based on the shape information. [Effects of the Invention]

[0006] According to this disclosure, it is possible to understand the changes in branch shape after cutting. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 shows an example of the overall configuration of the shape acquisition system according to the embodiment. [Figure 2] Figure 2 is a block diagram showing an example configuration of the first imaging device, the second imaging device, and the information processing device. [Figure 3] Figure 3 is a sequence diagram showing an example of the operation determination process and shape acquisition process performed by the processing unit. [Figure 4] Figure 4 shows an example of characteristic points for the worker, cutting tool, and crop, which are determined based on the second image data. [Figure 5] Figure 5 shows an example of an image taken immediately before a worker performs a pruning action. [Figure 6] Figure 6 shows an example of map information (environmental map) created by the processing unit. [Figure 7] Figure 7 shows a portion of the characteristic point cloud of a crop after its secondary branches have been pruned. [Modes for carrying out the invention]

[0008] First, the details of the embodiment will be listed and explained. [Summary of the Embodiment]

[0009] (1) The shape acquisition system disclosed herein is a shape acquisition system for acquiring the shape of branches after cutting a branch included in a crop. The system comprises a first imaging device for imaging the crop and a processing device for performing processing based on first image data obtained by the first imaging device. The processing device performs processing to acquire shape information of the crop over time, with the base of the main stem of the crop as the reference point, based on the first image data, and acquisition processing to acquire coordinate information indicating the shape of the branch after cutting, based on the shape information. According to the above configuration, by acquiring shape information of the crop and coordinate information indicating the shape of the branches after cutting, it is possible to understand the changes in the shape of the branches after cutting. Furthermore, since the reference point for shape information is set at the base of the main stem of the crop, the relative position of the reference point to the crop will not change significantly unless the crop is pulled up from the roots. This allows for improved accuracy of both the shape information and the coordinate information based on that shape information.

[0010] (2) In the shape acquisition system described in (1) above, if a second imaging device is further provided to image the worker performing the cutting of the crop, the acquisition process preferably includes a first process of determining whether the worker has performed the cutting operation of the branch using a cutting tool based on the second image data from the second imaging device, and a second process of acquiring the coordinate information based on the shape information after the cutting operation from the shape information acquired over time, if it is determined that the cutting operation has been performed. In this case, the first process can determine whether or not the operator performed a cutting operation, and shape information after the cutting operation can be appropriately obtained.

[0011] (3) In the shape acquisition system described in (2) above, if the crop includes a plurality of branches, the second process may, when it is determined that the cutting operation has been performed, include a process to identify the cut branch from among the plurality of branches based on a comparison result between the shape information before the cutting operation and the shape information after the cutting operation, from the shape information acquired over time, and a process to acquire the coordinate information of the cut branch. In this case, the coordinate information of the branch cut from among the plurality of branches possessed by the crop can be obtained.

[0012] (4) Also, in the shape acquisition system of (1) or (2) above, the shape information after the cutting operation may be the shape information after a predetermined period has elapsed from the timing determined to be the timing when the cutting operation was performed, among the shape information acquired over time. In this case, the probability of obtaining shape information indicating the branch in the state after the cutting operation is completed can be increased after the cutting operation.

[0013] (5) In any one of the shape acquisition systems of (1) to (4) above, the branch is a sub-branch extending from a main branch connected to the main trunk, and the coordinate information may include coordinates indicating the branch point where the branch is connected to the main branch and coordinates indicating the position where the branch is cut. In this case, the coordinate information of the sub-branch to be cut can be obtained.

[0014] (6) Also, in any one of the shape acquisition systems of (1) to (5) above, the first imaging device may have a first mounting tool mounted on the front chest of the operator. In this case, the first imaging device is mounted on the front chest of the operator.

[0015] (7) Also, in any one of the shape acquisition systems of (1) to (6) above, the second imaging device may have a second mounting tool mounted on the head of the operator. In this case, the second imaging device is mounted on the head of the operator.

[0016] (8) From another perspective, the present disclosure is a processing device that acquires the shape of a branch after cutting of a branch included in a crop. This processing device performs processing for acquiring shape information of the crop over time with reference to the root of the main trunk of the crop based on first image data from a first imaging device that images the crop, Based on the shape information, an acquisition process is performed to obtain coordinate information indicating the shape of the branch after cutting.

[0017] (9) Furthermore, this disclosure from other perspectives is: This is a shape acquisition method for obtaining the shape of branches after they have been cut from a crop. A step of acquiring shape information of the crop over time, with the base of the main stem of the crop as the reference point, based on first image data obtained by a first imaging device that images the crop; The process includes an acquisition step of acquiring coordinate information indicating the shape of the branch after cutting, based on the shape information.

[0018] (10) Furthermore, this disclosure from other perspectives is: This is a computer program that causes a computer to perform a shape acquisition process to obtain the shape of branches after they have been cut from a crop. On the computer, A step of acquiring shape information of the crop over time, with the base of the main stem of the crop as the reference point, based on first image data obtained by a first imaging device that images the crop; Based on the shape information, the system performs an acquisition step to obtain coordinate information indicating the shape of the branch after cutting.

[0019] [Details of the embodiment] Preferred embodiments will be described below with reference to the drawings. Furthermore, at least some of the embodiments described below may be combined in any way.

[0020] [Regarding the overall configuration of the shape acquisition system] Figure 1 shows an example of the overall configuration of the shape acquisition system according to the embodiment. In Figure 1, the shape acquisition system 1 has the function of acquiring the shape of crops planted in the field. A crop A planted in a field has, for example, a main stem A1, multiple main branches A2, and multiple secondary branches A3. The main stem A1 extends upward from the ground G in the field. The base A10 of the main stem A1 is the point where the main stem A1 and the ground G intersect. Multiple main branches (two in the diagram) A2 extend horizontally from the tip (upper end) of the main trunk A1. Multiple secondary branches A3 extend upward from the main branch A2. Multiple secondary branches A3 are aligned in the direction in which the main branch A2 extends. Shape acquisition system 1 acquires the shape of crop A. More specifically, shape acquisition system 1 acquires coordinate information indicating the branch shape of the secondary branch A3 of the cut crop A. The coordinate information indicating the branch shape of secondary branch A3 acquired by shape acquisition system 1 is the coordinate information of secondary branch A3 that is connected to the main branch A2 after being cut, and not the coordinate information of the cut piece that is separated from crop A by being cut.

[0021] The shape acquisition system 1 includes a first imaging device 2, a second imaging device 4, and an information processing device 6. The information processing device 6, the first imaging device 2, and the second imaging device 4 are connected to each other via a network 7 such as a LAN or the Internet, enabling them to communicate with one another. The first imaging device 2 and the second imaging device 4 are attached to worker M. Worker M is a worker who performs the task of cutting branches of crop A. Therefore, worker M holds a cutting tool T, such as pruning shears. Branch cutting includes pruning, leaf removal, and harvesting. In this embodiment, the case of pruning among the branch cutting methods will be described below. In this embodiment, the pruning work and pruning actions refer to the work and actions of pruning multiple secondary branches A3.

[0022] The first imaging device 2 comprises a device body 2a and a first attachment 2b. The first attachment 2b is attached to the upper body of worker M. When the first attachment 2b is attached to worker M, the device body 2a is fixed to the front of worker M's chest. The device body 2a has the function of imaging the area in front of worker M, including crop A. The second imaging device 4 comprises a device body 4a and a second attachment 4b. The second attachment 4b is attached to the head of the worker M. When the second attachment 4b is attached to the worker M, the device body 4a is fixed to the worker M's head. The device body 4a has the function of imaging the area around the worker M, including the worker M and the crop A.

[0023] The information processing device 6 is a computer such as a server. The information processing device 6 has the function of performing motion analysis of worker M based on image data captured by the second imaging device 4. Furthermore, the information processing device 6 has the function of creating an environmental map of the worker M's surroundings and estimating its own position using VSLAM (Visual Simultaneous Localization and Mapping). Specifically, the information processing device 6 extracts feature points based on the image data captured by the first imaging device 2, and creates an environmental map and estimates its own position based on the changes in these feature points. Furthermore, the information processing device 6 acquires the shape of crop A identified by the environmental map and obtains coordinate information indicating the branch shape of the secondary branch A3 of the pruned crop A.

[0024] Figure 2 is a block diagram showing an example configuration of the first imaging device 2, the second imaging device 4, and the information processing device 6. In Figure 2, the first imaging device 2 (device body 2a) includes a processing unit 8, a storage unit 10, a communication unit 12, a pair of cameras 14, and a bus 15. The bus 15 connects each part to the others.

[0025] The pair of cameras 14 capture images in front of the worker M. The pair of cameras 14 constitute a stereo camera. The pair of cameras 14 each output the captured image as first image data. The pair of cameras 14 capture images of crop A and the area around crop A. Therefore, the first image data output from the pair of cameras 14 includes images of crop A. The communication unit 12 has the function of communicating with the information processing device 6 via the network 7. The communication unit 12 has, for example, the function of a wireless LAN terminal or a wireless communication terminal in a mobile communication system.

[0026] The processing unit 8 is a variety of processors suitable for computer control, such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), etc. The storage unit 10 includes, for example, flash memory, hard disk, SSD (Solid State Drive), ROM (Read Only Memory), etc. The memory unit 10 stores computer programs to be executed by the processing unit 8, as well as other necessary information. The processing unit 8 realizes the various processing functions of the first imaging device 2 by executing the computer programs stored on a computer-readable, non-transient recording medium such as the memory unit 10.

[0027] The processing unit 8 has the function of controlling a pair of cameras 14 and a communication unit 12. The processing unit 8 causes the pair of cameras 14 to capture images in front of the worker M and obtains first image data. The processing unit 8 captures images at predetermined time intervals and provides the information processing unit 6 with a plurality of first image data sets arranged in a time series. Time information indicating the time of capture is attached to each of the plurality of first image data sets. Therefore, the capture time of each of the plurality of first image data sets can be recognized by the time information attached to the first image data set.

[0028] The second imaging device 4 (device body 4a) includes a processing unit 18, a storage unit 20, a communication unit 22, a camera 24, and a bus 25. The bus 25 connects each of the parts to each other.

[0029] Camera 24 is an omnidirectional camera capable of capturing images in all directions in a single shot. Camera 24 has a fisheye lens, which guides the surrounding scenery to the image sensor, capturing an omnidirectional image. Camera 24 outputs the captured image as second image data. The imaging range of camera 24 is set to the area around worker M's hands. The imaging range is set to include a part of worker M and crop A when worker M is performing pruning work. In other words, the camera 24 captures an omnidirectional image centered on worker M's hands. Therefore, the second image data output from camera 24 includes an image of worker M, an image of the cutting tool T, and an image of crop A.

[0030] The communication unit 22 has the function of communicating with the information processing device 6 via the network 7. The communication unit 22 has, for example, the function of a wireless LAN terminal or a wireless communication terminal in a mobile communication system.

[0031] The processing unit 18 is a variety of processors suitable for computer control, such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), etc. The storage unit 20 includes, for example, flash memory, hard disk, SSD (Solid State Drive), ROM (Read Only Memory), etc. The memory unit 20 stores computer programs and necessary information for the processing unit 18 to execute. The processing unit 18 realizes the various processing functions of the second imaging device 4 by executing the computer programs stored on a computer-readable, non-transient recording medium such as the memory unit 20.

[0032] The processing unit 18 has the function of controlling the camera 24 and the communication unit 22. The processing unit 18 causes the camera 24 to capture images of the hands of worker M and obtains second image data. The processing unit 18 captures images at predetermined time intervals and provides the information processing unit 6 with a plurality of second image data sets arranged in a time series. Time information indicating the time of capture is attached to each of the plurality of second image data sets. Therefore, the capture time of each of the plurality of second image data sets can be recognized by the time information attached to the second image data set.

[0033] The information processing device 6 includes a processing unit 28, a storage unit 30, a communication unit 32, and a bus 35. The bus 35 connects each unit to the others. The communication unit 32 has the function of communicating with the first imaging device 2 and the second imaging device 4 via the network 7. The communication unit 32 has the function of, for example, a wireless LAN terminal or a wireless communication terminal in a mobile communication system.

[0034] The processing unit 28 is a variety of processors suitable for computer control, such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), etc. The storage unit 30 includes, for example, flash memory, hard disk, SSD (Solid State Drive), ROM (Read Only Memory), etc. The memory unit 30 stores computer programs and necessary information for the processing unit 28 to execute. The processing unit 28 realizes the various processing functions of the first imaging device 2 by executing the computer programs stored on a computer-readable, non-transient recording medium such as the memory unit 30.

[0035] Furthermore, the memory unit 30 stores the first learning model 30a, the second learning model 30b, and the work performance database 30c, etc. The first learning model 30a is used in the motion determination process (explained later) to determine the characteristic points of worker M and crop A, etc. The second learning model 30b is used in the shape acquisition process (explained later) to determine the characteristic points of crop A. The work performance database 30c is a database for registering the coordinate information of pruned secondary branches A3. The coordinate information of secondary branches A3 is obtained through shape acquisition processing.

[0036] The processing unit 28 has the function of issuing commands to and controlling the first imaging device 2 and the second imaging device 4. Furthermore, the processing unit 28 has the function of executing the operation determination process and the shape acquisition process, which will be described later. The processing unit 28 executes these processes using the first image data and the second image data. The motion determination process analyzes the actions of worker M and determines whether or not worker M performed a pruning action on crop A. The shape acquisition process is a process that acquires coordinate information indicating the branch shape of secondary branch A3 of crop A after pruning.

[0037] Figure 3 is a sequence diagram showing an example of the operation determination process and shape acquisition process performed by the processing unit 28. The processing unit 28 executes the operation determination process and the shape acquisition process in parallel. First, worker M puts on the first imaging device 2 and the second imaging device 4. Then, worker M inputs an operation to the switch that starts processing on the shape acquisition system 1, and begins pruning crop A.

[0038] When the switch receives an input from operator M, the processing unit 28 of the information processing device 6 starts the shape acquisition process (step S1). Furthermore, once the processing unit 28 has started the shape acquisition process, it outputs start information indicating that the shape acquisition process has begun (step S2).

[0039] While the shape determination process and motion determination process are being executed, the information processing device 6 causes the first imaging device 2 and the second imaging device 4 to take images. Therefore, while the shape determination process and motion determination process are being executed, the first imaging device 2 provides the information processing device 6 with a plurality of first image data arranged in time series. Similarly, the second imaging device 4 provides the information processing device 6 with a plurality of second image data arranged in time series. The information processing device 6 stores the given plurality of first image data and plurality of second image data in the storage unit 30.

[0040] Furthermore, while the shape acquisition process is being performed, the processing unit 28 uses VSLAM to create an environmental map of the area in front of the worker M and to estimate its own position. Based on the first image data, the processing unit 28 extracts feature points and creates an environmental map and estimates its own position based on the changes in these feature points. The first image data includes images from a pair of cameras 14 that constitute a stereo camera. Therefore, spatial understanding, such as the identification of the location of feature points, becomes easier for the processing unit 28. The processing unit 28 acquires point cloud data consisting of feature points of surrounding objects and creates an environmental map and estimates its own position based on the point cloud data. The environmental map also includes information about objects around (in front of) worker M. Therefore, the environmental map includes information about the shape of crop A. The created environmental map is stored in the storage unit 30 as map information.

[0041] When start information is output during the shape acquisition process, the processing unit 28 starts the operation determination process and determines the characteristic points of the worker M, the cutting tool T, and the crop A based on multiple second image data (step S3). The processing unit 28 determines multiple joint points as characteristic points of worker M and also determines the skeleton based on the second image data, and generates 3D information for each of the multiple joint points. Furthermore, the processing unit 28 also determines the feature points of the cutting tool T and crop A based on the second image data, and generates three-dimensional information of the feature points of the cutting tool T and three-dimensional information of the feature points of crop A. In this way, the processing unit 28 generates three-dimensional information of feature points for the worker M, the cutting tool T, and the crop A.

[0042] Figure 4 shows examples of characteristic points for worker M, cutting tool T, and crop A, which are determined based on the second image data. In Figure 4, characteristic points and skeletal lines are superimposed on the image based on the second image data. The processing unit 28 detects the image region of worker M included in the second image data, and estimates the feature point cloud m10 and skeletal line m20 of worker M based on this image region. Similarly, the processing unit 28 detects the image region of crop A included in the image and estimates the feature point cloud a10 and skeletal line a20 of crop A based on this image region. The processing unit 28 also detects the image region of cutting tool T included in the image and estimates the feature point cloud t10 and skeletal line t20 of cutting tool T based on this image region.

[0043] The processing unit 28 uses a machine learning method to extract image regions of worker M, cutting tool T, and crop A from the second image data, and obtains feature point clouds m10, a10, t10 and skeletal lines m20, a20, t20. The first learning model 30a stored in the memory unit 30 has been trained to output feature point clouds m10, a10, t10 and skeletal lines m20, a20, t20 from the second image data. The processing unit 28 uses the machine learning-trained first learning model 30a to obtain the feature point clouds m10, a10, t10 and the skeleton lines m20, a20, t20.

[0044] The feature point cluster m10 includes multiple joint points m11 corresponding to the shoulder joint, elbow joint, wrist (hand) of worker M. These joint points m11 are connected by skeletal lines m20 based on worker M's body shape. The feature point group t10 includes the tip point t11 ​​and the base point t12 of the cutting tool T. The tip point t11 ​​corresponds to the cutting edge of the cutting tool T. The base point t12 corresponds to the base of the cutting tool T. The skeletal line t20 connects the tip point t11 ​​and the base point t12.

[0045] The feature point group a10 includes the root point a11, the first branch point a12, multiple second branch points a13, and multiple endpoint points a14. The root point a11 corresponds to the root point A10 of crop A. In other words, the root point a11 corresponds to the point where the main stem A1 and the ground G intersect. The first branching point a12 corresponds to the point where the two main branches A2 connect to the main trunk A1. The second branching point a13 corresponds to the point where the secondary branch A3 connects to the main branch A2. The multiple endpoints a14 correspond to the endpoints of the multiple secondary branches A3. The structural line a20 includes the main trunk line a21, the main branch line a22, and the secondary branch line a23. Main line a21 is the line connecting the base point a11 and the first branch point a12, and corresponds to main line A1. Main branch line a22 is the line corresponding to main branch A2. The secondary branch line a23 is the line connecting the second branch point a13 and the endpoint a14, and corresponds to the secondary branch A3. The processing unit 28 generates each feature point as three-dimensional information.

[0046] As shown in Figure 3, in step S3, the characteristic points of the worker M, the cutting tool T, and the crop A are determined. Based on the determined characteristic points and skeletal lines, the processing unit 28 performs a motion analysis of the worker M and determines whether or not the worker M performed a pruning motion (step S6). The processing unit 28 grasps the movement of each feature point in each of the multiple second image data and performs motion analysis of the operator M and the cutting tool T. The processing unit 28 determines that worker M has performed a pruning operation if it determines that worker M is gripping the cutting tool T and that the cutting tool T has crossed a part of crop A. The processing unit 28 repeats step S6 until it determines that worker M has performed a pruning operation. Once it determines that worker M has performed a pruning operation, the processing unit 28 outputs determination information (step S7).

[0047] Figure 5 shows an example of an image taken immediately before worker M performs a pruning action. Figure 5 shows worker M approaching the longest of the multiple secondary branches A3, A31 (Figure 1), with the cutting tool T to prune it. When the processing unit 28 determines that the skeletal line t20 of the cutting tool T and the secondary branch line a231 (a23) of the secondary branch A31 intersect, it determines that the worker M has performed a pruning operation.

[0048] The processing unit 28 outputs the time information attached to the second image data when it is determined that the skeletal line t20 and the secondary branch line a231 have intersected, along with the determination information.

[0049] As shown in Figure 3, while the pruning operation is being determined in the operation determination process, the processing unit 28 identifies the base of the main stem A1 of crop A based on the map information in the shape acquisition process (step S4). The processing unit 28 refers to the map information and identifies the base A10 of crop A from the information about the shape of crop A contained in the map information. Next, the processing unit 28 acquires shape information of crop A with the root A10 as the origin. The processing unit 28 acquires shape information of crop A with the root A10 as the origin for each of the multiple first image data. In this way, the processing unit 28 acquires shape information of crop A over time (step S5).

[0050] Figure 6 shows an example of map information (environmental map) created by the processing unit 28. In Figure 6, feature points and skeletal lines are superimposed on the map information. The processing unit 28 acquires the characteristic points of an object in front of worker M as coordinate points and creates map information. The map information includes point cloud data consisting of multiple coordinate points. The map information includes at least a ground area GS showing the shape of the ground G in front of worker M, and a crop area AS showing the shape of crop A extending from the ground G.

[0051] The processing unit 28 detects the crop region AS included in the map information and, based on this crop region AS, obtains the feature point cloud s10 and the skeletal line s20 of crop A. The processing unit 28 uses a machine learning method to obtain a feature point cloud s10 and a skeleton line s20 from the crop region AS included in the map information. The second learning model 30b stored in the memory unit 30 has been trained to output a feature point cloud s10 and a skeleton line s20 from map information. The processing unit 28 uses the machine learning-trained second learning model 30b to obtain the feature point cloud s10 and the skeleton line s20. The feature point cloud and skeleton lines of crop A obtained based on the second image data are shown as feature point cloud a10 and skeleton line a20, respectively, as shown in Figure 4. However, the feature point cloud and skeleton lines of crop A obtained based on map information (first image data) are shown as feature point cloud s10 and skeleton line s20, respectively, as shown in Figure 6.

[0052] The feature point group s10 includes a root point s11, a first branch point s12, multiple second branch points s13, and multiple tip points s14. The root point s11 corresponds to the root point A10 of crop A. In other words, the root point s11 corresponds to the point where the main stem A1 and the ground G intersect. The first branching point s12 corresponds to the point where the two main branches A2 connect to the main trunk A1. The second branching point s13 corresponds to the point where the secondary branch A3 connects to the main branch A2. The multiple endpoints s14 correspond to the endpoints of the multiple secondary branches A3. The skeletal line s20 is a line that connects each feature point according to the shape of crop A. The processing unit 28 generates the feature point cloud s10 as three-dimensional information.

[0053] The processing unit 28 obtains shape information of crop A based on the feature point cloud s10. The processing unit 28 converts the feature point cloud s10, which has three-dimensional information, into coordinates with the root point s11 as the origin. As a result, the processing unit 28 obtains shape information of crop A with the root A10 as the origin. In other words, the shape information includes the feature point cloud s10 converted into coordinates with the root point s11 as the origin. Furthermore, each of the multiple shape information pieces acquired over time is accompanied by the time information attached to the original data, the first image data.

[0054] As shown in Figure 3, the processing unit 28 acquires shape information of crop A (step S5) and waits for judgment information from the operation judgment process (step S8). When the operation determination process outputs determination information indicating that worker M has performed a pruning operation (step S7), the processing unit 28 compares the shape information after a predetermined period from the time the pruning operation was performed with the shape information before the pruning operation (step S9).

[0055] In this way, by using shape information obtained after a predetermined period has elapsed from the time it is determined that pruning has been performed, the probability of obtaining shape information that shows the secondary branch A3 in the state after pruning has been completed can be increased. The processing unit 28 treats the time indicated by the time information attached to the determination information as the timing when the pruning operation was performed. The predetermined period is, for example, 0.5 seconds.

[0056] The processing unit 28 compares shape information based on the first image data captured 0.5 seconds after the timing of the pruning operation with shape information based on the first image data captured 0.5 seconds before the timing of the pruning operation. The processing unit 28 determines whether crop A has been pruned or not based on the comparison in step S10.

[0057] Figure 7 shows a portion of the characteristic point cluster s10 of crop A after the secondary branch A3 has been pruned. In Figure 7, the secondary branch A31 (Figure 1) has been pruned. The skeletal line s201 corresponding to secondary branch A31 is a straight line connecting the tip point s141 and the second branching point s131. Therefore, the positions of tip point s142 and tip point s141 are significantly different. Tip point s142 is tip point s14 connected to skeletal line s201 before the pruning operation. Tip point s141 is tip point s14 connected to skeletal line s201 after the pruning operation. In other words, the shape information before pruning and the shape information after pruning show a difference in the position of the tip of the secondary branch A31.

[0058] If the comparison in step S9 does not show any differences as shown in Figure 7, the processing unit 28 determines that crop A was not pruned in step S10 in Figure 3 and proceeds to step S6. Therefore, the processing unit 28 again determines in the operation determination process whether or not the pruning operation was performed (step S6). In this case, the processing unit 28 also acquires the shape information of crop A in the shape acquisition process (step S5) and waits for the determination information from the operation determination process (step S8).

[0059] If the comparison in step S9 reveals a difference as shown in Figure 7, the processing unit 28 determines that crop A was pruned in step S10 in Figure 3 and proceeds to step S11. The processing unit 28 obtains the coordinate information of the secondary branch A31 where a difference was observed in the position of the tip. In other words, the processing unit 28 identifies the sub-branch A31, which showed a difference in the position of its tip in Figure 7, as the pruned sub-branch A3 (step S11), and obtains the coordinate information of the pruned sub-branch A31 (step S12). The coordinate information of the pruned secondary branch A31 refers to the shape information of crop A, with the root A10 as the origin, and includes the coordinates that indicate the shape of the secondary branch A3, which has been identified as being pruned.

[0060] The processing unit 28 obtains the coordinate information of the pruned secondary branch A31, specifically the coordinates of the tip point s141 where a positional difference was observed, and the coordinates of the second branch point s131 connected to the tip point s141 via the skeletal line s201. In this case, the coordinates of the tip point s141 indicate the position where the secondary branch A31 was pruned. The processing unit 28 can determine the length and shape of the pruned secondary branch A31 based on the coordinates of the tip point s141 and the coordinates of the second branching point s131.

[0061] Upon obtaining the coordinate information of the pruned secondary branch A3, the processing unit 28 determines in the action determination process whether or not the pruning operation has been performed again (step S6), obtains the shape information of crop A in the shape acquisition process (step S5), and waits for the determination information from the action determination process (step S8). As a result, the processing unit 28 repeatedly obtains the coordinate information of the pruned secondary branch A3.

[0062] The coordinate information of the pruned secondary branch A3 is registered in the work performance database stored in the memory unit 30 (step S20). The coordinate information registered in the work performance database is used for recording work performance, quantifying the pruning work performed by worker M, and accumulating work know-how based on the quantified work content.

[0063] The processing unit 28 of this embodiment performs the following: a process to acquire shape information of crop A over time based on first image data, with the base A10 of the main stem A1 of crop A as the reference point (step S5 in Figure 3); and an acquisition process to acquire coordinate information indicating the branch shape after pruning based on the shape information (steps S6 to S12 in Figure 3). With the above configuration, by obtaining shape information of crop A and coordinate information indicating the branch shape of secondary branch A3 after pruning, it is possible to understand the changes in branch shape after pruning. Furthermore, since the reference point for the shape information is set at the base A10 of the main stem A1 of crop A, the relative position of the reference point to crop A will not change significantly unless crop A is pulled up from the base A10. This improves the accuracy of the shape information and the coordinate information based on the shape information. The base A, which is the reference point for the shape information, is the point where the main stem A1 and the ground G intersect.

[0064] Furthermore, the acquisition process includes a first process (step S6) that determines whether or not the worker M performed a pruning operation using the cutting tool T based on the second image data from the second imaging device 4, and a second process (steps S7 to S12) that, if it is determined that a pruning operation has been performed, acquires coordinate information based on the shape information after the pruning operation from the shape information acquired over time. This allows the first process to determine whether or not worker M performed a pruning operation, and to appropriately acquire shape information after the pruning operation.

[0065] Furthermore, the second process may include, upon determining that a pruning operation has been performed, a process to identify the pruned secondary branch A3 from among multiple branches based on a comparison of the shape information before the pruning operation and the shape information after the pruning operation, among the shape information acquired over time (step S11), and a process to acquire the coordinate information of the pruned secondary branch A3 (step S12). In this case, the coordinate information of the pruned secondary branch A3 can be obtained from among the multiple secondary branches A3 of crop A.

[0066] 〔others〕 It should be noted that the embodiments disclosed herein are illustrative in all respects and not restrictive. In the above embodiment, the case of pruning among the cutting of branches was described, but leaf removal and harvesting can be performed in the same manner as pruning.

[0067] Furthermore, although the above embodiment illustrates a case where the first imaging device 2 has a pair of cameras 14 that constitute a stereo camera, LiDAR (Light Detection And Ranging) may be used instead of the pair of cameras 14. Also, a monocular camera may be used as long as it can be used for VSLAM. Furthermore, although the above embodiment illustrates a case where the second imaging device 4 has an omnidirectional camera, other types of cameras may be used as long as they can broadly capture the hands of the worker M.

[0068] Furthermore, in the above embodiment, an example was given in which the coordinate information of the pruned secondary branch A3 is obtained based on shape information (feature point cloud s10) obtained from map information. However, in addition to the shape information including the feature point cloud s10, the coordinate information of the pruned secondary branch A3 may also be obtained using the feature point cloud a10 of crop A obtained in the motion determination process. The feature point cloud a10 of crop A obtained in the motion determination process includes information indicating the shape of the pruned secondary branch A3 after the pruning operation. Therefore, the feature point cloud a10 of crop A can be used to identify the pruned secondary branch A3 and to understand the shape of the secondary branch A3 after cutting. Therefore, the coordinate information obtained based on the shape information including the feature point cloud s10 can be verified using the information contained in the feature point cloud a10 of crop A, thereby further improving the accuracy of the coordinate information.

[0069] In the above embodiment, an example was given of acquiring and processing coordinate information showing the shape of the branch after cutting, with the root A10 as the origin. However, the coordinate information showing the shape of the branch after cutting can also be configured to be linked with location information such as GPS (Global Positioning System) or GNSS (Global Navigation Satellite System). This makes it possible to distinguish each crop and manage the branch shape after cutting (pruning result) and the yield for each crop in a linked manner. As a result, it is possible to understand the impact that operations such as pruning and leaf removal have on the yield.

[0070] Furthermore, in the above embodiment, the first imaging device 2 and the second imaging device 4 provided the first image data and the second image data to the information processing device 6, and the information processing device 6 performed the motion determination process and the shape acquisition process. However, for example, the motion determination process may be performed by the processing unit 18 of the second imaging device 4, and the other processes may be performed by the information processing device 6. In this case, the processing unit 18 and the information processing device 6 constitute the processing devices in the shape acquisition system 1. Alternatively, the operation determination process may be performed by the processing unit 18 of the second imaging device 4, while other processes may be performed by the processing unit 8 of the first imaging device 2. In this case, the processing unit 8 and the processing unit 18 constitute the processing unit in the shape acquisition system 1. Furthermore, all processing may be performed by either the processing unit 8 of the first imaging device 2 or the processing unit 18 of the second imaging device 4. In this case, the processing unit that performs all processing among processing units 8 and 18 constitutes the processing unit in the shape acquisition system 1.

[0071] The scope of the present invention is indicated by the claims, not in the sense described above, and is intended to include all modifications within the meaning and scope of the claims, equivalent to the claims. [Explanation of Symbols]

[0072] 1. Shape Acquisition System 2. First Imaging Device 2a Device body 2b First attachment device 4. Second Imaging Device 4a Main unit of the device 4b Second fitting device 6. Information Processing Device 7 Network 8 Processing Unit 10 Storage section 12 Communications Department 14 Cameras 15 bus 18 Processing Unit 20 Memory section 22 Communications Department 24 cameras 25 buses 28 Processing Unit 30 Storage section 30a First Learning Model 30b Second Learning Model 30c Work Performance Database 32 Communications Department 35 bus A crop A1 Manager A10 Root A2 main branch A3, A31 secondary branches AS crop area G ground GS ground area M worker T cutting tool a10 Feature point cloud a11 Root point a12 First branching point a13 Second junction a14 tip point a20 Skeletal lines a21 main line a22 main branch line a23, a231 secondary line m10 Feature Point Cloud m11 articular point m20 Skeleton Line s10 Feature point cloud s11 Root point s12 First branching point s13, s131 Second branching point s14, s141, s142 tip s20, s201 Skeleton lines t10 Feature point cloud t11 tip t12 base point t20 Skeletal line

Claims

1. A shape acquisition system for acquiring the branch shape of a crop after cutting, A first imaging device for imaging the crop, The system comprises a processing device that performs processing based on first image data obtained by the first imaging device, The aforementioned processing apparatus is A process to acquire shape information of the crop over time, with the base of the main stem of the crop as the reference point, based on the first image data, Based on the shape information, the following steps are performed: an acquisition process to obtain coordinate information indicating the branch shape after cutting. Shape acquisition system.

2. The system further includes a second imaging device for imaging a worker performing the aforementioned crop cutting, The aforementioned acquisition process is, A first process that determines whether the worker performed the cutting operation on the branch using a cutting tool based on the second image data from the second imaging device, When it is determined that the cutting operation has been performed, the process includes a second step of obtaining the coordinate information based on the shape information after the cutting operation from the shape information acquired over time. The shape acquisition system according to claim 1.

3. The crop includes a plurality of the aforementioned branches, The second process described above is: When it is determined that the cutting operation has been performed, a process is performed to identify the cut branch from among the multiple branches based on a comparison of the shape information acquired over time, specifically the shape information before the cutting operation and the shape information after the cutting operation. The process includes obtaining the coordinate information of the cut branch. The shape acquisition system according to claim 2.

4. The shape information after the cutting operation is the shape information acquired over time, specifically the shape information obtained after a predetermined period has elapsed from the time when it was determined that the cutting operation had been performed. The shape acquisition system according to claim 2.

5. The aforementioned branch is a secondary branch extending from the main branch connected to the main trunk. The coordinate information includes coordinates indicating the branching point where the branch connects to the main branch, and coordinates indicating the position where the branch was cut. A shape acquisition system according to any one of claims 1 to 4.

6. The first imaging device has a first attachment that is worn on the chest of the worker. A shape acquisition system according to any one of claims 2 to 4.

7. The second imaging device has a second attachment that is worn on the head of the worker. A shape acquisition system according to any one of claims 2 to 4.

8. A processing device for obtaining the branch shape of a branch after cutting a branch contained in a crop, A process for acquiring shape information of the crop over time, with the base of the main stem of the crop as the reference point, based on first image data obtained by a first imaging device that images the crop, Based on the shape information, the following steps are performed: an acquisition process to obtain coordinate information indicating the branch shape after cutting. Processing device.

9. A method for obtaining the shape of a branch after cutting a branch contained in a crop, A step of acquiring shape information of the crop over time, with the base of the main stem of the crop as the reference point, based on first image data obtained by a first imaging device that images the crop; The acquisition step includes obtaining coordinate information indicating the branch shape after cutting, based on the shape information. Shape acquisition method.

10. A computer program that causes a computer to perform a shape acquisition process to obtain the shape of branches after they have been cut from a crop, to the computer A step of acquiring shape information of the crop over time, with the base of the main stem of the crop as the reference point, based on first image data obtained by a first imaging device that images the crop; The procedure involves obtaining coordinate information that indicates the shape of the branch after cutting, based on the shape information mentioned above. Computer program.

Images