Information provision methods, server devices, information processing devices, information processing systems, programs, and storage media

Abstract

To provide a technology that allows for easy verification of work results by visualizing and displaying the results at the work site. [Solution] An information provision method for providing information to work-related parties, including workers performing work on a work site, supervisors supervising the workers, managers of the work site, or owners of the work site, includes an acquisition step of acquiring the status of the work site, a generation step of generating display information to be displayed on the display unit of a terminal used by the work-related parties based on the status of the work site, and a display step of displaying the display information on the display unit.

Description

Technical Field

[0001] The present invention relates to an information providing method, a server device, an information processing device, an information processing system, a program, and a storage medium.

Background Art

[0002] Patent Document 1 discloses identifying the content of a defect occurring at a work site based on a photographed image of the work site and controlling an autonomous work machine based on the identified defect content.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technique described in Patent Document 1, it is difficult to visualize the work result, and it is necessary for the workers involved in the work to take images at the site or conduct on-site confirmation and submit a work report after the work.

[0005] The present invention has been made in view of the above problems, and provides a technique that enables easy confirmation of the work result by visually displaying the work result at the work site.

Means for Solving the Problems

[0006] An information providing method according to an aspect of the present invention is an information providing method for providing information to a worker involved in work who performs work on a work site, a supervisor who supervises the worker, an administrator of the work site, or an owner of the work site, the information providing method including: an acquisition step of acquiring the state of the work site; A generation step of generating display information to be displayed on the display unit of a terminal used by the person in charge of the work, based on the state of the work site, The present invention is characterized by including a display step of displaying the aforementioned display information on the display unit. [Effects of the Invention]

[0007] According to the present invention, the visualization and display of work results makes it possible to easily confirm the work results at the work site. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing an example configuration of an information processing system according to one embodiment. [Figure 2] A diagram showing an example of the hardware configuration of a work machine according to one embodiment. [Figure 3] A diagram showing an example of the hardware configuration of a server device according to one embodiment. [Figure 4] A diagram showing an example of the hardware configuration of an information processing device according to one embodiment. [Figure 5] A diagram showing an example of the functional configuration of a server device according to one embodiment. [Figure 6] A diagram illustrating the processing procedure in an information processing system according to one embodiment. [Figure 7] A diagram illustrating the processing procedure in an information processing system according to one embodiment. [Figure 8] A diagram showing an example of a processing sequence in an information processing system according to one embodiment. [Figure 9] A figure showing an example of display information according to one embodiment. [Figure 10] A figure showing an example of display information according to one embodiment. [Figure 11] A figure showing an example of display information according to one embodiment. [Figure 12] A figure showing an example of display information according to one embodiment. [Figure 13] A figure showing an example of display information according to one embodiment. [Figure 14]A figure showing an example of display information according to one embodiment. [Figure 15] A figure showing an example of display information according to one embodiment. [Figure 16] A diagram illustrating the mounting position of a state sensor according to one embodiment. [Modes for carrying out the invention]

[0009] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features from the multiple features described in the embodiments may be combined arbitrarily. Furthermore, identical or similar configurations will be given the same reference numeral, and redundant descriptions will be omitted.

[0010] <System Configuration> FIG. 1 is a diagram showing a configuration example of an information processing system according to the present embodiment. In FIG. 1, reference numeral 10 denotes a server device (information processing device). Reference numeral 20 denotes a work machine, such as a lawn mower or a turf mower, but is not limited thereto. The work machine 20 according to the present embodiment is a ride-on work machine, but may also operate as an autonomous work machine without being operated by an operator. Further, the work machine 20 may be an unmanned autonomous work machine. As the coordinate system of the work machine 20, Fr indicates the front, and Rr indicates the rear. Further, U indicates upward, and D indicates downward. Reference numerals 30a to 30c denote information processing devices, which are terminal devices possessed or owned by a user. Reference numeral 30a denotes a personal computer, reference numeral 30b denotes a smartphone, and reference numeral 30c denotes a tablet terminal. The user can use at least one of these. Any one of the information processing devices 30a to 30c is also referred to as an information processing device 30. The information processing device 30 is an information processing device (information providing device) that provides information to a worker who performs work on a work site, a supervisor who supervises the worker, a manager of the work site, or an owner of the work site. Further, the server device 10 may operate as an information processing device that provides information to a worker. Reference numeral 40 denotes a network. The server device 10, the work machine 20, and the information processing device 30 are communicably connected via the network 40.

[0011] <Hardware Configuration> Figure 2 shows an example of the hardware configuration of the work machine 20 according to this embodiment. The hardware configuration of the work machine 20 is a two-dimensional hardware configuration, and in the coordinate system shown in Figure 2, Fr indicates the front and Rr indicates the rear. Also, Rt indicates the right side and Lt indicates the left side. The work machine 20 is equipped with a right front wheel 201, a left front wheel 202, a right rear wheel 203, and a left rear wheel 204. In this embodiment, the rear wheels are the drive wheels. The work machine 20 further includes a front sensor 211, blade motors 212a to 212c, blades 213a to 213c, an ECU 214, a battery 215, a GNSS sensor 216, a communication device 217, a travel motor 218a to 218b, a state sensor 219, and a rear sensor 220. The blade motors 212a to 212c are collectively referred to as blade motor 212. The blades 213a to 213c are collectively referred to as blade 213. The travel motors 218a to 218b are collectively referred to as the travel motor 218. The work machine 20 is also equipped with a storage device 221 and a display unit 222 (display). The storage device 221 is one or more memories that store various information such as image information. The display unit 222 displays various information to the operator of the work machine 20.

[0012] The forward sensor 211 is, for example, a stereo camera that photographs the area in front of the work machine 20. The stereo camera uses two optical cameras to acquire a visible image and can obtain depth information of the lawn before mowing by utilizing the parallax between the two optical cameras. In the stereo camera, the two cameras are positioned far apart in the width direction (left and right direction) of the work machine 20, and each captures the same scene from a different viewpoint. By analyzing the difference (parallax) between these two images, the distance (depth) of each point is calculated. Here, depth information is information about the distance to an object or surface, and it is possible to understand the position and shape of an object in three-dimensional space. By obtaining depth information of the lawn, it becomes possible to obtain detailed information such as the height and unevenness of the lawn. The stereo camera in the forward sensor 211 uses two optical cameras to acquire depth information of the lawn before mowing, and by using the acquired depth information, a three-dimensional image (three-dimensional image model) of the lawn before mowing can be generated.

[0013] The blade motor 212 drives the blade 213. The ECU 214 is an Electronic Control Unit, which controls various operations of the working machine 20. The battery 215 is a drive source of the working machine 20. In this embodiment, a configuration is provided with three sets of the blade 213 and the blade motor 212 that drives the blade, but the number thereof is not limited.

[0014] The GNSS sensor 216 is a sensor that receives satellite signals of the Global Navigation Satellite System. The position of the working machine 20 can be measured using the received signals of the GNSS sensor 216. The communication device 217 communicates with other devices by wire and / or wirelessly. The traveling motor 218a drives the right rear wheel 203. The traveling motor 218b drives the left rear wheel 204. Each traveling motor can operate separately and can perform a turning operation on the spot.

[0015] The state sensor 219 is a sensor for detecting the state of the working area (for example, the state of the soil). The state sensor 219 may include a soil sensor that monitors the state of the soil. A soil sensor is a sensor that can monitor and visualize the state (EC (electrical conductivity)) of the soil in agriculture and the like. There is a correlation between the EC value and the nutrient concentration in the soil, and the higher the nutrient content in the soil, the higher the EC (electrical conductivity). The nutrient state of the land can be evaluated from the detection value of the soil sensor.

[0016] As shown in FIG. 16, for example, the state sensor 219 is a rod-shaped sensor that extends in the left-right direction so as to point rearward and downward in the traveling direction of the working machine 20 on the lower surface at the rear of the working machine 20, and can detect the state of the position where the working machine 20 has traveled. For example, by extending from the right rear wheel 203 to the left rear wheel 204, it becomes possible to detect the state of a wide range of working areas in one travel. However, the state sensor 219 is not limited to extending in the left-right direction, and may extend in a diagonal direction or a direction parallel to the traveling direction of the working machine 20.

[0017] The rear sensor 220 is composed of, for example, a stereo camera that photographs the rear of the work machine 20, and an NIR (Near Infrared) camera (hereinafter also referred to as the NIR camera or near-infrared camera). The stereo camera of the rear sensor 220 has the same configuration as the stereo camera of the front sensor 211. That is, the stereo camera in the rear sensor 220 can acquire a visible image using two optical cameras, and can acquire depth information of the lawn after mowing by utilizing the parallax between the two optical cameras. By using the acquired depth information, a three-dimensional image (three-dimensional image model) of the lawn after mowing can be generated.

[0018] A near-infrared (NIR) camera is a camera that detects light in the near-infrared (NIR) region, which has a longer wavelength than visible light, and can detect information that is invisible to the naked eye. Plants such as grass have the property of strongly reflecting near-infrared light and absorbing red visible light. NIR cameras are equipped with sensors for detecting near-infrared light, and the sensor of an NIR camera can acquire an image of near-infrared light reflected from the grass (hereinafter also called an NIR image). A filter that removes visible light and allows only near-infrared light to pass through may also be used in a near-infrared (NIR) camera.

[0019] By using the image information (visible image) acquired by the stereo camera of the rear sensor 220 and the image information (near-infrared image) acquired by the NIR camera, an index (NDVI: Normalized Difference Vegetation Index) indicating the quality of vegetation, such as the distribution and growth status of the grass after mowing, can be obtained. For example, the image information (visible image) captured by the stereo camera of the rear sensor 220 includes images showing R (red), G (green), and B (blue) wavelengths. By using the R image showing the R (red) wavelength and the NIR image from the visible image, an index (NDVI) indicating the quality of vegetation, such as the distribution and growth status of the grass after mowing, can be obtained.

[0020] Specifically, the following formula can be used to obtain an index (NDVI) indicating the quality of vegetation. NDVI = (NIR - RED) / (NIR + RED) NIR: Wavelength (reflectance) of near-infrared light in NIR images RED: Red wavelength (reflectance) in visible images By combining NDVI with location information of the work area acquired by the GNSS sensor 216, an image (NDVI image) can be generated that superimposes the distribution of the index (NDVI) in the work area. The NDVI image is an image that visualizes the distribution of the index (NDVI) which indicates the quality of the grass vegetation in the work area. NDVI is represented by a normalized value between -1 and 1, and is characterized by a higher NDVI value when the vegetation is dense (when the grass vegetation is good). By generating an NDVI image of the work area, it is possible to visualize areas with good vegetation and areas with poor vegetation. This NDVI data may also be acquired as information indicating the condition of the work area. Areas with low NDVI values ​​can be determined to be areas with poor grass growth (areas with poor grass vegetation), and areas with high NDVI values ​​can be determined to be areas with good grass growth (areas with good grass vegetation).

[0021] The above description describes a configuration in which the NIR camera is included in the rear sensor 220, but the configuration is not limited to this, and the NIR camera may be included in the state sensor 219. The state sensor 219 may include a sensor (or camera) for detecting at least one of the following: the mowing status of the grass, the growth status of the grass, or the health status of the grass, and the forward sensor 211 and / or the rear sensor 220 may also be referred to as the state sensor 219. In other words, the state sensor 219 may be a sensor for detecting at least one of the following as the state of the work area: the mowing status of the grass, the growth status of the grass, the health status of the grass, or the soil condition.

[0022] Figure 3 shows an example of the hardware configuration of the server device 10 according to this embodiment. The server device 10 includes a CPU 101, a storage device 102, and a communication unit 103. The control operation of the server device 10 is realized by the CPU 101 reading and executing a computer program stored in the storage device 102. The CPU 101 may be one or more CPUs. The storage device 102 is one or more memories that store various kinds of information. For example, it stores information received from other devices and computer programs that are read and executed by the CPU 101. The communication unit 103 has the function of communicating with other devices via wired or wireless connection through the network 40.

[0023] Figure 4 shows an example of the hardware configuration of the information processing device 30 according to this embodiment. The information processing device 30 includes a CPU 301, a storage device 302, a communication unit 303, a display unit 304, and an operation input unit 305. The control operation of the information processing device 30 is realized by the CPU 301 reading and executing a computer program stored in the storage device 302. The CPU 301 may be one or more CPUs. The storage device 302 is one or more memories that store various kinds of information. For example, it stores information received from other devices and computer programs that are read and executed by the CPU 301.

[0024] The communication unit 303 has the function of communicating with other devices via wired or wireless connection through the network 40. The display unit 304 displays various information to the user. The operation input unit 305 receives operation input from the user. The operation input unit 305 can be configured with a keyboard, mouse, touch panel, etc.

[0025] <Functional Configuration> Next, with reference to Figure 5, an example of the functional configuration of the server device 10 according to this embodiment will be described. The server device 10 includes a state acquisition unit 1001, a display information generation unit 1002, a display control unit 1003, an image information acquisition unit 1004, a planning unit 1005, a notification information generation unit 1006, and a notification unit 1007. These functional configurations may be realized by the CPU 101 executing a predetermined program, or they may be configured as integrated circuits or the like, as long as they perform similar functions.

[0026] The status acquisition unit 1001 acquires the status of the work site. The status of the work site may be acquired by a sensor or camera mounted on the work machine 20 performing work at the work site. Alternatively, the status of the work site may be acquired by a sensor or camera fixed to the work site. Furthermore, the status of the work site may be acquired by a sensor or camera mounted on an autonomous aircraft flying over the work site. The status acquisition unit 1001 can acquire the status of the work site by receiving it from the work machine 20, the autonomous aircraft, or a fixedly installed sensor or camera.

[0027] The image information acquisition unit 1004 acquires image information of the work area. For example, it receives and acquires image information (captured images) of the work area taken by the front sensor 211 and / or rear sensor 220 of the work machine 20. The image information of the work area may include, for example, visible images and depth information of the lawn before mowing acquired by the front sensor 211, and visible images and depth information of the lawn after mowing acquired by the rear sensor 220. The image information of the work area may also include near-infrared images (NIR images) of the lawn acquired by an NIR camera.

[0028] The display information generation unit 1002 generates display information for display on the display unit 304 of the information processing device 30 (a terminal device used by workers) based on the image information of the work area acquired by the image information acquisition unit 1004. The display information generation unit 1002 generates a 3D image (3D image model) of the lawn before mowing using the visible image and depth information of the lawn before mowing acquired by the front sensor 211. The display information generation unit 1002 also generates a 3D image (3D image model) of the lawn after mowing using the visible image and depth information of the lawn after mowing acquired by the rear sensor 220. The state of the work area acquired by the state acquisition unit 1001 includes color information of the work area, and the display information generation unit 1002 generates a 3D image model that is colored based on the color information of the work area.

[0029] The display information generation unit 1002 can generate a 3D image (3D image model) for the entire area of ​​the work site or for a partial area of ​​the work site designated by the work personnel. Furthermore, the display information generation unit 1002 can generate a 2D image (2D image model) for a predetermined vertical cross-section (cutting plane: XZ plane in Figure 11) at a position designated by the work personnel within the 3D image (3D image model). Additionally, the display information generation unit 1002 can generate a 2D image (2D image model) for a predetermined horizontal cross-section (cutting plane: XY plane in Figure 11) at a position designated by the work personnel within the 3D image (3D image model). The work site status acquired by the status acquisition unit 1001 includes color information of the work site, and the display information generation unit 1002 generates a 2D image model colored based on the work site color information.

[0030] Furthermore, the display information generation unit 1002 uses image information (visible image) acquired by the stereo camera of the rear sensor 220 and image information (near-infrared image) acquired by the NIR camera to generate an index (NDVI) indicating the quality of vegetation, such as the distribution and growth status of the grass after mowing. The display information generation unit 1002 also combines the 3D positional information of the work area acquired by the GNSS sensor 216 with the index (NDVI) to generate an image (NDVI image) in which the distribution of the index (NDVI) indicating the quality of vegetation, such as the distribution and growth status of the grass after mowing, is superimposed on the work area. The display information generation unit 1002 can also generate an NDVI image for the entire area of ​​the work area or for a partial area of ​​the work area designated by the work personnel.

[0031] The display information generation unit 1002 generates display information for display on the display unit 304 of the information processing device 30 based on the status of the work site acquired by the status acquisition unit 1001, and may also generate display information for display on the display unit 304 of the information processing device 30 in conjunction with the image information of the work site acquired by the image information acquisition unit 1004.

[0032] The display control unit 1003 transmits the display information generated by the display information generation unit 1002 to the information processing device 30, causing it to be displayed on the display unit 304 of the information processing device 30.

[0033] The planning unit 1005 plans the work at the work site based on the image information of the work site acquired by the image information acquisition unit 1004 and the state of the work site acquired by the state acquisition unit 1001. The notification information generation unit 1006 generates notification information to notify the information processing device 30 (a terminal device used by the work personnel) of the work planned by the planning unit 1005. The notification unit 1007 notifies the information processing device 30 (a terminal device used by the work personnel) of the notification information generated by the notification information generation unit 1006. Notification can be made by transmitting the notification information to the information processing device 30.

[0034] <Processing> Figures 6 and 7 show the processing procedures performed by the information processing system according to this embodiment. Figure 8 shows an example of a processing sequence in the information processing system. In S610, the implement 20 travels to the work area and performs lawn mowing work at the work area based on the control of the ECU 214. In S620, the front sensor 211 and rear sensor 220 of the implement 20 acquire image information of the work area (F801 in Figure 8). In this step, the state sensor 219 of the implement 20 may acquire the state of the work area (for example, the state of the soil) along with the image information of the work area. In the following description, the image information of the work area may include the state of the work area.

[0035] In S630, the ECU 214 of the work machine 20 determines whether to upload the image information acquired by the front sensor 211 and the rear sensor 220 to the server device 10. If the image information is not to be uploaded to the server device 10 (S630-NO), the image information is saved in the storage device 221 of the work machine 20 and it waits in a state of waiting for the image information to be uploaded. On the other hand, if the image information is to be uploaded to the server device 10 (S630-YES), the process proceeds to S660. After the image information upload is complete, the image information saved in the storage device 221 of the work machine 20 is deleted (S650). Upon completion of the upload, the image information acquisition unit 1004 of the server device 10 acquires the image information of the work site. The image information acquisition unit 1004 receives and acquires the image information (captured image) of the work site taken by the front sensor 211 and / or rear sensor 220 of the work machine 20 from the work machine 20 (F802 in Figure 8). In this step, upon completion of the upload, the status acquisition unit 1001 of the server device 10 acquires the status of the work site transmitted from the work machine 20 along with the image information.

[0036] In S660, the CPU 101 of the server device 10 notifies the work-related parties, including the workers performing the work at the work site, the supervisors supervising the workers, the site managers, or the site owners, that the lawn mowing work by the work machine 20 has been completed (F803 in Figure 8).

[0037] In S670, the worker decides whether to check the mowing results. If they do not want to check the mowing results (S670-NO), they wait for instructions from the worker. On the other hand, if they want to check the mowing results (S670-YES), the worker inputs a confirmation instruction from the operation input unit 305. The confirmation instruction input from the operation input unit 305 is transmitted to the server device 10 (F804 in Figure 8).

[0038] In S680, the display information generation unit 1002 of the server device 10 generates display information for display on the display unit 304 of the information processing device 30 based on image information of the work area acquired by the image information acquisition unit 1004. The display information for display on the display unit 304 may include visible images captured by the front sensor 211 and the rear sensor 220. The display information for display on the display unit 304 may also include a three-dimensional image of the lawn (three-dimensional image model) and an image (NDVI image) superimposed with the distribution of an index (NDVI) indicating the quality of vegetation, such as the distribution and growth status of the lawn after mowing. The three-dimensional image of the lawn (three-dimensional image model) is a three-dimensional image model virtually modeled using visible images. The display information may also include a two-dimensional image (two-dimensional image model) of the lawn at a predetermined cross-section of the three-dimensional image model.

[0039] Furthermore, the display information includes multiple evaluation values ​​obtained for each of the multiple evaluation items as the status of the work site, and the display information generation unit 1002 generates the multiple evaluation items and multiple evaluation values ​​obtained as the status of the work site as display information. The multiple evaluation values ​​obtained for each of the multiple evaluation items may be quantitative evaluation values ​​obtained by image processing of the visible image. Alternatively, the work personnel may manually input the multiple evaluation values ​​for each of the multiple evaluation items.

[0040] The display information generation unit 1002 generates display information in the form of a shape whose size is proportional to the magnitude of multiple evaluation values ​​for each of the multiple evaluation items. Examples of shapes whose size is proportional to the magnitude of multiple evaluation values ​​for each of the multiple evaluation items may include radar charts, bar graphs, line graphs, etc. Here, the quantitative evaluation values ​​for each of the multiple evaluation items may include, for example, evaluation items such as uniformity of lawn color, lawn color, lawn soil condition, lawn height after mowing, percentage of lawn covering the ground (number of lawn growths in a predetermined area), and regularity of the mowed surface. Furthermore, NDVI acquired using visible images and near-infrared images may also be included as an evaluation item in the display information. Note that the multiple evaluation items are not limited to these and can be changed or added according to the settings of the work personnel. By visualizing the multiple evaluation items in a shape whose size is proportional to the magnitude of multiple evaluation values, it becomes possible to provide multifaceted evaluation results.

[0041] In S690, the display control unit 1003 of the server device 10 transmits the display information generated by the display information generation unit 1002 to the information processing device 30 (F806 in Figure 8), and displays it on the display unit 304 of the information processing device 30.

[0042] Next, the process proceeds to S710 in Figure 7. In S710, the information processing device 30 displays the display information transmitted from the server device 10 on the display unit 304 (F807 in Figure 8).

[0043] In S720, the worker refers to the display information (visible image of the grass, NDVI image, etc.) displayed on the display unit 304 and decides whether to examine the area of ​​interest in detail. If detailed examination of the area of ​​interest is not performed (S720-NO), the system waits for instructions from the worker. On the other hand, if detailed examination of the area of ​​interest is performed (S720-YES), the worker inputs a detailed examination instruction from the operation input unit 305. When a detailed examination instruction is input from the operation input unit 305, the CPU 301 of the information processing device 30 uses the display information transmitted from the server device 10 to display a display corresponding to the detailed examination instruction (for example, a 3D image (3D image model), a 2D image (2D image model), a radar chart, etc.) on the display unit 304. The CPU 301 can combine and display 3D image models and 2D image models, or superimpose an NDVI image on top of a 3D image model, based on instructions from the worker.

[0044] In S740, the worker refers to the displayed image (e.g., a visible image of the grass, a 3D image model, a 2D image model (image of a vertical cross-section), an NDVI image, etc.) displayed on the display unit 304 to determine if there is an area that requires attention in the next operation. If there is no area of ​​attention for the next operation (S740-NO), the process ends. On the other hand, if there is an area of ​​attention for the next operation (S740-YES), in S750, the worker inputs setting information for the area of ​​attention for the next operation from the operation input unit 305. The setting information for the area of ​​attention for the next operation input from the operation input unit 305 is transmitted to the server device 10 (F808 in Figure 8).

[0045] The planning unit 1005 of the server device 10 plans the next work at the work site based on image information of the work site, the status of the work site, and setting information of the area of ​​focus for the next work transmitted from the information processing device 30 (F809 in Figure 8). The notification information generation unit 1006 generates notification information to notify the information processing device 30 (a terminal device used by work personnel) of the work planned by the planning unit 1005 (F810 in Figure 8). Here, the notification information may include a flag that can identify the area to be paid attention to in the next work. The notification information may also include an alert that notifies when the work machine 20 approaches the area to be paid attention to while the work machine 20 is traveling on the work site. The notification unit 1007 notifies the information processing device 30 (a terminal device used by work personnel) of the notification information generated by the notification information generation unit 1006 (S660 in Figure 6, F811 in Figure 8). Furthermore, the notification unit 1007 may also notify the work machine 20 of the notification information generated by the notification information generation unit 1006 (F813 in Figure 8).

[0046] The notification information is displayed on the information processing device 30 before the next work begins (F812 in Figure 8). Workers can check in advance whether there are any areas requiring attention at the work site. After the work is completed, they can refer to the displayed information (visible image, NDVI image, 3D image model, 2D image model, and radar chart, etc.) to determine whether the condition of the grass in the areas requiring attention has improved.

[0047] Furthermore, in the next operation, if the work machine 20 approaches the area of ​​interest 1501, an alert 1502 may be displayed as a pop-up on the display unit 222 of the work machine 20 to notify that the work machine 20 is approaching the area of ​​interest 1501. Although the work machine 20 is an autonomous work machine that performs work autonomously in the work area, if an operator is on board and operating the work machine 20, the pop-up display of the alert 1502 on the display unit 222 allows the operator to confirm in advance that the work machine is approaching the area of ​​interest 1501, making it possible to operate the work machine 20 in a more careful manner during the lawn mowing operation.

[0048] <Example Display> Figures 9 to 15 show examples of display information according to this embodiment. The state acquisition unit 1001 may acquire a first state, which is the state of the work site at a first time, and a second state, which is the state of the work site at a second time after the first time. The display area of ​​the display unit 304 that displays the display information has a first display area that displays information indicating the first state, and a second display area that displays information indicating the second state. The first display area and the second display area can be displayed on the same screen of the display unit 304. The first display area and the second display area may be displayed individually by swiping the screen of the display unit 304. The display information includes geographic information representing the geographic location of the work site, and the display information displayed on the display unit 304 is generated such that information indicating the state of the work site is superimposed on the geographic information. The display information is generated such that information indicating the state of the work site is superimposed on image information.

[0049] As shown in Figure 9, the display area 901 of the display unit 304 of the information processing device 30 (tablet terminal 30c) displays a visible image 902 of the work area captured by the stereo camera of the rear sensor 220 and an NDVI image 903 acquired using the NIR camera of the rear sensor 220. In the visible image 902 of the work area, A to D represent the four work areas of the work area.

[0050] In NDVI image 903, the white areas represent areas with good grass vegetation (NDVI = 1). Areas 911-913 within NDVI image 903 represent areas with an NDVI lower than 1. Area 911 in work area B shows the distribution area with the lowest NDVI among the three areas (911-913), area 912 in work area C shows the distribution area with the second lowest NDVI among the three areas (911-913), and area 913 in work area D shows the distribution area with the third lowest NDVI among the three areas (911-913). In Figure 9, for the sake of simplicity, NDVI is classified into four areas: three areas (911-913) and areas with good vegetation (white areas). However, it is also possible to further subdivide and display NDVI within the range of -1 to 1.

[0051] When a user (e.g., a worker) 904 specifies work area B, which includes the area 911 with the lowest NDVI among the three areas (911-913), in order to check the details, a 3D image (3D image model) 1011 of the specified work area B is displayed in the display area 901 of the display unit 304, as shown in Figure 10. The 3D image model is a virtual model generated based on the visible image, and in the 3D image (3D image model) 1011, "GD" indicates the topography (unevenness) of the ground of the work site (grassland). The topography GD can be obtained by combining the position information of the work site acquired by the GNSS sensor 216 when the work machine 20 is working with the visible image, and the topography GD is realistically modeled in the generation of the 3D image model. The 3D image (3D image model) 1011 can be rotated and displayed at any angle around each of the three-dimensional axes (XYZ axes) by the user 904. The 3D image model of the grass can be generated in a modifiable manner based on the specified viewpoint direction. Furthermore, the work area is not limited to a single area; multiple work areas may be specified, or the entire work area, including work areas A through D, can be specified.

[0052] When a user (e.g., a worker) 904 specifies an arbitrary position in the 3D image (3D image model) 1011 to check details, the 2D image (2D image model) 1012 of the specified position (AA) is displayed in the display area 901 of the display unit 304, as shown in Figure 11. The 2D image (2D image model) 1012 is a model of the cross-section of the specified position (AA). In the 2D image (2D image model) 1012, "GD" indicates the undulations (unevenness) of the grassy ground as viewed from the direction of the AA cross-section (perpendicular to the paper plane). The 2D image (2D image model) 1012 may also display the average height L1 of the grass after mowing. In addition to the average height L1 of the grass, quantitative evaluation values ​​such as the uniformity of the grass color and the regularity of the grass color and mowed surface at the specified AA cross-section may also be displayed in combination with the 2D image model.

[0053] Figure 11 shows an example where a 3D image model 1011 and a 2D image model 1012 are displayed in a single display area 901, but the example is not limited to this. For example, the user 904 may specify any multiple positions in the 3D image (3D image model) 1001, and the 2D image models at the specified multiple positions (cross-sections) may be displayed together with the 3D image model 1011 in a single display area 901.

[0054] To confirm the details, for example, as shown in Figure 12, an NDVI image may be superimposed on the 3D image model 100. In the display example shown in Figure 12, the NDVI image 903 is displayed below the display area 901, and the 3D image model 1011 of work area B specified by the user (worker) is displayed above the display area 901. On the 3D image model 1011, areas 911 in work area B with low NDVI are superimposed. By superimposing the 3D image model and the NDVI image (or NDVI distribution area) in this way, workers can easily visually identify areas with poor grass vegetation within the work area.

[0055] To confirm the details, for example, as shown in Figure 13, an image (e.g., a radar chart) visualizing the quantitative evaluation values ​​obtained by image processing of the visible image 902 may be displayed. In the display example shown in Figure 13, the visible image 902 is displayed above the display area 901, and the radar chart 1301 is displayed below the display area 901. In the radar chart 1301 shown in Figure 13, examples of evaluation items include the uniformity of the grass color, the color of the grass, the soil condition of the grass, the height of the grass after mowing, the percentage of grass covering the ground (the number of grasses growing in a given area), and the regularity of the mowed surface. However, the evaluation items of the radar chart are not limited to these and can be changed or added according to the settings of the work personnel. In the radar chart 1301, the evaluation values ​​of the evaluation items are evaluated using four grades: Excellent, Good, Average, and Poor. However, the classification of evaluation values ​​is not limited to this example and can be added or changed. Furthermore, while the radar chart 1301 shown in Figure 13 visualizes the overall evaluation value of the work area, the display is not limited to the example in Figure 13. For example, radar charts may be displayed for each partial work area A to D within the work area, as shown in Figure 14. In Figure 14, 1401 is an example of the radar chart for work area A, and 1402 is an example of the radar chart for work area B. By displaying radar charts for each work area in a single display area 901, the evaluation values ​​of the evaluation items for each work area can be compared relatively.

[0056] Figure 15 shows an example of the display of notification information shown on the information processing device 30 before the start of the next work. 1501 indicates the area that the work personnel have set as the area of ​​focus for the next work based on the results of the previous work that has been completed. For example, if the work personnel have set area 911 in work area B as the area of ​​focus for the next work in the NDVI image 903, the area of ​​focus 1501 will be displayed at the position corresponding to area 911 in work area B. A flag (for example, a "!" display) that can identify the area of ​​focus 1501 that requires attention in the next work may be displayed within the area of ​​focus 1501. In addition, when the work machine 20 approaches the area of ​​focus 1501, an alert 1502 that notifies that the work machine 20 has approached the area of ​​focus 1501 may be displayed as a pop-up on the display area 901 of the information processing device 30. Alternatively, the alert 1502 that notifies that the work machine 20 has approached the area of ​​focus may be displayed as a pop-up on the display unit 222 of the work machine 20. For example, when an operator is riding in the work machine 20 and performing the operation, an alert 1502 will pop up on the display unit 222 of the work machine 20, allowing the operator to confirm in advance that they are approaching the area of ​​interest 1501, and enabling them to operate the work machine 20 in a more careful manner during the lawn mowing operation.

[0057] [Differentiation] In the above embodiment, an example was described in which the server device 10 performs the main processing, but the system is not limited to this example. The work machine 20 may also perform the processing of generating display information. In that case, the display information generated by the work machine 20 may be transmitted to the information processing device 30 via the server device 10 or directly without going through the server device 10. Similarly, the work machine 20 may plan work, generate notification information, or acquire image information.

[0058] In that case, the work content planned by the work machine 20, the notification information generated by the work machine 20, or the image information acquired by the work machine may be transmitted to the information processing device 30 via the server device 10 or directly without going through the server device 10. Furthermore, the information processing device 30 may perform processes such as generating display information, planning work, generating notification information, and acquiring image information. Each of the processes in the above-described embodiment can be performed by the work machine 20, the server device 10, or the information processing device 30, and no single device is required to perform all of the processes.

[0059] Furthermore, when the planning unit 1005 plans work at a work site based on image information of the work site acquired by the image information acquisition unit 1004 and the state of the work site acquired by the state acquisition unit 1001, it may also take weather forecast information into consideration. Based on the current state of grass growth at the work site and weather forecast information, the planning unit 1005 may predict the state of grass growth and, for example, plan to carry out the work in 10 days or half a month and notify (suggest) the information processing device 30.

[0060] Furthermore, although the above-described embodiment describes an example in which a server device 10 connected to the work machine 20 via a network 40 mainly performs processing, the embodiment is not limited to this example. For example, the above embodiment may be applied to an example in which multiple work machines 20 are connected to each other in a communicative manner, with one work machine 20 operating as a master work machine and the other work machines 20 operating as slave work machines. In other words, the master work machine may perform the processing of the above embodiment (the processing mainly performed by the server device 10).

[0061] Furthermore, although the above-described embodiment shows an example in which the server device 10, the work machine 20, and the information processing device 30 are connected to each other via a network 40, the embodiment is not limited to this example. For example, an edge server may be provided instead of the server device 10, and configured to collect only the data necessary for the cloud. This can prevent all data from being concentrated in the cloud. Specifically, the edge server may acquire data from the work machine 20, execute the processing of the above-described embodiment, and save the acquired data and the processing results on the edge server to the cloud.

[0062] For example, the edge server may be configured to generate display information based on the status of the work site and save the generated display information to the cloud. This allows the information processing device 30 to retrieve the necessary data from the cloud. In this way, by utilizing edge computing, data processing can be performed quickly and securely on the edge server side, while only the necessary data is sent to the cloud, thereby reducing the amount of communication and alleviating the network load.

[0063] <Summary of Embodiments> 1. The information provision method according to the above embodiment is: A method for providing information to persons involved in work, including workers performing work at a work site, supervisors supervising said workers, managers of said work sites, or owners of said work sites, The acquisition step involves acquiring the status of the work site, A generation step of generating display information to be displayed on the display unit of a terminal used by the person in charge of the work, based on the state of the work site, The process includes a display step of displaying the aforementioned display information on the display unit.

[0064] 2. In the information provision method according to the above embodiment, The state of the work site acquired in the acquisition step includes a visible image of the grass of the work site taken by a stereo camera installed on a work machine performing work at the work site, and a near-infrared image taken by a near-infrared camera.

[0065] 3. In the information provision method according to the above embodiment, The generation step involves obtaining an NDVI index indicating the quality of the vegetation on the lawn of the work site using the red wavelength in the visible image and the near-infrared wavelength in the near-infrared image. Using the location information of the work site measured by a sensor installed on the work machine and the NDVI index, an NDVI image is generated by superimposing the distribution of the NDVI index in the work site.

[0066] 4. In the information provision method according to the above embodiment, The generation step generates a three-dimensional image model of the grass at the work site using depth information of the grass at the work site obtained using the parallax of the stereo camera.

[0067] 5. In the information provision method according to the above embodiment, The generation step generates the three-dimensional image model of the grass in a modifiable manner based on the direction of a specified viewpoint.

[0068] 6. In the information provision method according to the above embodiment, The generation step generates a two-dimensional image model showing a cross-section of the three-dimensional image model at a specified position in the three-dimensional image model.

[0069] 7. In the information provision method according to the above embodiment, The two-dimensional image model includes a two-dimensional image model showing a vertical cross-section of the three-dimensional image model, and a two-dimensional image model showing a horizontal cross-section of the three-dimensional image model.

[0070] 8. In the information provision method according to the above embodiment, The generation step generates an image by superimposing the distribution of the NDVI index onto the three-dimensional image model.

[0071] 9. In the information provision method according to the above embodiment, The state of the work site acquired in the acquisition step includes color information of the work site. The generation step generates the three-dimensional image model, which is colored based on the color information of the work site.

[0072] 10. In the information provision method according to the above embodiment, The generation step involves obtaining multiple evaluation values, each evaluated for a set of multiple evaluation items, as the state of the work site. Multiple evaluation items and multiple evaluation values ​​are generated as the display information.

[0073] 11. In the information provision method according to the above embodiment, The generation step generates the display information in which each of the multiple evaluation items is represented by a shape whose size is proportional to the magnitude of the multiple evaluation values.

[0074] 12. In the information provision method according to the above embodiment, The acquisition step described above is: A step of obtaining a first state which is the state of the work site at a first time, The process includes the step of obtaining a second state, which is the state of the work site at a second time after the first time.

[0075] This makes it easy to recognize changes in the condition of the work site at different times.

[0076] 13. In the information provision method according to the above embodiment, The display area for displaying the aforementioned display information is: A first display area for displaying information indicating the first state, It has a second display area for displaying information indicating the second state.

[0077] This makes it possible to display the status of the work site at different times.

[0078] 14. In the information provision method according to the above embodiment, The first display area and the second display area are displayed within the same screen of the display unit.

[0079] This makes it easy to compare changes in the work site conditions at different times.

[0080] 15. In the information provision method according to the above embodiment, The first display area and the second display area are displayed individually by swiping the screen of the display unit.

[0081] 16. In the information provision method according to the above embodiment, The aforementioned display information includes geographic information representing the geographic location of the work site, The display information is generated such that information indicating the state of the work site is superimposed on the geographic information.

[0082] 17. In the information provision method according to the above embodiment, The process further includes a second acquisition step of acquiring image information of the work site, The generation step is, The display information is generated so that information indicating the state of the work site is superimposed on the image information.

[0083] 18. In the information provision method according to the above embodiment, A planning step of planning work at the work site based on the acquired state of the work site, A second generation step of generating notification information for notifying the terminal of the work at the work site planned in the planning step, The aforementioned notification information includes information indicating areas that require attention during work at the work site.

[0084] 19. In the information provision method according to the above embodiment, The method further includes a notification step of notifying the terminal of the notification information generated in the second generation step.

[0085] According to the information provision method of the above embodiment, it becomes possible to provide a technology that makes it possible to easily confirm work results by visually displaying the work results at the work site.

[0086] 20. In the server device according to the above embodiment, A server device (10) that provides information to work-related parties, including workers performing work at a work site, supervisors supervising said workers, managers of said work sites, or owners of said work sites, An acquisition means (1001) for acquiring the state of the work site, Based on the aforementioned conditions of the work site, a generation means (1002) generates display information to be displayed on the display unit (304) of a terminal (30) used by the person involved in the work, The system includes a display control means (1003) for causing the display information to be displayed on the display unit.

[0087] According to the server device of the above embodiment, it is possible to provide a technology that makes it possible to easily check work results by visually displaying the work results at the work site.

[0088] 21. In the information processing apparatus according to the above embodiment, An information processing device (30) that provides information to persons involved in work, including workers performing work at a work site, supervisors supervising said workers, managers of said work sites, or owners of said work sites, An acquisition means for acquiring the state of the work site, A generation means for generating display information to be displayed on the display unit of the information processing device based on the state of the work site, The system includes a display control means for causing the display information to be displayed on the display unit.

[0089] According to the information processing device of the above embodiment, it is possible to provide a technology that makes it possible to easily confirm work results by visualizing and displaying the work results at the work site.

[0090] 22. In the information processing system according to the above embodiment, An information processing system that provides information to work-related parties, including workers performing work at a work site, supervisors supervising said workers, managers of said work sites, or owners of said work sites, An acquisition means for acquiring the state of the work site, A generation means for generating display information to be displayed on the display unit of a terminal used by the person in charge of the work, based on the state of the work site, The system includes a display control means for causing the display information to be displayed on the display unit.

[0091] According to the information processing system of the above embodiment, it is possible to provide a technology that makes it possible to easily confirm work results by visualizing and displaying the work results at the work site.

[0092] 23. The program according to the above embodiment is: This is a program for causing a computer to execute the information provision method of the above embodiment.

[0093] This makes it possible to implement information delivery methods as a program.

[0094] 24. The storage medium according to the above embodiment is This is a storage medium containing a program that causes a computer to execute the information provision method of the above embodiment.

[0095] This makes it possible to implement information delivery methods using storage media.

[0096] <Other Embodiments> Furthermore, a program that implements one or more functions described in each embodiment is supplied to a system or device via a network or storage medium, and one or more processors in the computer of the system or device can read and execute this program. The present invention can also be realized in this manner.

[0097] The invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of the gist of the invention. [Explanation of Symbols]

[0098] 10: Server device, 20: Work machine, 30: Information processing device, 1001: Status acquisition unit, 1002: Display information generation unit, 1003: Display control unit, Image information acquisition unit 1004, 1005: Planning unit, 1006: Notification information generation unit, 1007: Notification unit

Claims

1. A method for providing information to persons involved in work, including workers performing work at a work site, supervisors supervising said workers, managers of said work sites, or owners of said work sites, The acquisition step involves acquiring the status of the work site, A generation step of generating display information to be displayed on the display unit of a terminal used by the person in charge of the work, based on the state of the work site, A display step in which the display information is displayed on the display unit, A method of providing information characterized by including the following.

2. The information provision method according to claim 1, characterized in that the state of the work site acquired in the acquisition step includes a visible image of the lawn of the work site taken by a stereo camera and a near-infrared image taken by a near-infrared camera, which are installed on a work machine performing work at the work site.

3. The generation step involves obtaining an NDVI index indicating the quality of the vegetation on the lawn of the work area using the red wavelength in the visible image and the near-infrared wavelength in the near-infrared image. The information provision method according to claim 2, characterized in that an NDVI image is generated by superimposing the distribution of the NDVI index in the work area using the location information of the work area measured by a sensor provided on the work machine and the NDVI index.

4. The information provision method according to claim 3, characterized in that the generation step generates a three-dimensional image model of the grass at the work site using depth information of the grass at the work site obtained using the parallax of the stereo camera.

5. The information provision method according to claim 4, characterized in that the generation step generates the three-dimensional image model of the lawn in a modifiable manner based on the direction of a specified viewpoint.

6. The information provision method according to claim 4 or 5, characterized in that the generation step generates a two-dimensional image model showing a cross-section of the three-dimensional image model at a specified position in the three-dimensional image model.

7. The information provision method according to claim 6, characterized in that the two-dimensional image model includes a two-dimensional image model showing a vertical cross-section of the three-dimensional image model and a two-dimensional image model showing a horizontal cross-section of the three-dimensional image model.

8. The information provision method according to any one of claims 4 to 7, characterized in that the generation step generates an image in which the distribution of the NDVI index is superimposed on the three-dimensional image model.

9. The state of the work site acquired in the acquisition step includes color information of the work site. The information provision method according to any one of claims 4 to 8, characterized in that the generation step generates the three-dimensional image model which is colored based on the color information of the work site.

10. The generation step involves obtaining multiple evaluation values, each evaluated for a set of multiple evaluation items, as the state of the work site. The information provision method according to any one of claims 1 to 9, characterized in that it generates a plurality of evaluation items and a plurality of evaluation values ​​as the display information.

11. The information provision method according to claim 10, characterized in that the generation step generates a shape as display information that is proportional in size to the magnitude of the multiple evaluation values ​​for each of the multiple evaluation items.

12. The acquisition step described above is: A step of obtaining a first state which is the state of the work site at the first time, A step of obtaining a second state which is the state of the work site at a second time after the first time, The information provision method according to any one of items 1 to 11, characterized by including the following:

13. The display area for displaying the aforementioned display information is: A first display area for displaying information indicating the first state, A second display area for displaying information indicating the second state, The information provision method according to claim 12, characterized by having the following features.

14. The information provision method according to claim 13, characterized in that the first display area and the second display area are displayed within the same screen of the display unit.

15. The information provision method according to claim 14, characterized in that the first display area and the second display area are displayed individually by swiping the screen of the display unit.

16. The aforementioned display information includes geographic information representing the geographic location of the work site, The information provision method according to any one of claims 1 to 15, characterized in that the display information is generated such that information indicating the state of the work site is superimposed on the geographic information.

17. The acquisition step includes a second acquisition step of acquiring image information of the work site, The generation step is, The information provision method according to any one of claims 1 to 16, characterized in that the display information is generated such that information indicating the state of the work site is superimposed on the image information.

18. A planning step of planning work at the work site based on the acquired state of the work site, A second generation step of generating notification information for notifying the terminal of the work at the work site planned in the planning step, The information provision method according to any one of claims 1 to 17, characterized in that the notification information includes information indicating areas that require attention during work at the work site.

19. The information provision method according to claim 18, further comprising a notification step of notifying the terminal of the notification information generated in the second generation step.

20. A server device that provides information to work-related parties, including workers performing work at a work site, supervisors supervising said workers, managers of said work sites, or owners of said work sites, An acquisition means for acquiring the state of the work site, A generation means for generating display information to be displayed on the display unit of a terminal used by the person in charge of the work, based on the state of the work site, A display control means for displaying the aforementioned display information on the display unit, A server device characterized by being equipped with the following features.

21. An information processing device that provides information to persons involved in work, including workers performing work at a work site, supervisors supervising said workers, managers of said work sites, or owners of said work sites, An acquisition means for acquiring the state of the work site, A generation means for generating display information to be displayed on the display unit of the information processing device based on the state of the work site, A display control means for displaying the aforementioned display information on the display unit, An information processing device characterized by comprising:

22. An information processing system that provides information to work-related parties, including workers performing work at a work site, supervisors supervising said workers, managers of said work sites, or owners of said work sites, An acquisition means for acquiring the state of the work site, A generation means for generating display information to be displayed on the display unit of a terminal used by the person in charge of the work, based on the state of the work site, A display control means for displaying the aforementioned display information on the display unit, An information processing system characterized by comprising the following features.

23. A program for causing a computer to execute the information provision method described in any one of claims 1 to 19.

24. A storage medium storing a program for causing a computer to execute the information provision method described in any one of claims 1 to 19.

Images