Information presentation device

Abstract

An operator of a work machine is prompted with an overview image of a work site even when the number of imaging devices at the work site is insufficient. The present disclosure provides an information prompting device that generates and outputs a terrain image related to a work site, the information prompting device including: a storage device that stores a program for performing predetermined image processing on a terrain image; a processor that reads the program from the storage device, generates a terrain image, the processor performing the following processing: processing that uses an image of the work site acquired by at least one imaging device disposed at the work site to acquire a first overview image that represents a current terrain of the work site; processing that acquires a second overview image that represents an original terrain of the work site acquired by a method other than imaging by an imaging device disposed at the work site; and image synthesis processing that generates a first composite overview image by superimposing the first overview image and the second overview image (see FIG. 6).

Description

Technical Field

[0001] This invention relates to an information prompting device. Background Technology

[0002] In the past, at construction sites where multiple machines such as work machinery and transport vehicles exist, information prompting devices have been known to enable operators of work machinery to identify the locations of other work machinery and transport vehicles present in the vicinity. For example, the information prompting device described in Patent Document 1 determines the positions of multiple vehicles based on images captured by multiple cameras installed at the construction site, and outputs a signal that overlays the positions of other vehicles among the determined vehicles that are less than a threshold distance from any specified vehicle onto an overhead image of the construction site and displays it.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-151968 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] However, in the structure described in Patent Document 1, for example, from the viewpoint of setup cost, it is sometimes difficult to synthesize an overhead image when the number of imaging devices that can be used at the construction site is limited. This is because it is impossible to synthesize an image that covers the entire area of ​​the construction site. Consequently, it is impossible to efficiently manage multiple vehicles at the construction site.

[0008] This disclosure was made in view of the following situation, and proposes a technique to provide operators of construction machinery with an overhead view of the work site even when the number of cameras at the construction site is insufficient.

[0009] Methods for solving problems

[0010] To address the aforementioned issues, the present invention provides an information prompting device that generates and outputs a terrain image related to a construction site. The information prompting device comprises: a storage device storing a program for performing predetermined image processing on the terrain image; and a processor that reads the program from the storage device, generates the terrain image, and performs the following processing: processing to obtain a first overhead image representing the current terrain of the construction site from images of the construction site acquired by at least one camera installed at the construction site; processing to obtain a second overhead image representing the original terrain of the construction site acquired by a method other than photography by the camera installed at the construction site; and image compositing processing to overlay the first overhead image and the second overhead image to generate a first composite overhead image.

[0011] Further features of this disclosure will become clear from the description and accompanying drawings. Furthermore, this disclosure is accomplished and implemented through elements and combinations of various elements, as well as through detailed descriptions and appended claims. The description in this specification is merely exemplary and does not limit the claims or application of this disclosure in any way.

[0012] Invention Effects

[0013] According to this disclosure, even when the number of cameras at the construction site is insufficient, it is possible to generate and provide overhead images of the work site for the operators of the machinery. Attached Figure Description

[0014] Figure 1 This is a diagram showing a general structural example of a machine tool.

[0015] Figure 2A This is a diagram showing the movable range 24 of the hydraulic excavator 1.

[0016] Figure 2B This is a diagram showing the movable range 19 of the wheel loader 36.

[0017] Figure 3 This is a block diagram illustrating an example of the functional structure of the information display device in this embodiment.

[0018] Figure 4 This is a diagram showing an example of an overhead image 2 obtained by the original terrain information acquisition unit 26.

[0019] Figure 5 This is an example of an overhead image 1 obtained by the current terrain information acquisition unit 25.

[0020] Figure 6 This is a diagram illustrating an example of an overhead image 3 generated by the overhead image synthesis unit 28 by overlaying the current terrain information (overhead image 1) with the original terrain information (overhead image 2).

[0021] Figure 7 This is a flowchart illustrating the detailed process of generating the overhead image 3 in this embodiment.

[0022] Figure 8 This is a block diagram illustrating a functional structure example of the information display device in the second embodiment.

[0023] Figure 9A This is a diagram (1) used to illustrate the calculation method of blind spot 40 (the presence or absence of blind spot, the location of blind spot, and the extent (size) of blind spot).

[0024] Figure 9BThis is a diagram (2) used to illustrate the calculation method of blind spot 40 (the presence or absence of blind spot, the location of blind spot, and the extent (size) of blind spot).

[0025] Figure 9C This is a diagram (3) used to illustrate the calculation method of blind spot 40 (the presence or absence of blind spot, the location of blind spot, and the extent (size) of blind spot).

[0026] Figure 10 This is a diagram illustrating an example of an overhead view 4 (an overhead view generated by adding blind spots to an overhead view 3) according to the second embodiment.

[0027] Figure 11 This is a flowchart illustrating the detailed process of generating the overhead image 4 in the second embodiment.

[0028] Figure 12 This is a block diagram illustrating a functional structure example of the information prompting device according to the third embodiment.

[0029] Figure 13 This is an example of an aerial view 48 from a vehicle.

[0030] Figure 14 This is an example of an overhead view of the synthesis of the third embodiment.

[0031] Figure 15 This is a flowchart illustrating the detailed process of generating the overhead image 5 in the third embodiment.

[0032] Figure 16 This is a diagram illustrating a structural example of the overhead view providing system of this embodiment (first to third embodiments). Detailed Implementation

[0033] The first to third embodiments of this disclosure describe information prompting techniques for generating and outputting overhead topographic images of construction sites. Hereinafter, embodiments of this disclosure will be described with reference to the accompanying drawings. In the drawings, functionally identical elements are sometimes shown using the same numbering system. Furthermore, the drawings illustrate specific embodiments and installation examples following the principles of this disclosure, but they are intended for understanding this disclosure and are not intended to limit its interpretation.

[0034] In this embodiment, a thorough and detailed description has been provided for those skilled in the art to implement this disclosure. However, other installations and methods are also possible, and it should be understood that structural changes and substitutions of various elements can be made without departing from the scope and spirit of the technical concept of this disclosure. Therefore, the following description is not intended to be limited to this interpretation.

[0035] Furthermore, as described below, embodiments of this disclosure can be implemented by software running on a general-purpose computer, or by dedicated hardware or a combination of software and hardware.

[0036] (1) First implementation method

[0037] <Structure example of an overhead image provisioning system>

[0038] Figure 16 This is a diagram illustrating a structural example of an overhead image providing system (also called an information providing system) according to this embodiment (including the second and third embodiments described later).

[0039] The overhead image providing system includes: a work machine 1 with a communication device; at least one shooting device (camera) 33A to 33B set at the construction site 31; a computer (information prompting device) that generates and outputs various overhead images of the construction site 31; a drone (equivalent to an original terrain information acquisition unit 26) for taking overhead images of the original terrain of the construction site 31 (the terrain before the placement of building materials and work machine 1); a display unit (display device) 29 for displaying the generated overhead images; and a communication device 49.

[0040] In this system, an aerial view (hereinafter referred to as aerial view 2) taken by a drone (equivalent to the original terrain information acquisition unit 26) is used, but satellite photos obtained by satellites or the like can also be used as aerial view 2.

[0041] In addition, Figure 16 In this configuration, the display unit (display device) 29 is remotely configured with the computer (information display device), but it can also be part of the computer (information display device). This computer (information display device) can be located outside the construction site 31 (including a remote location) so that a manager located far from the construction site 31 can view the overhead view, or it can be configured in the driver's seat of the work machinery 1 so that the operator can view the overhead view (or both). Alternatively, the information display device can be configured outside the construction site 31, with only the display unit 29 installed on the work machinery 1. In the case where the information display device and the display unit 29 are configured separately, such as... Figure 16 As shown, they can also be configured to communicate with each other via a network.

[0042] <Example of the structure of operating machinery>

[0043] Figure 1 This is a diagram illustrating a general structural example of a working machine. In this embodiment, a hydraulic excavator 1 will be used as an example of a working machine for explanation.

[0044] The hydraulic excavator 1 is configured such that an upper traveling body 15 is mounted on a lower traveling body 14 via a rotating mechanism 16. The rotating mechanism 16 includes a hydraulic motor that causes the upper traveling body 15 to rotate clockwise or counterclockwise. A boom 8 is mounted on the upper traveling body 15. The boom 8 swings vertically relative to the upper traveling body 15 via a hydraulically driven boom cylinder 5. A stick 9 is mounted at the front end of the boom 8. The stick 9 swings back and forth relative to the boom 8 via a hydraulically driven stick cylinder 6. A bucket 10 is mounted at the front end of the stick 9. The bucket 10 swings relative to the stick 9 via a hydraulically driven bucket cylinder 7. A cab 2 for accommodating the operator is also mounted on the upper traveling body 15. Furthermore, in this specification, the boom 8, stick 9, and bucket 10 are sometimes collectively referred to as the front working section 23. The front working section 23 rotates together with the upper traveling body 15 around a rotation center 17. Furthermore, the distance from the rotation center 17 to the front end of the front working section 23 is varied by swinging the boom 8, stick 9, and bucket 10. Here, the upper traveling body 15 rotates around the rotation center 17, and the accessory 18 is mounted on the upper traveling body 15.

[0045] The boom 8 swings up and down around a swing center parallel to the y-axis. A stick 9 is installed at the front end of the boom 8, and a bucket 10 is installed at the front end of the stick 9.

[0046] The vehicle position detection unit 44 uses a device such as GNSS that can obtain its own position, and the vehicle azimuth angle calculation unit 43 calculates the azimuth angle of the vehicle.

[0047] <Moveable range of the working machinery>

[0048] Figure 2A and Figure 2B This is a diagram showing an example of the movable range of a working machine. Figure 2A This indicates the movable range of the hydraulic excavator 1, 24. Figure 2B This indicates the movable range of the wheel loader 36 as 19.

[0049] exist Figure 2A In the case of the longest possible front working section, the movable range 24A of the front working section 23 of the hydraulic excavator 1 becomes the range that the front working section 23 can reach when the upper rotating body rotates 360 degrees. Furthermore, the movable range 24B of the lower traveling body 14 becomes the range that the excavator body can reach, for example, within T seconds when the hydraulic excavator 1 is traveling. The movable range combining the movable ranges 24A and 24B is defined as the movable range 24 of the hydraulic excavator 1. Additionally, in... Figure 2B In this context, the movable range 19 of the wheel loader 36 is defined as the range within T seconds that it can move in the forward and backward direction when the steering wheel is turned to its left and right limits.

[0050] <Example of the functional structure of an information prompting device>

[0051] Figure 3 This is a block diagram illustrating an example of the functional structure of the information display device in this embodiment.

[0052] The information prompting device includes a current terrain information acquisition unit 25, a raw terrain information acquisition unit 26, an overhead image processing unit 27, and a display unit (display device) 29. The overhead image processing unit 27 includes an overhead image synthesis unit 28. This information prompting device is, for example, composed of a computer, in which a processor reads programs for implementing each processing unit (current terrain information acquisition unit, raw terrain information acquisition unit, overhead image processing unit, etc.) from a memory (storage device; hereinafter the same), and executes them in the internal memory, thereby implementing each processing unit. In addition, the information prompting device can be configured in the driver's seat of the operating machine 1, or it can be configured in a remote location (e.g., a management office) separate from the operating machine 1 and the shooting device 33.

[0053] The existing terrain is captured by a camera or other shooting device 33 installed at the construction site 31. When the existing terrain image is received, the existing terrain information acquisition unit 25 merges them into an overhead image (overhead image 1) and sends it to the overhead image synthesis unit 28.

[0054] The original terrain information acquisition unit 26 acquires aerial images (aerial images of the original terrain of the construction site 31) taken by drones or satellites, or images drawn based on design drawings (aerial images of the original terrain: aerial image 2).

[0055] The overhead image compositing unit 28 composites an image by overlaying an overhead image 2 of the construction site 31, obtained by the original terrain information acquisition unit 26, with an overhead image 1 generated based on the current terrain information. The overhead image compositing unit 28 then sends the composited overhead image (overhead image 3) to the display unit 29.

[0056] Display unit 29 is a display device such as a monitor held by the operator of the hydraulic excavator (operating machinery) 1, or on-site workers, construction managers, and on-site supervisors, and / or a display terminal such as a smartphone, that displays the overhead image 3 synthesized by overhead image synthesis unit 28.

[0057] <Example of aerial image 2>

[0058] Figure 4This diagram illustrates an example of an overhead image 2 acquired by the original terrain information acquisition unit 26. Site personnel use equipment such as drones capable of capturing overhead images to take images before construction. Furthermore, the original terrain image can also be used to depict the site based on design drawings. In the original terrain image, the construction site 31 is divided into the area inside the construction site 31, the area outside the construction site 32, and fixed obstacles 30. This division is performed before construction begins and does not necessarily require a high update frequency. The division method needs to be set by site personnel or construction managers, but learning functions such as deep learning can also be used, employing algorithms that automatically identify the boundaries between the fixed obstacles 30 and the construction site 31. Additionally, the coordinates of the reference points for the construction site 31 are determined during the acquisition / description of the overhead image.

[0059] <Example of aerial image 1>

[0060] Figure 5 This diagram shows an example of an overhead image 1 obtained by the current terrain information acquisition unit 25. At least one camera or other imaging device 33 is positioned at a location and height within or near the construction site 31, capable of capturing images of the construction site 31. The construction site 31 is then photographed using the imaging device 33. The operator pre-registers the position (coordinates), height, field of view, and line of sight of the imaging device 33, or automatically registers data from a device capable of self-positioning, such as GNSS, with the imaging device 33 to the current terrain information acquisition unit 25, either manually or automatically.

[0061] Furthermore, the current terrain information acquisition unit 25 (the aforementioned computer) pre-stores (in the memory) the deformation parameters of the images captured by each imaging device 33. The image deformation parameters are data that can be determined based on the installation position of the imaging device 33 and the viewing direction. However, the method for generating the overhead view 1 is not limited to this. For example, a three-dimensional image of the construction site 31 can be generated by performing triangulation on multiple images acquired by the current terrain information acquisition unit 25, and an overall overhead view (overhead view 1) can be generated based on the obtained three-dimensional image.

[0062] <Example of aerial image 3>

[0063] Figure 6 This is a diagram illustrating an example of an overhead image 3 generated by the overhead image synthesis unit 28 by overlaying the current terrain information (overhead image 1) with the original terrain information (overhead image 2).

[0064] Based on the position information of each shooting device 33, the overhead image 1 of the current terrain generated by the current terrain information acquisition unit 25 is compared with the reference point of the original terrain information (overhead image 2), and the position of the composite image is adjusted to generate an overhead image 3.

[0065] <Detailed information on image generation and processing from an aerial perspective>

[0066] Figure 7 This is a flowchart illustrating the detailed process of generating the overhead image 3 in this embodiment. In each step described below, each processing unit (current terrain information acquisition unit 25, original terrain information acquisition unit 26, and overhead image synthesis unit 28) is referred to as the action subject. However, as mentioned above, their programs are read from memory and implemented by the processor of the computer (information prompting device), so the processor can also be referred to as the action subject.

[0067] (i) Step 101

[0068] The original terrain information acquisition unit 26 acquires an aerial view (aerial view image 2) captured by a drone or satellite, or an image drawn based on a design drawing, and sends the aerial view image 2 to the aerial view image synthesis unit 28. Furthermore, the aerial view image 2 may be an image pre-stored in a database or storage device, or it may be an image re-captured or obtained by an operator, construction manager, etc., and input into the information display device.

[0069] (ii) Step 102

[0070] The current terrain information acquisition unit 25 is captured by at least one imaging device 33 and receives images of the current terrain transmitted from there.

[0071] (iii) Step 103

[0072] The current terrain information acquisition unit 25 merges the image received in step 102 (images from multiple imaging devices 33) with a predetermined reference point (e.g., any pixel or region of any image) to convert it into an overhead image (overhead image 1). Then, the current terrain information acquisition unit 25 sends the generated overhead image 1 to the overhead image synthesis unit 28.

[0073] (iv) Step 104

[0074] The overhead image synthesis unit 28 synthesizes or converts the overhead image 2 received from the original terrain information acquisition unit 26 and the overhead image 1 received from the current terrain information acquisition unit 25 to generate a synthesized overhead image (overhead image 3), and sends it to the display unit 29.

[0075] (v) Step 105

[0076] Display unit 29 displays an overhead view 3 on the display screen. As a result, the operator or construction manager, the operator of the hydraulic excavator (operating machinery) 1, the surrounding workers and / or the on-site supervisor can have an overhead view of the construction site.

[0077] In this way, by overlaying an aerial image of the current terrain onto an aerial image of the original terrain, the aerial image can be supplemented when the information in the aerial image of the current terrain is insufficient.

[0078] (2) Second Implementation

[0079] Given the limited number of cameras 33 available at the construction site 31, there is a higher probability of blind spots 40 appearing at the work site. In such cases, if the operator of the hydraulic excavator (operating machinery) 1 is not informed of the existence of blind spots 40, it may be difficult to perform actions (operations) that take into account the possibility of obstacles (other operating machinery, etc.) appearing from the blind spots 40 of the fixed obstacle 30.

[0080] In view of this point, the second embodiment provides an information prompting device and a method for prompting the operator of the hydraulic excavator (operating machinery) 1 when there is a blind spot (blind spot 40 described below) at the construction site 31.

[0081] use Figures 8 to 11 The information prompting device according to the second embodiment of this disclosure will be described. Furthermore, in Figures 8 to 11 In, with Figures 1 to 7 The same reference numerals denote the same parts or processing units, therefore, further descriptions are omitted.

[0082] In the first embodiment, an overhead image of the current terrain is superimposed on an overhead image of the original terrain. However, in the second embodiment, the information prompting device further includes: a blind spot calculation unit 39, which calculates the blind spot 40 of the overhead image generated by the current terrain information acquisition unit 25; and a determination unit 38, which determines that the information in the overhead image of the current terrain is insufficient based on the calculated distance to the blind spot 40 and the area of ​​the blind spot 40.

[0083] <Example of the functional structure of an information prompting device>

[0084] Figure 8 This is a block diagram illustrating an example of the functional configuration of the information prompting device according to the second embodiment. In this information prompting device, the blind spot calculation unit 39 calculates the area of ​​the grid outside the shooting range of the shooting device 33 based on the overhead image (overhead image 1) of the current terrain information acquisition unit 25, for example, by dividing the overhead image 1 into a grid. Alternatively, the range of the blind spot 40 can be estimated in detail based on the position, height, line of sight, and viewing angle of the shooting device 33.

[0085] The determination unit 38 determines whether the information in the overhead view image 1 is insufficient based on the result calculated by the blind spot calculation unit 39. Specifically, if the area of ​​the blind spot 40 in the construction site 31 exceeds a certain threshold of the overall area of ​​the construction site 31, the determination unit 38 determines that the information (the information required for the safe operation of the work machinery 1 (vehicle body)) is insufficient. Thus, for example, when the work machinery (hydraulic excavator) 1 is operating near a blind spot, there is a possibility that an obstacle suddenly appears from the blind spot 40 of the fixed obstacle 30 and comes into contact with the front work unit 23 or the counterweight 11. Therefore, warnings can be issued in advance, mainly during rotation and travel operations. Furthermore, the threshold for the area of ​​the blind spot 40 can be arbitrarily set by the construction manager. Furthermore, if the position (coordinates) of the working machine 1 can be estimated using GNSS or a camera 33, it can also be determined that the required information (information necessary for the safe operation of the working machine 1 (vehicle body)) is insufficient if the distance to the working machine 1 and the blind spot 40 is below a threshold, for example, within the movable range 24 of the working machine 1 and the area of ​​the corresponding blind spot 40 is above a certain value.

[0086] In the overhead image (overhead image 3) synthesized by combining overhead image 1 and overhead image 2, display unit 29 further displays overhead image 4 (overhead image taking into account blind spots), which is generated by performing display processing such as smearing the blind spots 40. This allows for visualization of the blind spots 40 of the construction site 31 by operators, construction managers, surrounding workers, and / or on-site supervisors. Furthermore, if the determination unit 38 determines that the required information is insufficient, the display unit 29 displays a warning such as "Insufficient Information". In addition, display unit 29 can implement sound-based warnings in addition to display devices such as monitors and display terminals such as smartphones.

[0087] <Calculations regarding blind spot 40>

[0088] use Figures 9A to 9C This section details the calculation method for blind spot 40 (the presence or absence of a blind spot, its location, and its width (size)). First, when using a camera or other shooting device 33 to photograph a fixed obstacle 30, the line connecting the ends 30A and 30B of the fixed obstacle 30 to the center point of the shooting device 33 becomes the blind spot line 41, and the inner side of the blind spot line 41 becomes the blind spot 40. Next, as... Figure 9BAs shown, by equally dividing the dead angle 40 into a grid or into units of each pixel (thus determining the coordinates of the dead angle), the area of ​​the dead angle 40 can be represented by the area corresponding to the number of quadrilaterals. Therefore, to more rigorously determine the area of ​​the dead angle 40, a finer grid division is required. Furthermore, if the area inside the dead angle line 41 of each quadrilateral (grid) (the part hidden as a dead angle) exceeds a threshold, the grid is determined to be a quadrilateral constituting the dead angle 40. This threshold can be arbitrarily set.

[0089] In addition, when a camera or other shooting device 33 is installed at a position higher than the fixed obstacle 30, the line connecting the upper part 30C of the fixed obstacle and the center point of the shooting device 33 becomes the blind spot line 41, which can reduce the blind spots in the depth section.

[0090] As described above, by determining the blind spot 40 in three dimensions, it is possible to accurately determine whether the operating machinery 1 can operate safely.

[0091] <Example of aerial image 4>

[0092] Figure 10 This is a diagram illustrating an example of an overhead view 4 (an overhead view generated by adding blind spots to an overhead view 3) according to the second embodiment.

[0093] like Figure 10 As shown, the display unit 29 displays the overhead image 4 generated by overlaying the current terrain information acquisition unit 25's blind spot 40 onto the overhead image 3 on the display screen of the information prompting device. Furthermore, if the area of ​​the blind spot 40 is large, the display unit 29 displays a "warning" on the display screen, warning of the possibility of other obstacles (not shown) appearing from the blind spot 40.

[0094] <Detailed information on image generation and processing from an aerial perspective>

[0095] Figure 11 This is a flowchart illustrating the detailed process of generating the overhead image 4 based on the second embodiment. In each step described below, each processing unit (current terrain information acquisition unit 25, original terrain information acquisition unit 26, blind spot calculation unit 39, overhead image synthesis unit 28, and determination unit 38) is considered as the action subject. However, as mentioned above, their programs are equivalent to being read from memory by the processor of the computer (information prompting device), so the processor can also be considered as the action subject.

[0096] (i) Step 201

[0097] The original terrain information acquisition unit 26 acquires an aerial view (aerial view image 2) captured by a drone or satellite, or an image drawn based on a design drawing, and sends the aerial view image 2 to the aerial view image synthesis unit 28. Furthermore, the aerial view image 2 may be an image pre-stored in a database or storage device, or it may be an image re-captured or acquired by an operator or construction manager and input into the information display device.

[0098] (ii) Step 202

[0099] The current terrain information acquisition unit 25 receives images of the current terrain captured and transmitted by at least one imaging device 33.

[0100] (iii) Step 203

[0101] The current terrain information acquisition unit 25 merges the image received in step 202 (images from multiple imaging devices 33) with a predetermined reference point (e.g., any pixel or region of any image) to convert it into an overhead image (overhead image 1). Then, the current terrain information acquisition unit 25 sends the generated overhead image 1 to the blind spot calculation unit 39.

[0102] (iv) Step 204

[0103] The presence or absence of blind spot in the calculation unit 39 and the blind spot in the calculation unit 40 (refer to) Figures 9A to 9C ), the area of ​​the blind spot 40 (refer to) Figure 9A ~C) and the distance to the blind spot 40 (distance: for example, the movable range 24A of the working machines 1A and 1B or the shortest distance from the movable range 24 to the blind spot 40). Furthermore, the blind spot calculation unit 39 sends the information of the blind spot 40 (presence, area and distance) together with the overhead view image 1 to the overhead view image synthesis unit 28.

[0104] (v) Step 205

[0105] The overhead image synthesis unit 28 synthesizes or converts the overhead image 2 received from the original terrain information acquisition unit 26 and the overhead image 1 received from the blind spot calculation unit 39 (or can be directly received from the current terrain information acquisition unit 25) to generate a synthesized overhead image (overhead image 3). Then, it synthesizes the blind spot 40 received from the blind spot calculation unit 39 with the overhead image 3 to generate an overhead image 4 (an overhead image with blind spots added). Furthermore, the overhead image synthesis unit 28 sends the information of the overhead image 4 and the blind spot 40 (e.g., the presence or absence of blind spots, their area, and distance, at least the area of ​​the blind spot) to the determination unit 38.

[0106] (vi) Step 206

[0107] The determination unit 38 determines, based on the information received from the overhead image synthesis unit 28, whether the necessary information (information required for the safe operation of the work machinery 1 (vehicle body)) can be obtained. For example, if the distance from the work machinery 1 to the blind spot 40 (the movable range 24A of the work machinery 1A and 1B, or the shortest distance from the movable range 24 to the blind spot 40) is greater than a predetermined threshold (distance threshold), and the area of ​​the blind spot 40 is less than the predetermined threshold, it is determined that the necessary information can be obtained. If it is determined that the necessary information can be obtained (if yes in step 206), the process proceeds to step 207. If it is determined that the necessary information cannot be obtained (if no in step 206), the process proceeds to step 208.

[0108] (vii) Step 207

[0109] The display unit 29 displays an overhead view 4 on the display screen. As a result, the construction manager, the operator of the hydraulic excavator (operating machinery) 1, the surrounding operators, and / or the on-site supervisor can have an overhead view of the construction site and can safely operate the operating machinery 1 at the construction site 31.

[0110] (viii) Step 208

[0111] Display unit 29 overlays "warning" (such as warning text and signs used to notify of danger) onto the overhead view 4 and displays it on the screen. As a result, the construction manager, the operator of the hydraulic excavator (operating machinery) 1, the surrounding operators, and / or the site supervisor can have an overhead view of the construction site and can grasp the dangers of operating the operating machinery 1 in the construction site 31.

[0112] As described above, by calculating the area of ​​the blind spots 40 at the construction site 31 and warning the operator, surrounding workers, and site supervisor of insufficient information, it is possible to implement actions that take into account the possibility of obstacles appearing from the blind spots 40 of the fixed obstacle 30. Furthermore, by displaying the blind spots 40 on the display unit 29, it is possible to prompt changes in the configuration of the current terrain information acquisition unit 25.

[0113] (3) Third implementation method

[0114] use Figures 12 to 15 The information prompting device of the third embodiment of this disclosure will be described. Furthermore, in Figures 12 to 15 In, with Figures 1 to 11 The same reference numerals denote the same parts or processing units, therefore, further descriptions are omitted.

[0115] In the second embodiment, a camera device 33 is deployed around the construction site 31 to determine whether the required information is insufficient based on the area of ​​the blind spots 40 within the current terrain, and to issue warnings to the operator, surrounding workers, and on-site supervisor. On the other hand, the third embodiment has the function of combining (overlaying) the data of the vehicle-mounted overhead image 48 generated based on the images captured by the vehicle-mounted camera device 33 with the current terrain information (overhead image 1), thereby reducing the area of ​​the blind spots 40.

[0116] <Example of the functional structure of an information prompting device>

[0117] Figure 12 This is a block diagram illustrating an example of the functional configuration of an information display device according to a third embodiment. For example... Figure 12 As shown, this information prompting device, in addition to Figure 8 In addition to the structural elements, it also includes a vehicle orientation angle calculation unit 43, a vehicle position detection unit 44, a vehicle overhead image generation unit 45, and a display switching unit 47.

[0118] The vehicle orientation angle calculation unit 43 obtains the orientation of the working machinery 1 in the construction site reference based on the position information from the vehicle position detection unit 44, and sends the orientation information to the vehicle-mounted overhead image correction unit 46. Additionally, the vehicle position detection unit 44 determines the position of the working machinery 1 using the construction site reference or a global coordinate reference, and sends the position information to the vehicle-mounted overhead image correction unit 46. The vehicle-mounted overhead image generation unit 45 sends the overhead image (hereinafter referred to as the vehicle-mounted overhead image 48) generated based on the vehicle-mounted imaging device 33 to the vehicle-mounted overhead image correction unit 46.

[0119] <Examples of aerial views from vehicles>

[0120] Figure 13 This diagram illustrates an example of a vehicle-mounted overhead image 48. Here, the vehicle-mounted overhead image 48 is synthesized from an overhead image of the existing terrain, with a different position and orientation based on the location and orientation of the working machinery 20. Therefore, the position of the synthesized vehicle-mounted overhead image 48 is first determined based on the positional relationship between the location information of the working machinery 1A and 1B and the reference point of the construction site 31.

[0121] Next, based on the orientation information of the working machines 1A and 1B under the construction site reference or global coordinate reference, the vehicle-mounted overhead image correction unit 46 rotates the image data by the amount corresponding to the orientation angle of the working machines. Then, according to the obtained positions of the working machines 1A and 1B, the rotated image data is projected onto the overhead image and synthesized, thereby enabling the vehicle-mounted overhead image 48 to be displayed on the overhead image of the construction site.

[0122] <Example of a composite aerial image>

[0123] Figure 14 This is an example of an overhead view of image 5 showing the synthesis of the third embodiment. Furthermore, Figure 14 The overhead view 5 shown becomes the following image: an image obtained by composite image 2 onto overhead view 1, reflecting the vehicle-mounted overhead view 48: from Figure 10 The image shown is obtained by removing the area covered by the vehicle-mounted overhead image 48 from the blind spot 40 of the overhead image 4.

[0124] By overlaying the blind spot 40 and the vehicle-mounted overhead view 48 onto the overhead view 1 of the current terrain obtained by the current terrain information acquisition unit 25, the blind spot 40 of the overhead view 1 obtained by the current terrain information acquisition unit 25 can be supplemented. Furthermore, the overhead view image synthesis unit 28 can generate an overhead view image 5 by further overlaying the overhead view image 2 from the original terrain information acquisition unit 26 onto the overhead view image from the blind spot calculation unit 39 (the image obtained by overlaying the blind spot 40 and the vehicle-mounted overhead view 48 onto the overhead view image 1).

[0125] In this way, by using the vehicle-mounted overhead view 48, the area of ​​the blind spot 40 can be reduced, and actions that take into account the possibility of obstacles appearing from the blind spot 40 of the fixed obstacle 30 can be implemented.

[0126] The display switching unit 47 can switch between the vehicle-mounted overhead image 48 and the overhead image 5 (surrounding overhead image) via a button or a monitor touch screen.

[0127] <Detailed information on image generation and processing from an aerial perspective>

[0128] Figure 15 This is a flowchart illustrating the detailed process of generating the overhead view image 5 according to the third embodiment. In each step described below, each processing unit (current terrain information acquisition unit 25, original terrain information acquisition unit 26, blind spot calculation unit 39, overhead view image synthesis unit 28, determination unit 38, vehicle body orientation angle calculation unit, vehicle body position detection unit 44, vehicle-mounted overhead view image generation unit 45, display switching unit 47) is taken as the action subject. However, as mentioned above, their programs are equivalent to being read from memory and implemented by the processor of the computer (information prompting device), so the processor can also be taken as the action subject.

[0129] (i) Step 301

[0130] The original terrain information acquisition unit 26 acquires an aerial view (aerial view image 2) captured by a drone or satellite, or an image drawn based on a design drawing, and sends the aerial view image 2 to the aerial view image synthesis unit 28. Furthermore, the aerial view image 2 may be an image pre-stored in a database or storage device, or it may be an image re-captured or obtained by an operator or construction manager and input into the information display device.

[0131] (ii) Step 302

[0132] The current terrain information acquisition unit 25 captures images of the current terrain by at least one imaging device 33 and receives images of the current terrain transmitted from there.

[0133] (iii) Step 303

[0134] The current terrain information acquisition unit 25 merges the image received in step 302 (images from multiple imaging devices 33) with a predetermined reference point (e.g., any pixel or region of any image) to convert it into an overhead image (overhead image 1). Then, the current terrain information acquisition unit 25 sends the generated overhead image 1 to the blind spot calculation unit 39.

[0135] (iv) Step 304

[0136] The vehicle-mounted overhead image correction unit 46 obtains an overhead image of the work machinery 1 from the vehicle-mounted overhead image generation unit 45, obtains the orientation angle information of the work machinery 1 from the vehicle body orientation angle calculation unit 43, and obtains the position information of the work machinery 1 from the vehicle body position detection unit 44. Then, based on the orientation angle and position information of the work machinery 1, the vehicle-mounted overhead image correction unit 46 corrects the overhead image of the work machinery 1 and sends the corrected overhead image to the overhead image synthesis unit 28.

[0137] (v) Step 305

[0138] The presence or absence of blind spot in the calculation unit 39 and the blind spot in the calculation unit 40 (refer to) Figures 9A to 9C ), the area of ​​the blind spot 40 (refer to) Figures 9A to 9C The information is calculated by the blind spot calculation unit 39, which calculates the blind spot information (existence, area, and distance) along with the overhead view image 1 and the distance to the blind spot 40 (distance: for example, the movable range 24A of the working machines 1A and 1B, or the shortest distance from the movable range 24 to the blind spot 40).

[0139] (vi) Step 306

[0140] The overhead image synthesis unit 28 synthesizes the overhead image 2 received from the original terrain information acquisition unit 26, the overhead image 1 received from the blind spot calculation unit 39, the blind spot 40, and the vehicle-mounted overhead image (corrected vehicle-mounted overhead image) 48 received from the vehicle-mounted overhead image correction unit 46 (by combining them) to generate an overhead image 5 reflecting the blind spot 40 and the vehicle-mounted overhead image (see reference). Figure 14 Then, the overhead image synthesis unit 28 sends the overhead image 5 to the determination unit 38.

[0141] (vii) Step 307

[0142] The determination unit 38 determines, based on the information received from the overhead image synthesis unit 28, whether the necessary information (information required for the safe operation of the work machinery 1 (vehicle body)) can be obtained. For example, if the distance from the work machinery 1 to the blind spot 40 (the movable range 24A of the work machinery 1A and 1B, or the shortest distance from the movable range 24 to the blind spot 40) is greater than a predetermined threshold (distance threshold) and the area of ​​the blind spot 40 is less than the predetermined threshold, it is determined that the necessary information can be obtained. If it is determined that the necessary information can be obtained (if yes in step 307), the process proceeds to step 308. If it is determined that the necessary information cannot be obtained (if no in step 307), the process proceeds to step 309.

[0143] (viii) Step 308

[0144] The display unit 29 displays an overhead view 5 on the display screen. As a result, the construction manager, the operator of the hydraulic excavator (operating machinery) 1, the surrounding operators, and / or the on-site supervisor can have an overhead view of the construction site and can safely operate the operating machinery 1 at the construction site 31.

[0145] (ix) Step 309

[0146] Display unit 29 overlays "warning" (such as warning text or a sign used to notify of danger) onto the overhead view 5 and displays it on the screen. As a result, construction managers, operators of hydraulic excavators (operating machinery) 1, surrounding operators, and / or on-site supervisors can have an overhead view of the construction site and can understand the dangers of operating the operating machinery 1 in the construction site 31.

[0147] As described above, by calculating the area of ​​the blind spots 40 at the construction site 31, and alerting operators, construction managers, surrounding workers, and / or site supervisors to any lack of necessary information, actions can be implemented that take into account the possibility of objects appearing from the blind spots 40 of the fixed obstacles 30. Furthermore, by displaying the blind spots 40 on the display unit 29, adjustments to the configuration of the current terrain information acquisition unit 25 can be made.

[0148] (4) Summary

[0149] (i) According to the first embodiment, the information prompting device overlays a first overhead image (overhead image 1) showing the current terrain of the construction site 31 from above and a second overhead image (overhead image 2) showing the original terrain of the construction site obtained by a method other than photography (e.g., drone or satellite photography) by the camera 33 installed at the construction site to generate a first composite overhead image (overhead image 3), and outputs it (displayed on the display screen of the display unit 29). In this way, since the overhead image 2, which is originally present in the current terrain image obtained by the camera 33, is used to generate the overall overhead image of the current construction site, even if the number of camera 33s installed at the construction site is small, a useful overhead image of the construction site can be provided to the operator.

[0150] (ii) According to the second embodiment, the information prompting device uses the first overhead view image (overhead view image 1) acquired by the imaging device 33 to determine the blind spot 40 in the construction site, reflects the image of the blind spot 40 onto the first composite overhead view image (overhead view image 3) to generate a second composite overhead view image (overhead view image 4), and outputs it. Thus, it is possible to have an overhead view of the possibility of other working machinery emerging from the blind spot 40, thereby enabling the working machinery to operate safely.

[0151] (iii) According to the third embodiment, the information prompting device reflects the vehicle-mounted overhead image generated based on the image captured by the vehicle-mounted camera device installed on the working machinery 1 onto the second composite overhead image (overhead image 4) to generate a third composite overhead image (overhead image 5), and outputs it. As a result, the area of ​​the blind spot 40 can be reduced, and the working machinery can be operated more safely.

[0152] (iv) The functions of the embodiments of this disclosure can also be implemented by software program code. In this case, a storage medium storing program code is provided to a system or device, and the computer (or CPU or MPU) of the system or device reads the program code stored in the storage medium. In this case, the program code read from the storage medium itself implements the functions of the embodiments described above, and the program code itself and the storage medium storing the program code constitute this disclosure. Examples of storage media for supplying such program code include floppy disks, CD-ROMs, DVD-ROMs, hard disks, optical disks, optical discs, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0153] Alternatively, based on the instructions in the program code, the OS (operating system) running on the computer may perform part or all of the actual processing, thereby achieving the functions of the aforementioned embodiments. Furthermore, after the program code read from the storage medium is written into the computer's memory, the computer's CPU or other processor may perform part or all of the actual processing based on the instructions in the program code, thereby achieving the functions of the aforementioned embodiments.

[0154] Furthermore, the program code of the software that implements the functions of the implementation method can also be distributed via the network and stored in a storage unit such as a hard disk or memory of the system or device, or in a storage medium such as a CD-RW or CD-R. When in use, the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage unit or the storage medium.

[0155] Finally, the processes and techniques described herein are not inherently related to any particular device, and it should be understood that installation can be achieved through any appropriate combination of components. Furthermore, various types of general-purpose devices can be used according to the teachings described herein. It is also beneficial to construct specialized devices to perform the steps described herein.

[0156] This disclosure is described in connection with specific examples, but these are not intended to be limiting in any way, but are for illustrative purposes. Those skilled in the art will understand that various combinations of hardware, software, and firmware are suitable for implementing the techniques of this disclosure. For example, the described software can be installed in a wide range of program or scripting languages ​​such as assembly, C / C++, perl, shell, or Java (registered trademark).

[0157] Furthermore, in the described embodiments, the control lines and information lines represent what is necessary for the description, and may not necessarily represent all control lines and information lines on the product. All structures can also be interconnected.

[0158] The various embodiments of this disclosure have been described in detail above. However, the scope of this disclosure is not limited to the content of each embodiment, and various design changes can be made without departing from the scope of the claims. For example, the embodiments described in detail are for the purpose of easily understanding and illustrating the technology of this disclosure, and are not limited to having all the described structures. In addition, a part of the structure of a certain embodiment can be replaced with the structure of another embodiment, and it is also possible to add the structure of another embodiment to the structure of a certain embodiment. Furthermore, for a part of the structure of each embodiment, other structures can be added, deleted, or replaced.

[0159] Explanation of reference numerals in the attached figures

[0160] 1. Operating machinery

[0161] 25. Current Topographic Information Acquisition Department

[0162] 26. Original Topographic Information Acquisition Department

[0163] 27. Overlooking the Image Processing Department

[0164] 28. Overlooking the image compositing department

[0165] 29. Display Unit (Display Device)

[0166] 30 Fixed Obstacles

[0167] 31 Construction Site

[0168] 32 Construction site outside

[0169] 33 filming devices

[0170] 38 Judgment Department

[0171] 39 Dead Angle Computing Unit

[0172] 43 Vehicle body orientation angle calculation unit

[0173] 44 Vehicle Position Detection Department

[0174] 45 Vehicle-mounted overhead image generation unit

[0175] 46-vehicle overhead image correction unit

[0176] 47 Display Switching Unit

[0177] 49 Communications Department (Communication Devices).

Claims

1. An information prompting device that generates and outputs topographic images related to a construction site, characterized in that, This information display device has the following features: A storage device that stores a program for performing predetermined image processing on the terrain image; The processor reads the program from the storage device and generates the terrain image. The processor performs the following processing: Processing of obtaining a first overhead image of the current terrain of the construction site from images of the construction site acquired by at least one camera device, wherein the at least one camera device is positioned at the construction site at a location and height capable of capturing images of the construction site; The processing of obtaining a second overhead image, which is obtained by a method other than shooting by the shooting device set up at the construction site, and shows the topography of the construction site before construction from an overhead perspective; Image compositing process that generates a first composite aerial image by superimposing the second aerial image onto the first aerial image. The processor also performs the following processing: based on the first overhead image, it performs processing to determine information about blind spots in the construction site. In the image compositing process, the processor further generates an image of the blind spot based on information used to determine the blind spot, and generates a second composite top-down image that reflects the image of the blind spot in the first composite top-down image. When the shooting device is positioned at a height higher than the fixed obstacle in the construction site, the processor determines the blind spot line as the line connecting the upper part of the fixed obstacle and the center point of the shooting device.

2. The information prompting device according to claim 1, characterized in that, The processor uses the first overhead view to calculate the presence or absence of the blind spot, the location of the blind spot, and the area of ​​the blind spot as information for determining the blind spot.

3. The information prompting device according to claim 2, characterized in that, The processor also performs the following processing: based on the information used to determine the blind spot, it determines whether the information required to enable the operating machinery to operate at the construction site can be obtained.

4. The information prompting device according to claim 3, characterized in that, When the distance from the working machinery to the blind spot is greater than a first threshold and the area of ​​the blind spot is less than a second threshold, the processor determines that it is able to obtain the information required to make the working machinery operate at the construction site.

5. The information prompting device according to claim 4, characterized in that, When the processor determines that it cannot obtain the information required for the machine to operate, it outputs a warning.

6. The information prompting device according to claim 1, characterized in that, The processor also performs the following processing: generating a third composite overhead image by reflecting an image captured by a vehicle-mounted camera device installed on the work machinery operating at the construction site onto the second composite overhead image.

7. The information prompting device according to claim 6, characterized in that, The processor also responds to a display switching instruction by switching and outputting the third composite overhead image and the image captured by the vehicle-mounted camera.

Images