Information transmission system

Abstract

To provide an information transmission system which makes it easy to transmit information even in a noisy circumstance.SOLUTION: An information transmission system comprises: an acquisition unit which acquires an image captured by an imaging unit; an image recognition unit which recognizes a learned object, out of the image; and a control unit which on the basis of a result of the recognition by the image recognition unit, controls an optical projection device to make the optical projection device project an image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information transmission system. [Background technology]

[0002] Cranes used at work sites have been known (see, for example, Patent Document 1). At such work sites, various personnel may be present, such as a crane operator who operates the crane and ground workers who attach the crane's hoisting gear to an object to be hoisted (a load). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-518804 Summary of the Invention [Problem to be solved by the invention]

[0004] At such work sites, it may be desirable to transmit predetermined information (e.g., work instructions and safety warnings) to personnel such as crane operators and ground workers, depending on, for example, the state of the suspended load. One possible method for achieving this is to perform image recognition of objects (e.g., suspended loads and ground workers) present at the work site, generate audio based on the results of the image recognition, and transmit information to personnel through the generated audio. However, in work sites (environments) with high levels of noise, it may be difficult to transmit information through audio.

[0005] The present invention has been made in consideration of the above circumstances, and has as its object to provide an information transmission system that can easily transmit information even in a noisy environment. [Means for solving the problem]

[0006] In order to solve the above problem, the information transmission system according to aspect 1 of the present invention includes an acquisition unit that acquires an image captured by an imaging unit, an image recognition unit that recognizes a learned object from the image, and a control unit that controls an optical projection device to project an image on the optical projection device based on the recognition result by the image recognition unit.

[0007] Furthermore, in aspect 2 of the present invention, in the information transmission system of aspect 1, at least a portion of the projection surface onto which the image is projected is included in the imaging range of the imaging unit, and the control unit controls the optical projection device so that the optical projection device does not project the image based on the image captured during a projection period in which at least a portion of the image is reflected in the image.

[0008] Furthermore, in aspect 3 of the present invention, in the information transmission system of aspect 2, the image recognition unit is trained to be able to recognize the image, and the control unit controls the optical projection device so that, when the image acquired by the acquisition unit corresponds to a specific image of which at least a portion has been recognized by the image recognition unit, the image is not projected based on the specific image.

[0009] Furthermore, aspect 4 of the present invention relates to an information transmission system of aspect 3, further comprising a distortion correction unit, wherein the acquisition unit acquires three-dimensional information of the projection surface, the distortion correction unit calculates the shape of the projection surface based on the three-dimensional information of the projection surface, and corrects the image based on the calculated shape of the projection surface, and the image recognition unit recognizes the image based on the image corrected by the distortion correction unit.

[0010] Furthermore, aspect 5 of the present invention is an information transmission system according to any one of aspects 1 to 4, further comprising the imaging unit and the optical projection device, wherein the imaging unit captures the image from above downward, and the optical projection device projects the image from above downward. [Effects of the Invention]

[0011] According to the above aspect of the present invention, it is possible to provide an information transmission system that can easily transmit information even in a noisy environment. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating an example of an object to which an information transmission system according to an embodiment is applied; [Figure 2] 1 is a block diagram illustrating an example of a configuration of an information transmission system according to an embodiment. [Figure 3] 1A and 1B are diagrams illustrating an example of an image projected by an optical projection device according to an embodiment. [Figure 4] 2 is a diagram showing an example of a method for attaching an imaging unit, a three-dimensional sensor, and an optical projection device according to the embodiment to the crane shown in FIG. 1. FIG. [Figure 5] 10A and 10B are diagrams illustrating an example of image distortion according to the shape of the projection surface. [Figure 6] 10A and 10B are diagrams illustrating an example of distortion correction by a distortion correction unit according to an embodiment. [Figure 7] 3A to 3C are diagrams illustrating an example of an image acquired by an image acquisition unit and image recognition by an image recognition unit according to the embodiment. [Figure 8] 10 is a flowchart illustrating an example of processing performed by the information transmission system according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] An information transmission system according to an embodiment of the present invention will be described below with reference to the drawings.

[0014] <Overview of crane configuration and information transmission system> The information transmission system 100 according to this embodiment is applied to an object such as a crane (crane facility), and is a system for transmitting information related to work using the object to personnel. A crane (crane facility) 1 shown in FIG. 1 is an example of an object to which the information transmission system 100 is applied. However, the specific configuration of the crane is not limited to the example in FIG. 1 and can be changed as appropriate. Furthermore, the object to which the information transmission system 100 is applied does not have to be a crane and can be changed as appropriate.

[0015] Hereinafter, the direction along gravity (i.e., the vertical direction) will be referred to as the gravity direction Z. The upward direction in the gravity direction Z will be represented by the +Z direction and simply referred to as "upward." The downward direction in the gravity direction Z will be represented by the -Z direction and simply referred to as "downward." Furthermore, a direction that intersects (e.g., is perpendicular to) the gravity direction Z will be referred to as the first direction X. One direction in the first direction X will be represented by the +X direction and referred to as "rightward." The direction opposite to the +X direction will be represented by the -X direction and referred to as "leftward." A direction that intersects (e.g., is perpendicular to) both the gravity direction Z and the first direction X will be referred to as the second direction Y. One direction in the second direction Y will be represented by the +Y direction and referred to as "forward." The direction opposite to the +Y direction will be represented by the -Y direction and referred to as "backward." The first direction X and the second direction Y may be directions perpendicular to the gravity direction Z (ie, horizontal directions).

[0016] The crane 1 shown in FIG. 1 is a device that hoists a load L using a hoisting tool 22 (described below). The load L is an object to be hoisted by the crane 1, and may be, for example, a plate-shaped member. In the following, an example will be described in which the load L is a plate-shaped steel material that is rectangular in plan view. However, the type and shape of the load L can be changed as appropriate.

[0017] The crane 1 includes a pair of rails 2, a pair of girders 3, a pair of moving parts 4, a hoisting device 5, a rope 6, a sheave 7, a hook 8, a trolley 10, and a hooking mechanism 20. The hooking mechanism 20 includes a rope 21 and a pair of lifting devices (hackers) 22. The crane 1 is operated by a crane operator U' operating the operating part 9.

[0018] Each of the pair of rails 2 extends in the second direction Y. The pair of rails 2 are arranged at an interval in the first direction X. The rails 2 guide the movement of the moving part 4 in the second direction Y (described later).

[0019] Each of the pair of girders 3 extends in a first direction X. The pair of girders 3 are spaced apart in a second direction Y. The girders 3 guide the movement of the trolley 10 in the first direction X (described later).

[0020] The pair of moving parts 4 are provided at both ends of the pair of girders 3 in the first direction X. More specifically, one of the pair of moving parts 4 is provided so as to connect the right side (+X side) ends of the pair of girders 3 together, and the other of the pair of moving parts 4 is provided so as to connect the left side (-X side) ends of the pair of girders 3 together.

[0021] The moving part 4 (and the girder 3 connected thereto) moves along the rails 2 in the second direction Y in response to an operation performed on the operation part 9 by, for example, a crane operator U'. Specifically, a drive mechanism (not shown) having a motor or the like for moving the moving part 4 may be provided on the crane 1, and the drive mechanism may be driven in response to an operation performed on the operation part 9 by the crane operator U'.

[0022] The trolley 10 is installed on the pair of girders 3 so as to straddle the pair of girders 3. The trolley 10 may or may not have a cab for the crane operator U'.

[0023] The trolley 10 moves in the first direction X along the girder 3 in response to an operation performed on the operation unit 9 by, for example, a crane operator U'. Specifically, a drive mechanism (not shown) having a motor or the like for moving the trolley 10 may be provided on the crane 1, and the drive mechanism may be driven in response to an operation performed on the operation unit 9 by the crane operator U'.

[0024] The hoisting device 5 is provided on a trolley 10. A rope 6 having a sheave 7 connected to the end thereof is attached to the hoisting device 5. The hoisting device 5 has a motor and the like (not shown) for winding up and winding down the rope 6. Note that "winding up" refers to the operation of driving the motor so that the sheave 7 moves upward. "Winding down" refers to the operation of driving the motor so that the sheave 7 moves downward.

[0025] The hoisting device 5 hoists and lowers the rope 6 in accordance with, for example, an operation performed by a crane operator U' on the operation unit 9. Specifically, a motor or the like provided in the hoisting device 5 may be driven in accordance with an operation performed by the crane operator U' on the operation unit 9.

[0026] The hanging mechanism 20 is a mechanism for hanging (hooking) the suspended load L. The hanging mechanism 20 is hung from a hook 8 provided on the sheave 7. Specifically, the center portion of a rope 21 of the hanging mechanism 20 is hung from the hook 8. A pair of suspenders 22 are connected to both ends of the rope 21.

[0027] Each hoisting tool 22 has a base 22b and a pair of claws 22a. Each claw 22a is a part that comes into contact with the underside of the load L to support the load L when the load L is hung on the hanging mechanism 20.

[0028] In this embodiment, the claw 22a of one of the pair of suspenders 22 is hooked to one long side of the load L, and the claw 22a of the other of the pair of suspenders 22 is hooked to the other long side of the load L. That is, when the load L is hung on the hanging mechanism 20, the pair of suspenders 22 face each other in the short direction of the load L. Furthermore, in each suspender 22, the pair of claws 22a are arranged spaced apart in the horizontal direction (the longitudinal direction of the load L). Therefore, when the load L is hung on the hanging mechanism 20, each of the four claws 22a of the pair of suspenders 22 contacts the underside of the load L to support the load L. However, the configuration of the hanging mechanism 20 (hanging mechanism 22) can be modified as appropriate as long as it is possible to hang the load L.

[0029] In this embodiment, the removal of the hoisting device 22 from the load L is performed by a ground worker (ground slinger) U. The ground worker U is a worker who performs work on the ground. The ground worker U is, for example, a different person from the crane operator U'.

[0030] The operation unit 9 accepts operations by the crane operator U'. The operation unit 9 may be a physical mechanism (levers, buttons, etc.) for operating the crane 1, or may be an information processing terminal for operating the crane 1, etc.

[0031] After the ground worker U hooks the load L to the hooking mechanism 20 (hanging tool 22), the crane operator U' operates the control unit 9 to wind up the rope 6, and the load L is lifted by the hooking mechanism 20 (hanging tool 22) and rises. In this state, the crane operator U' operates the control unit 9 to move the trolley 10 and / or the moving unit 4 (girder 3). Then, the load L suspended from the hooking mechanism 20 moves in the first direction X and / or the second direction Y. In other words, the load L moves in the first direction X as the trolley 10 moves in the first direction X, and moves in the second direction Y as the moving unit 4 (girder 3) moves in the second direction Y.

[0032] Hereinafter, the movement of the load L (crane 1) in the first direction X accompanying the movement of the trolley 10 may be referred to as "lateral movement." Furthermore, the movement of the load L (crane 1) in the second direction Y accompanying the movement of the moving part 4 (girder 3) may be referred to as "traveling." That is, in this embodiment, the first direction X coincides with the lateral movement direction of the load L (crane 1), and the second direction Y coincides with the traveling direction of the load L (crane 1).

[0033] The information transmission system 100 according to this embodiment performs image recognition on an image IM captured by the crane 1 as described above, and projects an image P based on the image recognition results. This allows information based on the image IM to be transmitted to personnel such as the ground worker U and the crane operator U'. Examples of information transmitted to personnel include information (i.e., an alarm) informing them of the risk of an accident occurring due to the drop (collapse) of the load L from the hoisting device 22, and information indicating the next task or action to be taken (i.e., instruction information). For example, an image IM including the load L, the hoisting device 22, and the ground worker U is acquired, and the load L, the hoisting device 22, and the ground worker U are image-recognized from the image IM. Then, an image P representing an alarm, instruction information, or the like is projected based on the information obtained through image recognition.

[0034] Hereinafter, various components of the information transmission system 100 that perform the image recognition and projection of the image P, and the processing performed by the information transmission system 100 will be described in detail. In the following description, the surface onto which the image P is projected will be referred to as the projection surface S (see, for example, FIG. 5). The projection surface S may be, for example, the floor or wall surface of the site where the crane 1 is installed. However, the projection surface S (i.e., the position onto which the image P is projected) can be changed as appropriate.

[0035] <Configuration of information transmission system> 2 is a block diagram showing an example of the configuration of an information transmission system 100 according to this embodiment. The information transmission system 100 according to this embodiment includes a terminal device 200, an imaging unit 301, a three-dimensional sensor 302, and an optical projection device 303.

[0036] The imaging unit 301 captures an image IM (see, for example, FIG. 7) including an imaging target in the crane 1. The imaging target may include, for example, a ground worker U, a hoisting device 22, and a suspended load L. The imaging unit 301 may include, for example, at least one camera for capturing the image IM. In this embodiment, the imaging range of the imaging unit 301 includes at least a portion of the projection surface S (image P) (see, for example, FIG. 5). Furthermore, the imaging unit 301 according to this embodiment captures the image IM from above downward (see, for example, FIGS. 4 and 5). The imaging unit 301 outputs the captured image IM to an image acquisition unit 211 (described later) included in the terminal device 200.

[0037] The three-dimensional sensor 302 measures three-dimensional information of a projection surface S (see, for example, FIG. 5) onto which the image P is projected. The three-dimensional information of the projection surface S includes information on the three-dimensional position of the projection surface S and information on the three-dimensional shape of the projection surface S. The three-dimensional sensor 302 may, for example, have a LiDAR that measures the distance to an object using reflected light obtained by irradiating the object with light. The three-dimensional sensor 302 outputs the measured three-dimensional information to a three-dimensional information acquisition unit 212 (described below) included in the terminal device 200.

[0038] The optical projection device 303 irradiates light onto the projection surface S and projects an image P onto the projection surface S in accordance with control by a control unit 240 (described later) of the terminal device 200. The optical projection device 303 may include, for example, a light (such as a laser light source or an LED light source) or a projector. The image P may be, for example, graphic information, text information, or information containing both. The specific form of the image P can be changed as appropriate as long as it is projected by irradiating light and is visually recognizable. Similarly, the specific form of the optical projection device 303 for projecting the image P can be changed as appropriate.

[0039] FIG. 3 is a diagram showing an example of an image P projected by the optical projection device 303. In the example of FIG. 3, the optical projection device 303 is a plurality of lights, and the image P includes a plurality of circles arranged in a circle. In the example of FIG. 3, the optical projection device 303 projects the image P from above downward onto the periphery of the suspended load L. That is, in the example of FIG. 3, the position of the projection plane S is selected so that at least a portion of the suspended load L (the upper surface of the suspended load L) is included in the projection plane S. Even when the hoisting device 22 is lifted by the crane 1, it is generally positioned below the eyes of the ground worker U. Therefore, by projecting the image P as an information transmission means onto the periphery of the suspended load L (the upper surface of the suspended load L) in this manner, the image P as an information transmission means can be more easily seen by the ground worker U.

[0040] The imaging unit 301, three-dimensional sensor 302, and optical projection device 303 are provided (attached) to the crane 1. The imaging unit 301, three-dimensional sensor 302, and optical projection device 303 may be provided, for example, on the trolley 10. As shown in FIG. 1, the ground worker U, the hoisting device 22, and the suspended load L are located near directly below the trolley 10. Therefore, by providing the imaging unit 301 on the trolley 10, it is possible to easily capture an image IM including these objects. Similarly, by providing the three-dimensional sensor 302 on the trolley 10, it is possible to easily measure three-dimensional information of the projection surface S (e.g., the periphery of the suspended load L). Furthermore, by providing the optical projection device 303 on the trolley 10, it is possible to easily project an image P onto the periphery of the suspended load L (the top surface of the suspended load L).

[0041] 4 is a diagram showing an example of a method for attaching the imaging unit 301, three-dimensional sensor 302, and optical projection device 303 to the trolley 10. However, the method for attaching the imaging unit 301, three-dimensional sensor 302, and optical projection device 303 to the crane 1 (trolley 10) can be changed as appropriate.

[0042] As shown in Fig. 4, in the information transmission system 100 according to this embodiment, a frame-shaped mounting jig J is attached to the trolley 10. When the mounting jig J is attached to the trolley 10, the mounting jig J extends downward from the trolley 10, and the lower end of the mounting jig J is located below the lower end of the girder 3. An imaging unit 301, a three-dimensional sensor 302, and an optical projection device 303 are attached to a portion of the mounting jig J located below the girder 3 so as to face downward. Note that the imaging unit 301 in the illustrated example includes two cameras.

[0043] By positioning the imaging unit 301 below the girder 3, it is possible to prevent the girder 3 from entering the imaging range of the imaging unit 301. Similarly, by positioning the three-dimensional sensor 302 below the girder 3, it is possible to prevent the girder 3 from entering the detection range of the three-dimensional sensor 302. Furthermore, by positioning the optical projection device 303 below the girder 3, it is possible to prevent the light from the optical projection device 303 from being blocked by the girder 3, resulting in missing image P.

[0044] The terminal device 200 (see FIG. 2) is configured using an information processing terminal such as a personal computer. The terminal device 200 may be provided on the crane 1 (e.g., the trolley 10) or may be provided outside the crane 1. As shown in FIG. 2, the terminal device 200 includes an acquisition unit 210, a storage unit 220, a processing unit 230, and a control unit 240. The acquisition unit 210 includes, for example, an image acquisition unit 211 and a three-dimensional information acquisition unit 212.

[0045] The image acquisition unit 211 acquires an image IM (see, for example, FIG. 7) including an image capture target. The image capture target may include, for example, a ground worker U, a hoisting device 22, and a suspended load L. The image acquisition unit 211 acquires, for example, the image IM output from the imaging unit 301. The image acquisition unit 211 also outputs the acquired image IM to the processing unit 230.

[0046] The three-dimensional information acquisition unit 212 acquires three-dimensional information of the projection surface S. The three-dimensional information acquisition unit 212 acquires, for example, the three-dimensional information of the projection surface S output from the three-dimensional sensor 302. Furthermore, the three-dimensional information acquisition unit 212 outputs the acquired three-dimensional information of the projection surface S to the processing unit 230.

[0047] The storage unit 220 is realized by, for example, a random access memory (RAM), a read only memory (ROM), a flash memory, an SD card, a register, a hard disk drive, etc. The storage unit 220 stores data used by the processing unit 230 and the control unit 240. The storage unit 220 stores data required when the processing unit 230 and the control unit 240 perform processing. The storage unit 220 stores, for example, an image recognition model 221 and communication information output data 222.

[0048] The image recognition model 221 is data indicating parameters, connection structure, function characteristics, etc. of a trained model used for processing by the processing unit 230 (image recognition unit 232). The image recognition model 221 is trained to recognize trained objects included in an image IM when the image IM is input.

[0049] A "trained object" is an object that has been trained by the image recognition model 221 to be capable of image recognition. In this embodiment, the "trained object" includes the above-described image targets (ground worker U, sling 22, and suspended load L). The image recognition model 221 according to this embodiment includes a first trained model (not shown) that has been trained to output the position of the ground worker U, the position of the sling 22, and the position of the suspended load L when an image IM is input. The first trained model may be trained to output the status of the ground worker U in addition to (or instead of) the position of the ground worker U. In other words, the first trained model may be trained to output at least one of the position and the status of the ground worker U. Note that the "status of the ground worker U" may include information such as the orientation of the ground worker U and the status of the work of the ground worker U (e.g., whether or not the ground worker U is attaching the sling 22 to the suspended load L). Hereinafter, "at least one of the position and state of the ground worker U, the position of the sling 22, and the position of the suspended load L" may be collectively referred to as "recognition target information." Hereinafter, image recognition that recognizes recognition target information from image IM may be referred to as "first image recognition." The first trained model is a trained model for performing the first image recognition.

[0050] Furthermore, the image recognition model 221 according to this embodiment is trained to be capable of second image recognition in addition to the first image recognition described above. The "second image recognition" is image recognition that recognizes an image P (more specifically, the position and / or presence or absence of an image P) from an image IM. The image recognition model 221 according to this embodiment includes a second trained model (not shown) that is trained to output the position and / or presence or absence of an image P when an image IM is input. The second trained model may be trained to recognize a part of the image P when the part of the image P is reflected in the image IM.

[0051] The image recognition model 221 (first trained model and second trained model) may be created using, as training data, a set of multiple pieces of input information and output information associated one-to-one with the multiple pieces of input information. Here, the "input information" includes the image IM. The "output information" includes recognition target information and the position and / or presence or absence of the image P. The image recognition model 221 may be created by the information transmission system 100 (terminal device 200). Alternatively, the image recognition model 221 may be created by a learning device provided separately from the information transmission system 100, and the created image recognition model 221 may then be stored in the storage unit 220. The specific method for training (machine learning) the image recognition model 221 is not particularly limited, and, for example, a neural network (deep learning) such as CNN or RNN may be used. Furthermore, learning (machine learning) may be performed using the method disclosed in Japanese Patent Application Laid-Open No. 2022-102930.

[0052] The communication information output data 222 is data used for processing by the processing unit 230 (communication information output unit 233). The communication information output data 222 is data for outputting communication information based on the recognition target information. Here, the "communication information" is information relating to what kind of image P should be projected by the optical projection device 303 (i.e., what information should be communicated by the image P). Note that the communication information may also be information indicating that the image P should not be projected.

[0053] For example, the communication information output data 222 may be information indicating conditions for linking recognition target information with communication information. In other words, the communication information output data 222 may be used to output communication information on a rule-based basis when recognition target information is input. For example, the communication information output data 222 may include information indicating conditions such as "when the position of the ground worker U and the position of the suspended load L are in a predetermined positional relationship, an image P related to an alarm is projected" or "when the ground worker U is in a predetermined state, an image P instructing the next work content is projected." However, the condition information included in the communication information output data 222 can be changed as appropriate.

[0054] Alternatively, the transmission information output data 222 may be a trained model that has been trained to output transmission information when recognition target information is input. In this case, the transmission information output data 222 may be, for example, data indicating parameters, connection structure, function characteristics, etc. of the trained model used in processing by the transmission information output unit 233. In addition to the recognition target information, an image IM or information related to three-dimensional information (height distribution information) of the load L may be input to the transmission information output data 222. The "information related to the three-dimensional information (height distribution information) of the load L" may be, for example, the three-dimensional information (height distribution information) of the load L itself, or information calculated based on the three-dimensional information (height distribution information) of the load L. For example, the "information related to the three-dimensional information (height distribution information) of the load L" may be deflection information indicating the deflection of the load L. The deflection information may be calculated by a calculation unit (not shown) provided in the processing unit 230. For example, the deflection information of the suspended load L may include information indicating the amount of deflection of the suspended load L and the shape of the deflection. The three-dimensional information of the suspended load L may be acquired by the three-dimensional sensor 302 and the three-dimensional information acquisition unit 212. The communication information output data 222 may include multiple trained models. The trained models as the communication information output data 222 may be created by a method similar to that of the image recognition model 221 described above.

[0055] The processing unit 230 includes, for example, a distortion correction unit 231, an image recognition unit 232, a transmission information output unit 233, and a generation unit 234. Some or all of the functions of the processing unit 230 and the control unit 240 are realized by, for example, a processor such as a CPU (Central Processing Unit) executing a program (software) stored in its own storage unit. Furthermore, some or all of the functions of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device such as an HDD (Hard Disk Drive) or flash memory, or may be stored in a removable storage medium such as a DVD or CD-ROM and installed by inserting the storage medium into a drive device.

[0056] The control unit 240 controls the optical projection device 303 to project the image P. The control unit 240 controls the optical projection device 303 based on, for example, a control signal output by the processing unit 230 (generation unit 234).

[0057] The distortion correction unit 231 calculates the shape of the projection surface S based on the three-dimensional information of the projection surface S acquired from the three-dimensional information acquisition unit 212. Then, the distortion correction unit 231 corrects the image IM based on the calculated shape of the projection surface S.

[0058] If the projection surface S is a flat surface (hereinafter also referred to as an "ideal projection surface") that is perpendicular to the projection direction of the optical projection device 303, no distortion (change in shape) occurs in the image P projected onto the projection surface S. On the other hand, if the projection surface S is not an ideal projection surface, the shape of the image P projected onto the projection surface S changes depending on the shape of the projection surface S. In other words, distortion occurs in the image P depending on the shape of the projection surface S. FIG. 5 is a diagram showing an example of distortion of the image P depending on the shape of the projection surface S.

[0059] In FIG. 5, the projection direction of the optical projection device 303 coincides with the direction of gravity Z. Therefore, a flat surface extending in the horizontal direction corresponds to an "ideal projection surface." However, the projection surface S shown in FIG. 5 is an inclined surface tilted relative to the horizontal direction and does not correspond to an ideal projection surface. More specifically, the projection surface S in FIG. 5 is an inclined surface that is inclined so that its position in the direction of gravity Z changes as its position in one direction in the horizontal direction (hereinafter referred to as the "inclined direction") changes. An image P projected onto such a projection surface S has a shape that is elongated in the inclined direction of the projection surface S.

[0060] The distortion correction unit 231 performs image processing (correction) on the image IM so as to remove such distortion from the image P. FIG. 6 is a diagram showing an example of distortion correction by the distortion correction unit 231 according to this embodiment. The left side of FIG. 6 shows the image P projected onto the projection surface S, which has a shape stretched in the tilt direction as described above. The distortion correction unit 231 compresses the image IM (image P) in the tilt direction based on the calculated shape of the projection surface S, and outputs an image (corrected image) IM' (right side of FIG. 6). The image (corrected image) P' included in the corrected image IM' has a shape from which distortion has been removed. In other words, the corrected image P' matches the image P when the optical projection device 303 projects the image P onto an ideal projection surface.

[0061] In the examples shown in Figures 5 and 6, the projection surface S is an inclined surface, but regardless of the shape of the projection surface S, the distortion of the image P can be similarly removed by calculating the shape of the projection surface S from the three-dimensional information of the projection surface S and correcting the image IM based on the calculated shape.

[0062] The image recognition unit 232 (see FIG. 2) recognizes a trained object from the image IM. That is, the image recognition unit 232 performs first image recognition by inputting the image IM into a first trained model, and outputs recognition target information. The image recognition unit 232 also recognizes an image P from the image IM corrected by the distortion correction unit 231 (i.e., corrected image IM'). That is, the image recognition unit 232 performs second image recognition by inputting the corrected image IM' into a second trained model, and outputs the position and / or presence or absence of the image P.

[0063] 7 is a diagram showing an example of an image IM acquired by the image acquisition unit 211 according to this embodiment, and an example of image recognition (first image recognition) by the image recognition unit 232. In the example shown in FIG. 7, the image recognition unit 232 outputs, as recognition target information, the position B1 of the ground worker U, the position B2 of the sling 22, and the position B3 of the suspended load L in the form of a bounding box, and also outputs the orientation D of the ground worker U. Note that, in the example of FIG. 7, the image IM does not include an image P, and the image recognition unit 232 does not recognize (detect) the image P.

[0064] The transmission information output unit 233 (see FIG. 2) outputs transmission information based on the recognition target information output from the image recognition unit 232 and the transmission information output data 222. For example, if the transmission information output data 222 is for outputting transmission information on a rule basis, the transmission information output unit 233 outputs the transmission information by comparing the recognition target information output from the image recognition unit 232 with the transmission information output data 222. If the transmission information output data 222 is a trained model, the transmission information output unit 233 outputs the transmission information by inputting the recognition target information output from the image recognition unit 232 into the trained model.

[0065] The generation unit 234 generates a control signal for the control unit 240 to control the optical projection device 303 based on the transmission information output from the transmission information output unit 233. When the image P includes graphic information, the control signal generated by the generation unit 234 may include, for example, information corresponding to the shape, color, etc. of the graphic. When the image P includes text information, the control signal generated by the generation unit 234 may be, for example, information corresponding to the content, font, color, etc. of the text.

[0066] The control signal may be information for continuing to project the image P for a predetermined period of time. The control signal may be information for flashing the image P at a predetermined flashing rate for a predetermined period of time. The control signal may be information for continuing to project the image P for a predetermined period of time and momentarily ceasing the projection of the image P at a predetermined timing (momentarily turning off the optical projection device 303). In other words, the control signal may be information for controlling the projection period during which the image P is projected and the non-projection period during which the image P is not projected. In this embodiment, since at least a portion of the projection surface S is included in the imaging range of the imaging unit 301, at least a portion of the image P is reflected in the image IM (corrected image IM') captured during the projection period.

[0067] The content of the control signal generated by the generation unit 234 varies depending on the content of the transmission information. As a result, the control unit 240 controls the optical projection device 303 based on the result of recognition by the image recognition unit 232, and causes the optical projection device 303 to project an image P. For example, the control unit 240 causes the optical projection device 303 to project a different image P depending on the result of recognition by the image recognition unit 232.

[0068] However, if at least a part of the image P is reflected in the image IM, the accuracy of the first image recognition by the image recognition unit 232 may be reduced. For example, if the ground worker U is wearing a circular helmet or the like and the image P contains a circular light, the image recognition unit 232 may erroneously recognize the circular light in the image P as the ground worker U. If such an erroneous recognition occurs, the generation unit 234 may generate a control signal based on the erroneous recognition, which may cause the optical projection device 303 to project an image P that is different from the original intention. This may lead to erroneous communication of information.

[0069] To solve this problem, the above-mentioned image recognition unit 232 may be configured not to perform the first image recognition if at least a part of the image P is recognized from the corrected image IM' in the second image recognition. In other words, the image recognition unit 232 may be configured not to perform the first image recognition if the corrected image IM' corresponds to a specific image. Here, the specific image is an image in which at least a part of the image P is recognized by the image recognition unit 232 (i.e., an image in which at least a part of the image P is reflected).

[0070] According to this configuration, when the image IM acquired by the image acquisition unit 211 corresponds to a specific image (i.e., when the image IM is captured during the projection period), the generation unit 234 does not newly generate a control signal for causing the optical projection device 303 to project an image P based on the specific image. Therefore, when the image IM acquired by the image acquisition unit 211 corresponds to a specific image (i.e., when the image IM is captured during the projection period), the control unit 240 controls the optical projection device 303 so as not to newly project an image P based on the specific image. This makes it possible to prevent the projection of an image P based on an image IM that may be erroneously recognized, and to prevent the erroneous transmission of information.

[0071] <Processing performed by the information transmission system> Next, an example of processing performed in the information transmission system 100 according to this embodiment will be described. Fig. 8 is a flowchart showing an example of processing performed in the information transmission system 100 according to this embodiment. The processing shown in Fig. 8 is started, for example, when a user issues a command to start processing. For example, the command to start processing may be issued when a crane operator U' operates the operation unit 9 or when a user operates the terminal device 200.

[0072] (Step S101) First, the processing of step S101 is performed. In the processing of step S101, the image acquisition unit 211 acquires an image IM including an imaged object, which is imaged by the imaging unit 301. The imaged object may include, for example, a ground worker U, a hoisting device 22, and a suspended load L. The image acquisition unit 211 outputs the acquired image IM to the processing unit 230. Furthermore, the three-dimensional information acquisition unit 212 acquires three-dimensional information of the projection surface S acquired by the three-dimensional sensor 302. The three-dimensional information acquisition unit 212 outputs the acquired three-dimensional information of the projection surface S to the processing unit 230. After the processing of step S101 is performed, the processing of step S102 is performed.

[0073] (Step S102) In the processing of step S102, the distortion correction unit 231 calculates the shape of the projection surface S based on the output of the processing of step S101 (i.e., the three-dimensional information of the projection surface S). Then, the distortion correction unit 231 corrects the image IM based on the calculated shape of the projection surface S. After the processing of step S102 is performed, the processing of step S103 is performed.

[0074] (Step S103) In the processing of step S103, the image recognition unit 232 performs second image recognition on the output of the processing of step S102 (i.e., corrected image IM'). That is, the image recognition unit 232 inputs the corrected image IM' into the second trained model, thereby outputting the position and / or presence or absence of image P. Since the image IM is corrected in the processing of step S102 and distortion of the image P is eliminated, image recognition of the image P can be performed with high accuracy. After the processing of step S103 is performed, the processing of step S104 is performed.

[0075] (Step S104) In the processing of step S104, it is determined whether or not at least a part of the image P is reflected in the image IM (corrected image IM') based on the output of the processing of step S103. This determination may be made, for example, by the processing unit 230 (image recognition unit 232). If it is determined that at least a part of the image P is reflected in the image IM (corrected image IM') (step S104; YES), the processing of step S101 is performed again. If it is not determined that at least a part of the image P is reflected in the image IM (corrected image IM') (step S104; NO), the processing of step S105 is performed.

[0076] (Step S105) In the processing of step S105, the image recognition unit 232 performs first image recognition on the output of the processing of step S101 (i.e., image IM). That is, the image recognition unit 232 inputs the image IM into the first trained model, thereby outputting recognition target information. After the processing of step S105 is performed, the processing of step S106 is performed.

[0077] (Step S106) In the processing of step S106, the communication information output unit 233 outputs communication information based on the output of the processing of step S105 (i.e., recognition target information) and the communication information output data 222. For example, if the communication information output data 222 is for outputting communication information on a rule basis, the communication information output unit 233 outputs communication information by comparing the recognition target information output from the image recognition unit 232 with the communication information output data 222. If the communication information output data 222 is a trained model, the communication information output unit 233 outputs communication information by inputting the recognition target information output from the image recognition unit 232 into the trained model. After the processing of step S106 is performed, the processing of step S107 is performed.

[0078] (Step S107) In the processing of step S107, the generation unit 234 generates a control signal for the control unit 240 to control the optical projection device 303 based on the output of the processing of step S106 (i.e., the transmission information). As described above, the content of the control signal generated by the generation unit 234 differs depending on the content of the transmission information. After the processing of step S107 is performed, the processing of step S108 is performed.

[0079] (Step S108) In the processing of step S108, the control unit 240 controls the optical projection device 303 based on the output of the processing of step S107 (i.e., the control signal). As a result, the control unit 240 controls the optical projection device 303 based on the result of recognition by the image recognition unit 232 to cause the optical projection device 303 to project an image P. By performing the processing of step S104 described above, in the processing of step S108, projection of the image P based on the specific image (more specifically, resulting from image recognition of the imaging target from the specific image) is not performed. This makes it possible to prevent erroneous transmission of information resulting from erroneously recognizing at least a part of the image P as the imaging target.

[0080] (Step S109) In the processing of step S109, it is determined whether or not to repeat the processing of steps S101 to S108. This determination may be made by, for example, the processing unit 230. For example, it may be determined to end the repetition when a command to end the repetition is received from the user, and it may be determined not to end the repetition when a command to end the repetition is not received from the user. It may be determined to end the repetition when the work of the crane 1 has finished, and it may be determined not to end the repetition when the work of the crane 1 has not finished. For example, the image recognition unit 232 may be configured to detect the end of the work of the crane 1 through image recognition. It may be determined to end the repetition when the end of the work of the crane 1 is detected, and it may be determined not to end the repetition when the end of the work of the crane 1 is not detected.

[0081] If it is determined that the repetition should be ended (step S109; YES), the processing of the flowchart is ended. If it is determined that the repetition should not be ended (step S109; NO), the processing of steps S101 to S108 is repeated. By repeating the processing of steps S101 to S108 in this manner, it is possible to continue projecting an appropriate image P and transmit information throughout a series of operations performed by the crane 1, for example. The series of operations performed by the crane 1 may include, for example, hoisting the load L, moving the load L (i.e., traversing and / or traveling), and lowering the load L.

[0082] <Summary> According to the present embodiment described above, information is transmitted to personnel (for example, a ground worker U or a crane operator U') by the optical image P. This makes it easy to transmit information even in a noisy environment. Furthermore, the image P is not projected due to an image IM (specific image) that includes at least a portion of the image P. This makes it possible to prevent erroneous transmission of information due to the misidentification of at least a portion of the image P as the imaging target.

[0083] <Modification> The technical scope of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present invention.

[0084] For example, in the above embodiment, an example has been described in which the first image recognition is not performed when at least a portion of the image P is recognized from the image IM (corrected image IM') in the second image recognition, but this is not limiting. For example, the first image recognition may be performed when at least a portion of the image P is recognized in the second image recognition. Even in this case, the same effect as in the above embodiment can be obtained by configuring the generation unit 234 not to generate a new control signal to cause the optical projection device 303 to project the image P when at least a portion of the image P is recognized from the image IM (corrected image IM') in the second image recognition. In other words, it is possible to prevent erroneous transmission of information due to erroneous recognition of at least a portion of the image P as the imaging target.

[0085] Furthermore, in the above embodiment, the projection period and the non-projection period are detected (identified) based on the second image recognition, but this is not limited thereto. For example, the control unit 240 may detect the projection period and the non-projection period by referring to the content of the control signal from the generation unit 234. However, in cases where the image P blinks (i.e., switching between the projection period and the non-projection period) at high speed, it may be difficult to synchronize the detection of the projection period and the non-projection period based on the control signal with whether or not the image P is included in the actually acquired image IM. Therefore, the configuration of the above embodiment in which the projection period and the non-projection period are detected based on whether or not the image P is reflected in the image IM, as detected by image recognition, is more preferable.

[0086] Furthermore, in the above embodiment, the image IM is corrected based on the three-dimensional information of the projection surface S before the second image recognition is performed, but this is not limited to this. That is, the image IM does not have to be corrected based on the three-dimensional information of the projection surface S. However, if such correction is not performed, distortion corresponding to the shape of the projection surface S will remain in the image P, which may reduce the accuracy of recognition of the image P by the second image recognition. For this reason, the configuration of the above embodiment in which the image IM is corrected before the second image recognition is performed is more preferable.

[0087] The imaging direction of the imaging unit 301 and the projection direction of the image P by the optical projection device 303 can be changed as appropriate. However, capturing an image IM from above downward is more preferable because it allows capturing an overview of the work situation by the ground worker U and allows obtaining an image IM that makes it easy to grasp the work situation.

[0088] Furthermore, the terminal device 200 may be implemented using a plurality of terminal devices.

[0089] In addition, it is possible to replace the components in the above-described embodiments with well-known components as appropriate, and the above-described embodiments and variations may be combined as appropriate, without departing from the spirit of the present invention. [Explanation of symbols]

[0090] 100... Information transmission system 210... Acquisition unit 231... Distortion correction unit 232... Image recognition unit 240... Control unit 301... Image capture unit 303... Optical projection device P... Image S... Projection surface IM... Image

Claims

1. an acquisition unit that acquires an image captured by the imaging unit; an image recognition unit that recognizes a learned object from the image; and a control unit that controls the optical projection device to project an image based on a result of the recognition by the image recognition unit. Information transmission system.

2. At least a part of the projection surface onto which the image is projected is included in an imaging range of the imaging unit, the control unit controls the optical projection device so as not to project the image based on the image captured during a projection period in which at least a part of the figure is reflected in the image. The information transmission system according to claim 1 .

3. the image recognition unit has been trained to be able to recognize the image, the control unit controls the optical projection device so as not to project the image based on the specific image when the image acquired by the acquisition unit corresponds to a specific image of which at least a part is recognized by the image recognition unit. The information transmission system according to claim 2 .

4. Further comprising a distortion correction unit, the acquisition unit acquires three-dimensional information of the projection plane, the distortion correction unit calculates a shape of the projection surface based on three-dimensional information of the projection surface, and corrects the image based on the calculated shape of the projection surface; the image recognition unit recognizes the image based on the image corrected by the distortion correction unit. The information transmission system according to claim 3.

5. The imaging unit and the optical projection device are further included, the imaging unit captures the image from above downward, the optical projection device projects the image downward from above; An information transmission system according to any one of claims 1 to 4.

Images