Information processing device, information processing system, information processing method and program

Abstract

To easily identify the positional relationship between an object and a reference position.SOLUTION: A processing apparatus being an example of an information processing apparatus comprises an image creation unit being an example of image creation means that, based on a photographic image, creates an identification image indicating the position of an object included in the photographic image, and the image creation unit determines whether to create the identification image or changes a display form in displaying the identification image according to the distance between the object and a reference position.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing system, an information processing method, and a program. [Background technology]

[0002] Patent Document 1 describes a crane operation monitoring system that includes a camera that is positioned at least horizontally away from the space in which the crane hook can move and that captures images of the space at least horizontally; an identification unit that identifies the load suspended from the hook and the person contained in the image generated by the camera; a determination unit that determines whether or not a dangerous situation exists based on the positional relationship between the load and the person identified in the image; and an alarm output unit that outputs an alarm if a dangerous situation is determined to exist. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-093890 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to easily identify the positional relationship between an object and a reference position. [Means for solving the problem]

[0005] The information processing device according to the present invention comprises: Includes loads suspended by crane hooks The image generating means generates an identification image indicating the position of an object included in a photographed image based on the photographed image, and the image generating means generates an identification image indicating the distance from a reference position of the object. , and whether the load is elevated. In response to the request, the CPU 101 determines whether to generate the identification image or changes the display format when displaying the identification image. [Effects of the Invention]

[0006] According to the present invention, the positional relationship of the object to the reference position can be easily identified. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is an overall configuration diagram of a communication system according to an embodiment of the present invention; [Figure 2] FIG. 2 is a hardware configuration diagram of a processing device and a management server according to the present embodiment. [Figure 3] FIG. 1 is a hardware configuration diagram of a data acquisition device according to an embodiment of the present invention. [Figure 4] FIG. 1 is a functional block diagram of a communication system according to an embodiment of the present invention. [Figure 5] FIG. 10 is a conceptual diagram illustrating an example of a teacher data management table according to the present embodiment. [Figure 6] FIG. 10 is a sequence diagram illustrating an example of processing according to the present embodiment. [Figure 7] 10 is a flowchart illustrating an example of a task determination process according to the present embodiment. [Figure 8] 10 is a flowchart illustrating an example of an image generation process according to the present embodiment. [Figure 9] 10 is a flowchart illustrating a second example of image generation processing according to the present embodiment. [Figure 10] 10 is a flowchart illustrating a third example of an image generation process according to the present embodiment. [Figure 11] 10 is a flowchart illustrating a fourth example of an image generation process according to the present embodiment. [Figure 12] FIG. 10 is an explanatory diagram of a display image showing a state determined to be before work according to the present embodiment. [Figure 13] FIG. 10 is an explanatory diagram of a display image showing a state in which it is determined that work is in progress according to the present embodiment. [Figure 14] FIG. 10 is an explanatory diagram of a display image showing a state in which it has been determined that ground cutting is in progress according to the present embodiment. [Figure 15] FIG. 10 is an explanatory diagram of a display image showing a state in which it has been determined that a load is being suspended according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] 1 is a diagram showing the overall configuration of a communication system that is an example of an information processing system according to an embodiment of the present invention. The communication system 1 of this embodiment is constructed by a data acquisition device 9, a communication terminal 10, a processing device 30 that is an example of an information processing device, and a management server 50 that is an example of a communication terminal and an intermediary device.

[0009] The data acquisition device 9, the communication terminal 10, the processing device 30, and the management server 50 can communicate with each other via a communication network 100. The communication network 100 is constructed using the Internet, a mobile communication network, a LAN (Local Area Network), etc. The communication network 100 may include not only wired communication networks but also wireless communication networks such as 3G (3rd Generation), WiMAX (Worldwide Interoperability for Microwave Access), and LTE (Long Term Evolution).

[0010] The data acquisition device 9 and the processing device 30 are provided at the work site of the crane 210, and may each be configured integrally with the crane 210. An operator at the site can check the work content at the site using the processing device 30. Also, a manager at a remote location can check the work content at the site using the communication terminal 10.

[0011] The data acquisition device 9 includes a photographing device 7 and a distance sensor 8. The photographing device 7 is provided on the crane 210, and photographs the hook 221 of the crane 210, the cargo P, and subjects such as surrounding objects and people to acquire photographed images. The distance sensor 8 is provided on the crane 210, and measures the distance to the subject photographed by the photographing device 7 to acquire distance information.

[0012] The image capturing device 7 may capture time-lapse images or videos, in which case the distance sensor 8 acquires distance information in synchronization with the frame rate of the image capturing device 7.

[0013] The distance sensor 8 is a ToF (Time of Flight) sensor, which measures the distance from a light source to an object by irradiating the object with laser light from the light source and measuring the scattered and reflected light.

[0014] Specifically, the distance sensor 8 is a LiDAR (Light Detection and Ranging) sensor. LiDAR is a method of measuring the time of flight of light using pulses, but as another method of ToF sensors, distance may be measured using a phase difference detection method. In the phase difference detection method, a measurement range is irradiated with laser light amplitude-modulated at a fundamental frequency, and the reflected light is received and the phase difference between the irradiated light and the reflected light is measured to determine the time, and the distance is calculated by multiplying this time by the speed of light.

[0015] The data acquisition device 9 may include a stereo camera instead of the image capture device 7 and the distance sensor 8, and may acquire captured images and distance information using the stereo camera.

[0016] <Hardware configuration> 2 is a diagram showing the hardware configuration of the processing device and management server according to this embodiment. Each piece of hardware configuration in the processing device 30 is indicated by a reference number in the 300s. Each piece of hardware configuration in the management server 50 is indicated by a reference number in the 500s in parentheses. The communication terminal 10 also has a hardware configuration similar to that of the processing device 30 and management server 50, but a description thereof will be omitted.

[0017] The processing device 30 includes a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, a HD (Hard Disk) 304, a HDD (Hard Disk Drive) 305, a recording medium 306, a media I / F 307, a display 308, a network I / F 309, a keyboard 311, a mouse 312, a CD-RW (Compact Disc-Rewritable) drive 314, and a bus line 310.

[0018] Of these, the CPU 301 controls the overall operation of the processing device 30. The ROM 302 stores programs used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301. The HD 304 stores various data such as programs. The HDD 305 controls the reading and writing of various data from the HD 304 under the control of the CPU 301. The media I / F 307 controls the reading and writing (storage) of data from a recording medium 306 such as a flash memory. The display 308 displays various information such as a cursor, menus, windows, characters, and images. The network I / F 309 is an interface for data communication using the communication network 100. The keyboard 311 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The mouse 312 is a type of input means for selecting and executing various instructions, selecting processing targets, moving the cursor, etc. The CD-RW drive 314 controls the reading and writing of various data from a CD-RW 313, which is an example of a removable recording medium.

[0019] The management server 50 also includes a CPU 501, a ROM 502, a RAM 503, a HD 504, a HDD 505, a recording medium 506, a media I / F 507, a display 508, a network I / F 509, a keyboard 511, a mouse 512, a CD-RW drive 514, and a bus line 510. These components have the same configuration as the above-described components (CPU 301, ROM 302, RAM 303, HD 304, HDD 305, recording medium 306, media I / F 307, display 308, network I / F 309, keyboard 311, mouse 312, CD-RW drive 314, and bus line 310), and therefore description thereof will be omitted.

[0020] It should be noted that a CD-R drive or the like may be used instead of the CD-RW drive 314 (514). The processing device 30 and management server 50 may each be constructed by a single computer, or may be constructed by multiple computers in which each section (function, means, or storage section) is divided and arbitrarily assigned.

[0021] ○Hardware configuration of data acquisition device○ 3 is a diagram showing an example of the hardware configuration of a data acquisition device 9. The data acquisition device 9 includes the image capture device 7 and the sensor device 8 as shown in FIG. 1, as well as a controller 900 that controls the processing or operation of the data acquisition device 9.

[0022] The controller 900 includes an imaging device I / F (Interface) 901, a sensor device I / F 902, a bus line 910, a CPU (Central Processing Unit) 911, a ROM (Read Only Memory) 912, a RAM (Random Access Memory) 913, a HD (Hard Disk) 914, an HDD (Hard Disk Drive) controller 915, a network I / F 916, a DVD-RW (Digital Versatile Disk Rewritable) drive 918, a media I / F 922, an external device connection I / F 923, and a timer 924.

[0023] Of these, the imaging device I / F 901 is an interface for transmitting and receiving various data or information to and from the imaging device 7. The sensor device I / F 902 is an interface for transmitting and receiving various data or information to and from the sensor device 8. The bus line 910 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 911 shown in FIG. 3.

[0024] Furthermore, the CPU 911 controls the overall operation of the data acquisition device 9. The ROM 912 stores programs used to drive the CPU 911, such as the IPL. The RAM 913 is used as a work area for the CPU 911. The HD 914 stores various data such as programs. The HDD controller 915 controls the reading and writing of various data from and to the HD 914 under the control of the CPU 911. The network I / F 916 is an interface for data communication using the communication network 100. The DVD-RW drive 918 controls reading and writing of various data from and to a DVD-RW 917, which is an example of a removable recording medium. The medium is not limited to a DVD-RW, and may be a DVD-R or a Blu-ray (registered trademark) Disc. The media I / F 922 controls reading and writing (storing) of data from and to a recording medium 921, such as a flash memory. The external device connection I / F 923 is an interface for connecting an external device, such as an external PC 930. The timer 924 is a measuring device having a time measurement function. The timer 924 may be a computer-implemented software timer.

[0025] FIG. 4 is a functional block diagram of the communication system according to this embodiment.

[0026] <Functional configuration of the processing device> The processing device 30 has a communication unit 31, a reception unit 32, a display control unit 33, a determination unit 34, an image generation unit 35, a detection unit 36, and a storage / readout unit 39. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 2 operates in response to an instruction from the CPU 301 in accordance with a program loaded from the HD 304 onto the RAM 303. The processing device 30 also has a storage unit 3000 constructed by the RAM 303 and HD 304 shown in Fig. 2. The storage unit 3000 is an example of a storage means.

[0027] <Functional configuration of the processing device> The communication unit 31 is an example of a transmission means, and is realized by instructions from the CPU 301 shown in Figure 2 and the network I / F 309, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network 100.

[0028] The reception unit 32 is an example of a reception means, and is realized mainly by commands from the CPU 301 shown in FIG. 2, as well as the keyboard 311 and mouse 312, and receives various inputs from the user.

[0029] The display control unit 33 is an example of a display control means, and is realized by instructions from the CPU 301 shown in FIG. 2, and causes the display 308, which is an example of a display unit, to display various images and screens.

[0030] The determination unit 34 is an example of a determination means, and is realized by the processing of the CPU 301, and performs various determinations.

[0031] The image generating unit 35 is an example of an image generating means, and is realized by the processing of the CPU 301, and performs image generation processing.

[0032] The detection unit 36 ​​is an example of a detection means, and is realized by the processing of the CPU 301, and performs various detections.

[0033] 2, and by the HDD 305, media I / F 307, and CD-RW drive 314, to perform processes such as storing various data in the storage unit 3000, recording medium 306, and CD-RW 313, and reading various data from the storage unit 3000, recording medium 306, and CD-RW 313. The storage unit 3000, recording medium 306, and CD-RW 313 are examples of storage means.

[0034] The storage unit 3000 stores a teacher data management DB 3001 that is configured by a teacher data management table.

[0035] The training data management DB 3001 stores and manages training images for detecting objects from images.

[0036] <Management server functional configuration> The management server 50 has a communication unit 51, a reception unit 52, a display control unit 53, a processing unit 54, and a storage / readout unit 59. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 2 operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 onto the RAM 503. The management server 50 also has a storage unit 5000 that is constructed by the RAM 503 and HD 504 shown in Fig. 2. The storage unit 5000 is an example of storage means.

[0037] <Management server functional configuration> The communication unit 51 is an example of a transmission means, and is realized by instructions from the CPU 501 shown in Figure 2 and the network I / F 509, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network 100.

[0038] The reception unit 52 is an example of a reception means, and is realized mainly by commands from the CPU 501 shown in FIG. 2, as well as the keyboard 511 and mouse 512, and receives various inputs from the user.

[0039] The display control unit 53 is an example of a display control means, and is realized by instructions from the CPU 501 shown in FIG. 2, and causes the display 508, which is an example of a display unit, to display various images and screens.

[0040] The processing unit 54 is realized by instructions from the CPU 501 shown in FIG. 2, and performs various processes such as image generation processing.

[0041] The determination unit 55 is realized by instructions from the CPU 501 shown in FIG. 2, and makes various determinations.

[0042] The setting unit 57 is realized by commands from the CPU 501 shown in FIG. 2, and performs various settings and decisions.

[0043] 2, and is executed by the HDD 505, the media I / F 507, and the CD-RW drive 514, to store various data in the storage unit 5000, the recording medium 506, and the CD-RW 513, and to read various data from the storage unit 5000, the recording medium 506, and the CD-RW 513. The storage unit 5000, the recording medium 506, and the CD-RW 513 are examples of storage means.

[0044] In the storage unit 5000, a teacher data management DB 5001 configured by a teacher data management table is constructed.

[0045] The training data management DB 5001 stores and manages training images for detecting objects from images.

[0046] <Functional configuration of communication terminal> The communication terminal 10 has a communication unit 11, a reception unit 12, a display control unit 13, a determination unit 14, and a storage / readout unit 15. Each of these units is a function or means realized by one of the components shown in Fig. 2 being loaded from the HD onto the RAM and operating in accordance with an instruction from the CPU in accordance with a program for the communication terminal. The communication terminal 10 also has a storage unit 16 constructed by the ROM and HD as shown in Fig. 2.

[0047] <Functional configuration of communication terminal> The communication unit 11 is mainly realized by the processing of the CPU for the network I / F, and transmits and receives various data or information to and from other devices via the communication network 100.

[0048] The reception unit 12 is mainly realized by CPU processing of a keyboard or a pointing device, and receives various selections or inputs from the user.

[0049] The display control unit 13 is an example of a display control means, and is mainly realized by processing of the CPU, and causes a display, which is an example of a display unit, to display various images and screens.

[0050] The determination unit 14 is realized by processing of the CPU, and performs various determinations.

[0051] The storage / reading unit 15 is executed by instructions from the CPU as shown in FIG. 2, as well as the HDD, media I / F, and CD-RW drive, and stores various data (or information) in the storage unit 16 and reads various data (or information) from the storage unit 16.

[0052] <Functional configuration of data acquisition device> The data acquisition device 9 has a communication unit 91, a determination unit 92, an imaging device control unit 93, a sensor device control unit 94, a captured image data acquisition unit 95, a sensor data acquisition unit 96, a time data acquisition unit 97, a request acceptance unit 98, and a storage / readout unit 99. Each of these units is a function or means realized when any of the components shown in Fig. 3 operates in response to an instruction from the CPU 911 in accordance with a program for the data acquisition device that has been loaded from the HD 914 onto the RAM 913. The data acquisition device 9 also has a storage unit 9000 constructed by the ROM 912 and HD 914 shown in Fig. 3.

[0053] <Functional configuration of the data acquisition device> The communication unit 91 is mainly realized by processing of the CPU 911 on the network I / F 916, and communicates various data or information with other devices via the communication network 100. The communication unit 91 transmits, for example, data acquired by the captured image data acquisition unit 95 and the sensor data acquisition unit 96 to the processing device 30. The determination unit 92 is realized by processing of the CPU 911, and makes various determinations.

[0054] The imaging device control unit 93 is mainly realized by the processing of the CPU 911 for the imaging device I / F 901, and controls the imaging processing by the imaging device 7. The sensor device control unit 94 is mainly realized by the processing of the CPU 911 for the sensor device I / F 902, and controls the data acquisition processing for the sensor device 8.

[0055] The captured image data acquisition unit 95 is mainly realized by processing by the CPU 911 for the imaging device I / F 901, and acquires captured image data related to an image captured by the imaging device 7. The sensor data acquisition unit 96 is mainly realized by processing by the CPU 911 for the sensor device I / F 902, and acquires sensor data that is the detection result by the sensor device 8. The time data acquisition unit 97 is mainly realized by processing by the CPU 911 for the timer 924, and acquires time data that indicates the time when data was acquired by the captured image data acquisition unit 95 or the sensor data acquisition unit 96.

[0056] The request receiving unit 98 is mainly realized by the processing of the CPU 911 for the external device connection I / F 923, and receives a predetermined request from the external PC 930 or the like.

[0057] The storage / readout unit 99 is mainly realized by the processing of the CPU 911 , and stores various data (or information) in the storage unit 9000 and reads out various data (or information) from the storage unit 9000 .

[0058] FIG. 5 is a conceptual diagram showing an example of a teacher data management table according to this embodiment.

[0059] The setting information management table is a table for managing teacher images for detecting objects from images. A teacher data management DB 3001 configured with a teacher data management table such as that shown in Fig. 5 is constructed in the storage unit 3000. A teacher data management DB 5001 configured with a teacher data management table such as that shown in Fig. 5 is also constructed in the storage unit 5000. In this teacher data management table, the type name of the object to be detected and the teacher image are associated and managed for each type number.

[0060] FIG. 6 is a sequence diagram showing an example of processing according to this embodiment.

[0061] The photographing device 7 of the data acquisition device 9 is provided on the crane 210 and photographs the hook 221 of the crane 210, the cargo P, and surrounding objects, people, and other subjects to obtain photographed images, and the distance sensor 8 measures the distance to the subject photographed by the photographing device 7 to obtain distance information (step S1).

[0062] The communication unit 91 of the data acquisition device 9 transmits the acquired photographed image and distance information to the processing device 30, and the communication unit 31 of the processing device 30 receives the photographed image and distance information transmitted from the data acquisition device 9 (step S2).

[0063] Based on the captured image received in step S2, the detection unit 36 ​​detects the type of object contained in the captured image by image matching processing using the teacher image shown in the teacher data management DB 3001 (step S3).

[0064] The determination unit 34 determines the current work content based on the distance information received in step S2 and the object information indicating the object detected in step S3 (step S4).

[0065] Specifically, the determination unit 34 determines the work content, such as whether work is being carried out before or during work, whether the cargo P suspended from the hook 221 is swinging, and whether the worker is safe.

[0066] The image generating unit 35 generates an identification image indicating the position of the object included in the captured image based on the captured image and distance information received in step S2, the object information detected in step S3, and the work content determined in step S4 (step S5). Step S5 is an example of an image forming step. Details of the identification image will be described later.

[0067] The display control unit 33 causes the display 308 to display a display image in which the identification image is superimposed on the captured image (step S6). When the image capturing device 7 acquires time lapse images or videos, the display control unit 33 causes the display image to be displayed as time lapse images or videos. The display control unit 33 may also cause the display 308 to display the work content determined in step S4 together with the display image.

[0068] The communication unit 31 of the processing device 30 transmits the captured image and the identification image as separate data to the management server 50, and the communication unit 51 of the management server 50 receives the captured image and the identification image transmitted from the processing device 30 (step S7).

[0069] Here, when the photographing device 7 acquires time-lapse images or videos, the communication unit 31 of the processing device 30 transmits to the management server 50 not all of the generated identification images and the corresponding photographed images or videos, but only some of the photographed images and identification images.

[0070] As an example, the communication unit 31 transmits the captured image and the identification image to the management server 50 only when a predetermined condition is met, such as when it is determined in step S4 that the worker is not safe.

[0071] Furthermore, the communication unit 31 may transmit the work content determined in step S4 to the management server 50 in association with the photographed image and the identification image.

[0072] The processing unit 54 of the management server 50 generates a composite image by superimposing the identification image on the photographed image received in step S7, and the display control unit 53 displays the composite image on the display 508 (step S8).

[0073] The communication unit 51 of the management server 50 transmits the composite image generated in step S8 to the communication terminal 10, and the communication unit 11 of the communication terminal 10 receives the composite image transmitted from the management server 50 (step S9). The communication unit 51 of the management server 50 may transmit the work content determined in step S4 to the communication terminal 10 in association with the captured image and the identification image.

[0074] The display control unit 13 of the communication terminal 10 displays the composite image on the display (step S10).

[0075] The storage / reading unit 59 of the management server 50 updates the teacher images managed in the teacher data management DB 5001 based on new teacher images stored in the media I / F, CD-RW drive, etc., and the results of learning the captured images and identified images received in step S7 (step S11).

[0076] The communication unit 51 transmits the data of the teacher image updated in step S11 to the processing device 30, and the communication unit 31 of the processing device 30 receives the data of the teacher image transmitted from the management server 50 (step S12).

[0077] The storage / readout unit 39 of the processing device 30 updates the teacher images managed in the teacher data management DB 3001 based on the teacher image data received in step S12 (step S13). This makes it possible to make the teacher images managed in the teacher data management DB 3001 the same as the teacher images managed in the teacher data management DB 5001.

[0078] FIG. 7 is a flowchart showing an example of the task determination process according to this embodiment, and shows the process corresponding to step S4 in FIG.

[0079] The detection unit 36 ​​of the processing device 30 detects the position in the captured image of the object detected in step S3 shown in FIG. 6 (step S21).

[0080] The detection unit 36 ​​detects the three-dimensional position of the object detected in step S21 based on the distance information received in step S2 (step S22).

[0081] The detection unit 36 ​​compares the three-dimensional position of the object detected in step S22 with the three-dimensional position of the object detected during the previous work judgment for that object, and detects movement information of the object, including the amount of movement, movement speed, movement direction, etc. (step S23).

[0082] Based on the position information of the hook 221 and the luggage P detected in step S22, the judgment unit 34 judges whether the distance between them is within a predetermined range (step S24), and if it is outside the predetermined range, it determines that work has not yet begun (step S25).

[0083] If it is determined in step S24 that the distance between the hook 221 and the luggage P is within the predetermined range, the determination unit 34 determines whether the movement amount of the hook 221 or the luggage P detected in step S23 is within a threshold range (step S26).

[0084] If the determination unit 34 determines in step S26 that the amount of movement of the hook 221 or the luggage P is outside the threshold range, it determines that the luggage P is swinging (step S27).

[0085] If the judgment unit 34 determines in step S26 that the movement amount of the hook 221 or the luggage P is within the threshold range, it determines that work is in progress and determines the specific work content based on the three-dimensional position of the object detected in step S21 and the movement information detected in step S23 (step S28).

[0086] For example, the judgment unit 34 determines that the cargo P is being sling-loaded if the position of the cargo P before the work and the position of the cargo P during the work do not change, determines that the cargo P is being lifted from the ground if the position of the cargo P before the work and the position of the cargo P during the work are within a predetermined value (for example, 30 cm), determines that the cargo P is being lifted if the position of the cargo P before the work and the position of the cargo P during the work exceed a predetermined value, and determines that the cargo P is being unloaded if the direction of movement of the cargo P before the work is a downward direction.

[0087] Based on the position information of the hook 221 or luggage P detected in step S22 and the worker, the judgment unit 34 judges whether the distance between the worker and the hook 221 or luggage P is within a predetermined range (step S29), and if it is outside the predetermined range, it determines that it is safe (step S30).

[0088] On the other hand, if it is determined in step S28 that the lifting is in progress or that a load is being suspended, and the distance between the worker and the hook 221 or the load P is within a predetermined range, the determination unit 34 determines that there is danger (step S31).

[0089] FIG. 8 is a flowchart showing an example of image generation processing according to this embodiment, and shows processing corresponding to step S5 in FIG.

[0090] 7, when the determination unit 34 determines that the distance between the luggage P and the hook 221 is within a predetermined range (step S41), the image generation unit 35 generates an identification image indicating the position of the luggage P (step S42). As an example, the image generation unit 35 generates an identification image in the shape of a frame surrounding the luggage P as a first display form.

[0091] That is, the image generation unit 35 determines whether to generate an identification image depending on the distance of the baggage P from the hook 221. In this case, the baggage P is an example of an object, and the position of the hook 221 is an example of a reference position.

[0092] The display control unit then displays on the display a display image in which an identification image is superimposed on the captured image, allowing the operator or manager to easily identify whether or not the luggage P is close to the hook 221, i.e., whether or not work is in progress to attach the luggage P to the hook 221.

[0093] 7, when the determination unit 34 determines that the luggage P has risen (step S43) and determines that the distance by which the luggage P has risen is less than a predetermined value (step S44), the image generation unit 35 changes the identification image indicating the position of the luggage P to a second display form (step S45). As an example, the image generation unit 35 generates an identification image in which the inside of a frame shape surrounding the luggage P is filled in blue.

[0094] On the other hand, if the determination unit 34 determines in step S44 that the distance that the luggage P has risen is equal to or greater than the predetermined value, the image generation unit 35 changes the identification image indicating the position of the luggage P to a third display form (step S46). As an example, the image generation unit 35 generates an identification image in which the inside of a frame shape surrounding the luggage P is filled in red.

[0095] That is, the image generation unit 35 changes the display mode when displaying the identification image depending on the distance between the luggage P hung by the hook 221 and the luggage P before it was hung by the hook 221, in other words, the height of the luggage P. In this case, the luggage P hung by the hook 221 is an example of the target object, and the position of the luggage P before it was hung by the hook 221 is an example of the reference position.

[0096] The display control unit then displays on the display a display image in which an identification image is superimposed on the captured image, allowing the operator or manager to easily identify whether the height of the cargo P is within a specified range, i.e., whether the cargo is being lifted from the ground or is being suspended.

[0097] FIG. 9 is a flowchart showing a second example of the image generation process according to this embodiment.

[0098] When the determination unit 34 determines that the distance between the hook 221 and the luggage P is within a predetermined range (step S51), the image generation unit 35 generates an identification image indicating the position of the luggage P (step S52). As an example, the image generation unit 35 generates an identification image in the shape of a frame surrounding the luggage P as a first display form.

[0099] Then, when the determination unit 34 determines that the baggage P has risen (step S53) and that the distance between the baggage P and the obstacle is outside a predetermined range (step S54), the image generation unit 35 changes the identification image indicating the position of the baggage P to a second display form (step S55). As an example, the image generation unit 35 generates an identification image in which the inside of a frame shape surrounding the baggage P is filled in blue.

[0100] Here, the determination unit 34 identifies an object located on an extension of the movement direction of the luggage P as an obstacle, based on the movement information of the luggage P detected in step S23 of FIG.

[0101] On the other hand, if the determination unit 34 determines in step S54 that the distance between the baggage P and the obstacle is within the predetermined range, the image generation unit 35 changes the identification image indicating the position of the baggage P to a third display form (step S56). As an example, the image generation unit 35 generates an identification image in which the inside of a frame shape surrounding the baggage P is filled in red.

[0102] That is, the image generation unit 35 changes the display mode when displaying the identification image depending on the distance from the obstacle to the luggage P. In this case, the luggage P is an example of an object, and the position of the obstacle is an example of a reference position.

[0103] The display control unit then displays a display image on the display in which an identification image is superimposed on the captured image, allowing the operator or manager to easily identify whether the distance between the luggage P and the obstacle is within a specified range.

[0104] FIG. 10 is a flowchart showing a third example of the image generation process according to this embodiment.

[0105] When the determination unit 34 determines that the distance between the hook 221 and the luggage P is within a predetermined range (step S61), the image generation unit 35 generates an identification image indicating the position of the luggage P (step S62). As an example, the image generation unit 35 generates an identification image in the shape of a frame surrounding the luggage P as a first display form.

[0106] 7, when the determination unit 34 determines that the luggage P is shaking (step S63), the image generation unit 35 changes the identification image indicating the position of the luggage P to the second display form (step S64). As an example, the image generation unit 35 generates an identification image in which the inside of a frame shape surrounding the luggage P is filled in red.

[0107] That is, the image generation unit 35 changes the display mode when displaying the identification image depending on the distance of the luggage P before and after the movement, in other words, the swaying of the luggage P. In this case, the luggage P after the movement is an example of the target object, and the position of the luggage P before the movement is an example of the reference position.

[0108] The display control unit then displays a display image in which the identification image is superimposed on the captured image, thereby enabling the operator or manager to easily identify the swaying of the luggage P.

[0109] FIG. 11 is a flowchart showing a fourth example of the image generation process according to this embodiment.

[0110] 7, when the detection unit 36 ​​detects the three-dimensional position of the person M (step S71), the image generation unit 35 generates an identification image indicating the position of the person M (step S72). As an example, the image generation unit 35 generates an identification image in the shape of a frame surrounding the person M as the first display form.

[0111] When the determination unit 34 determines that the distance between the hook 221 and the baggage P is within a predetermined range (step S73) and determines that the distance between the person M and the hook 221 is outside the predetermined range (step S74), the image generation unit 35 changes the identification image indicating the position of the person M to a second display form (step S75). As an example, the image generation unit 35 generates an identification image in which the inside of a frame shape surrounding the person M is filled with green.

[0112] That is, the image generation unit 35 changes the display mode when displaying the identification image depending on the distance between the person M and the hook 221 of the crane 210 that attaches the luggage P or the luggage P attached to the hook 221. In this case, the person M is an example of an object, and the position of the hook 221 of the crane 210 that attaches the luggage P or the luggage P attached to the hook 221 is an example of a reference position.

[0113] The display control unit then displays on the display a display image in which an identification image is superimposed on the captured image, allowing the operator or manager to easily identify whether the distance between person M and the hook 221 of the crane 210 that attaches the cargo P, or the cargo P attached to the hook 221, is outside a specified range.

[0114] Then, if the determination result of step S74 is within a predetermined range and the determination unit 34 determines that the luggage P has not risen (step S76), the image generation unit 35 changes the identification image indicating the position of the person M to a third display form (step S77). As an example, the image generation unit 35 generates an identification image in which the inside of a frame shape surrounding the person M is filled in blue.

[0115] On the other hand, if the determination unit 34 determines in step S76 that the luggage P is rising, the image generation unit 35 changes the identification image indicating the position of the person M to a fourth display form (step S78). As an example, the image generation unit 35 generates an identification image in which the inside of a frame shape surrounding the person M is filled in red.

[0116] That is, the image generation unit 35 changes the display mode when displaying the identification image depending on the distance between the person M and the luggage P attached to the hook 221, and the distance between the luggage P attached to the hook 221 and the luggage P before it is hung by the hook 221, in other words, whether the luggage P is rising or not. In this case, the position of the distance between the luggage P before it is hung by the hook 221 is an example of the second reference position.

[0117] The display control unit then displays on the display a display image in which an identification image is superimposed on the captured image, allowing the operator or manager to easily identify whether the distance between person M and the luggage P attached to hook 221 is within a specified range and whether the luggage P is rising.

[0118] In addition, the image generation unit 35 may further change the display format of the identification image indicating the position of person M based on the distance the luggage has risen, i.e., whether the height of the luggage is above a predetermined value, in addition to whether the luggage is rising or not, and may change the display format of the identification image indicating the position of person M based on whether the height of the luggage is above a predetermined value, instead of whether the luggage is rising or not.

[0119] 12 to 15 show a state in which the display control unit 33 displays a display image in which an identification image is superimposed on a captured image on the display screen 1000 of the display 308 in step S6 in Fig. 6. Note that the display screens in steps S8 and S10 are the same as those in Fig. 12, and therefore descriptions thereof will be omitted.

[0120] FIG. 12 is an explanatory diagram of a display image showing a state determined to be before work according to this embodiment.

[0121] FIG. 12 shows a state in which the distances of the respective loads P1 to P3 from the hook 221 are outside the predetermined range, and in step S25 of FIG. 7, the determining unit 34 determines that work is not yet underway.

[0122] The display control unit 33 displays on the display screen 1000 a display image in which an identification image including the hook identification image 221W and the person identification images MW1 to MW3 is superimposed on a captured image including the hook 221, people M1 to M3, and luggage P1 to P3.

[0123] Here, the hook identification image 221W is an identification image indicating the position of the hook 221 generated by the image generating unit 35 in step S6 of FIG.

[0124] This allows operators and administrators to easily identify the location of the hook 221.

[0125] Furthermore, each of the person identification images MW1 to MW4 is an identification image indicating the position of each of the people M1 to M4 generated by the image generation unit 35 in step S72 of Figure 11, and has a first display form in the shape of a frame surrounding each of the people M1 to M4.

[0126] This allows the operator and manager to easily identify the positions of the people M1 to M4.

[0127] Then, in step S41 of FIG. 8, the determination unit 34 determines that the distances of the packages P1 to P3 from the hook 221 are outside the predetermined range, and therefore the image generation unit 35 does not generate identification images of the packages.

[0128] This allows the operator and manager to easily identify that the work of attaching the luggage P to the hook 221 has not yet begun.

[0129] FIG. 13 is an explanatory diagram of a display image showing a state in which it has been determined that slinging is in progress according to this embodiment.

[0130] Figure 13 shows a state following the state shown in Figure 12, in which the distance of the cargo P1 from the hook 221 is within a predetermined range and the cargo P is not rising, and in step S25 of Figure 7, the judgment unit 34 determines that slinging is in progress.

[0131] 8, the determination unit 34 determines that the distance between the hook 221 and the luggage P is within a predetermined range and that the luggage P is not rising, so in step S42 of Fig. 8, the image generation unit 35 generates a luggage identification image PW1 indicating the position of the luggage P1 in a first display form in the shape of a frame surrounding the luggage P1. Then, the display control unit 33 displays on the display screen 1000 a display image in which the luggage identification image PW1 is superimposed on the captured image of the luggage P1.

[0132] This allows the operator and manager to easily identify that the cargo P1 is being sling-attached to the hook 221.

[0133] Furthermore, for person M4, in step S74 of FIG. 11, the judgment unit 34 determines that the distance from the hook 221 is outside the specified range, and therefore in step S75 of FIG. 11, the image generation unit 35 changes the identification image indicating the position of person M4 to a second display form in which the inside of the frame shape is filled in green.

[0134] On the other hand, for people M1 to M3, in steps S74 and S76 of Figure 11, the judgment unit 34 judges that the distance from the hook 221 is within a predetermined range and that the object P has not risen, so in step S77 of Figure 11, the image generation unit 35 changes the identification image indicating the positions of people M1 to M3 to a third display form in which the inside of a frame shape is filled with blue.

[0135] This allows the operator and manager to easily recognize that the person M4 is located far away from the baggage P1 attached to the hook 221, but the people M1 to M3 are located close to the baggage P1.

[0136] FIG. 14 is an explanatory diagram of a display image showing a state in which it has been determined that ground cutting is in progress according to this embodiment.

[0137] Fig. 14 shows a state in which the luggage P1 has risen by less than a predetermined value following the state shown in Fig. 13, and the determination unit 34 determines that the luggage P1 is lifting off the ground in step S28 of Fig. 7. Also, in step S31 of Fig. 7, the determination unit 34 determines that there is danger.

[0138] Here, in step S44 of Figure 8, the judgment unit 34 determines that the distance that the luggage P1 has risen is less than a predetermined value, so in step S45 of Figure 8, the image generation unit 35 changes the identification image indicating the position of the luggage P1 to a second display form in which the inside of the frame shape is filled in blue.

[0139] This allows the operator and manager to easily identify that ground breaking is in progress.

[0140] Furthermore, for person M2, in steps S74 and S76 of Figure 11, the judgment unit 34 determines that the distance from the hook 221 is within a predetermined range and that the luggage P1 has risen, so in step S78 of Figure 11, the image generation unit 35 changes the identification image indicating the position of person M2 to a fourth display form in which the inside of a frame shape is filled in red.

[0141] On the other hand, for people M1 and M3, in step S74 of Figure 11, the judgment unit 34 determines that the distance from the hook 221 is outside the specified range, so in step S75 of Figure 11, the image generation unit 35 changes the identification image indicating the positions of people M1 and M3 to a second display form in which the inside of the frame shape is filled in green.

[0142] This allows the operator and manager to easily identify that persons M1, M3, and M4 are in safe positions relative to the load P1 attached to the hook 221, but that person M2 is in a dangerous position.

[0143] FIG. 15 is an explanatory diagram of a display image showing a state in which it has been determined that a load is being suspended according to this embodiment.

[0144] Fig. 15 shows a state in which the distance that the cargo P1 has risen is equal to or greater than a predetermined value, following the state shown in Fig. 14, and the determination unit 34 determines that the cargo is being suspended in step S28 of Fig. 7. Also, in step S30 of Fig. 7, the determination unit 34 determines that it is safe.

[0145] Here, in step S44 of Figure 8, the judgment unit 34 determines that the distance that the luggage P1 has risen is greater than or equal to a predetermined value, so in step S46 of Figure 8, the image generation unit 35 changes the identification image indicating the position of the luggage P1 to a third display form in which the inside of the frame shape is filled in red.

[0146] This allows the operator and manager to easily identify that a load is being lifted.

[0147] Furthermore, for person M2, in step S74 of FIG. 11, the judgment unit 34 determines that the distance from the hook 221 is outside the specified range, and therefore in step S75 of FIG. 11, the image generation unit 35 changes the identification image indicating the position of person M4 to a second display form in which the inside of the frame shape is filled in green.

[0148] This allows the operator and manager to easily identify that the people M1 to M4 are in a safe position relative to the baggage P1 attached to the hook 221.

[0149] ●Summary● As described above, the processing device 30, which is an example of an information processing device according to one embodiment of the present invention, includes the image generation unit 35, which is an example of an image generation means that generates, based on the captured image, an identification image indicating the position of an object included in the captured image, and the image generation unit 35 determines whether to generate an identification image or changes the display format when displaying the identification image, depending on the distance of the object from a reference position.

[0150] This makes it easy to identify whether the object is close to the reference position.

[0151] When the object is a person M and the reference position is the hook 221 of the crane 210 attaching the cargo P or the position of the cargo P attached to the hook 221, it is possible to easily identify whether the person M is close to the cargo P attached to the hook 221.

[0152] When the object is a load P and the reference position is the position of the hook 221 of the crane 210, it is possible to easily identify whether or not work is being performed to attach the load P to the hook 221.

[0153] When the object is a cargo P suspended by the hook 221 of the crane 210 and the reference position is the position of the cargo P before it is suspended by the hook 221, the height of the cargo P suspended by the hook 221 can be easily identified.

[0154] The image generating unit 35 generates the identification image so that the display form varies depending on the distance of the object from the reference position and the distance of the reference position from the second reference position.

[0155] This makes it easy to identify whether the object is close to the reference position and whether the reference position is close to the second reference position.

[0156] When the object is a person M, the reference position is the position of the luggage P suspended by the hook 221 of the crane 210, and the second reference position is the position of the luggage P before it is suspended by the hook 221, it is easy to identify whether the person M is close to the luggage P suspended by the hook 221 and whether the luggage P suspended by the hook 221 has risen.

[0157] The image capturing device 100 includes a display control unit 33, which is an example of a display control means, that displays the captured image and a display image including the identification image on a display 308, which is an example of a display unit.

[0158] This makes it easy to determine from the display content of the display 308 whether the object is close to the reference position or not.

[0159] The processing device 3 includes a communication unit 31, which is an example of a transmission means for transmitting the captured image and the identification image to a management server 50, which is an example of a communication terminal capable of communicating with the processing device 3.

[0160] This allows the management server 50 to display a display image including the captured image and the identification image, and if the captured image and the identification image are sent separately, the management server 50 can learn the position of the object shown in the identification image.

[0161] The image generating unit 35 may generate a composite image including the photographed image and the identification image, and the communication unit 31 may transmit the composite image to the management server 50.

[0162] A communication system 1, which is an example of an information processing system according to one embodiment of the present invention, includes a processing device 3 and a management server 50, which is an example of a communication terminal capable of communicating with the processing device 3. The processing device 3 includes an image generation unit 35 that generates an identification image indicating the position of an object contained in the captured image based on the captured image, and a communication unit 31 that transmits the captured image and the identification image to the management server 50. The image generation unit 35 determines whether to generate an identification image or changes the display format when displaying the identification image depending on the distance from a reference position of the object. The management server 50 includes a communication unit 51, which is an example of a receiving means that receives the captured image and the identification image, and a display control unit 53, which is an example of a display control means that displays a display image including the captured image and the identification image on a display 508, which is an example of a display unit.

[0163] A communication system 1, which is another example of an information processing system according to one embodiment of the present invention, includes a processing device 3, a management server 50 which is an example of an intermediary device capable of communicating with the processing device 3, and a communication terminal 10 which is capable of communicating with the intermediary device. The processing device 3 includes an image generation unit 35 which generates an identification image indicating the position of an object contained in the captured image based on the captured image, and a communication unit 31 which transmits the captured image and the identification image to the management server 50. The image generation unit 35 determines whether to generate an identification image or changes the display format when displaying the identification image depending on the distance from a reference position of the object. The management server 50 includes a communication unit 51 which is an example of a transmitting means which receives the captured image and the identification image and transmits a composite image including the captured image and the identification image to the communication terminal 10. The communication terminal 10 includes a communication unit 11 which is an example of a receiving means which receives the composite image, and a display control unit 13 which is an example of a display control means which displays the composite image on a display 508 which is an example of a display unit.

[0164] An information processing method according to one embodiment of the present invention includes an image generation step of generating an identification image indicating the position of an object contained in a captured image based on the captured image, and the image generation step includes a process of determining whether to generate an identification image or changing the display format when displaying the identification image depending on the distance of the object from a reference position.

[0165] A program according to an embodiment of the present invention causes a computer to execute the above information processing method. [Explanation of symbols]

[0166] 1. Communication systems (examples of information processing systems) 100 Communication Network 7. Imaging equipment 8 Distance Sensor 9 Data acquisition equipment 210 Crane 221 Hook 221W Hook frame 10. Communication terminals 11 Communication unit (an example of a receiving means) 12 Reception unit (an example of reception means) 13 Display control unit (an example of a display control means) 14. Determination unit (an example of a determination means) 15 Memory / readout unit (an example of memory control means) 16 Memory unit (an example of memory means) 30 Processing device (an example of an information processing device) 31 Communication unit (an example of a transmission means) 32 Reception unit (an example of reception means) 33 Display control unit (an example of a display control means) 34 Determination unit (an example of a determination means) 35 Image generation unit (an example of image generation means) 36 Detection unit (an example of a detection means) 39 Memory / read unit (an example of memory control means) 308 Display (Example of display unit) 3000 storage section 3001 Teacher data management DB (an example of teacher data management means) 50 Management server (an example of a communication terminal or intermediary device) 51 Communication unit (an example of a transmitting means or a receiving means) 52 Reception unit (an example of reception means) 53 Display control unit (an example of display control means) 54 Processing unit (an example of processing means) 59 Memory / readout unit (an example of memory control means) 508 Display (an example of a display unit) 5000 storage section 5001 Teacher data management DB (an example of a teacher data management means) 1000 display screen P, P1~P3 Luggage P1W~P3W Baggage frame (example of identification information) M1~M3 people M1W~M3W Person frame (example of identification information)

Claims

1. An image generating means for generating, based on a photographed image including a load suspended by a crane hook, an identification image indicating the position of an object included in the photographed image, The image generation means determines whether to generate the identification image or changes the display format when displaying the identification image depending on the distance of the object from a reference position and whether the luggage is rising.

2. 2. The information processing apparatus according to claim 1, wherein the object is a person, and the reference position is a hook of a crane suspending a load or a position of the load suspended by the hook.

3. 2. The information processing device according to claim 1, wherein the object is a load, and the reference position is the position of a hook of a crane.

4. 2. The information processing apparatus according to claim 1, wherein the object is a load suspended by a hook of a crane, and the reference position is the position of the load before it is suspended by the hook.

5. An information processing device described in any one of claims 1 to 4, wherein the image generation means generates the identification image so that it has a different display form depending on the distance of the object from the reference position and the distance of the reference position from a second reference position.

6. 6. The information processing device according to claim 5, wherein the object is a person, the reference position is the position of a load suspended by a hook of a crane, and the second reference position is the position of the load before it was suspended by the hook.

7. 7. The information processing apparatus according to claim 1, further comprising a display control means for displaying, on a display unit, a display image including the photographed image and the identification image.

8. 8. The information processing apparatus according to claim 1, further comprising a transmitting means for transmitting the captured image and the identification image to a communication terminal capable of communicating with the information processing apparatus.

9. An information processing system including an information processing device and a communication terminal capable of communicating with the information processing device, The information processing device includes: an image generating means for generating, based on a photographed image including a load suspended by a hook of a crane, an identification image indicating the position of an object included in the photographed image; a transmitting means for transmitting the captured image and the identification image to the communication terminal, the image generating means determines whether to generate the identification image or changes the display format of the identification image depending on the distance of the object from a reference position and whether the luggage is rising; The communication terminal receiving means for receiving the photographed image and the identification image; a display control means for displaying a display image including the photographed image and the identification image on a display unit; An information processing system comprising:

10. An information processing system including an information processing device, an intermediary device capable of communicating with the information processing device, and a communication terminal capable of communicating with the intermediary device, The information processing device includes: an image generating means for generating, based on a photographed image including a load suspended by a hook of a crane, an identification image indicating the position of an object included in the photographed image; a transmitting means for transmitting the captured image and the identification image to the intermediary device, the image generating means determines whether to generate the identification image or changes the display format of the identification image depending on the distance of the object from a reference position and whether the luggage is rising; The intermediary device receiving means for receiving the photographed image and the identification image; a transmitting means for transmitting a composite image including the captured image and the identification image to the communication terminal, The communication terminal receiving means for receiving the composite image; a display control means for displaying the composite image on a display unit; An information processing system comprising:

11. An image generation step of generating an identification image indicating the position of an object included in a photographed image including a load suspended by a hook of a crane, based on the photographed image, The image generation step is an information processing method that includes a process of determining whether to generate the identification image or changing the display format when displaying the identification image depending on the distance of the object from a reference position and whether the luggage is rising.

12. A program for causing a computer to execute the information processing method according to claim 11.

Images