Information processing systems and information processing programs

Abstract

[Problem] To provide an information processing system capable of coordinating data of a plurality of terminal devices without using a time stamp. [Solution] An information processing system 1 comprising: a mobile camera 100 including a wireless communication unit 101 for performing communication by means of a fifth-generation mobile communication system 200, a sensor including a camera, and a clocking unit for receiving a signal from a global positioning satellite system to obtain clocking information; a server 400 for performing communication with the mobile camera 100 via the fifth-generation mobile communication system 200; and a data processing unit 402 which, if a communication delay between the mobile camera 100 and the server 400 is less than or equal to a predetermined time, coordinates a plurality of data items, assuming that the data of the mobile camera 100 is of the same clock time.

Description

Technical Field

[0001] The present invention relates to an information processing system and an information processing program.

Background Art

[0002] In recent years, a system has been constructed in which a plurality of terminal devices are connected by a network and data is transmitted and received between them.

[0003] When attempting to link the data of a plurality of terminal devices, each terminal device needs to add timekeeping information as a timestamp to the data and align the times of the plurality of data using the timestamp.

[0004] Also, as a technique for synchronizing between a plurality of devices, for example, there is Patent Document 1. The technique of Patent Document 1 uses a time signal from a Global Navigation Satellite System (GNSS) as a common synchronization clock in mobile communication, which is one of the representative networks, among a plurality of mobile communication base stations.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, while conventional technology can achieve phase synchronization between base stations in mobile communications, it does not consider the synchronization of multiple terminal devices connected to the base station. Therefore, with conventional technology, when linking data from multiple terminal devices, pre-processing to align the timestamps attached to the data on each terminal device is still necessary. As a result, conventional technology has the problem that pre-processing takes time and data cannot be linked immediately.

[0007] Therefore, the object of the present invention is to provide an information processing system and information processing program that can link data from multiple terminal devices without using timestamps. [Means for solving the problem]

[0008] The above objectives of the present invention are achieved by the following means.

[0009] (1) A terminal device having a wireless communication unit that performs communication using a fifth-generation mobile communication system, sensors including a camera, and a timing unit that receives signals from a global positioning satellite system and converts them into timing information, A server that communicates with the terminal device via the fifth-generation mobile communication system, If the communication delay between the terminal device and the server is less than or equal to a predetermined time, the data from the terminal device is treated as data from the same time, and a data processing unit links multiple pieces of data together. An information processing system having the following features.

[0010] (2) The information processing system according to (1) above, wherein the data processing unit selects from among a plurality of terminal devices a terminal device to link the data with based on the location information of each terminal device obtained from the global positioning satellite system.

[0011] (3) The information processing system according to (1) or (2) above, wherein the data is image data captured by the camera.

[0012] (4) The timing unit is an information processing system according to any one of (1) to (3) above, which updates the timing information within the predetermined time.

[0013] (5) The above data is image data, The information processing system according to any one of (1) to (4) above, wherein the predetermined time is less than the time of one frame.

[0014] (6) A terminal device having sensors including a camera and receiving signals from a global positioning satellite system as timing information, and a server, communicate via a fifth-generation mobile communication system. If the communication delay between the terminal device and the server is less than or equal to a predetermined time, the data from the terminal device is considered to be data from the same time, and the process involves linking multiple pieces of data. An information processing program that causes a computer to execute something.

[0015] (7) The information processing program according to (6) above, wherein the step of linking the multiple data sets includes a step of selecting a terminal device from among the multiple terminal devices based on the location information of each terminal device obtained from the global positioning satellite system to link the data.

[0016] (8) The data is image data captured by the camera, as described in (6) or (7) above, the information processing program.

[0017] (9) An information processing program according to any one of (6) to (8) above, which has a step of updating the timing information within the predetermined time.

[0018] (10) The above data is image data, The information processing program described in any one of (6) to (9) above, wherein the predetermined time is less than the time of one frame. [Effects of the Invention]

[0019] In the present invention, the terminal device and the server communicate with each other via a fifth-generation communication system, and the terminal device performs internal timing based on signals from GNSS. As a result, the present invention can associate data obtained from a plurality of terminal devices without using a time stamp.

Brief Description of Drawings

[0020] [Figure 1] It is a block diagram showing a schematic configuration of an information processing system according to an embodiment. [Figure 2] It is a block diagram for explaining the functions of the information processing system according to an embodiment. [Figure 3] It is a block diagram showing an example of the hardware configuration of a mobile camera. [Figure 4] It is a block diagram showing an example of the hardware configuration of a fixed camera. [Figure 5] It is a block diagram showing an example of the hardware configuration of a server. [Figure 6] It is a flowchart showing the procedure of processing by a server.

Embodiments for Carrying Out the Invention

[0021] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same reference numerals are given to the same elements, and redundant descriptions are omitted. Also, the dimensional ratios in the drawings are exaggerated for the convenience of explanation and may be different from the actual ratios.

[0022] (Information Processing System) FIG. 1 is a block diagram showing a schematic configuration of an information processing system according to an embodiment. FIG. 2 is a block diagram for explaining the functions of the information processing system according to an embodiment.

[0023] As shown in FIGS. 1 and 2, the information processing system 1 according to the embodiment includes a mobile camera 100, a fifth-generation mobile communication system 200 (also referred to as a high-speed mobile communication system), a fixed camera 300, and a server 400.

[0024] (Mobile camera) The mobile camera 100 is the first terminal device. This embodiment has a plurality of mobile cameras 100a, 100b, and 100c. In this embodiment, the plurality of mobile cameras 100a, 100b, and 100c are simply referred to as mobile camera 100 unless otherwise distinguished or when referring to them collectively.

[0025] The mobile camera 100 is connected to the server 400 via the 5G communication system 200. Therefore, the mobile camera 100 is also referred to as an edge device or edge terminal, and is an electronic device that can be used in IoT (Internet of Things). The mobile camera 100 is, for example, a handheld camera, and is a camera that can be moved freely. Alternatively, the mobile camera 100 may be a mobile terminal device with camera functionality, such as a smartphone or tablet computer. These mobile cameras 100 are moved manually. The mobile camera 100 may also be mounted on a vehicle that is moved under human control.

[0026] Furthermore, the mobile camera 100 may be configured as an autonomously mobile device. The autonomously mobile camera 100 moves, for example, when its destination location is instructed by the server 400. The autonomously mobile camera 100 may be, for example, a robot, an aerial drone, or an underwater drone. Alternatively, the autonomously mobile camera 100 may be attached to a robotic arm and moved by the robotic arm.

[0027] The mobile camera 100 may be a visible light camera that takes pictures using visible light, as well as a night vision camera or an infrared camera (including a thermal camera).

[0028] As shown in Figure 2, the functions of the mobile camera 100 include a wireless communication unit 101, a first camera control unit 102, a first camera imaging unit 103, a first GNSS unit 105, a first camera clock unit 106, and storage 107.

[0029] The wireless communication unit 101 performs 5G communication with the 5G communication system 200. The wireless communication unit 101 transmits data from the mobile camera 100 to the server 400 via the 5G communication system 200. Here, the data is mainly image data captured by the mobile camera 100. The wireless communication unit 101 also receives data transmitted from the server 400 via the 5G communication system 200.

[0030] The first camera control unit 102 controls the mobile camera 100. The first camera control unit 102 transmits the image data captured by the first camera imaging unit 103 to the server 400 via the wireless communication unit 101.

[0031] In this embodiment, the integration and analysis of multiple data sets using artificial intelligence (AI) is performed by a server 400, which will be described later. For the integration and analysis of multiple data sets using AI, a machine learning model (AI model) that has been pre-trained by AI is used.

[0032] However, the AI ​​may perform the integration and analysis of multiple data sets. In that case, the first camera control unit 102 becomes a computer that causes the mobile camera 100 to perform specific functions.

[0033] When the first camera control unit 102 is instructed to perform AI-based integration and analysis of multiple data sets, the program for these tasks is sent from the server 400 in response to a request from the first camera control unit 102 or based on a decision made by the server 400. The program sent from the server 400 is stored in the memory or storage 107 within the FPGA (described later), read out as needed, and various analyses and processes are executed. Therefore, when performing AI-based integration and analysis of multiple data sets, the first camera control unit 102 functions as a data processing unit.

[0034] The AI ​​model is provided as a control program and / or logic data, based on the hardware configuration of the mobile camera 100. The hardware configuration of the mobile camera 100 will be described later, but for example, if the hardware configuration is CPU-based, it is provided as a control program. If the hardware configuration is rewritable hardware such as an FPGA, it is provided as logic data. Logic data is sometimes referred to as programmable logic. In the case of an FPGA, some or all of the settings may be provided as a control program.

[0035] The first camera imaging unit 103 is equipped with an image sensor for shooting video. The image sensor may be, for example, a visible light image sensor (video recording camera) or an infrared image sensor. The first camera imaging unit 103 captures visible light images, infrared images, etc., using these image sensors. The first camera imaging unit 103 captures images according to the timing of the clock signal supplied from the first camera clock unit 106. The captured video is transmitted as image data to the server 400 via the wireless communication unit. Alternatively, the captured images may be stored as image data in the storage 107.

[0036] The first camera imaging unit 103 may be equipped with a zoom lens. The magnification of the zoom lens is changed by the control of the first camera control unit 102. Alternatively, the magnification of the zoom lens may be changed by a person (user) as appropriate.

[0037] The mobile camera 100 may be equipped with sensors other than a camera (image sensor). These sensors may include, for example, acoustic sensors that detect sound (such as microphones), altitude sensors that detect altitude (elevation), barometric pressure sensors, underwater depth sensors (water pressure sensors), vibration sensors, compass sensors, angle sensors, temperature sensors, voltage sensors, current sensors, and power sensors. The data detected by these sensors is transmitted to the server 400 as needed. The data detected by the sensors is also stored in the storage 107 as needed.

[0038] The first GNSS unit 105 is, as is well known, a satellite-based positioning system. The first GNSS unit 105 recognizes the coordinates of the current position of the mobile camera 100. The first GNSS unit 105 is equipped with a GNSS receiver that receives signals (radio waves) in Coordinated Universal Time (UTC) from GNSS satellites.

[0039] The first GNSS unit 105 transmits the positioning results to the first camera control unit 102. This allows the first camera control unit 102 to determine the current position of the mobile camera 100 and, if necessary, transmit the current position to the server 400. At this time, the first camera control unit 102 also transmits the shooting direction of the mobile camera 100 to the server 400.

[0040] Examples of global positioning satellite systems include the US GPS (Global Positioning System), Japan's Quasi-Zenith Satellite System (QZSS), Russia's GLONASS, and the European Union's Galileo.

[0041] The first camera clock unit 106 is a timing unit. The first camera clock unit 106 generates a clock signal used for control within the mobile camera 100 from the UTC signal received by the first GNSS unit 105. The clock signal is timing information (also called time information). The clock signal is generated from the UTC signal received by the first GNSS unit 105. The generated clock signal has an error of less than 1 μsec, and the time obtained from the clock signal also has a similar error with respect to UTC. Therefore, image data captured by the mobile camera 100 is captured at timing based on the UTC signal, transmitted to the server 400, and stored in storage 107 or the like as needed.

[0042] The first camera clock unit 106 updates its clock signal at predetermined intervals using the UTC signal obtained from the first GNSS unit 105. The predetermined interval is preferably less than the time of one frame, for example, in accordance with the frame rate of the image data (video). Specifically, for example, in the case of 30fps, the predetermined interval is preferably less than 33msec. When multiple image data are linked, if the time difference of 30fps image data is 33msec or more, a synchronization error of one or more frames will occur. Therefore, in order to prevent time differences in the image data, it is preferable to update the generated clock signal in accordance with UTC at intervals of less than the time of one frame, as described above.

[0043] Storage 107 is a memory unit. Storage 107 stores image data. In addition, if the AI ​​model is running on the mobile camera 100, storage 107 may store the control program and / or logical data of the AI ​​model.

[0044] Storage 107 is, for example, a storage medium such as eMMC (embedded MultiMediaCard), SSD (Solid State Drive), or HDD (Hard Disk Drive). Alternatively, storage 107 may be a portable storage medium such as a memory card.

[0045] Furthermore, it is not necessary to store all of the above data in storage 107. The capacity of storage 107 should be determined according to the cost of the mobile camera 100, and the content to be stored should be determined and changed accordingly. Also, storage 107 does not have to be provided.

[0046] Figure 3 is a block diagram showing an example of the hardware configuration of the mobile camera 100.

[0047] In this embodiment, the case of FPGA will be used as an example.

[0048] As shown in Figure 3, the mobile camera 100 includes a SoCFPGA 110, a first camera imaging unit 103, a 5G communication interface 150, an operation display unit 160, a first GNSS unit 105, a first camera clock unit 106, and storage 107. Each unit is connected by a bus 180. The first camera imaging unit 103, the first GNSS unit 105, the first camera clock unit 106, and storage 107 have already been described.

[0049] The SoCFPGA110 primarily functions as the first camera control unit 102. The SoCFPGA110 is a semiconductor device (including semiconductor devices formed by joining multiple chips) in which an FPGA whose processing content can be rewritten is formed as a system on a single chip. The SoCFPGA110 is sometimes referred to as a Programmable SoC. The SoCFPGA110 has functions such as a CPU (Central Processing Unit) which is an arithmetic element, and memory elements such as ROM (Read Only Memory) and RAM (Random Access Memory) which are formed on a single chip (or multiple chips having multiple of these functions are integrated). The SoCFPGA110 may also be equipped with accelerators such as a GPU / DSP. For this reason, the mobile camera 100 is a computer.

[0050] Such a SoCFPGA110 stores the control programs and / or logic data necessary for operation (including rewriting gate circuits in the FPGA), and when these are executed, the functions of each part of the mobile camera 100 are realized. In addition, the SoCFPGA110 executes AI processing when the logic data necessary for AI processing is written to it.

[0051] The 5G communication interface 150 is a wireless communication unit 101 for communicating with the 5G communication system 200, and is composed of a communication module chip. The 5G communication interface 150 may also be integrated as SoCFPGA110. In addition to the 5G communication interface, the mobile camera 100 may also be provided with network interfaces such as Ethernet®, IEEE1394, and wireless communication interfaces such as Bluetooth® and IEEE802.11.

[0052] The operation display unit 160 is, for example, a touch panel display that displays various information and accepts various inputs from the user. The mobile camera 100 may also be configured with input buttons and a monitor. The mobile camera 100 may also be configured so that an image viewer functions as the display unit.

[0053] Furthermore, the mobile camera 100 is not limited to SoCFPGA110; it may also be an FPGA other than an SoC, or it may be configured with an independent CPU, RAM, and ROM connected by a bus.

[0054] (Communication system) The 5G communication system 200 has wireless communication capabilities using 5G and controls communication between the mobile camera 100 and the server 400.

[0055] The 5G communication system 200 is a known 5G communication system, and although not shown in the figures, it has wireless communication control functions, relay processing functions, etc., and connects the mobile camera 100 and the server 400 via 5G communication.

[0056] The 5G communication system 200 of this embodiment is referred to as private 5G, local 5G, etc., and is used only by specific users. Furthermore, it is preferable that multiple terminal devices are connected to the same base station along with the server 400 in the 5G communication system 200 of this embodiment.

[0057] In a 5G communication system, communication delay within the same base station is guaranteed to be less than 10 msec. The time for one frame of image data is 33 msec at a frame rate of 30 fps and 16 ms at 60 fps. Therefore, when multiple terminal devices transmit image data to server 400 via the 5G communication system, the communication delay will be less than the time for one frame.

[0058] (Fixed camera) The fixed camera 300 is a second terminal device. This embodiment has a plurality of fixed cameras 300a, 300b, and 300c. In this embodiment, the plurality of fixed cameras 300a, 300b, and 300c are simply referred to as fixed camera 300 unless otherwise distinguished or referred to collectively.

[0059] The fixed camera 300 is connected to the server 400. The fixed camera 300's shooting operation is controlled, for example, from the server 400 or from a control computer other than the server 400. The fixed camera 300 may also be able to change its orientation (up / down and left / right pan angles). It is preferable that the orientation of the fixed camera 300 be controlled remotely. The fixed camera 300 may also be equipped with a zoom function.

[0060] The fixed camera 300 may be a visible light camera that takes pictures using visible light, as well as a night vision camera or an infrared camera (including a thermal camera).

[0061] As shown in Figure 2, the fixed camera 300 has the following functions: a wired communication unit 301, a second camera control unit 302, a second camera imaging unit 303, a second GNSS unit 305, and a second camera clock unit 306.

[0062] Since each part of these fixed cameras 300 has the same functions as the first camera control unit 102, first camera imaging unit 103, first GNSS unit 105, and first camera clock unit 106 of the first camera described earlier, a detailed explanation of these will be omitted.

[0063] Therefore, the second camera imaging unit 303 of the fixed camera 300 captures video at the timing of a clock signal based on the UTC signal, similar to the mobile camera 100. It is preferable that the frame rate and resolution of the image data be the same as those of the mobile camera 100. By making the frame rate and resolution the same, they can be coordinated without causing processing delays. The frame rate and resolution of the image data may be different from those of the mobile camera 100. If the frame rate and / or resolution differ between the fixed camera 300 and the mobile camera 100, it is preferable to perform processing to match them before executing AI processing or 3D visualization.

[0064] The fixed camera 300 has a wired communication unit 301 for connection with the server 400. The fixed camera 300 may also be equipped with a wireless communication unit instead of, or together with, the wired communication unit 301, and connect to the server 400 by wireless communication.

[0065] The second camera control unit 302 of the fixed camera 300 transmits the captured image data to the server 400 via the wired communication unit 301.

[0066] Furthermore, the fixed camera 300 may also have an AI-based analysis processing function, similar to the mobile camera 100.

[0067] Figure 4 is a block diagram showing an example of the hardware configuration of the fixed camera 300.

[0068] The fixed camera 300 is also configured using an FPGA. As shown in Figure 4, the fixed camera 300 has a SoCFPGA 310, a second camera imaging unit 303, a communication interface 370, a second GNSS unit 305, and a second camera clock unit 306. Each unit is connected by a bus 380. These hardware components are the same as those of the mobile camera 100, so their description is omitted. In this embodiment, the image data captured by the fixed camera 300 is always transmitted to the server 400. For this reason, the fixed camera 300 does not have a storage 107. However, it is not limited to this, and the fixed camera 300 may also be provided with a storage 107.

[0069] The communication interface 370 functions as a wired communication unit 301. The communication interface 370 may be, for example, a network interface conforming to standards such as Ethernet® (wired LAN (Local Area Network)), PCI Express, USB, IEEE1394, or HDMI® (High-Definition Multimedia Interface). Alternatively, wireless communication interfaces other than 5G, such as Bluetooth® or IEEE802.11, may be used for the communication interface 370. Furthermore, a 5G communication system may also be used for communication between the fixed camera 300 and the server 400.

[0070] (server) Server 400 integrates image data transmitted from the mobile camera 100 and / or the fixed camera 300.

[0071] The server 400 includes a wireless communication unit 401, a data processing unit 402, an image data receiving unit 404, a GNSS unit 405, a server clock unit 406, and a server storage unit 407.

[0072] The wireless communication unit 401 performs 5G communication with the 5G communication system 200. The wireless communication unit 401 transmits data from the server 400 to the mobile camera 100 via the 5G communication system 200. The wireless communication unit 401 also receives data transmitted from the mobile camera 100 via the 5G communication system 200.

[0073] In this embodiment, the data processing unit 402 links multiple image data to create an AI-generated 3D visualization image (video or still image). In this embodiment, the data to be linked are image data received from the mobile camera 100 and / or image data received from the fixed camera 300.

[0074] The data processing unit 402 synchronizes these multiple image data in their received state without any preprocessing such as aligning the timing information. As already explained, the multiple image data are captured based on the UTC signal received from GNSS. Therefore, even if the multiple image data are used as is, without preprocessing such as aligning timestamps as in the conventional method, they will be synchronized.

[0075] Furthermore, the data processing unit 402 performs AI-based analysis on the image data. This AI analysis includes, for example, recognizing people or objects within an image (frame), recognizing faces and skeletons of people, recognizing the movement of people (or objects), and determining the attributes of people (or objects). Here, "object" includes gas (airflow), flames, water flows, etc. The AI ​​model may be modified depending on the processing content.

[0076] The GNSS unit 405 and server clock unit 406 in server 400 generate a clock signal for use within server 400 from the GNSS UTC signal, similar to the mobile camera 100 and the fixed camera 300. Therefore, their functions are the same as those of the first GNSS unit and the first clock unit already described, and a detailed explanation will be omitted. In server 400 as well, by generating a clock signal from the GNSS UTC signal, synchronization between the terminal device and the entire information processing system 1 can be achieved without performing any processing to align the time axis.

[0077] Furthermore, it is not necessary to use the GNSS UTC signal to link multiple image data. When image data is linked, the multiple image data itself is synchronized by using the GNSS UTC signal. Therefore, linking multiple image data is possible even without synchronization with the clock signal within server 400.

[0078] The server storage 407 stores the data processed by the data processing unit 402. The server storage 407 also stores image data received from the mobile camera 100 and / or the fixed camera 300.

[0079] The image data receiving unit 404 receives image data from the fixed camera 300.

[0080] Figure 5 is a block diagram showing an example of the hardware configuration of Server 400.

[0081] Server 400 is a computer. As shown in Figure 5, Server 400 has a CPU 410, ROM 420, RAM 430, server storage 407, 5G communication interface 450, operation display unit 460, communication (non-5G) interface 470, GNSS unit, and server clock unit 406. Each unit is connected by a bus 480. The GNSS unit and server clock unit 406 have already been described.

[0082] The CPU 410 executes the functions of each part of the server 400 described above by running programs recorded in the ROM 420 and the server storage 407.

[0083] ROM420 stores various programs and data.

[0084] RAM430 temporarily stores programs and data as a working area.

[0085] Server storage 407 stores various programs, including the operating system, and various data. Server storage 407 also stores control programs and / or logical data.

[0086] For server 400, server storage 407 primarily uses high-capacity storage media such as HDDs. Alternatively, semiconductor storage media such as eMMC or SSDs may be used as server storage 407, either alongside or in place of HDDs.

[0087] The 5G communication interface 450 is a wireless communication unit 401 for communicating with the 5G communication system 200.

[0088] The operation display unit 460 is, for example, a touch panel display that displays various information and accepts various inputs from the user. The operation display unit 460 may also be connected to input devices such as a keyboard or mouse, and a monitor.

[0089] The communication (non-5G) interface 470 is the image data receiving unit 404. The communication (non-5G) interface 470 uses an interface of the same standard as the communication interface 370 of the fixed camera 300. The communication (non-5G) interface 470 may also be used to connect other computers other than the fixed camera 300.

[0090] (Processing procedure) Next, the processing procedure in the embodiment will be described.

[0091] The processing procedure is described below based on the following assumptions. Information processing system 1 is a gas leak monitoring system. The mobile camera 100 and fixed camera 300 used are infrared cameras capable of capturing images of gases with different components from air that have leaked into the air, particularly flammable gases. Server 400 uses AI processing of image data to determine whether or not a gas with a different component from air has leaked, and links the image data from mobile camera 100 and fixed camera 300. Of course, information processing system 1 in this embodiment is not limited to a gas leak monitoring system.

[0092] Figure 6 is a flowchart showing the processing procedure performed by server 400.

[0093] First, the server 400 acquires image data from the fixed camera 300 at a designated location (such as inside a facility) and monitors for gas leaks through AI analysis (S101). At this stage, the mobile camera 100 is not in operation.

[0094] If server 400 does not detect a gas leak (S102: NO), it returns to step S101 and continues monitoring.

[0095] On the other hand, if a gas leak is detected (S102: YES), the server 400 instructs at least one mobile camera 100 to photograph the site (S103). The instruction to photograph with the mobile camera 100 may be displayed on the operation display unit 460 of the server 400, or the server 400 may send an instruction to the mobile camera 100 and display it on the operation display unit 160 of the mobile camera 100. Alternatively, the instruction to photograph with the mobile camera 100 may be displayed on another computer connected to the server 400 (including smartphones and tablet devices).

[0096] Next, server 400 receives image data transmitted from mobile camera 100 (S104).

[0097] Next, when the server 400 receives image data from multiple mobile cameras 100, it obtains the location information of each mobile camera 100 and selects the mobile camera 100 suitable for 3D visualization (S105).

[0098] Server 400 creates a 3D visualization image from at least two image data. One of the image data used is from a fixed camera 300, and the other is from a mobile camera 100. On the other hand, images suitable for 3D visualization need to be captured from directions that intersect each other with respect to the object. For this reason, for example, the shooting directions of the two cameras preferably intersect at a 45-degree angle, and more preferably at a 90-degree angle. The shooting direction is the direction the camera is facing when the object is approximately in the center of the image.

[0099] To this end, the server 400 acquires and compares the location information and shooting direction of one fixed camera 300 that detected the gas leak with the current location and shooting direction of the mobile camera 100. The server 400 then selects one mobile camera 100 whose shooting direction intersects the shooting direction of the fixed camera 300 at preferably 90 degrees ± 45 degrees.

[0100] In step S105, a pair of fixed camera 300 and mobile camera 100 is selected, but the system is not limited to this. In step S105, the server 400 may select two or more mobile cameras 100 that are in a position and shooting direction suitable for 3D visualization.

[0101] Next, the server 400 links the two image data from the selected fixed camera 300 and the mobile camera 100 and performs 3D visualization processing (S106). At this time, the server 400 treats the two image data as data from the same time and does not perform any preprocessing such as time alignment using timestamps. Data from the same time means that multiple data are flowing at the same time. For example, if the frame rates of multiple image data (videos) are the same, the time difference in which one frame progresses is extremely small for data from the same time, and in this embodiment, it falls within the error range of the UTC signal from GNSS.

[0102] Multiple image data are captured according to the timing of the UTC signal from the GNSS. Therefore, server 400 can synchronize both image data even if they are linked at the same time they are received, eliminating the need for time synchronization processing.

[0103] Next, the server 400 determines whether 3D visualization is impossible or not (S107). As already explained, when trying to obtain a good 3D visualization image, there are certain constraints on the camera position and shooting direction. Also, there are cases where the gas leak condition may not be visible. Cases where the gas leak condition may not be visible include, for example, when the gas flows out of the shooting range of the mobile camera 100 and / or the fixed camera 300, when the gas flows into the shadow of an object, or when the gas leak is resolved. In such cases, the server 400 becomes unable to perform 3D visualization processing. S107 is a step to determine whether 3D visualization processing is impossible in such cases.

[0104] In S107, if 3D visualization processing is not impossible (S107:NO), server 400 returns to S106 and continues 3D visualization processing.

[0105] On the other hand, if 3D visualization becomes impossible (S107: YES), the server 400 then determines whether the gas leak has been resolved or not (S108). One reason for 3D visualization failure is that the gas leak may not be visible. If the gas leak is not visible, it may mean that the gas leak has been resolved in the first place. S108 is a step to make a determination assuming such a case where the gas leak has been resolved.

[0106] As previously explained, this embodiment can also use an autonomously moving mobile camera 100. In that case, the procedure may include a step in which, if 3D visualization processing becomes impossible, the server 400 instructs the mobile camera 100 to move to a location where gas leaks can be detected.

[0107] If the gas leak is not resolved in S108 (S108: NO), the server 400 returns to step S105 and selects a camera suitable for 3D visualization again. The server 400 then continues processing.

[0108] On the other hand, if the gas leak is resolved (S108: YES), the server 400 then returns to S101 and continues processing unless there is an instruction to terminate processing (S109: NO). At this time, it instructs the mobile camera 100, which was taking pictures, to terminate its operation. On the other hand, if there is an instruction to terminate processing (S109: YES), the server 400 terminates the gas leak monitoring process (end).

[0109] According to the embodiment described above, the following actions and effects are achieved.

[0110] According to this embodiment, the mobile camera 100 and the server 400 communicate using a fifth-generation communication system. The communication delay within the same base station of the 5G communication system is less than 10ms, which is shorter than the time required for one frame of image data.

[0111] Furthermore, both the mobile camera 100 and the fixed camera 300 perform internal timing based on the UTC signal from GNSS. Therefore, both the mobile camera 100 and the fixed camera 300 can acquire accurate timing information from the moment they start up. As a result, the data from the mobile camera 100 and the fixed camera 300 will be from the same time. In addition, since this embodiment updates the clock signal (timing information) at predetermined intervals, the mobile camera 100 and the fixed camera 300 can be operated with absolute timing information.

[0112] As a result, this embodiment eliminates (or simplifies) the need for synchronization between data when linking multiple data sets.

[0113] Therefore, this embodiment allows data to be linked without using timestamps. In this embodiment, it is not necessary to add timestamps to image data. However, timestamps may be added so that a person (user) can check the passage of time in the image (video).

[0114] Furthermore, in this embodiment, there is no need to synchronize the time when later stitching together image data obtained through continuous shooting.

[0115] Furthermore, this embodiment does not require time synchronization when switching to data from a terminal device different from the one previously synchronized, allowing for immediate switching and integration of the data being used.

[0116] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. The conditions and numerical values ​​used in the description of the embodiments are for illustrative purposes only, and the present invention is not limited to these conditions and numerical values.

[0117] Although the above-described embodiment illustrates an example using multiple mobile cameras 100 and multiple fixed cameras 300, the information processing system 1 of the present invention may be composed of multiple mobile cameras 100 and a server 400. Furthermore, the information processing system 1 of the present invention may consist of multiple mobile cameras 100 and multiple fixed cameras 300, or only multiple mobile cameras 100. In such a configuration, for example, one of the multiple mobile cameras 100 will be equipped with a data processing unit 402 and function as the server 400.

[0118] Furthermore, while 3D visualization processing was given as an example of linking multiple data in the embodiment, the example is not limited to this. Examples of linking multiple data include connecting and combining multiple image data. Also, linking multiple data does not necessarily require AI processing; it could simply involve connecting or combining image data.

[0119] Furthermore, the information processing program according to the present invention can also be implemented using dedicated hardware circuits. This information processing program can also be provided via computer-readable recording media such as USB (Universal Serial Bus) memory or DVD (Digital Versatile Disc)-ROM (Read Only Memory), or it can be provided online via a network such as the Internet, regardless of the recording media. When provided online, this data input control program is recorded on a recording media such as a magnetic disk in a computer connected to the network.

[0120] Furthermore, the present invention can be modified in various ways based on the configuration described in the claims, and these modifications also fall within the scope of the present invention.

[0121] This application is based on Japanese Patent Application No. 2021-119632, filed on July 20, 2021, and its disclosures are referenced and incorporated as a whole. [Explanation of Symbols]

[0122] 1. Information processing system, 100 mobile cameras, 101, 401 Wireless Communication Section, 102 First camera control unit, 103 First camera imaging unit, 105 1st GNSS Division, 106 First camera clock section, 107 storage, 150, 450 5G communication interface, 200 5G communication systems, 300 fixed cameras, 301 Wired Communications Department, 302 Second camera control unit, 303 Second camera imaging unit, 305 2nd GNSS Division, 306 Second camera clock section, 400 servers, 402 Data Processing Unit, 404 Image data receiving unit, 405 GNSS Department, 406 Server clock section, 407 Server Storage.

Claims

1. A terminal device having a wireless communication unit that performs communication using a fifth-generation mobile communication system, sensors including a camera, and a timing unit that receives signals from a global positioning satellite system and converts them into timing information, A server that communicates with multiple terminal devices via the fifth-generation mobile communication system, If the communication delay between each of the aforementioned terminal devices and the server is less than or equal to a predetermined time, the data from the terminal devices is treated as data from the same time, and a data processing unit links multiple such data, It has, The aforementioned data is image data captured by the aforementioned camera, The aforementioned fifth-generation mobile communication system is a local 5G system. If the communication between the terminal device and the server is within the same base station, it is determined that the communication delay between the terminal device and the server is less than or equal to a predetermined time, and the data from the terminal device is considered to be data from the same time, and The aforementioned data processing unit is an information processing system that creates a 3D visualization image by linking multiple image data received simultaneously.

2. The information processing system according to claim 1, wherein the data processing unit selects from among a plurality of terminal devices a terminal device to link the data with, based on the location information of each terminal device obtained from the global positioning satellite system.

3. The information processing system according to claim 1 or 2, wherein the timing unit updates the timing information within the predetermined time.

4. The information processing system according to any one of claims 1 to 3, wherein the predetermined time is less than the time of one frame.

5. The process involves a stage in which multiple terminal devices, each equipped with sensors including cameras and receiving signals from a global positioning satellite system as timing information, communicate with a server via a fifth-generation mobile communication system. If the communication delay between each of the aforementioned terminal devices and the server is less than or equal to a predetermined time, the data from the terminal devices is considered to be data from the same time, and the process involves linking multiple sets of data. The aforementioned data is image data captured by the aforementioned camera, The aforementioned fifth-generation mobile communication system is a local 5G system. If the communication between the terminal device and the server is within the same base station, it is determined that the communication delay between the terminal device and the server is less than or equal to a predetermined time, and the data from the terminal device is considered to be data from the same time, and In the stage of linking the aforementioned data, a 3D visualization image is created by linking multiple image data received simultaneously. An information processing program that causes a computer to perform a task.

6. The information processing program according to claim 5, wherein the step of linking the plurality of data includes a step of selecting a terminal device from among the plurality of terminal devices based on the location information of each terminal device obtained from the global positioning satellite system to link the data.

7. The information processing program according to claim 5 or 6, further comprising the step of updating the timing information within the predetermined time.

8. The information processing program according to any one of claims 5 to 7, wherein the predetermined time is less than the time of one frame.

Images