Monitoring device, monitoring system, and monitoring method

Abstract

A monitoring device according to the present invention comprises a sensor element that detects monitoring targets, a conversation unit that, on the basis of detection results for the monitoring targets, uses generic LLM data to generate conversation data related to a prescribed monitoring target, and a determination unit that determines a monitoring target for the monitoring device. The conversation unit transmits / receives conversation data to / from one or more other monitoring devices that are different from the monitoring device, and, on the basis of a plurality of conversation data transmitted / received to / from the other monitoring devices, the determination unit determines a monitoring target that is to be monitored by the monitoring device.

Description

Monitoring device, monitoring system, and monitoring method

[0001] The present disclosure relates to a monitoring device, a monitoring system, and a monitoring method.

[0002] Patent Literature 1 discloses a method performed by an object tracking system. The method includes detecting at least one time-series channel information (CI) of a wireless multipath channel affected by a current movement of an object being tracked within a location, the at least one TSCI being detected from a wireless signal transmitted over the wireless multipath channel between a type 1 heterogeneous wireless device at a first position within the location and a type 2 heterogeneous wireless device at a second position within the location. The method includes continuously calculating a distance of the current movement of the object based on a similarity score between a pair of temporally adjacent CIs of the at least one TSCI associated with the current movement of the object, monitoring the spatiotemporal information of the object based on at least one of the at least one TSCI, a time parameter associated with the current movement, and past spatiotemporal information of the object, and the distance, and tracking the object based on the spatiotemporal information.

[0003] Japanese Patent Application Publication No. 2023-75094

[0004] In recent years, there has been a demand for optimizing devices by having them think and act for themselves to detect dynamic changes in the environment and objects, and then achieve a specific purpose assigned to the device (in this case, tracking an object) based on the detection results.

[0005] In Patent Document 1, multiple different devices share information about their own devices with each other to track a moving object. However, when each device is freely movable and the relative positions of the devices change dynamically, it is difficult to share information between the devices.

[0006] The present disclosure has been devised in view of the above-described conventional circumstances, and aims to provide a monitoring device, a monitoring system, and a monitoring method that realize efficient monitoring of an object using a plurality of freely movable monitoring devices.

[0007] The present disclosure provides a monitoring device that is mobile and monitors a monitoring target, comprising: a sensor element that detects the monitoring target; a dialogue unit that generates dialogue data regarding a specific monitoring target using general-purpose LLM data based on the detection result of the monitoring target; and a determination unit that determines the monitoring target of the monitoring device, wherein the dialogue unit transmits and receives the dialogue data to and from one or more other monitoring devices different from the monitoring device, and the determination unit determines the monitoring target to be monitored by the monitoring device based on the plurality of dialogue data transmitted and received between the other monitoring devices.

[0008] The present disclosure also provides a monitoring system including a plurality of monitoring devices each having a sensor element for detecting a monitoring target, each being movable and monitoring the same or different monitoring targets, wherein a first monitoring device among the plurality of monitoring devices detects the monitoring target using the sensor element, and when detecting a change in the monitoring environment in which the monitoring target is monitored based on the detection result of the monitoring target, generates first dialogue data regarding the monitoring of the specified monitoring target using general-purpose LLM data and transmits the first dialogue data to one or more second monitoring devices different from the first monitoring device, the second monitoring device generates second dialogue data for determining a third monitoring device among the second monitoring devices that will monitor the specified monitoring target based on the first dialogue data and the detection result of the monitoring target by the sensor element of the second monitoring device, and transmits the second dialogue data to the first monitoring device, and the third monitoring device moves so as to be able to detect the specified monitoring target and monitors the specified monitoring target based on the first dialogue data.

[0009] The present disclosure also provides a monitoring method performed by a plurality of devices that have sensor elements for detecting a monitoring target, are movable, and monitor the monitoring target, wherein, when a change in the monitoring environment in which the monitoring target is monitored is detected based on the detection results of the monitoring target, dialogue data regarding the monitoring of the monitoring target is generated using general-purpose LLM data, a plurality of pieces of dialogue data are transmitted and received between the plurality of devices, and one of the plurality of devices is determined to monitor the monitoring target based on the plurality of dialogue data, and the determined device is caused to monitor the monitoring target.

[0010] According to the present disclosure, it is possible to realize efficient monitoring of an object using a plurality of movable monitoring devices.

[0011] FIG. 1 is a diagram showing an example of a system configuration of a monitoring system according to embodiment 1. Block diagram showing an example of the internal configuration of a master and a sensor according to embodiment 1. Sequence diagram showing an example of an initial setup procedure for a monitoring system according to embodiment 1. Sequence diagram showing an example of a procedure for changing settings for a monitoring system according to embodiment 1. Diagram explaining an example of a change in settings for a sensor ... showing an example of an interaction between a master and a sensor according to embodiment 1. Diagram showing an example of a system configuration of a monitoring system according to embodiment 2. Block diagram showing an example of the internal configuration of a sensor according to embodiment 2. Sequence diagram showing an example of an initial setup procedure for a monitoring system according to embodiment 2. Sequence diagram showing an example of a procedure for changing settings for a monitoring system according to embodiment 2. Diagram showing an example of an interaction between a sensor according to embodiment 2.

[0012] Hereinafter, with reference to the accompanying drawings as appropriate, detailed descriptions of embodiments specifically disclosing a monitoring device, a monitoring system, and a monitoring method according to the present disclosure will be provided. However, unnecessary detailed descriptions may be omitted. For example, detailed descriptions of well-known matters or redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter recited in the claims.

[0013] (Embodiment 1) First, an overview of a monitoring system 100 according to embodiment 1 will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the system configuration of the monitoring system 100 according to embodiment 1. Note that the monitoring system 100 shown in Fig. 1 is an example and is not limited to this.

[0014] The surveillance system 100 is a system that uses multiple sensors SS1 to SSN that are movable within a surveillance area to monitor surveillance targets (for example, objects or people) within the surveillance area or to detect detection targets (for example, the state of objects or people). Note that "movement" here is not limited to movement on the ground, but may also include movement in the air.

[0015] In this disclosure, as an example, a description will be given of a case in which a drone (an example of a sensor) capable of flying over a parking lot, a site, or a yard (an example of a logistics base) that is a monitoring area and equipped with a camera (an example of a sensor element) is provided to monitor trucks (an example of a monitoring target) passing through roads within the yard. The monitoring system 100 includes a master MS and multiple sensors SS1 to SSN.

[0016] The master MS is configured to be able to accept instructions (operations) from an administrator AD who manages the monitoring area. The master MS is also wirelessly connected to each of the multiple sensors SS1 to SSN to transmit and receive data. The master MS may be implemented by a personal computer (hereinafter referred to as "PC"), a laptop computer, a tablet terminal, a smartphone, or the like. Note that, although the present disclosure illustrates an example in which each of the multiple sensors SS1 to SSN is connected to one master MS, the monitoring system 100 may also include a repeater or the like that relays data communication between the master MS and one or more sensors.

[0017] The wireless communication referred to here is realized by a wireless LAN such as Wi-Fi (registered trademark), a wireless WAN, a mobile communication network such as 4G or 5G, or other network configuration capable of wireless communication.

[0018] The master MS executes settings for implementing the monitoring functions of each of the multiple sensors SS1 to SSN based on instructions from the administrator AD. Furthermore, the master MS generates optimization instructions for implementing monitoring of all monitoring targets monitored by each of the sensors SS1 to SSN based on the detection results transmitted from any of the sensors SS1 to SSN, and transmits the optimization instructions to each of the sensors.

[0019] Each of the multiple sensors SS1 to SSN is wirelessly connected to the master MS and other sensors, and transmits and receives various data, such as detection results and natural language information. Each of the multiple sensors SS1 to SSN is implemented by a device (e.g., an unmanned aerial vehicle such as a drone) having sensor elements 24 (e.g., a camera, a microphone, a radar, etc.) that can detect environmental changes in the area monitored by the sensor SS1 to SSN, monitor a monitoring target (e.g., an object or a person, etc.) within the monitoring area, or detect the state of the detection target (e.g., an object or a person, etc.). Note that, for ease of explanation, this disclosure will describe an example in which each of the multiple sensors SS1 to SSN is implemented by a camera, but the sensors SS1 to SSN may also be implemented by combining different sensing devices, such as a camera and a microphone.

[0020] Each of the multiple sensors SS1 to SSN executes a dialogue with at least one other sensor using natural language processing based on an optimization instruction transmitted from the master MS to determine a monitoring target or a destination for monitoring the monitoring target. The multiple sensors SS1 to SSN move based on the destination determined through the dialogue, and monitor the monitoring target. The determined destination may be the location of the monitoring target or the location of the sensor that detected the monitoring target.

[0021] Next, an example of the internal configuration of each of the master MS and the multiple sensors SS1 to SSN will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the internal configuration of the master MS and the sensors SS1 to SSN in embodiment 1. Note that since each of the multiple sensors SS1 to SSN in this disclosure has the same configuration, only the internal configuration of sensor SS1 will be described.

[0022] The master MS includes a communication interface (hereinafter referred to as “I / F”) 10 , a processor 11 , and a memory 12 .

[0023] The communication I / F 10 is connected to the sensors SS1 to SSN so that data can be sent and received between them. The communication I / F 10 transmits various data output from the processor 11 to each of the sensors SS1 to SSN. The communication I / F 10 also outputs to the processor 11 various data (various information) such as operation details accepted by an input operation by the administrator AD or detection results transmitted from any of the sensors SS1 to SSN.

[0024] The processor 11 is configured using, for example, a Central Processing Unit (hereinafter referred to as "CPU"), a Digital Signal Processor (hereinafter referred to as "DSP"), or a Field Programmable Gate Array (hereinafter referred to as "FPGA"), and performs various processes and controls in cooperation with the memory 12. Specifically, the processor 11 references programs and data stored in the memory 12 and executes the programs to realize the functions of a Large Language Models (hereinafter referred to as "LLM") processing unit 111.

[0025] The LLM processing unit 111 generates optimization instructions for realizing the functions (roles) of each of the multiple sensors SS1 to SSN based on the general-purpose LLM data (large-scale language model) stored in the general-purpose LLM holding unit 121 and the specific knowledge data of the master MS stored in the specific knowledge holding unit 122, and transmits the instructions to each of the corresponding multiple sensors.

[0026] The memory 12 includes a read-only memory (hereinafter referred to as "ROM") and a random access memory (hereinafter referred to as "RAM"). The ROM stores programs that define the processing (operations) of the processor 11 and data referenced when the programs are executed. The RAM is a work memory used when the processor 11 executes its processing (operations) and temporarily stores data or information generated or acquired during each process. The memory 12 includes a general-purpose LLM storage unit 121 and a specific knowledge storage unit 122.

[0027] The general-purpose LLM storage unit 121 stores general-purpose LLM data, which includes a large-scale language model used by the processor 11 to generate dialogue data.

[0028] The specific knowledge holding unit 122 is used by the processor 11 and stores specific knowledge data for the master MS for realizing the functions of the master MS.

[0029] The specific knowledge data for the master MS here includes data regarding the layout (map) of the site where the sensor is used, data regarding the operation plan, data regarding the site operating procedures, or specific knowledge data dedicated to each of the sensors SS1 to SSN.

[0030] Data relating to operation plans is data that indicates plans for operations to be carried out within a monitoring area, such as production plans and process plans (for example, unloading at 1 pm, loading at 2 pm, etc.).

[0031] The data regarding each sensor SS1 to SSN includes data regarding the relationship between the sensors (for example, whether or not there is cooperation between the sensors in monitoring the monitored object), information about the area that the sensors SS1 to SSN track or monitor, or the functions of the sensors SS1 to SSN (for example, the size of the angle of view, the range of motion of the pan angle and tilt angle, etc.).

[0032] The input unit 13 is a user interface that can accept input operations by the administrator AD, and is realized by, for example, a keyboard, a mouse, a touch panel, etc. The input unit 13 converts the content of the accepted input operation into an electrical signal and outputs it to the processor 11.

[0033] The sensor SS1 includes a communication I / F 20, a processor 21, a memory 22, a moving mechanism 23, a sensor element 24, and a position receiver 25. The moving mechanism 23 is not an essential component and may be omitted. The sensor SS1 may have any configuration or function depending on the sensing device that realizes the sensor and the role of the sensor (i.e., the object to be monitored or detected by the sensor, etc.).

[0034] The communication I / F 20 connects the master MS and the other sensors SS2 to SSN so that data can be sent and received between them. The communication I / F 20 transmits various data output from the processor 21 to the master MS or the sensors SS2 to SSN. The communication I / F 20 also outputs various data (various information) transmitted from the master MS or the sensors SS2 to SSN to the processor 21.

[0035] The processor 21 is configured using, for example, a CPU, a DSP, an FPGA, or a Graphics Processing Unit (GPU), and performs various processes and controls in cooperation with the memory 22. Specifically, the processor 21 references the programs and data stored in the memory 22 and executes the programs to realize the functions of the LLM processing unit 211, the movement control unit 212, etc.

[0036] The LLM processing unit 211 recognizes the role of the sensor SS1 and determines the purpose of the monitoring performed by the sensor SS1 based on the specific knowledge data of the sensor itself (i.e., dedicated to the sensor SS1) and the general-purpose LLM data (large-scale language model) stored in the specific knowledge holding unit 222. For example, if the role of the sensor SS1, which is a camera, is to "monitor a parking lot," the LLM processing unit 211 determines that the purpose is to determine a position and altitude that maximizes the sensing area (i.e., image capture range) of the sensor SS1 within the range in which the monitored object can be detected. Furthermore, for example, if the role of the sensor SS1, which is a camera, is to "monitor vehicles in a parking lot," the LLM processing unit 211 determines that the purpose is to determine the movement operation (e.g., the destination position or movement speed) of the sensor SS1 to track and monitor the monitored vehicle so that the monitored vehicle fits within the sensing area (i.e., image capture range) of the sensor SS1.

[0037] Here, the LLM processing unit 211 uses general-purpose LLM data (large-scale language model) to recognize the role assigned to its own sensor, thereby enabling it to more accurately determine the monitoring purpose of sensor SS1 and the monitoring target or monitoring location of sensor SS1, and to realize more flexible dialogue (i.e., more advanced collaboration) in real time with other sensors to determine the sensor responsible for monitoring a specific monitoring target.

[0038] The LLM processing unit 211 generates dialogue data to be executed between the master MS or other sensors using a large-scale language model stored in the general-purpose LLM holding unit 221 based on information about the sensor itself (e.g., the current location or function of the sensor SS1) contained in the specific knowledge data of the sensor SS1, the current environment of the sensor, and the purpose of monitoring the sensor.

[0039] Furthermore, when the LLM processing unit 211 acquires dialogue data transmitted from the master MS or another sensor, or information about the other sensor (for example, the installation location, function, or monitoring status of the other sensor), it analyzes the dialogue content included in the dialogue data by natural language processing. Based on the analysis result, the monitoring purpose of the own sensor, the current environment of the own sensor, and the analyzed dialogue content, the LLM processing unit 211 determines whether to maintain the current status of or add a monitoring target, or determines a destination based on the determined monitoring target.

[0040] As a result, the LLM processing unit 211 can perform recognition using general-purpose LLM data (large-scale language model) and request support for monitoring targets that cannot be fully monitored, and when support is requested, analyze the status of its own sensor and other sensors based on dialogue and determine (judge) how to monitor the currently monitored targets and the targets that cannot be fully monitored using the limited sensors.

[0041] The movement control unit 212 controls the movement mechanism 23. The movement control unit 212 controls the movement mechanism 23 based on the position of each monitoring target detected by the sensor element 24 or changed or added through interaction, and tracks the monitoring target.

[0042] The memory 22 includes a ROM and a RAM. The ROM stores programs that define the processing (operations) of the processor 21 and data referenced when the programs are executed. The RAM is a work memory used when the processor 21 executes its processing (operations) and temporarily stores data or information generated or acquired during each process. The memory 22 includes a general-purpose LLM storage unit 221 and a specific knowledge storage unit 222.

[0043] The general-purpose LLM storage unit 221 stores general-purpose LLM data, which includes a large-scale language model used by the processor 21 to generate dialogue data.

[0044] The specific knowledge storage unit 222 is used by the processor 21 and stores specific knowledge data for the sensor SS1 to realize the function of the sensor itself (i.e., the sensor SS1). The specific knowledge data for the sensor SS1 includes data indicating the role of the sensor SS1 or data related to the function of the sensor SS1 (for example, the possible flight speed, the possible flight altitude, the angle of view, the movable range of the pan angle and the tilt angle, etc.).

[0045] As described above, the data indicating the role is used to control the moving mechanism 23 or the sensor element 24 so that the sensor SS1 can monitor the monitored object or detect the detected object in the optimization process of tracking and monitoring the monitored object executed by the sensor SS1. The data indicating the role is, for example, language data such as detecting the movement (detection object) of a person (monitoring object) or monitoring a wider area (monitoring object) within a factory (monitoring area).

[0046] Note that multiple pieces of data indicating the role may be set, and different roles may be set depending on, for example, the time period, date and time, season, etc. This allows the sensor SS1 to change its role as appropriate, for example, tracking and monitoring each truck using the parking lot during the day, and patrolling above the parking lot at night to monitor (detect) suspicious people or vehicles in the parking lot, and can also execute control that is more suitable for its current role as appropriate.

[0047] The movement mechanism 23 is a mechanism that controls the movement of the sensor itself or the sensing direction of the sensor element 24. The movement mechanism 23 in the present disclosure is realized by, for example, a mechanism that realizes flight control of a propeller, engine, etc. that causes the sensor SS1 to fly. Note that the movement mechanism 23 may further include a pan mechanism or tilt mechanism that can change the angle of view of the sensor element 24 (camera).

[0048] The sensor element 24 is at least one element capable of monitoring a monitoring target or detecting a detection target. In the present disclosure, the sensor element 24 is an image sensor, a lens, or the like. The sensor element 24 may have multiple elements, and for example, if the camera is a compound eye camera, the sensor element 24 may be realized by two image sensors and lenses. The sensor element 24 outputs the monitoring or detection results to the master MS or another device (e.g., a device operated by a person performing monitoring work in the monitoring area), etc.

[0049] The position receiver 25 receives signals transmitted from, for example, multiple satellites that make up the Global Positioning System (GPS) and outputs the signals to the processor 21. The processor 21 measures the current position of the sensor SS1 based on the multiple signals acquired by the position receiver 25.

[0050] Next, an example of an initial setup procedure for the monitoring system 100 will be described with reference to Fig. 3. Fig. 3 is a sequence diagram showing an example of an initial setup procedure for the monitoring system 100 according to embodiment 1. The initial setup procedure shown in Fig. 3 is a procedure for installing various types of data that is executed before and after each of the multiple sensors SS1 to SSN is installed and before each of the multiple sensors SS1 to SSN starts monitoring the monitoring target.

[0051] The administrator AD performs an operation (that is, an installation operation) to install general-purpose LLM data (large-scale language model) in the master MS and each of the sensors SS1 to SSN (St11).

[0052] The master MS installs the general-purpose LLM data (large-scale language model) based on the administrator AD operation, and stores it in the general-purpose LLM holding unit 121 of the memory 12 (St12).

[0053] Each of the sensors SS1 to SSN installs general-purpose LLM data (large-scale language model) based on an installation operation by the administrator AD and stores the data in the general-purpose LLM storage unit 221 of the memory 22 (St13). Note that the installation operation here may be the connection of an external storage medium such as a Universal Serial Bus (USB, registered trademark) memory or an SD card (registered trademark) connected by the administrator AD so that data can be read, or the transmission of the general-purpose LLM data via a wired or wireless network line by the administrator AD.

[0054] After installing the general-purpose LLM data in the master MS and each of the sensors SS1 to SSN, the administrator AD performs an operation to install the specific knowledge data for the master MS (that is, an installation operation) (St14).

[0055] The master MS installs the specific knowledge data for the master MS based on the administrator AD operation and stores it in the specific knowledge holding unit 122 of the memory 12 (St15).

[0056] The master MS determines, for each of the sensors SS1 to SSN, the specific knowledge data to be installed in each of the sensors SS1 to SSN from the specific knowledge data for the master MS based on the role of each of the sensors SS1 to SSN (monitoring target or detection target) etc. (St16). The master MS associates a control command requesting installation of the specific knowledge data with the specific knowledge data to be installed in each sensor, and transmits the control command to each of the corresponding sensors (St17).

[0057] Each of the plurality of sensors SS1 to SSN installs the specific knowledge data transmitted from the master MS and stores it in the specific knowledge holding unit 222 of the memory 22 (St18).

[0058] Next, an example of a procedure for changing the settings of the monitoring system 100 will be described with reference to Fig. 4. Fig. 4 is a sequence diagram showing an example of a procedure for changing the settings of the monitoring system 100 according to the first embodiment. The procedure for changing the settings shown in Fig. 4 is a procedure for adding or changing a monitoring target. In the description of Fig. 4, for ease of understanding, an example will be described in which the sensor that detects a change in the monitoring environment is sensor SS1 and the sensor that changes the monitoring target is sensor SS2.

[0059] 4 shows an example in which the interaction sensor group is determined based on the distance between sensors based on the current positions of the sensors, but the process of determining the interaction sensor group is not limited to the above-described method. For example, the master MS may determine the interaction sensor group using any information such as the current positions of the sensors, the sensor functions (size of the angle of view, range of movement of the pan angle and tilt angle), the interrelationship between the sensors (whether or not they cooperate), or the role of each sensor.

[0060] The setting change procedure is started by an instruction from the administrator AD (step St21) or by detection of an environmental change by any of the sensors SS1 to SSN (step St22).

[0061] The process of step St21 is executed, for example, when there is a change in the layout of the monitoring area, when there is a change in the roles of the sensors SS1 to SSN, or periodically (every week, month, etc.).

[0062] Furthermore, the processing of step St22 is executed when there is a change in the monitoring system of the monitored area due to a change in the layout of the monitored area, the addition, relocation, or removal of sensors (sensing devices), or when a change in the environment is detected. The change in the environment here includes the current monitoring environment of the sensor, the environment of the monitored area in which the sensor is located, etc. The monitoring environment may be changed, for example, when it is determined (estimated) that it will become difficult to monitor a target that has not yet been tracked and monitored by the sensor due to an addition or change in the monitored targets, or due to the direction or speed of movement of each monitored target, or when each sensor has a designated area for tracking and monitoring, and it is determined (estimated) that the target that the sensor is tracking and monitoring will move beyond this area to another area.

[0063] In step St21, when the administrator AD requests optimization of one or more sensors, or when the administrator AD requests a change in the role (optimization purpose) of sensor SS1, the administrator AD executes an operation (request operation) to request optimization processing (St211). Here, the administrator AD may select at least one sensor to be optimized, or may select all sensors SS1 to SSN. Furthermore, when the administrator AD requests a change in the role (optimization purpose) of sensor SS1, the administrator AD includes information about the new role and requests optimization based on this new role.

[0064] In step St22, the multiple sensors SS1 to SSN analyze the monitoring results or detection results of the sensor elements 24, and if they determine (detect) that there is a change in the environment of the sensing area they are monitoring (St221), they generate dialogue data using natural language processing to notify the user of the environmental change in the sensing area and send it to the master MS (St222).

[0065] When the master MS receives a request for optimization processing via an administrator AD operation or receives interaction data transmitted from any of the sensors, the master MS acquires information such as the current positions of all sensors SS1 to SSN based on the specific knowledge data for the master MS (St23). The master MS determines an interaction sensor group by grouping each of the multiple sensors, including sensor SS1, based on information such as the distances between all sensors SS1 and each of sensors SS2 to SSN (St24). For example, the master MS in the present disclosure determines sensors SS2 to SS5, whose distances from sensor SS1 are equal to or less than a predetermined distance, to be part of the interaction sensor group based on the current positions of each sensor.

[0066] The master MS associates a control command requesting optimization processing with information on sensors SS2 to SS5 included in the interaction sensor group, and transmits the control command to sensor SS1 (St25). Note that the master MS may also transmit a control command requesting optimization processing to each sensor included in the interaction sensor group (here, sensors SS1 to SS5).

[0067] The sensor SS1 generates dialogue data for consultation about the monitoring of the monitored object by natural language processing based on the information of the group of interactive sensors transmitted from the master MS, the current environment of the sensor itself, and the role of the sensor itself, and transmits the generated dialogue data and information about the sensor itself (for example, the current location of the sensor SS1 or the monitoring status of the monitored object by the sensor SS1 (i.e., the number and location of the monitored object)) to the other sensors SS2 to SS5 included in the group of interactive sensors. The sensor SS1 acquires dialogue data transmitted from the other sensors and including information about the other sensors or suggestions about the monitoring of the monitored object based on the environment of the other sensors.

[0068] The sensor SS1 transmits and receives dialogue data (i.e., dialogue) with other sensors as described above (St26), and determines to maintain the current monitoring target of the sensor based on the dialogue content of the dialogue data, and determines a destination of the sensor based on the determined monitoring target (St27). Note that if there is no change in the monitoring target, the sensor SS1 continues to track the existing monitoring target, and determines a destination where the existing monitoring target can be tracked and monitored (St27).

[0069] In addition, sensor SS2, another sensor included in the interactive sensor group, sets or adds as a new monitoring target the monitoring target that sensor SS1 has determined cannot be tracked or monitored based on the interaction including sensor SS1, and determines the destination of its own sensor based on the determined monitoring target (St27).

[0070] The sensors SS1 and SS2 drive the movement mechanism 23 to move to the determined destination (St28).

[0071] In step St26 described above, the sensor SS1 acquires the role of its own sensor based on its own specific knowledge data, and its current position and the monitoring status of the monitored objects (e.g., the number, position, speed, or acceleration of the monitored objects) based on the monitoring results (detection results) by the sensor element 24. Based on this referenced information, the sensor SS1 generates dialogue data including dialogue content for fulfilling the role of its own sensor, and executes dialogue with other sensors included in the group of interactive sensors based on the transmission and reception of dialogue data.

[0072] For example, if the role of sensors SS1 to SSN is to track and monitor vehicles (targets), sensors SS1 to SSN will engage in dialogue with other sensors to determine which of multiple vehicles each sensor will track and monitor, and based on the content of the dialogue, will determine which vehicle will be tracked and monitored, and move the determined vehicle to a position where it can be tracked.

[0073] As described above, the monitoring system 100 according to the first embodiment can acquire the latest position information of each of a plurality of sensors that are freely movable and whose relative positions change dynamically via the master MS, and can determine a conversation partner for each sensor based on the acquired sensor position information. This allows the monitoring system 100 to execute a conversation regarding the monitoring target that each sensor is tracking and monitoring, based on the status of each sensor acquired from other sensors through the conversation (e.g., the status of the environment, the status related to the monitoring being performed by each sensor).

[0074] Furthermore, the monitoring system 100 according to the first embodiment can execute optimization of the monitoring targets by the sensors SS1 to SSN when the administrator AD requests optimization processing or when the sensors detect an environmental change in the sensing area. This allows the monitoring system 100 to automate the optimization of the monitoring targets of the sensors SS1 to SSN in monitoring the monitoring targets, thereby reducing the personnel costs and time required to monitor all monitoring targets.

[0075] Furthermore, the monitoring system 100 according to the first embodiment can change the targets monitored by the sensors SS1 to SSN in real time even when static environmental changes (e.g., layout changes) or dynamic environmental changes (e.g., changes in the number of targets monitored depending on the time of day, or changes in the locations of targets monitored) occur. This allows the monitoring system 100 to more effectively prevent the sensors SS1 to SSN from missing targets when monitoring targets.

[0076] If the sensor SS1 determines that it is difficult to monitor all the monitoring targets through the optimization dialogue executed within the target sensor group, the sensor SS1 may execute the optimization process again by generating a request to increase the number of sensors to be determined as the target sensor group and transmitting the request to the master MS. Furthermore, when the optimization process is executed again, the master MS may relax the conditions for the sensors to be determined as the interaction sensor group (e.g., the distance from the sensor SS1, whether or not the sensing areas overlap, etc.).

[0077] As a result, even if the monitoring system 100 cannot monitor all of the monitored objects due to the interaction of multiple sensors included in the target sensor group, by increasing the number of sensors included in the interactive sensor group, it is possible to automatically monitor all of the monitored objects using multiple sensors SS1 to SSN.

[0078] Next, examples of changing the monitoring target of sensor SS2 will be described with reference to Fig. 5 to Fig. 8. Fig. 5 is a diagram illustrating an example of changing the settings of sensor SS2. Fig. 6 is a diagram illustrating an example of changing the settings of sensor SS2. Fig. 7 is a diagram illustrating an example of a dialogue between the master MS and sensors SS1, SS2, and SS3 in embodiment 1. Fig. 8 is a diagram illustrating an example of changing the settings of sensor SS2.

[0079] Sensors SS1 to SS3 shown in FIG. 5 are cameras that track and monitor trucks entering through a gate until they are parked in a designated parking spot in order to monitor where each truck is parked in a parking lot (not shown). To monitor (image) truck Tg, sensors SS1 to SS3 wait near the gate, and when they detect a truck entering, they image the detected truck while tracking it. Angle of view AR1 indicates the sensing area of ​​sensor SS1. Angle of view AR2 indicates the sensing area of ​​sensor SS2. Angle of view AR3 indicates the sensing area of ​​sensor SS3.

[0080] 5 detects the track Tg1 from within the field of view, and then starts tracking and monitoring the detected track Tg1.

[0081] 6 continue to track and monitor truck Tg1. Sensor SS3 detects truck Tg2, which entered after truck Tg1, and therefore begins tracking and monitoring truck Tg2. Note that sensor SS3 may interact with sensor SS1 or sensor SS2 and determine that sensors SS1 and SS2 will track and monitor truck Tg1, and sensor SS3 will track and monitor truck Tg2.

[0082] After starting to track and monitor truck Tg2, sensor SS3 further detects trucks Tg3 and Tg4 that enter following truck Tg2. Sensor SS3 performs tracking and monitoring by moving so that each of trucks Tg2 to Tg4 falls within its sensing area.

[0083] Here, based on the detection results of each of trucks Tg2 to Tg4 by sensor element 24, sensor SS3 determines that truck Tg2 is about to leave the sensing area and that it is not possible to track and monitor trucks Tg2 to Tg4 with just its own sensor; in other words, the current environment in which trucks Tg2 to Tg4 are being tracked and monitored will change to an environment in which it will no longer be possible to track and monitor truck Tg2, and only trucks Tg3 and Tg4 will be tracked and monitored.

[0084] The sensor SS3 generates dialogue data COM11 that notifies the master MS of a change in the environment in which the target is tracked and monitored, and requests instructions from the master MS, and transmits the dialogue data COM11 to the master MS (not shown). The sensor SS3 shown in Fig. 7 uses general-purpose LLM data to generate dialogue data COM11, "There are too many trucks near the gate and I can't capture them all! Help me!"

[0085] Based on the notification sent from sensor SS3 and the dialogue data COM11, the master MS acquires the position information of all sensors SS1 to SS3, information on the sensing area, etc. The master MS calculates the distance between sensor SS3 and sensors SS1 and SS2, and determines whether the calculated distances are equal to or less than a predetermined distance. The master MS designates sensors SS1 and SS2 as an interaction sensor group, and transmits dialogue data COM12 including an optimization instruction and information on the determined interaction sensor group to sensor SS3. The master MS shown in FIG. 7 uses the general-purpose LLM data to generate dialogue data COM12 saying, "Please cooperate with No. 1 and No. 2 and request assistance so that you can track the truck as closely as possible."

[0086] The sensor SS3 starts a dialogue with the sensors SS1 and SS2 based on the dialogue sensor group transmitted from the master MS. The sensor SS3 transmits and receives dialogue data COM13, COM14, and COM15 (i.e., dialogue) with the sensors SS1 and SS2, and determines that the tracks to be monitored by the sensor are tracks Tg3 and Tg4.

[0087] The sensor SS3 shown in FIG. 7 uses the general-purpose LLM data to generate dialogue data COM13 for requesting assistance from the sensors SS1 and SS2: "There are too many trucks near the gate and I can't capture them all! Help me!"

[0088] The sensor SS1 also interprets the contents of the dialogue data COM13 using natural language processing. The sensor SS1 generates dialogue data COM14 using general-purpose LLM data based on the contents of the dialogue data COM13, the specific knowledge data for the sensor SS1, and the current state of the sensor SS1. The sensor SS1 generates dialogue data COM14 "Rescue has been received from No. 3. What should we do?", which is directed to the sensor SS2, which is included in the dialogue sensor group and is other than the sensor SS3.

[0089] The sensor SS2 interprets the contents of the dialogue data COM14 using natural language processing. The sensor SS2 generates dialogue data COM15 using general-purpose LLM data based on the contents of the dialogue data COM14, the specific knowledge data for the sensor SS2, and the current state of the sensor SS2. The sensor SS2 is included in the dialogue sensor group and generates dialogue data COM15 directed to the sensor SS1, such as "This truck is the one you've been photographing all the way from the gate, so I'll leave this to you. I'll go." The dialogue data COM14 and COM15 may also be received by the sensor SS3.

[0090] As a result of the interaction (ie, each of the interaction data COM13-COM15), sensor SS1 continues to track and monitor truck Tg1.

[0091] Based on the result of the dialogue, sensor SS3 acquires its own position information or the position information and speed information of truck Tg2 and transmits it to sensor SS2. Also, based on the result of the dialogue, sensor SS3 performs tracking and monitoring of trucks Tg3 and Tg4. Note that sensor SS3 may continue monitoring truck Tg2 or measuring and transmitting the position information of truck Tg2 within the sensing area until sensor SS2 arrives at a position where it can monitor (image) truck Tg2, or may end monitoring truck Tg2 and prioritize tracking and monitoring trucks Tg3 and Tg4.

[0092] Sensor SS2 determines a position for monitoring (imaging) truck Tg2, i.e., a movement destination, based on the position information of sensor SS3 acquired from sensor SS3, or the position information, speed information, etc. of truck Tg2. Sensor SS2 determines the movement destination to be "near the gate" based on the dialogue data COM13, or the position of sensor SS1 itself or the position of truck Tg2 transmitted from sensor SS1, and starts movement to track and monitor truck Tg2.

[0093] 8, sensor SS1 continues to track and monitor track Tg1, sensor SS2 is tracking and monitoring track Tg2, and sensor SS3 is tracking and monitoring tracks Tg3 and Tg4.

[0094] As described above, the monitoring system 100 can track and monitor all of the monitoring targets by adjusting the monitoring targets of each of the multiple sensors SS1 to SS3 among them based on the environmental conditions observed by each of the sensors SS1 to SS3 by having the sensors included in the interaction sensor group interact with each other. In other words, even if there is a dynamic change in the environment (monitoring target), the monitoring system 100 can track and monitor the monitoring targets by responding in real time to the dynamic change detected by the multiple sensors included in the interaction sensor group.

[0095] Furthermore, by allowing the sensors included in the interactive sensor group to interact with each other, the monitoring system 100 allows the sensors to share their status (e.g., specific knowledge data, information about the current position of the sensor, or information about the target being tracked) and automatically optimizes the number of targets that each sensor will track and monitor, the locations at which each sensor will track and monitor the targets, etc. For example, as shown in Figure 8, sensors SS1 to SS3 can determine the sensor that will track and monitor the target based on the position, speed, etc. of each target.

[0096] (Embodiment 2) The monitoring system 100 according to embodiment 1 has shown an example in which the sensors included in the interaction sensor group are determined by a master MS connected to all sensors SS1 to SSN so as to be able to communicate data with them. The monitoring system 100A according to embodiment 2 will explain an example in which the sensors included in the interaction sensor group are determined by a sensor that has received an optimization instruction from an administrator AD or a sensor that has detected a change in the environment within its own sensing area.

[0097] In the description of the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description of the same components will be omitted.

[0098] First, an overview of a monitoring system 100A according to embodiment 2 will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of the system configuration of the monitoring system 100A according to embodiment 2. Note that the monitoring system 100A shown in Fig. 9 is merely an example, and is not limiting.

[0099] The monitoring system 100A is a system that uses multiple sensors SS1A to SSNA that are movable within a monitoring area to monitor monitoring targets (for example, objects or people) within the monitoring area or to detect detection targets (for example, the state of objects or people). The monitoring system 100A includes each of the multiple sensors SS1A to SSNA.

[0100] Each of the multiple sensors SS1A to SSNA is configured to be able to receive instructions from an administrator AD. Each of the multiple sensors SS1A to SSNA is also connected to other sensors so as to be able to communicate wirelessly with each other, and transmits and receives various data such as detection results and natural language information. For ease of understanding, this disclosure describes an example in which each of the multiple sensors SS1A to SSNA is implemented by a camera, but they may also be implemented by combining different sensing devices, such as a camera and a microphone.

[0101] Each of the multiple sensors SS1A to SSNA executes a dialogue with at least one other sensor using natural language processing based on an optimization instruction transmitted from the master MS to determine a monitoring target or a destination for monitoring the monitoring target. The multiple sensors SS1A to SSNA move based on the destination determined through the dialogue, and monitor the monitoring target. The determined destination may be the location of the monitoring target or the location of the sensor that detected the monitoring target.

[0102] Next, an example of the internal configuration of each of the multiple sensors SS1A to SSNA will be described with reference to Fig. 10. Fig. 10 is a block diagram showing an example of the internal configuration of the sensors SS1A to SSNA in embodiment 2. Note that since each of the multiple sensors SS1A to SSNA in the present disclosure has the same configuration, only the internal configuration of sensor SS1A will be described.

[0103] The sensor SS1A includes a communication I / F 20A, a processor 21A, a memory 22, a moving mechanism 23, a sensor element 24, and a position receiver 25. The sensor SS1A may have any configuration or function depending on the sensing device that realizes the sensor, the object that the sensor monitors, or the role of the sensor.

[0104] The communication I / F 20A connects a device (not shown) that is the sender of a control command (optimization command) based on an administrator AD operation with other sensors SS2A to SSNA so that data can be sent and received. The communication I / F 20A sends various data output from the processor 21A to each of the sensors SS2A to SSNA. The communication I / F 20A also outputs various data (various information) sent from the sensors SS2A to SSNA to the processor 21A.

[0105] Here, the device (not shown) that sends the control command (optimization command) based on the administrator AD operation may be, for example, a PC, notebook PC, tablet terminal, or smartphone, and may be a device that can view the monitoring results or detection results of sensors SS1A to SSNA, or may be a device owned by the administrator AD.

[0106] The processor 21A is configured using, for example, a CPU, DSP, FPGA, or GPU, and performs various processes and controls in cooperation with the memory 22. Specifically, the processor 21A references the programs and data stored in the memory 22 and executes the programs to perform the functions of the processor 21A.

[0107] Next, an example of an initial setup procedure for the monitoring system 100A will be described with reference to Fig. 11. Fig. 11 is a sequence diagram showing an example of an initial setup procedure for the monitoring system 100A according to embodiment 2. The initial setup procedure shown in Fig. 11 is a procedure for installing various types of data that is executed before and after each of the multiple sensors SS1A to SSNA is installed and before each of the multiple sensors SS1A to SSNA starts monitoring the monitoring target.

[0108] The administrator AD performs an operation (that is, an installation operation) to install general-purpose LLM data (large-scale language model) in each of the sensors SS1A to SSNA (St11).

[0109] Each of the sensors SS1A to SSNA installs the general-purpose LLM data (large-scale language model) based on the installation operation by the administrator AD, and stores it in the general-purpose LLM holding unit 221 of the memory 22 (St13A).

[0110] The administrator AD determines the specific knowledge data to be installed in each of the sensors SS1A to SSNA for each sensor (St14A).

[0111] Each of the sensors SS1A to SSNA installs the specific knowledge data for the sensor designated by the administrator AD and stores it in the specific knowledge holding unit 222 of the memory 22 (St18A).

[0112] Next, an example of a procedure for changing the settings of the monitoring system 100A will be described with reference to Fig. 12. Fig. 12 is a sequence diagram showing an example of a procedure for changing the settings of the monitoring system 100A according to embodiment 2. The procedure for changing the settings shown in Fig. 12 is a procedure for adding or changing a monitoring target.

[0113] 12, for ease of understanding, an example will be described in which the sensor that detects a change in the monitoring environment is sensor SS1A and the sensor that changes the monitoring target is sensor SS2A. Furthermore, the processing of step St22 shown in FIG. 12 is the same as the processing of step St22 shown in FIG. 4, and therefore description thereof will be omitted.

[0114] The setting change procedure is started by an instruction from the administrator AD (step St21A) or by detection of an environmental change by any of the sensors SS1A to SSNA (step St22).

[0115] The process of step St21A is executed, for example, when there is a change in the layout of the monitoring area, when there is a change in the roles of the sensors SS1A to SSNA, or periodically (every week, month, etc.).

[0116] In step St21A, if the administrator AD requests optimization of the monitoring target by one or more sensors, the administrator AD executes an operation (request operation) to request the optimization process (St211). If the sensor SS1A receives a control command (optimization instruction) based on the administrator AD operation, the sensor SS1A starts the optimization process (St212A).

[0117] When the sensor SS1A receives a request for optimization processing via an administrator AD operation or detects a change in the environment of its sensing area, it generates dialogue data notifying the detection of the environmental change of its own sensor and transmits it to each of the other sensors (St31). Note that the other sensors to which the dialogue data is transmitted are sensors located in positions where the sensor SS1A can communicate wirelessly, and there must be at least one such sensor.

[0118] When each of the other sensors acquires the dialogue data transmitted from the sensor SS1A, it acquires its own current location information. The other sensor generates a control command notifying that it has received the dialogue data, associates the acquired current location information with the control command, and transmits the control command to the sensor SS1A. The other sensor may further associate its own specific knowledge data with the control command and transmit the control command to the sensor SS1A.

[0119] The sensor SS1A receives a control command from each of the other sensors notifying that it has received the conversation data, and the current position information of each of the other sensors (St32). The sensor SS1A calculates the distance between the sensor SS1A and each of the other sensors (St33).

[0120] The sensor SS1A determines that the other sensors whose distances to the sensor SS1A are within a predetermined distance are included in the interaction sensor group (St34).

[0121] Sensor SS1A generates dialogue data for consultation about monitoring of the monitored object using natural language processing based on the current environment of its own sensor and its role, and transmits the generated dialogue data and information about its own sensor (for example, the current location of sensor SS1A or the monitoring status of the monitored object by sensor SS1A (i.e., the number and location of the monitored object)) to sensors SS2A to SS5A, which are other sensors included in the group of interactive sensors. Sensor SS1A acquires dialogue data transmitted from the other sensors and including information about the other sensors or suggestions about monitoring of the monitored object based on the environment of the other sensors.

[0122] The sensor SS1A transmits and receives dialogue data (i.e., dialogue) with other sensors as described above (St35), and determines to maintain the current state of the monitoring target of the sensor based on the dialogue content of the dialogue data, and determines the destination of the sensor based on the determined monitoring target (St36). Note that if there is no change in the monitoring target, the sensor SS1A continues to track the existing monitoring target, and determines the destination of the sensor to be a location where the existing monitoring target can be tracked and monitored (St36).

[0123] In addition, sensor SS2A, which is another sensor included in the interactive sensor group, sets or adds as a new monitoring target the monitoring target that sensor SS1A has determined cannot be tracked or monitored based on the interaction including sensor SS1A, and determines the destination of its own sensor based on the determined monitoring target (St36).

[0124] In step St36 described above, the sensor SS1A acquires the role of its own sensor based on its own specific knowledge data, its current position, and the monitoring status of its own sensor's monitoring targets (e.g., the number, position, speed, or acceleration of the monitoring targets) based on the monitoring results (detection results) by the sensor element 24. Based on this referenced information, the sensor SS1A generates dialogue data including dialogue content for fulfilling the role of its own sensor, and executes dialogue with other sensors included in the dialogue sensor group based on the transmission and reception of dialogue data.

[0125] As described above, the monitoring system 100A according to the second embodiment can acquire the latest position information of each of a plurality of sensors whose positional relationships change dynamically by communicating with each other through communication between the sensors, and can determine a communication partner for each sensor based on the acquired sensor position information. This allows the monitoring system 100A to execute a dialogue regarding the monitoring target that each sensor is tracking and monitoring, based on the status of each sensor (e.g., the status of the environment, the status of the monitoring being performed by each sensor) acquired from the other sensors through dialogue.

[0126] Furthermore, the monitoring system 100A according to the second embodiment can change the targets monitored by the sensors SS1A to SSNA in real time even when static environmental changes (e.g., layout changes) or dynamic environmental changes (e.g., changes in the number of targets monitored depending on the time of day, or changes in the locations of targets monitored) occur. This allows the monitoring system 100A to more effectively prevent the sensors SS1A to SSNA from missing targets when monitoring targets.

[0127] If the sensor SS1A determines that it is difficult to monitor all of the monitoring targets through the optimization dialogue executed within the target sensor group, the sensor SS1A may perform the optimization process again by increasing the number of sensors to be determined as the target sensor group. Furthermore, when the optimization process is executed again, the sensor SS1A may relax the conditions for the sensors to be determined as the interaction sensor group (e.g., the distance between the sensor SS1A and the sensor SS1A, whether or not the sensing areas overlap, etc.).

[0128] As a result, even if the monitoring system 100A cannot monitor all of the monitoring targets due to the interaction of multiple sensors included in the target sensor group, by increasing the number of sensors included in the interaction sensor group, it can automatically monitor all of the monitoring targets using multiple sensors SS1A to SSNA.

[0129] Next, an example of changing the monitoring target of the sensor SS1A will be described with reference to Figures 5 to 6, 8, and 13. Figure 13 is a diagram showing an example of a dialogue between the sensors SS1A, SS2A, and SS3A in the second embodiment.

[0130] Note that sensors SS1A, SS2A, and SS3A shown in Figure 13 correspond to sensors SS1, SS2, and SS3, respectively, in the explanations of Figures 5, 6, and 8 above, and since the roles of sensors SS1 to SS3 and the monitored objects that are changed are similar, explanations thereof will be omitted here.

[0131] Based on the detection results of each of trucks Tg2 to Tg4 by sensor element 24, sensor SS3A determines that truck Tg2 is about to leave the sensing area and that it is not possible to track and monitor trucks Tg2 to Tg4 with just its own sensor; in other words, the current environment in which trucks Tg2 to Tg4 are being tracked and monitored will change to an environment in which it will no longer be possible to track and monitor truck Tg2, and only trucks Tg3 and Tg4 will be tracked and monitored.

[0132] When the sensor SS3A detects a change in the environment within the sensing area, it acquires the position information of each of the other sensors (i.e., the sensors SS1A and SS2A).The sensor SS3A determines that the acquired position information of each of the sensors SS1A and SS2A is within a predetermined distance from the sensor SS3A and includes the sensor SS3A in the interaction sensor group.

[0133] The sensor SS3A generates a notification that the environment in which the monitored object is tracked and monitored has changed, along with conversation data COM21, and transmits these to each of the sensors SS1A and SS2A included in the conversation sensor group. The sensor SS3A shown in Figure 13 uses general-purpose LLM data to generate conversation data COM21, "There are too many trucks near the gate and I can't capture them all! Help me!"

[0134] The sensor SS1A interprets the above-mentioned contents of the dialogue data COM21 using natural language processing. The sensor SS1A generates dialogue data COM22 using general-purpose LLM data based on the contents of the dialogue data COM21, the specific knowledge data for the sensor SS1A, and the current state of the sensor SS1A. The sensor SS1A generates dialogue data COM22 "Rescue has been received from No. 3. What should we do?", which is directed to the sensor SS2A, which is included in the dialogue sensor group and is other than the sensor SS3A.

[0135] The sensor SS2A interprets the content of the dialogue data COM22 using natural language processing. The sensor SS2A generates dialogue data COM23 using general-purpose LLM data based on the content of the dialogue data COM22, the specific knowledge data for the sensor SS2A, and the current state of the sensor SS2A. The sensor SS2 is included in the dialogue sensor group and generates dialogue data COM23 directed to the sensor SS1A, such as "This truck is the one you've been photographing all the way from the gate, so I'll leave this to you. I'll go." The dialogue data COM22 and COM23 may also be received by the sensor SS3.

[0136] As a result of the interaction (ie, each of the interaction data COM13-COM15), sensor SS1 continues to track and monitor truck Tg1.

[0137] Based on the result of the dialogue, sensor SS3 acquires its own position information or the position information and speed information of truck Tg2 and transmits it to sensor SS2. Also, based on the result of the dialogue, sensor SS3 performs tracking and monitoring of trucks Tg3 and Tg4. Note that sensor SS3 may continue monitoring truck Tg2 or measuring and transmitting the position information of truck Tg2 within the sensing area until sensor SS2 arrives at a position where it can monitor (image) truck Tg2, or may end monitoring truck Tg2 and prioritize tracking and monitoring trucks Tg3 and Tg4.

[0138] The sensor SS2 determines a position for monitoring (imaging) the truck Tg2, i.e., a movement destination, based on the position information of the sensor SS3 acquired from the sensor SS3, or the position information, speed information, etc. of the truck Tg2. The sensor SS2 determines the determined movement destination and starts movement to track and monitor the truck Tg2.

[0139] 8, sensor SS1 continues to track and monitor track Tg1, sensor SS2 is tracking and monitoring track Tg2, and sensor SS3 is tracking and monitoring tracks Tg3 and Tg4.

[0140] As described above, the monitoring system 100A can track and monitor all of the monitoring targets by having the sensors included in the interaction sensor group interact with each other and adjusting the monitoring targets of each of the multiple sensors SS1A to SS3A based on the environmental conditions observed by each of the sensors SS1A to SS3A. In other words, even if there are dynamic changes in the environment (monitoring targets), the monitoring system 100A can track and monitor the monitoring targets by responding in real time to the dynamic changes detected by the multiple sensors included in the interaction sensor group.

[0141] Furthermore, by allowing the sensors included in the interactive sensor group to interact with each other, the monitoring system 100A allows the sensors to share their status (e.g., specific knowledge data, information about the current location of the sensor, or information about the target being tracked) with each other, and can automatically optimize the number of targets that each sensor will track and monitor, the locations at which each sensor will track and monitor the targets, etc. For example, as shown in Fig. 13, sensors SS1A to SS3A can determine the sensor that will track and monitor the target based on the location or speed of each target.

[0142] (Additional Notes) The above description of each embodiment discloses the following techniques.

[0143] (Technology 1) A monitoring device (sensors SS1-SSN, SS1A-SSNA) that is movable and monitors a monitoring target, comprising: a sensor element 24 that detects the monitoring target; a dialogue unit (LLM processing unit 211) that generates dialogue data about a predetermined monitoring target (e.g., truck Tg2 shown in FIG. 6) using general-purpose LLM data based on the detection result of the monitoring target; and a determination unit (processor 21) that determines the monitoring target of the monitoring device, wherein the dialogue unit (LLM processing unit 211) transmits and receives the dialogue data to and from one or more other monitoring devices different from the monitoring device (i.e., other sensors included in a dialogue sensor group), and the determination unit (processor 21) determines the monitoring target to be monitored by the monitoring device based on the plurality of dialogue data transmitted and received between the other monitoring devices. This allows the sensors SS1 to SSN and SS1A to SSNA to determine the sensor for a specific monitoring target based on the status of each sensor (e.g., the environmental status, the status of the monitoring being performed by each sensor) acquired from each sensor through dialogue. In other words, the monitoring systems 100 and 100A can optimize the placement of sensors monitoring the monitoring target.

[0144] (Technology 2) The monitoring device (sensors SS1 to SSN, SS1A to SSNA) described in (Technology 1) is configured such that, when it is determined based on the detection result of the monitoring target that the monitoring target being monitored has deviated from the sensing area of ​​the sensor element 24, the dialogue unit (LLM processing unit 211) generates dialogue data regarding the monitoring of the monitoring target. As a result, when the sensors SS1 to SSN, SS1A to SSNA determine (estimate) that there is a monitoring target that has been missed based on the environment of the monitoring result (detection result) by the sensor element 24 of their own sensor, they execute a dialogue to determine the sensor that will monitor this monitoring target, thereby enabling tracking and monitoring of all monitoring targets.

[0145] (Technology 3) The determination unit (processor 21) determines whether or not the specified monitoring target can be monitored based on the dialogue data transmitted from the other monitoring device and the detection result of the monitoring target, The dialogue unit (LLM processing unit 211) generates dialogue data responding to the dialogue data transmitted from the other monitoring device using the general-purpose LLM data based on the determined whether or not the specified monitoring target can be monitored, The determination unit (processor 21) determines the monitoring target of the monitoring device based on the multiple dialogue data transmitted and received between the other monitoring device, The monitoring device (sensors SS1 to SSN, SS1A to SSNA) according to (Technology 1). As a result, the sensors SS1 to SSN, SS1A to SSNA can respond as to whether or not they can monitor the specified monitoring target, thereby determining which of the multiple sensors will monitor the monitoring target. In addition, when there are multiple sensors that can track and monitor a specified monitoring target, the sensors SS1 to SSN and SS1A to SSNA may determine the sensor that is closest to the specified monitoring target as the monitoring target.

[0146] (Technology 4) The monitoring device (sensors SS1 to SSN) according to any one of (Technology 1) to (Technology 3), wherein the dialogue unit (LLM processing unit 211) acquires a list of the other monitoring devices (interaction sensor group) to which the dialogue data is to be transmitted and received from an external device (master MS) communicably connected between the monitoring device and the other monitoring devices, and transmits and receives the dialogue data to and from the other monitoring devices included in the list. This allows the sensors SS1 to SSN to identify the sensor with which to conduct dialogue to monitor a predetermined monitoring target based on information from multiple sensors included in the interaction sensor group, even if the current locations of the sensors SS1 to SSN are unknown.

[0147] (Technology 5) The monitoring device (sensors SS1A to SSNA) according to any one of (Technology 1) to (Technology 3), wherein the dialogue unit (LLM processing unit 211) acquires location information of the other monitoring devices from the other monitoring devices connected to the monitoring device so as to be able to communicate wirelessly with the monitoring device, generates a list (group of dialogue sensors) of the other monitoring devices to which the dialogue data is to be sent and received based on the location information of the other monitoring devices, and executes transmission and reception of the dialogue data with the other monitoring devices included in the list. As a result, the sensors SS1A to SSNA can identify a group of dialogue sensors (multiple sensors) to engage in dialogue to monitor a predetermined monitoring target based on the positions of each sensor, even if the current positions of the sensors SS1A to SSNA are unknown.

[0148] (Technology 6) The other monitoring devices included in the list are other monitoring devices determined to be located at a predetermined distance or less between the monitoring device and the other monitoring devices (sensors SS1 to SSN, SS1A to SSNA) according to (Technology 4) or (Technology 5). This allows sensors SS1 to SSN, SS1A to SSNA to determine sensors capable of tracking and monitoring a predetermined monitoring target as sensors that are located at a predetermined distance or less, i.e., close to, a sensor that can detect the predetermined monitoring target. Therefore, the sensor monitoring the predetermined monitoring target can reduce the possibility of overlooking the predetermined monitoring target or searching for the predetermined monitoring target while moving to monitor the predetermined monitoring target.

[0149] (Technology 7) A monitoring system 100, 100A includes a plurality of monitoring devices (sensors SS1-SSN, SS1A-SSNA) that are movable and each monitor the same or different monitoring targets, each having a sensor element 24 for detecting a monitoring target, wherein a first monitoring device among the plurality of monitoring devices detects the monitoring target using the sensor element 24, and when detecting a change in the monitoring environment in which the monitoring target is monitored based on the detection result of the monitoring target, generates first dialogue data regarding the monitoring of a predetermined monitoring target (e.g., truck Tg2 shown in FIG. 6) using general-purpose LLM data and transmits the first dialogue data to one or more second monitoring devices different from the first monitoring device, and the second monitoring device generates second dialogue data for determining a third monitoring device (e.g., sensors SS2, SS2A shown in FIG. 6) among the second monitoring devices that will monitor the predetermined monitoring target based on the first dialogue data and the detection result of the monitoring target by the sensor element 24 of the second monitoring device, and transmits the second dialogue data to the first monitoring device, and the third monitoring device The monitoring system 100, 100A monitors the predetermined monitoring target by moving the predetermined monitoring target so as to be detectable based on the first dialogue data. This allows the monitoring system 100, 100A to determine the sensors related to the predetermined monitoring target based on the state of each sensor (e.g., the state of the environment, the state related to the monitoring being performed by each sensor) acquired from each sensor through dialogue. In other words, the monitoring system 100, 100A can optimize the placement of sensors monitoring the monitoring target.

[0150] (Technology 8) A monitoring method performed by multiple devices (sensors SS1-SSN, SS1A-SSNA) that are mobile and have sensor elements 24 for detecting monitoring targets and monitor the monitoring targets, the monitoring method includes: generating dialogue data related to monitoring the monitoring target (e.g., truck Tg2 shown in FIG. 6) using general-purpose LLM data when a change in the monitoring environment in which the monitoring target is monitored is detected based on the detection results of the monitoring target; transmitting and receiving multiple dialogue data between the multiple devices; determining which of the multiple devices will monitor the monitoring target (e.g., sensors SS2 and SS2A shown in FIG. 6) based on the multiple dialogue data; and having the determined device monitor the monitoring target. As a result, sensors SS1-SSN, SS1A-SSNA can determine which sensor is related to a specific monitoring target based on the status of each sensor (e.g., environmental status, status related to the monitoring performed by each sensor) acquired from each sensor through dialogue. In other words, the monitoring system 100, 100A can optimize the placement of sensors monitoring the monitoring target.

[0151] Although various embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that those skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components of the various embodiments described above may be combined in any manner without departing from the spirit of the invention.

[0152] This application is based on a Japanese patent application (Patent Application No. 2024-108894) filed on July 5, 2024, the contents of which are incorporated herein by reference.

[0153] The present disclosure is useful for a monitoring device, a monitoring system, and a monitoring method that realize efficient monitoring of an object using a plurality of freely movable monitoring devices.

[0154] 10, 20, 20A Communication I / F 11, 21, 21A Processor 12, 22 Memory 13 Input unit 23 Moving mechanism 24 Sensor element 25 Position receiver 100, 100A Monitoring system AD Administrator AR1, AR2, AR3 Field of view COM11, COM12, COM13, COM14, COM15, COM21, COM22, COM23 Conversation data MS Master SS1, SS2, SS3, SSN, SS1A, SS2A, SS3A, SSNA Sensor Tg1, Tg2, Tg3, Tg4 Track

Claims

1. A monitoring device that is freely movable and monitors a monitoring target, comprising: a sensor element that detects the monitoring target; a dialogue unit that generates dialogue data regarding a specific monitoring target using general-purpose LLM data based on the detection result of the monitoring target; and a determination unit that determines the monitoring target of the monitoring device, wherein the dialogue unit transmits and receives the dialogue data to and from one or more other monitoring devices different from the monitoring device, and the determination unit determines the monitoring target to be monitored by the monitoring device based on the multiple dialogue data transmitted and received between the other monitoring devices.

2. The monitoring device according to claim 1, wherein the dialogue unit generates dialogue data regarding the monitoring of the monitored object when it determines, based on the detection result of the monitored object, that the monitored object deviates from the sensing area of ​​the sensor element.

3. The monitoring device described in claim 1, wherein the determination unit determines whether or not to monitor the specified monitoring target based on the dialogue data transmitted from the other monitoring device and the detection result of the monitoring target, the dialogue unit uses the general-purpose LLM data to generate dialogue data responding to the dialogue data transmitted from the other monitoring device based on the determined whether or not to monitor the specified monitoring target, and the determination unit determines the monitoring target of the monitoring device based on the multiple dialogue data transmitted and received between the other monitoring device.

4. The monitoring device according to claim 1, wherein the dialogue unit acquires a list of the other monitoring devices to which the dialogue data is to be sent and received from an external device that is communicatively connected between the monitoring device and the other monitoring devices, and transmits and receives the dialogue data to and from the other monitoring devices included in the list.

5. The monitoring device according to claim 1, wherein the dialogue unit acquires location information of the other monitoring devices from the other monitoring devices connected to the monitoring device so as to be able to communicate wirelessly with the monitoring device, generates a list of the other monitoring devices to which the dialogue data is to be sent and received based on the location information of the other monitoring devices, and executes sending and receiving of the dialogue data with the other monitoring devices included in the list.

6. The monitoring device according to claim 4 or 5, wherein the other monitoring devices included in the list are other monitoring devices for which the distance between the monitoring device and the other monitoring devices is determined to be less than a predetermined distance.

7. A monitoring system comprising a plurality of monitoring devices each having a sensor element for detecting a monitoring target, each being movable and monitoring the same or different monitoring targets, wherein a first monitoring device among the plurality of monitoring devices detects the monitoring target using the sensor element, and when detecting a change in the monitoring environment in which the monitoring target is monitored based on the detection result of the monitoring target, generates first dialogue data regarding the monitoring of the monitoring target using general-purpose LLM data and transmits it to one or more second monitoring devices different from the first monitoring device, the second monitoring device generates second dialogue data for determining a third monitoring device among the second monitoring devices that will monitor the monitoring target based on the first dialogue data and the detection result of the monitoring target by the sensor element of the second monitoring device, and transmits it to the first monitoring device, and the third monitoring device moves so as to be able to detect the monitoring target and monitors the monitoring target based on the first dialogue data.

8. A monitoring method performed by a plurality of devices that have sensor elements for detecting a monitoring target, are movable, and monitor the monitoring target, wherein, when a change in the monitoring environment in which the monitoring target is monitored is detected based on the detection results of the monitoring target, dialogue data regarding the monitoring of the monitoring target is generated using general-purpose LLM data, multiple pieces of dialogue data are transmitted and received between the plurality of devices, and one of the plurality of devices is selected to monitor the monitoring target based on the multiple pieces of dialogue data, and the selected device is caused to monitor the monitoring target.

Images