Control method, control device, and control program

Abstract

A control server (10) is executed by a control device that controls task processing. The control server (10) acquires sensor data from a plurality of sensors. When a preset event is detected from the sensor data, the control server (10) activates a preset task. The control server (10) registers, in cooperation with a pod on which the task is executed, pod information including at least the pod name, the processing source, and the pod state, and controls change and / or stop of the pod processing content on the basis of the registered pod information, the event, and / or the processing result of the task.

Description

Control method, control device, and control program

[0001] The present invention relates to a control method, a control device, and a control program.

[0002] In recent years, in order to create smart cities, technologies have been proposed that connect to sensors such as multiple security cameras to perform monitoring and other functions (Non-Patent Document 1).

[0003] Semgrep, a security camera center that connects and manages multiple network cameras, [online], [Retrieved July 1, 2024], Internet <URL: https: / / www.trinity4e.com / network-camera / install-several-network-cameras.html>

[0004] These technologies require the use of machine learning models to analyze images of the monitored area captured by each camera, and for the cameras to work together to accomplish tasks.

[0005] However, in use cases such as smart cities, when monitoring and detecting events across multiple cameras installed in different locations, there are still challenges in optimizing the analysis of data captured by multiple cameras and linking them together to accomplish tasks without delay.

[0006] The present invention has been made in consideration of the above, and aims to provide a control method, a control device, and a control program that can achieve tasks without delay by linking multiple sensors.

[0007] In order to solve the above-mentioned problems and achieve the objectives, the control method of the present invention is a control method executed by a control device that controls task processing, and is characterized by including the steps of: acquiring sensor data from multiple sensors; launching a predetermined task when a predetermined event is detected from the sensor data; and registering pod information including at least the pod name, the source being processed, and the state of the pod in cooperation with the pod on which the task is executed, and controlling the change and / or stop of the processing content of the pod based on the registered pod information, the event, and / or the processing result of the task.

[0008] In addition, the control device of the present invention is a control device that controls task processing, and is characterized by having an acquisition unit that acquires sensor data from multiple sensors, a startup unit that starts a predetermined task when a predetermined event is detected from the sensor data, and a control unit that works in cooperation with the pod on which the task is executed to register pod information including at least the pod name, the source being processed, and the state of the pod, and controls changing and / or stopping the processing content of the pod based on the registered pod information, the event, and / or the processing result of the task.

[0009] In addition, the control program of the present invention causes a computer to perform the following steps: acquiring sensor data from multiple sensors; launching a predetermined task when a predetermined event is detected from the sensor data; and registering pod information including at least the pod name, the source being processed, and the state of the pod in cooperation with the pod on which the task is executed, and controlling the change and / or stop of the processing content of the pod based on the registered pod information, the event, and / or the processing result of the task.

[0010] According to the present invention, a task can be accomplished without delay by linking multiple sensors.

[0011] FIG. 1 is a diagram illustrating an outline of inference processing executed in an embodiment. FIG. 2 is a diagram illustrating an example of the configuration of a control system according to an embodiment. FIG. 3 is a diagram illustrating an example that can be realized in a control system according to an embodiment. FIG. 4 is a diagram illustrating an example of the configuration of the control server shown in FIG. 1. FIG. 5 is a diagram illustrating definitions of function states. FIG. 6 is a diagram illustrating function states. FIG. 7 is a diagram illustrating function pod information according to an embodiment. FIG. 8 is a diagram illustrating function collaboration according to an embodiment. FIG. 9 is a diagram illustrating processing of a specific task instruction. FIG. 10 is a flowchart illustrating an example of a processing procedure for control processing according to an embodiment. FIG. 11 is a flowchart illustrating an example of a processing procedure for control processing according to an embodiment. FIG. 12 is a flowchart illustrating an example of a processing procedure for control processing according to an embodiment. FIG. 13 is a flowchart illustrating an example of a processing procedure for control processing according to an embodiment. FIG. 14 is a diagram illustrating the processing flow of the control server shown in FIG. 4. FIG. 15 is a diagram illustrating the processing flow of the control server shown in FIG. 4. FIG. 16 is a diagram illustrating the processing flow of the control server shown in FIG. 4. FIG. 17 is a diagram illustrating the processing flow of the control server shown in FIG. 4. FIG. 18 is a diagram illustrating an example of a computer in which a control server is realized by executing a program.

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are designated by the same reference numerals.

[0013] [Embodiment] FIG. 1 is a diagram illustrating an outline of inference processing executed in an embodiment. In a control system according to an embodiment, a control server collects images from each camera, collects sensor data (e.g., images captured by the cameras), and when an event is detected based on the collected sensor data ((1) in FIG. 1), the control server starts processing a predetermined task. The task processing may be performed by signal processing or by inference processing using a pre-trained model ((2) in FIG. 1). The event and task may be of any type. For example, the event may be the detection of a person in sensor data. For example, the task may be identifying the location of the person photographed. The control server does not need to perform actual inference during periods when no events occur for each camera, thereby reducing processing resources, power consumption, and costs.

[0014] 2 is a diagram showing an example of the configuration of a control system according to an embodiment. The control system 1 according to the embodiment includes multiple cameras 20A to 20H, a control server 10 (control device), and a user terminal 30 that receives notifications of processing results from the control server 10.

[0015] The cameras 20A to 20H capture images of their respective monitoring areas, and transmit the captured images to the control server 10 along with the identification information of the camera that captured the image (e.g., camera ID) and the capture time. For example, the cameras 20A to 20D are installed at predetermined intervals. The control server 10 stores relationship information (such as positional relationships) between the cameras 20A to 20H in advance. The control system 1 may also include other sensors in addition to the cameras 20A to 20H. Of the cameras 20A to 20H, for example, the cameras 20A to 20D are installed in an arbitrary area, and the cameras 20E to 20G are installed in an area adjacent to the area of ​​the cameras 20A to 20D. The number of cameras is not limited to that shown in FIG. 2. When the cameras 20A to 20H are collectively referred to as cameras 20, they will be referred to as cameras 20.

[0016] The control server 10 uses an inference model to analyze images of the monitoring area captured by each of the cameras 20A to 20H, and performs tasks by linking processes (e.g., inference processes) based on the images captured by each of the cameras 20A to 20H. Tasks include, for example, tracking target people, cars, etc. on a city-by-city basis (across multiple cameras) in a smart city, detecting congestion, accidents, etc. across multiple cameras and notifying appropriate vehicles according to the direction of travel of surrounding vehicles, detecting dangerous bicycles or cars at intersections, etc., and notifying appropriate vehicles according to the direction of travel of surrounding vehicles to avoid collisions, and processing such as four-dimensional digitalization and traffic flow optimization.

[0017] In the embodiment, in order to realize each of the above processes, a predetermined task is set in advance for each event. Each task is provided with an inference model for executing the respective task. Furthermore, for example, a large inference model with high inference accuracy and a small inference model with low inference accuracy are set as the inference models. Furthermore, the tasks may be executed by signal processing. In this case, the signal processing is set to be a process with relatively low accuracy and low computational complexity, and a process with relatively high accuracy and high computational complexity.

[0018] For example, because attention needs to be paid to the behavior of bicycles at intersections, an event "bicycle moving through an intersection" is set. This event is the detection of a new image of a bicycle moving through the intersection in an image captured by a camera capturing the intersection. For example, a tracking task for detecting and tracking the bicycle is set for this event. Then, in conjunction with the tracking task, for example, a risk determination task for determining risk and a distance determination task are associated. Risk may be, for example, heading toward the intersection and / or traveling at a predetermined speed or greater. The distance determination task is a task for determining the distance between the bicycle and the camera that detected the bicycle moving through the intersection.

[0019] 3 is a diagram illustrating an example that can be implemented in the control system according to the embodiment. For example, when the control server 10 detects an event, such as a bicycle passing through an intersection, from an image captured by camera 20A ((1) in FIG. 3), it performs inference processing for a bicycle tracking task using the image captured by camera 20A ((2) in FIG. 3). At the same time, the control server 10 instructs cameras 20B, 20C, and 20D in the vicinity of the next location where the bicycle will appear to begin tracking tasks (arrows Y2-1 to Y2-3). This causes the inference processing for the bicycle tracking task to be executed using cameras 20B, 20C, and 20D ((3) in FIG. 3).

[0020] Then, the control server 10 executes the task and, based on the processing result, changes the behavior due to the event in the same source.

[0021] As a distance determination task, the control server 10 infers the distance between each of the cameras 20B, 20C, and 20D and the bicycle ((4) in FIG. 3). Here, we will explain a case where the bicycle appears large in the image from camera 20D and it is inferred that the distance between the bicycle and camera 20D has become closer (less than a predetermined distance). In this case, even a small inference model can track the bicycle with a predetermined accuracy. Therefore, the control server 10 changes the behavior of the event in the same source ((5) in FIG. 3) so that the inference processing is performed by changing from a large inference model to a small inference model for the image from camera 20D ((4) in FIG. 3). Note that even when the control server 10 is performing one task based on sensor data obtained from multiple cameras, it is possible to adaptively change the inference model used in a task that processes related data depending on the results of another task (the distance between the camera and the bicycle).

[0022] The control server 10 also performs a risk assessment task using images from the cameras 20A, 20B, 20C, 20D, etc. ((6) in FIG. 3). For example, if it is determined that the risk has disappeared because the bicycle has changed direction, the control server 10 stops the tracking task using the surrounding cameras (e.g., cameras 20A, 20B, 20C, 20D, 20H) ((7) in FIG. 3).

[0023] As the bicycle moves, the control server 10 instructs the camera 20F, which is located near the camera 20C, to execute a tracking task (arrow Y3 in FIG. 3). As a result, the inference process for the tracking task for the bicycle is executed using the camera 20F ((8) in FIG. 3).

[0024] Here, when the control server 10 detects an event that a speeding vehicle is present in the image from the camera 20E ((9) in FIG. 3), it instructs the cameras 20E and 20F to start a vehicle tracking task, including matching the vehicle license plate number, etc. This causes the cameras 20E and 20F to execute an inference process for the vehicle tracking task ((10) in FIG. 3, arrow Y4 in FIG. 3). The vehicle tracking task is associated with a distance determination task and a risk determination task.

[0025] In this case, the control server 10 starts and / or stops another tracking task based on a different event for the same source (camera 20F). In addition to the car tracking task (car number matching task) using images from camera 20F, the control server 10 also performs a bicycle tracking task using images from camera 20F ((5) in Figure 3).

[0026] The control server 10 controls the number of pods in response to events. A pod is a unit of function execution. Pods may also be individually identifiable. To accomplish tasks linked between processes based on images captured by each camera 20 as described above, the control server 10 pre-installs standby pods in addition to function pods that execute tasks. A standby pod is a pod that is prepared in advance to be assigned to one of the tasks and is in a standby state. An inference model is also located in the function execution unit (described below) within the function pod.

[0027] The control server 10 then built a system that can manage function information (pod information) that indicates the status of function pods and standby pods, realizing cooperation between function pods. In other words, the control server 10 built a configuration that can manage which source is the target, what processing task is issued, which pod is processing it, and which pod is standby. This allows the control server 10 to pinpoint the pod that corresponds to various instructions.

[0028] For example, the control server 10, in conjunction with events, task processing results, and / or pod status, identifies function pods that have already started processing but no longer need to process based on function information, and stops the processing of those function pods. This allows the control server 10 to immediately stop tasks that no longer need to be processed, achieving optimal resource distribution.

[0029] Furthermore, the control server 10 identifies function pods whose behavior is to be changed based on events, task processing results, and / or function information, and changes the inference model and model parameters in the processing pipeline of the identified function pod. In this way, the control server 10 optimizes the inference model or its model parameters used to accomplish a task in accordance with changes in various fluctuating events, etc. This allows the control server 10 to flexibly respond to a variety of inference processing use cases while reducing the amount of analysis and / or inference processing.

[0030] Furthermore, the control server 10 obtains the status of function pods and standby pods from function information, and automatically scales in (reducing) and scales out (expanding) the function pods according to the task processing volume while maintaining a certain number of standby pods. This allows the control server 10 to accomplish tasks without delay in conjunction with the processing status of events and tasks, while achieving low resource usage, low power consumption, and low cost throughout the system.

[0031] [Control Server] A specific description will be given of the control server 10. Fig. 4 is a diagram showing an example of the configuration of the control server 10 shown in Fig. 1 .

[0032] The control server 10 is realized by loading a predetermined program into a computer or the like including, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a CPU (Central Processing Unit), etc., and having the CPU execute the predetermined program. The control server 10 is also capable of communicating with other devices connected via a network or the like via a communication interface that transmits and receives various types of information to and from other devices.

[0033] The control server 10 includes a management unit 11 , a determination unit 12 , a control mechanism unit 13 , and a function mechanism unit 14 .

[0034] The management unit 11 manages control processing related to the control system 1. The management unit 11 includes an overall management unit 111, an external data linkage unit 112 (acquisition unit), an external database (DB) 113, a processing result DB 114, and function information 115 (pod information).

[0035] The overall management unit 111 defines the processing content, the relationships between functions, and the data processing flow, and deploys the necessary mechanisms.

[0036] The external data linking unit 112 acquires sensor data from multiple sensors (for example, the cameras 20). The external data linking unit 112 communicates with the cameras 20A to 20H or various devices such as communication devices that transmit images from the cameras 20A to 20H to acquire information about external input sources and link them within the control server 10. The external input sources include information about each camera, images captured by each camera (sensor data), and the like.

[0037] The external DB 113 stores input sources and metadata. The input sources include camera information and image data captured by a camera, and are stored by the external data linking unit 112. The input sources are, for example, uniform resource locators (URLs) of videos and streams. The camera information includes, for example, the location and model of the camera, as well as internal camera parameters. Note that a camera is a type of sensor, and image data is an example of sensor data. The metadata includes source information, task information, and the like, and are stored by the external data linking unit 112.

[0038] The processing result DB 114 stores the processing results (inference results) of the tasks executed by the functions 141A to 141N.

[0039] The function information 115 includes at least information indicating the name of the function 141, the pod name, task information, the source being processed, and the status of the function pod and standby pod. The function information 115 includes at least information indicating the name of the function 141 (described later), the pod name, task information, the source being processed, and the status of a function execution unit 1411 (described later) of the function 141 (described later). The function information 115 is registered by the function collaboration unit 132 (described later). The function information 115 is referenced by the function collaboration unit 132 and the autoscaler 142 (described later). As described later, idle, processing, and stopping are defined as the status of a function.

[0040] The determination unit 12 includes an event issuing unit 121 (activation unit).

[0041] The event publisher 121 detects a preset event from the sensor data. The event publisher 121 references the metadata in the external DB 113 and the inference results in the processing result DB 114. The event publisher 121 determines an event according to the use case based on the metadata and the inference results. To determine the event, the event publisher 121 determines which function should process which source. Then, the event publisher 121 issues an event and issues an instruction to the control mechanism unit 13 (described later).

[0042] The event issuing unit 121 instructs the function linking unit 132 (described later) to stop processing or change the behavior of which task of which source based on the event and / or task processing result. The processing result of a task may be determined based on the result of a predetermined task associated with this task.

[0043] The control mechanism unit 13 controls task execution of each function of the function mechanism unit 14. The control mechanism unit 13 controls the functions for processing the tasks based on the task processing results. The control mechanism unit 13 has a task instruction unit 131 (launch unit) and a function collaboration unit 132 (control unit).

[0044] When an event is detected, the task instructing unit 131 starts a predetermined task. The task instructing unit 131 issues a processing task to one of the functions 141A to 141N (described later) and instructs the function to start the task, depending on the content of the event issued by the event issuing unit 121. When a predetermined event is detected from the sensor data, the task instructing unit 131 issues a processing task and starts a task that has been predetermined for this event.

[0045] The function collaboration unit 132 collaborates with the pod in which the task is executed and registers function information 115 including at least the pod name, the source being processed, and the state of the pod. The function collaboration unit 132 controls the change and / or stop of the processing content of the pod based on the registered function information, event, and / or task processing result as a specific task instruction.

[0046] The function collaboration unit 132 acquires and holds information about functions and collaborates with other functional units by registering the information about the functions in the function information 115. The function information indicates the tasks and data associated with the functions, and also indicates whether the function is in an idle (waiting), processing (running), or stopping state.

[0047] The function collaboration unit 132 controls the stopping of pod processing by referring to the function information 115 based on instructions from the determination unit 12. The function collaboration unit 132 controls changes to the pod processing content (changes in behavior) by referring to the function information 115 based on instructions from the determination unit 12. Changes in behavior include changes or switching of inference models, inference model parameters, etc. Parameters include, for example, detection thresholds, detection classes, and judgment thresholds. Inference models, inference model parameters, etc. are switched in accordance with changes in the relationship between sensors (e.g., cameras) and events (e.g., analysis difficulty).

[0048] The function mechanism unit 14 executes each task and includes functions 141A to 141N and an autoscaler 142 (control unit).

[0049] The functions 141A to 141N analyze images captured by the cameras 20A to 20H and perform inference processing using the images. The functions 141A to 141N store the processing results (inference results) in the processing result DB 114. The tasks performed by the functions 141A to 141N depend on the use case. When referring to the functions 141A to 141N collectively, they are referred to as function 141.

[0050] The functions 141A to 141N have function execution units 1411A to 1411N and function control units 1412A to 1412N, respectively.

[0051] The function execution units 1411A to 1411N are processes that execute functions (for example, one or more main function containers). When the function execution units 1411A to 1411N are collectively referred to as function execution units 1411.

[0052] The function control units 1412A to 1412N are, for example, sidecar containers and function control processes.

[0053] The function control units 1412A to 1412N function as interfaces that acquire information and states of the function execution units 1411A to 1411N and control the function execution units 1411A to 1411N. The function control units 1412A to 1412N function as interfaces that register the information and states of the function execution units 1411A to 1411N. The information and state information of the function execution units 1411A to 1411N output from the function control units 1412A to 1412N is registered in the function information 115 by the function collaboration unit 132. When the function control units 1412A to 1412N are referred to collectively, they will be referred to as function control units 1412.

[0054] The autoscaler 142 automatically scales in and out function pods according to the task processing volume. The autoscaler 142 automatically scales in and out function pods while maintaining a constant number of standby pods. The autoscaler 142 maintains a constant number of standby pods based on function information, events, and / or task processing results.

[0055] [Function Status] As described above, function information is information that includes at least the name of the function 141, the pod name, task information, and the status of the function execution unit 1411. The status of the function execution unit 1411 (function status) can be, for example, processing, waiting, stopped, etc. In the embodiment, idle (waiting), processing, and stopping are defined as the function status. FIG. 5 is a diagram for explaining the definition of the function status. FIG. 6 is a diagram for explaining the function status.

[0056] As shown in FIG. 5, in this embodiment, idle, processing, and stopping are defined as function states.

[0057] Idle is a state in which the task queue is polled and awaiting a task. Processing is a state in which a task is being executed. Stopping is a state in which processing is stopped. When the autoscaler 142 instructs a function pod to start ((1) in Figure 6), the function pod starts and begins polling the task queue in idle state as a standby pod. Once a task is acquired, the corresponding pod transitions from idle to processing. Then, when the function collaboration unit 132 instructs a function to stop ((2) in Figure 6), this function pod transitions from processing to stopping ((3) in Figure 6) and then becomes idle.

[0058] [Function Information] Function information will be described below. Fig. 7 is a diagram illustrating function pod information according to an embodiment.

[0059] As shown in FIG. 7, the control server 10 manages the pod name (primary key), status, order, and metadata for each function pod.

[0060] podname is the name of the pod. status indicates the status of the auto-scaling function. order indicates the instruction, which can be "stop" (perform stop processing), "stop-done" (completion of stop processing), "change" (perform behavior change), or "change-done" (completion of behavior change). metadata is source information, task information, etc.

[0061] [Function Cooperation] Next, function cooperation will be described. Fig. 8 is a diagram illustrating function cooperation in the embodiment.

[0062] In this embodiment, function controllers 1412A to 1412N are implemented in each function pod. The function information (status) of the function execution unit 1411 of each function pod is periodically acquired by the corresponding function controllers 1412A to 1412N and reflected in the function collaboration unit 132 ((2) and (6) in FIG. 8). The pod name is registered after the pod is started. The order is registered when an instruction is given and when the operation corresponding to the instruction is completed. The metadata is registered after a task is acquired. If there is a difference with the cache inside the function collaboration unit 132, the function collaboration unit 132 updates the function information 115 using the information acquired by the function controllers 1412A to 1412N ((3) in FIG. 8).

[0063] The autoscaler 142 periodically refers to the function information 115 ((4) in Figure 8) and issues scaling instructions by deploying function pods ((1) and (5) in Figure 8) while maintaining a constant number of standby pods using scale metrics (indicators) described below.

[0064] In the example of FIG. 8, if you want to maintain two standby pods for a tracing function (left side of FIG. 8), when a tracing task is issued, one of the two standby pods originally on the left (for example, function 141E) becomes in-process (tracing pod1 (in-process)) to process the task, leaving only one standby pod. In this state, to maintain two standby pods, the autoscaler 142 starts an additional new standby pod (tracing processing standby pod2) (for example, function 141C). Therefore, two standby pods are maintained.

[0065] The following describes scale metrics. The control server 10 manages the pod status (idle, processing, or stopping) for each function in the function information 115. The control server 10 then references the status of each function pod in the function information 115 to always maintain the number of standby pods [externally specified (use case dependent)] for each function using formula (1), calculates the number of pods required for startup, and passes this information to the autoscaler 142. The pod information is referenced, and the number of standby pods to be maintained, specified according to the use case, and the number of function execution units 1411 that are processing are used to calculate the number of pods required for startup.

[0066]

[0067] The autoscaler 142 scales out if the calculated number of formula (1) is greater than the number of running pods. The autoscaler 142 scales in if the calculated number of formula (1) is less than the number of running pods.

[0068] Conventionally, various metrics have been provided as indicators of scale. There are methods for scaling using metrics such as the number of messages in a queue or the number of records in a database. However, with conventional methods, scaling in occurs when there are no more messages (i.e., tasks) in the queue, making it impossible to maintain standby pods that are meant to wait when there are no tasks. Furthermore, using the number of records in a database, as with conventional methods, requires task management in the database, and does not allow optimal scaling according to the processing volume of the actual tasks being run.

[0069] Therefore, as described above, the control server 10 uses the number of recorders in the DB as a metric and constructs a configuration for managing function information, thereby achieving optimal scale-out and scale-in while maintaining a constant number of standby pods.

[0070] In this way, the autoscaler 142 maintains the number of standby pods at a fixed number and performs autoscaling, including on standby pods, depending on the state of each function pod. Therefore, even if it takes time to start a new function pod, the autoscaler 142 in the control server 10 starts a fixed number of standby pods in advance, so that processing can begin quickly and without delay the moment a task instruction is received.

[0071] The control server 10 can specify the maximum number of pods that can be started by the autoscaler 142 for each function, so pods are not started indefinitely depending on the amount of processing tasks. In particular, when processing tasks for short videos, such as 1 second long, and latency requirements are not strict, it is desirable to have the same pod process multiple tasks in order to reduce resources and pod startup overhead. In this case, it is recommended to specify a maximum number of pods that is less than the amount of tasks.

[0072] Therefore, the control server 10 can achieve optimal resource allocation according to the amount of processing tasks, and can reduce power consumption and costs by avoiding unnecessary resource allocation and pod activation. Furthermore, by always maintaining a certain number of standby pods, the control server 10 can achieve real-time processing of tasks without delay. Therefore, the entire system can achieve low latency, low resource usage, low power consumption, and low cost.

[0073] [Stopping Functions / Changing Behavior] The control server 10 has built a system for managing function information and implemented a function collaboration (specific task instruction) mechanism. The control server 10 manages which source is the target, what processing task is issued, which pod is processing it, and the status of each pod. This allows the control server 10 to pinpoint the pod that corresponds to the instruction target.

[0074] The control server 10 pinpoints the pod that corresponds to the instruction target, enabling flexible control such as stopping a function that has already started processing or changing the behavior of a function that is currently processing. Note that a function that has already started processing can be all tasks or tasks of one or more corresponding processing sources. An example of stopping a function is stopping processing for cameras 20A and 20B. An example of changing the behavior of a function is changing a model (e.g., an inference model) or model parameters in the processing pipeline.

[0075] [Specific Task Instruction] A specific task instruction by function cooperation will be described below. Fig. 9 is a diagram for explaining the processing of a specific task instruction.

[0076] Each of the functions 141A to 141N implements a function control unit 1412A to 1412N in its function pod. The function control units 1412A to 1412N acquire function information [pod name, status, metadata, etc.] of the corresponding function execution units 1411A to 1411N, and transmit it to the function collaboration unit 132 ((1) in FIG. 9).

[0077] The function collaboration unit 132 registers the function information [pod name, status, metadata, etc.] sent from the function control units 1412A to 1412N in the function information 115 ((2) in FIG. 9).

[0078] The event issuing unit 121 instructs the function linkage unit 132 to stop processing of a function or change the behavior of a function depending on the detected event or task processing result ((3) in FIG. 9). The event issuing unit 121 instructs the function linkage unit 132 as to which source and which task to stop processing or change the behavior of.

[0079] In response to an instruction from the event publisher 121, the function collaboration unit 132 queries the function information 115 for the name of the pod that processes the source and task corresponding to the instruction ((4) in FIG. 9). Then, the function collaboration unit 132 changes the order field of the corresponding pod to "execute" ((4) in FIG. 9), and instructs the function controller (e.g., function controller 1412A) of the corresponding function (e.g., function 141A) to stop or change its behavior ((5) in FIG. 9).

[0080] In accordance with this instruction, for example, the function control unit 1412A stops the processing of the function execution unit 1411A or changes the behavior of the function execution unit 1411A ((6) in FIG. 9). After the function control unit 1412A has stopped the processing or changed the behavior, it returns a notice of completion to the function collaboration unit 132 ((7) in FIG. 9).

[0081] The function cooperation unit 132 changes the "order" column of the pod corresponding to the instruction to stop the current process or change the behavior to "completed" in the function information 115 ((8) in FIG. 9).

[0082] In this way, the control server 10 immediately stops processing tasks that no longer need to be processed, thereby achieving optimal resource distribution. The control server 10 can reduce power consumption and costs by avoiding unnecessary resource allocation and / or processing.

[0083] The control server 10 then optimizes the content of task processing as needed to adapt to changes, thereby reducing the amount of analysis and / or inference processing, and reducing power consumption and costs, thereby achieving low resource usage, low power consumption, and low costs throughout the entire system.

[0084] [Control Processing Procedure] Next, the control processing procedure according to the embodiment will be described with reference to a flowchart of FIG.

[0085] In the control server 10, the event issuing unit 121 acquires sensor data from multiple sensors, and issues an event when a preset event is detected from the sensor data (step S1). The control mechanism unit 13 starts a preset task by instructing one of the functions 141A to 141N according to the content of the event (step S2).

[0086] In the function 141 to which the task has been specified, the function execution unit 1411 executes the processing of the specified task (for example, inference processing) (step S3). The function control unit 1412 acquires pod information and status of the function execution unit 1411 to be controlled, and transmits them to the function collaboration unit 132. The function collaboration unit 132 registers the received information of the function execution unit 1411 in the function information 115 (step S4).

[0087] The autoscaler 142 performs auto-scaling of the function pods while maintaining the number of standby pods at a fixed number based on the function information, events, and / or task processing results (step S6). Note that the autoscaler 142 performs auto-scaling processing in parallel with step S3, as well as various processes in the control server 10.

[0088] The function 141 registers the processing result of the function execution unit 1411 in the processing result DB 114 (step S5). After step S5 is completed, the control server 10 returns to step S1, for example.

[0089] The event publisher 121 determines whether to stop the task being executed or change its behavior based on the function information, the event, and / or the task processing result (step S7).

[0090] When stopping a task (step S7: Stop), the function collaboration unit 132 refers to the function information 115, identifies the function pod to be stopped (step S8), and instructs the function control unit 1412 of the corresponding pod to stop the function 141.

[0091] Upon receiving the instruction to stop, the function execution unit 1411 stops processing (step S9) and returns the function pod to standby mode. The function control unit 1412 then acquires information and status of the function execution unit 1411 to be controlled and transmits this information to the function collaboration unit 132. The function collaboration unit 132 updates the function information 115 using the received information from the function execution unit 1411 (step S10).

[0092] In the case of behavior change (step S7: behavior change), the function collaboration unit 132 identifies the function pod to be changed (step S11). The function collaboration unit 132 issues a behavior change instruction (e.g., change of inference model) to the function control unit 1412 of the identified pod.

[0093] The function control unit 1412, having received the instruction to change the behavior, changes the behavior of the corresponding function execution unit 1411 (step S12). Then, the function control unit 1412 acquires information and status of the function execution unit 1411 to be controlled, and transmits this to the function collaboration unit 132. The function collaboration unit 132 updates the function information 115 using the received information of the function execution unit 1411 (step S13).

[0094] [Effects of the embodiment] As described above, the control server 10 implements a function collaboration (specific task instruction) mechanism that enables management of function pod information by providing a function control unit 1412 that controls the state and information acquisition of the function execution unit 1411 for each function 141. This enables the control server 10 to link processes based on sensor data from multiple sensors, stop functions that have already started processing, and change models (e.g., inference models) and model parameters in the processing pipeline.

[0095] Therefore, the control server 10 immediately stops tasks that no longer need to be processed, achieving optimal resource distribution. The control server 10 can reduce power consumption and costs by avoiding unnecessary resource allocation and / or processing. Furthermore, the control server 10 can reduce the amount of analysis and / or inference processing, etc., by optimizing the content of processing as needed in response to changes, thereby reducing power consumption and costs. This makes it possible to achieve low resource usage, low power consumption, and low costs throughout the entire system.

[0096] As described above, according to the embodiment, it is possible to perform processing only when necessary, not only within one source (e.g., a camera) but across all sources, depending on the event. It is also possible to flexibly respond to a variety of inference processing use cases in response to a variety of changing events. Furthermore, the embodiment enables flexible placement according to the characteristics and processing load of functional blocks. That is, the embodiment enables flexible design of the processing scope of each function according to the characteristics, processing load, and resource status of the processing pipeline, and flexible placement of the number of function pods accordingly.

[0097] Furthermore, the control server 10 maintains a fixed number of standby pods and performs auto-scaling, including standby pods. Therefore, even if it takes time to start a new function pod, the control server 10 can execute the task without delay the moment a task instruction is received by starting the standby pods in advance. Furthermore, the control server 10 can specify the maximum number of pods that can be started by the autoscaler 142 for each function, so pods are not started without limit depending on the amount of processing tasks. Therefore, the control server 10 can achieve optimal resource distribution according to the amount of processing tasks.

[0098] Therefore, in the embodiment, even if there are multiple cameras, sensors, etc. per city in a smart city, tasks such as overall analysis, tracking, and various judgments can be accomplished without delay by linking processes based on sensor data from multiple sensors. At the same time, in the embodiment, low resource usage, low power consumption, and low cost can be achieved throughout the entire system.

[0099] [Application Examples] Application examples of the embodiment will be described. Figures 11 to 13 are flowcharts showing an example of the processing procedure for control processing according to the embodiment. Figures 14 to 17 are diagrams explaining the processing flow of the control server 10 shown in Figure 4. Figures 11 to 17 show processing procedures for actually realizing the example of Figure 3.

[0100] When the management unit 11 detects an event of a bicycle moving through an intersection (be careful), it defines a flow to track the bicycle and determine the risk, and deploys the related mechanisms (step S21 in FIG. 11, (1) in FIG. 14).

[0101] The event publisher 121 references cameras near the intersection from the external DB 113 and issues, for example, the source URLs of the cameras 20A and 20H and a bicycle detection analysis task for each source (step S22 in FIG. 11, (2) in FIG. 14).

[0102] In the control mechanism unit 13, the task instructing unit 131 adds each camera source (camera source of cameras 20A, 20H) to the task queue of the bicycle detection function (step S23 in FIG. 11, (3) in FIG. 14).

[0103] For example, function 141A (bicycle detection function) acquires the camera source of camera 20A and performs inference processing for the bicycle detection task (step S24 in FIG. 11, (4-1) in FIG. 14). Function 141A registers the analysis results in the processing result DB 114 (step S24 in FIG. 11, (4-2) in FIG. 14). The function control unit 1412 acquires the pod information and state (processing) of the function execution unit 1411A to be controlled, and the function collaboration unit 132 updates the state of this function 141 in the function information 115 to processing.

[0104] In this case, the autoscaler 142 may refer to the function information 115, maintain a certain number of standby pods, and automatically scale out according to the amount of tasks queued in the task queue and the number of function pods currently processing the tasks ((5) in Figure 14).

[0105] The event publisher 121 refers to the processing result DB 114, acquires the processing result, and determines whether or not a bicycle is present (step S25 in FIG. 11, (6) in FIG. 15).

[0106] If a bicycle is present (step S25 in FIG. 11: bicycle present, (6) in FIG. 15), the event publisher 121 issues a tracking task to each of the source URLs of cameras 20B, 20C, and 20D in the vicinity of the location (camera 20A) where the bicycle is located (step S26 in FIG. 11, (6) in FIG. 15). If no bicycle is present (step S25 in FIG. 11: bicycle not present), the control server 10 returns to the processing of step S24.

[0107] In the control mechanism unit 13, the task instruction unit 131 adds a bicycle tracking function to the task queue for each source URL of the cameras 20B, 20C, and 20D (step S27 in FIG. 11, (7) in FIG. 15).

[0108] For example, functions 141B to 141D acquire the camera sources of cameras 20B, 20C, and 20D and perform inference processing for the bicycle tracking task (step S28 in FIG. 11, (8-1) in FIG. 15). Functions 141B to 141D register the processing results in the processing result DB 114 (step S28 in FIG. 11, (8-2) in FIG. 15). Function control units 1412B to 1412D acquire the pod information and status (processing) of the function execution units 1411B to 1411D to be controlled. The function collaboration unit 132 updates the status of these functions 141B to 141D in the function information 115 to "processing."

[0109] At this time, the autoscaler 142 may refer to the function information 115, maintain a certain number of standby pods, and automatically scale out according to the number of tasks queued in the task queue ((9) in FIG. 15).

[0110] The event issuing unit 121 acquires the processing result from the processing result DB 114 and instructs a risk assessment task based on the speed and direction of the bicycle. The task instructing unit 131 causes one of the functions 141 to execute the risk assessment task based on the images from the cameras 20B, 20C, and 20D and registers the processing result in the processing result DB 114.

[0111] The event publisher 121 determines whether or not there is a risk based on the processing result of the risk determination task (step S29). If there is a risk (step S29 in FIG. 11: risk exists), the control server 10 returns to step S26 and continues the tracking task.

[0112] If the results of camera 20D indicate that there is no risk (step S29 in Figure 11: no risk), the event issuing unit 121 issues an instruction to the function collaboration unit 132 to stop the processing for cameras 20B, 20C, and 20D that are currently undergoing tracking processing (step S41 in Figure 12, (10) in Figure 16).

[0113] The function collaboration unit 132 identifies the function pod that is processing the tracking task for cameras 20B, 20C, and 20D, and instructs the function control units 11412B to 1412D of the corresponding function pod to stop processing the tracking task (step S42 in Figure 12, (11) in Figure 16).

[0114] The function control units 1412B to 1412D that have received the stop instruction from the function collaboration unit 132 stop the corresponding functions 141B to 141D (step S43 in FIG. 12, (12) in FIG. 16). The function control units 1412B to 1412D continue to return the status (stopping) to the function collaboration unit 132 while the function stop processing is being executed. When the stop processing is complete, the function control units 1412B to 1412D return a completion for order change to the function collaboration unit 132. The function collaboration unit 132 updates the status of these functions 141B to 141D in the function information 115 to stopping, and updates the order to indicate that the stop processing has been completed.

[0115] At this time, the autoscaler 142 may refer to the function information 115 and automatically scale in while maintaining a fixed number of standby pods for each function ((13) in FIG. 16).

[0116] In addition, the event issuing unit 121 instructs a distance determination task to determine whether the distance between the bicycle and each of the cameras 20B, 20C, and 20D is closer or farther than a predetermined distance (step S30 in FIG. 11, (14) in FIG. 17). The predetermined distance is set according to the inference accuracy of the large inference model with high inference accuracy and the small inference model with low inference accuracy. For example, the predetermined distance is set to a distance at which a predetermined inference accuracy can be maintained in the small inference model. If the distance is farther than the predetermined distance (step S30: far), the control server 10 returns to step S26 in FIG. 11 and continues the tracking task.

[0117] If the distance between the camera 20D and the bicycle is closer than a predetermined distance (step S30 in Figure 11: Close), the event publisher 121 issues an instruction to change the behavior of the function 141D that executes the tracking task of the corresponding camera 20D (step S51 in Figure 13, (14) in Figure 17). The event publisher 121 issues an instruction to change the inference model, for example, from the large inference model X to the small inference model Y.

[0118] The function collaboration unit 132 receives an instruction to change the behavior (change the inference model), identifies the function pod that is processing the tracking task of camera 20D, and issues an instruction to change the inference model to the function control unit 1412D of the corresponding pod (step S52 in Figure 13, (15) in Figure 17).

[0119] The function control unit 1412D, which has received an inference model change instruction from the function collaboration unit 132, changes the inference model of the function execution unit 1411D from the large inference model X to the small inference model Y (step S53 in Figure 13, (16) in Figure 17), and continues processing. The function control unit 1412D returns to the function collaboration unit 132 a notice that the behavior change has been completed. The function collaboration unit 132 updates the order of this function 141D in the function information 115 to indicate that the behavior change has been completed.

[0120] In this way, the control server 10 performs the processes of steps S21 to S26 in Fig. 11, and upon detecting an event of a bicycle moving through an intersection from the image of camera 20A ((1) in Fig. 3), performs bicycle detection ((2) in Fig. 3). By performing the processes of steps S27 to S30 in Fig. 11, the control server 10 instructs cameras 20B, 20C, and 20D in the vicinity of the position where the bicycle will next appear to perform tracking processing (arrows Y2-1 to Y2-3, (3) in Fig. 3).

[0121] Furthermore, if the control server 10 determines that there is no longer any danger based on the processing result of the danger assessment task, it performs the processing of steps S41 to S44 in Figure 12 to instruct the surrounding cameras (e.g., cameras 20A, 20B, 20C, 20D, and 20H) to stop tracking processing.

[0122] Furthermore, if the distance between the bicycle and cameras 20B, 20C, and 20D becomes short based on the processing results of the distance determination task, the control server 10 switches to the small inference model and performs tracking by performing the processing of steps S51 to S53 in Figure 13. In this way, the control server 10 changes the behavior according to events within the same source.

[0123] In this way, the control server 10 detects an event, such as whether or not a bicycle is present, using a camera near an intersection, and then starts processing only the sources of the necessary cameras within the necessary time period, taking into consideration the cameras around the camera that detected the event and the direction and speed of the bicycle's movement, and stops processing for the relevant cameras the moment processing becomes unnecessary. This allows the control server 10 to narrow down the number of cameras performing tracking processing to a minimum for each time period.

[0124] The control server 10 then narrows down the number of cameras that are subject to tracking processing to a minimum for each time period, thereby making all cameras in the system the target of tracking, thereby achieving reductions in power consumption, resources, and costs.

[0125] Furthermore, the control server 10 can flexibly change the inference model depending on the difficulty of analysis, such as the distance to the bicycle, within the same source, thereby achieving further reductions in power consumption, resources, and costs.

[0126] 2 illustrates an example in which one control server 10 is provided, but the present invention is not limited to this. For example, the functions of the control server 10 may be distributed in stages to multiple servers (e.g., intermediate servers). In this case, the area in which the cameras 20 are installed may be divided, and an intermediate server may be provided for each area. If the intermediate server in the area where the tracking target moves does not have enough resources, processing may be performed by an intermediate server in another area or by the control server 10.

[0127] [System Configuration of the Embodiment] The control server 10 is a functional concept and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of the functions of the control server 10 is not limited to that shown in the figure, and all or part of the control server 10 can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc.

[0128] Furthermore, all or any part of the processes performed by the control server 10 may be realized by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a program analyzed and executed by the CPU and GPU. Furthermore, each process performed by the control server 10 may be realized as hardware using wired logic.

[0129] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters described above and illustrated can be changed as appropriate unless otherwise specified.

[0130] 18 is a diagram showing an example of a computer in which the control server 10 is realized by executing a program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0131] The memory 1010 includes a ROM 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0132] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, application programs 1092, program modules 1093, and program data 1094. That is, the programs that define the processes of the control server 10 are implemented as program modules 1093 in which code that can be executed by the computer 1000 is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configuration of the control server 10 are stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0133] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, CPU 1020 reads out program module 1093 or program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary and executes them.

[0134] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0135] Although the present invention has been described above as an embodiment, the present invention is not limited to the descriptions and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention.

[0136] 1 Control system 10 Control server 11 Management unit 12 Determination unit 13 Control mechanism unit 14 Function mechanism unit 20, 20A, 20B, 20C, 20D, 20E, 20F, 20G, 20H Camera 30 User terminal 111 Overall management unit 112 External data linkage unit 113 External database (DB) 115 Function information 121 Event issuing unit 131 Task instruction unit 132 Function linkage unit 141, 141A to 141N Function 1411, 1411A to 1411N Function execution unit 1412, 1412A to 1412N Function control unit 142 Autoscaler

Claims

1. A control method executed by a control device that controls task processing, comprising: a step of acquiring sensor data from a plurality of sensors; a step of starting a predetermined task when a predetermined event is detected from the sensor data; and a step of registering pod information including at least the pod name, the source being processed, and the state of the pod in cooperation with the pod on which the task is executed, and controlling the change and / or stop of the processing content of the pod based on the registered pod information, the event, and / or the processing result of the task.

2. The control method described in claim 1, further comprising a step of maintaining a fixed number of standby pods prepared in advance to be allocated to any task based on the pod information, the event, and / or the processing results of the task.

3. The pod has a function execution unit that executes the task, and a function control unit that is provided for each function execution unit and controls the function execution unit and acquires information indicating the state of the function execution unit, and the control method is characterized in that the control step registers the state of the function execution unit in a memory unit as information included in the pod information, and the control step controls changing and / or stopping the processing content of the pod based on the pod information, the event, and / or the processing result of the task.

4. The control method according to claim 3, characterized in that the state of the function execution unit is idle, processing, or stopping, and the control method further comprises the steps of: referring to the pod information, calculating the number of pods required for startup based on the number of standby pods to be maintained, which is specified according to the use case, and the number of function execution units that are processing; scaling out the pods if the calculated number of pods is greater than the number of pods currently running; and scaling in the pods if the calculated number is less than the number of pods currently running.

5. The control method according to claim 3, characterized in that it includes the steps of: determining a task to be stopped based on the processing results of the event and / or the task; identifying a pod that executes the task to be stopped based on the pod information, and instructing the function control unit of the identified pod to stop processing in the function execution unit of the identified pod; causing the function control unit to stop processing in the function execution unit and acquire information indicating the state of the function execution unit of the pod instructed to stop processing; and updating the pod information registered in a storage unit based on the state of the function execution unit acquired by the function control unit.

6. The control method according to claim 3, comprising the steps of: determining a task whose behavior is to be changed based on the processing results of the event and / or the task; identifying a pod that executes the task whose behavior is to be changed based on the pod information, and instructing the function control unit of the identified pod to change the behavior of the function execution unit of the identified pod; causing the function control unit to change the behavior of the function execution unit and acquiring information indicating the state of the function execution unit of the pod that was instructed to change the behavior; and updating the pod information registered in a storage unit based on the state of the function execution unit acquired by the function control unit.

7. A control device that controls task processing, comprising: an acquisition unit that acquires sensor data from a plurality of sensors; an initiation unit that starts a predetermined task when a predetermined event is detected from the sensor data; and a control unit that cooperates with a pod on which the task is executed to register pod information including at least the pod name, the source being processed, and the state of the pod, and controls changing and / or stopping the processing content of the pod based on the registered pod information, the event, and / or the processing result of the task.

8. A control program for causing a computer to execute the following steps: acquiring sensor data from a plurality of sensors; launching a predetermined task when a predetermined event is detected from the sensor data; and registering pod information including at least the pod name, the source being processed, and the state of the pod in cooperation with the pod on which the task is executed, and controlling the change and / or stop of the processing content of the pod based on the registered pod information, the event, and / or the processing result of the task.

Images