Image processing system and image processing program

Abstract

To provide an image processing system and image processing program that can appropriately perform the image processing desired by the user. [Solution] The image processing system includes process selection means for selecting a process corresponding to each of a plurality of tasks that constitute the image processing requested by the user, and process execution means for connecting and executing each of the processes selected by the process selection means. The image processing system may also include input means for the user to input a query corresponding to the desired image processing, and decomposition means for decomposing the tasks included in the query input by the input means into a plurality of known tasks.

Description

Technical Field

[0001] The present invention relates to an image processing system and an image processing program that realize image processing that meets user requirements.

Background Art

[0002] Techniques for detecting objects such as people shown in images captured by cameras and recognizing attributes such as what the detected objects are, the colors of the objects, and the types are widely used. Although pattern recognition techniques are also used, recently, a learned model (hereinafter referred to as "learning model") using a neural network (hereinafter referred to as "NN: Neural Network") or the like that is trained to output the detection result and recognition result of the objects shown in the input image data when the target image data is input has evolved. These learning models are learned for each type of object or learned for each imaging environment of the image and are refined and optimized so that detection or recognition can be performed more accurately according to the type of object shown in the image.

[0003] On the other hand, a foundation model that is trained using a dataset with an enormous amount of data that can handle objects in any field and can also handle language input and output is provided. The foundation model can execute various tasks such as image generation and natural language conversation for data in a wide range of fields. By using a foundation model that is widely trained to meet the requirements of such diverse fields, inferences and the like can be performed in a wide range of fields regardless of the analysis target.

[0004] Learning models that are learned for each type of object to be detected or for each environment have low versatility because they are tailored to a specific purpose and environment. On the other hand, systems using a foundation model are trained using a dataset having an enormous amount of data and have high versatility, but there are cases where the output with the accuracy desired by the user cannot be obtained for a specific purpose and environment.

[0005] Patent Document 1 proposes a method for selecting a character recognition model based on user input, arguing that using individual learning models, each trained according to specific features, is more accurate than using a common learning model (pre-trained model). Specifically, Patent Document 1 proposes selecting a model that is tailored to each user's unique handwriting style. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2024-023055 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] Numerous learning models are available for different data types across various fields. Even if it's possible to manually select a learning model, choosing the appropriate one requires a high level of skill and knowledge from the selector.

[0008] The present invention aims to provide an image processing system and an image processing program that can appropriately perform image processing desired by the user. [Means for solving the problem]

[0009] An image processing system according to one embodiment of the present disclosure includes a process selection means for selecting a process (hereinafter referred to as "each process") corresponding to each of a plurality of tasks constituting image processing requested by a user, and a process execution means for executing each of the processes selected by the process selection means in sequence.

[0010] In one embodiment of the image processing system described herein, the system can automatically select a process in response to a user's request and chain the processes together to perform image processing.

[0011] An image processing system in one embodiment of the present disclosure further comprises a configuration determination means that determines a configuration, which is setting information to be referenced when each process is executed, in accordance with at least one of the installation environment of the camera that is the source of the image input used for the image processing, and the size of the object to be processed that is captured in the image of the camera, and the process execution means may execute each process by referring to the configuration determined by the configuration determination means.

[0012] In one embodiment of the image processing system described herein, information about the camera that captures the image to be processed is acquired, and the system automatically determines the configuration based on the camera's installation environment and how the object is captured.

[0013] In an image processing system according to one embodiment of the present disclosure, the processes selectable by the process selection means include a process corresponding to an object detection task and a process corresponding to an object recognition task, and the detector used in the process corresponding to the object detection task and the recognizer used in the process corresponding to the object recognition task are modularized and may be freely connectable.

[0014] In one embodiment of the image processing system of this disclosure, there are processes corresponding to an object detection task and processes corresponding to an object recognition task as selectable items. The system can appropriately select the processes corresponding to the object detection task and / or the processes corresponding to the object recognition task according to the user's request, and perform image processing by chaining the processes together.

[0015] In one embodiment of the image processing system of this disclosure, the process execution means may input the execution results for each frame of a plurality of frame images used in the image processing into a language model for summarization to obtain a combined execution result for the plurality of frame images.

[0016] In one embodiment of the image processing system of this disclosure, the results of image processing performed in each appropriately selected and connected process are input to a language model for multiple frame images. This makes it possible to output sentences in natural language that explain the results of processes such as object detection and image recognition on images captured over a certain period of time.

[0017] In one embodiment of the present disclosure, the image processing system, selectable by the process selection means, includes a process for each of a plurality of frame images used in the image processing, and a process that inputs the execution results of each of the plurality of frame images together into a language model for summarization to obtain a combined execution result for the plurality of frame images. These processes may be modularized and freely connectable.

[0018] In one embodiment of the image processing system of this disclosure, the process execution means can execute processes for each frame image and processes for multiple frame images by appropriately combining and connecting them.

[0019] An image processing system in one embodiment of the present disclosure may include an input means for a user to input a query corresponding to a desired image processing, and a decomposition means for decomposing a task included in the query input by the input means into a plurality of known tasks.

[0020] In one embodiment of the image processing system described herein, instead of directly selecting a process corresponding to the text contained in the input query, the query is broken down into multiple tasks, allowing for the appropriate connection and execution of tasks using well-trained learning models. This results in more accurate results than using a base model.

[0021] In an image processing system according to an embodiment of the present disclosure, the decomposition means has a search function for searching for a document useful for decomposing the task included in the query into the plurality of known tasks, and a language model for task decomposition. By inputting the document useful for task decomposition searched by the search function together with the query input by the input means into the language model for task decomposition, the task included in the query may be decomposed into the plurality of known tasks.

[0022] In an image processing system according to an embodiment of the present disclosure, when decomposing a query input by a user into a plurality of tasks, an external useful document and a language model for task decomposition are used in order to decompose appropriately. As a result, it becomes possible to decompose into tasks that can obtain highly accurate results and execute an optimal process for each task. Here, the useful document is typically data such as a manual that defines rules for decomposing the task included in the query into a plurality of known tasks. In the image processing system of the present embodiment, the technology of RAG (Retrieval Augmented Generation) is applied. The simplest implementation method is to prepare the assumed user query and an example of the task decomposition result as additional documents of RAG.

[0023] In an image processing system according to an embodiment of the present disclosure, the decomposition means may decompose the task included in the query input by the input means into at least an object detection task and an object recognition task.

[0024] An image processing system according to an embodiment of the present disclosure decomposes a query input by a user into at least an object detection task and an object recognition task. In the object detection task and the object recognition task, since the processing is different, by distinguishing which image processing corresponding to each task is to be executed, it is possible to execute image processing with higher accuracy than the task based on the base model.

[0025] In the image processing system according to an embodiment of the present disclosure, the decomposition means may decompose the query input by the input means into at least the specification information of devices and the like used for executing the image processing.

[0026] In the image processing system according to an embodiment of the present disclosure, the specification information of a device, which is useful for selecting an appropriate process for the image to be processed, is acquired from a query input by a user. Thereby, for a device with less computing resources, a process with a light load rather than a process with an overly heavy load can be selected, and for a device with more computing resources, a process with higher accuracy even if the load becomes heavy can be selected.

[0027] In the image processing system according to an embodiment of the present disclosure, the decomposition means decomposes the task included in the query input by the input means into at least a task corresponding to a per-frame process and a task corresponding to a process for a plurality of frames.

[0028] The image processing system according to an embodiment of the present disclosure decomposes the query input by a user into at least a per-frame task and a task spanning a plurality of frames. Since the appropriate processes for the per-frame task and the task spanning a plurality of frames are different from each other, a more accurate output can be obtained by appropriate decomposition.

[0029] In the image processing system according to an embodiment of the present disclosure, the decomposition means decomposes the task included in the query input by the input means into at least a real-time processing task and an offline processing task.

[0030] The image processing system according to an embodiment of the present disclosure decomposes the task included in the query input by a user into at least a real-time processing task and an offline processing task. Since the appropriate processes for the real-time processing task and the offline processing task are different from each other, a more accurate output can be obtained by appropriate decomposition.

[0031] In one embodiment of the present disclosure, the image processing system performs the real-time processing tasks on an edge device and the offline processing tasks on a cloud device.

[0032] In one embodiment of the image processing system of this disclosure, offline processing is performed in the cloud and real-time processing is performed at the edge, thereby enabling the use of edge computing resources for real-time processing.

[0033] An image processing system according to one embodiment of the present disclosure further comprises: a determination means for determining whether or not all the information necessary for a query input by the input means is included; and a notification means for outputting information to prompt the user to input the information that has not yet been input from the necessary information using the input means, if, as a result of the determination by the determination means, all the necessary information is not included.

[0034] In one embodiment of the image processing system of this disclosure, a minimum set of information items that must be filled in is pre-configured, and if it is determined that there is insufficient information in the query, the user can be prompted to add further information. This allows for the acquisition of appropriate information and increased user satisfaction, rather than using an incomplete query to produce low-accuracy results and lower user satisfaction.

[0035] An image processing system in one embodiment of the present disclosure includes an input means for a user to input a query corresponding to desired image processing; a decomposition means for decomposing a task included in the query input by the input means into a plurality of known tasks; a process selection means for selecting a process corresponding to each of the plurality of tasks obtained by the decomposition means; and a process execution means for executing each of the processes selected by the process selection means in sequence.

[0036] In one embodiment of the image processing system described herein, instead of simply selecting a process corresponding to the wording in the query input by the user, the system appropriately decomposes the query into multiple tasks that meet the user's requirements. This allows for the appropriate selection of processes corresponding to tasks using well-trained learning models, and the combined execution of appropriate image processing by connecting the selected processes. This approach yields more accurate results than using a base model that supports diverse outputs across diverse fields, by combining appropriately trained learning models.

[0037] In one embodiment of the image processing system of the present disclosure, the decomposition means extracts from the query input by the input means information of the object to be detected by the object detection task included in the query and information of the object to be recognized by the object recognition task included in the query.

[0038] In one embodiment of the present disclosure, the image processing system can distinguish between what to detect and what to recognize from a query entered by the user, and appropriately select the process for the task corresponding to each to perform image processing.

[0039] An image processing program in one embodiment of the present disclosure causes a computer to function as a process selection means for selecting a process corresponding to each of a plurality of tasks constituting image processing requested by a user, and as a process execution means for concatenating and executing each of the processes selected by the process selection means.

[0040] An image processing program in one embodiment of the present disclosure causes a computer to function as an input means for a user to input a query corresponding to a desired image processing, and as a decomposition means for decomposing a task included in the query input by the input means into a plurality of known tasks.

[0041] An image processing program in one embodiment of the present disclosure causes a computer to function as an input means for a user to input a query corresponding to desired image processing; a decomposition means for decomposing a task included in the query input by the input means into a plurality of known tasks; a process selection means for selecting a process corresponding to each of the plurality of tasks obtained by the decomposition means; and a process execution means for concatenating and executing each of the processes selected by the process selection means. [Effects of the Invention]

[0042] According to this disclosure, by appropriately selecting multiple processes in response to user requests and connecting processes using learning models that have been accurately trained in specific fields and / or configurations, it is possible to obtain image processing results with higher accuracy than by using a base model that can handle a wide range of fields. [Brief explanation of the drawing]

[0043] [Figure 1] This is a schematic diagram of the image processing system according to the first embodiment. [Figure 2] This is a block diagram showing the configuration of an image processing device. [Figure 3] This is a block diagram of the server configuration. [Figure 4] The client's composition is defined in block. [Figure 5] This is a flowchart showing an example of the process selection procedure in the image processing system of the first embodiment. [Figure 6] This figure shows an example of a process selected in the image processing apparatus according to the first embodiment. [Figure 7] This is a functional block diagram of the image processing system according to the first embodiment. [Figure 8] This is a block diagram showing the server configuration of the second embodiment. [Figure 9] This is a flowchart showing an example of the process selection procedure in the image processing system of the second embodiment. [Figure 10]This figure shows an example of a process selected in the image processing apparatus according to the second embodiment. [Figure 11] This figure shows another example of a process selected in the image processing apparatus in the second embodiment. [Figure 12] This flowchart shows an example of the process selection procedure in the image processing system of the third embodiment. [Figure 13] This flowchart shows an example of the process selection procedure in the image processing system of the third embodiment. [Figure 14] This figure shows an example of a process selected in the image processing apparatus according to the third embodiment. [Figure 15] This is a functional block diagram of the image processing system according to the third embodiment. [Modes for carrying out the invention]

[0044] This disclosure will be described in detail with reference to drawings illustrating embodiments thereof.

[0045] [First Embodiment] Figure 1 is a schematic diagram of the image processing system 100 of the first embodiment. The image processing system 100 includes a camera 2, an image processing device 1 connected to the camera 2, a server 3 capable of communication with the image processing device 1, and a client 4 capable of communication with the server 3.

[0046] The image processing system 100 of the first embodiment is a system that selects a process to be executed by the image processing device 1 on an image captured by a camera 2 installed in a target space, and has the image processing device 1 execute the process. The process may be executed using a pre-trained learning model provided by an external service outside the system, or it may be executed using a learning model designed and trained by the image processing system 100. The image processing system 100 receives a request from the user via the client 4 to execute a process for which task, and selects according to the received request. The image processing system 100 may receive the user's request in natural language, specifying what to do with the camera 2, and use a language model to decide which process to select, or it may accept the request by allowing the user to directly select the process.

[0047] In the image processing system 100, a user request may be received through a user interface connected to the image processing device 1, and the image processing device 1 may perform a function to select a process corresponding to the task in response to the request. In the image processing system 100, a client 4 may receive a user request and send the request to the image processing device 1 via the network N, and the image processing device 1 may perform a function to select a process corresponding to the task in response to the user request. In the image processing system 100, a client 4 may receive a user request and send the request to the server 3 via the network N, and the server 3 may perform a function to select a process corresponding to the task in response to the user. In the following description, the configuration in which the server 3 performs the process selection function will be described.

[0048] The configuration for realizing such an image processing system 100 will be described in more detail. In Figure 1, camera 2 outputs image data using an image sensor that corresponds to visible light and / or near infrared. Camera 2 outputs image data in time series at a rate of several fps to tens of fps. Camera 2 sequentially transmits the image data to the image processing device 1 via a directly connected communication line or via a local network.

[0049] Image processing device 1 is a device that executes a process selected by the image processing system 100 on image data acquired from camera 2. Image processing device 1 outputs the results obtained by executing the selected process to client 4. Image processing device 1 may store the results in itself and make them readable by client 4, or it may transmit them to client 4 via network N. Image processing device 1 may also transmit the process execution results to server 3 via network N, have server 3 store them, and make them readable by client 4.

[0050] Server 3 stores in database 300 the learning models used in each process executed by the image processing device 1. Database 300 also includes learning models provided by external services outside the system. Server 3 reads the learning model corresponding to the selected process from database 300 and deploys it to the image processing device 1.

[0051] Database 300 can provide detection models that, based on the input image, are trained to detect whether a specific person or object is present, such as a model for detecting whether a person is present in the image, or a model for detecting whether a vehicle is present in the image, according to the characteristics of the object. Database 300 can provide recognition models that recognize the attributes of the detected person or object. Database 300 has attribute-specific models that recognize the age range of a person as an attribute. Database 300 has attribute-specific models that recognize attributes such as the type and part number of a detected object. Database 300 has object-specific models that recognize the clothing and accessories worn by the detected person. Database 300 may also have models that recognize the color or pattern of a detected object.

[0052] The person detection model, object detection model, and attribute recognition model are each modularized as detectors or recognizers, and may be provided so that one or more detectors or recognizers can be connected in any order.

[0053] Database 300 holds a language model trained to output a set of words with a high probability of occurrence as a natural sentence in response to an input query. Database 300 holds three language models: a Large Language Model (LLM) for use on devices with abundant computing resources, a Small Language Model (SLM) for use on devices with limited computing resources, and a medium-sized language model for use on devices with moderate computing resources. Database 300 also offers a Vision Language Model (VLM) that accepts image data in addition to queries, and a Multimodal Language Model that accepts audio data in addition to queries.

[0054] Database 300 holds multiple types of configuration data for models that detect people or objects from images, or models that recognize the attributes of detected people or objects, including setting information such as the size of the detection target area or the size of the recognition target area within the image. Database 300 holds this configuration data so that it can be provided according to the installation environment of camera 2, image size, or resolution.

[0055] In the first embodiment of the image processing system 100, the server 3 selects the learning model and configuration data stored in the database 300 according to the user's request and has the image processing device 1 execute them.

[0056] Server 3 may store the data transmitted from the image processing device 1 in association with data that identifies the target space. Server 3 may aggregate the transmitted data and create data that can be referenced by client 4.

[0057] The following describes the detailed configuration of the image processing device 1 and server 3 for realizing such an image processing system 100, as well as the details of the processing.

[0058] Figure 2 is a block diagram showing the configuration of the image processing device 1. The image processing device 1 uses an edge computer. In the following description, the image processing device 1 will be described as a single computer, but it may be configured to use multiple computers, with each computer handling a different process. The image processing device 1 comprises a processing unit 10, a storage unit 11, a first communication unit 12, and a second communication unit 13.

[0059] The processing unit 10 includes one or more processors such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), and NPU (Neural Processing Unit). The processing unit 10 includes memory, which is a temporary storage medium such as SRAM (Static Random Access Memory) and DRAM (Dynamic Random Access Memory). The processing unit 10 includes a timer and can obtain time information at each point in time from data from the timer. The processing unit 10 may be configured as a single hardware (SoC: System On a Chip) integrating the processor, memory, and furthermore, the storage unit 11, the first communication unit 12, and the second communication unit 13.

[0060] The processing unit 10 causes the processor to perform image processing based on the image processing program P1 stored in the memory unit 11 and the learning model selected and deployed from the database 300.

[0061] The storage unit 11 is a relatively large-capacity non-temporary storage medium such as a hard disk or flash memory. A portion of the storage unit 11 may be removable.

[0062] The storage unit 11 stores the program (program product) necessary for the processing unit 10 to execute processing, the results of the processing by the processing unit 10, and reference setting data. The setting data includes identification data for the device itself. The program product includes the OS (Operating System) program, the image processing program P1 that runs on the OS, and the learning model group M1. Details of the learning model group M1 will be described later.

[0063] The image processing program P1 stored in the storage unit 11 may be an image processing program P9 stored on a computer-readable storage medium 9 that the processing unit 10 reads and stores in the storage unit 11, or it may be one that is pre-stored at the time of shipment. The image processing program P1 stored in the storage unit 11 may also be one that the processing unit 10 downloads from the server 3 or another download server via the second communication unit 13 and stores in the storage unit 11.

[0064] The image processing program P1 stored in the memory unit 11 is configured to cause the computer to execute processes corresponding to each of multiple tasks, and it is possible to select which task's process to execute. The image processing program P1 may also be configured by obtaining program modules that execute processes corresponding to multiple tasks from the server 3 and combining those modules.

[0065] At least a portion of the learning model group M1 stored in the memory unit 11 is selected from the database 300. The configuration data stored in the memory unit 11 may include configuration data selected from the database 300. The learning model group M1 and configuration data selected from the database 300 and received by the processing unit 10 via the second communication unit may not be stored in the memory unit 11 but may be stored in a temporary storage medium (RAM) built into the processing unit 10.

[0066] The first communication unit 12 is a communication device that enables communication via a local network in the space where the camera 2 is installed. The first communication unit 12 may be a LAN network card or a CAN communication device. The first communication unit 12 may be a communication device that supports wireless networks such as WiFi or Bluetooth (registered trademark). The first communication unit 12 may include multiple communication devices that support various types of cameras 2. The first communication unit 12 may include an interface such as USB (Universal Serial Bus) that connects to the camera 2. The first communication unit 12 can be replaced by an interface that connects to the camera 2 via a coaxial cable or other serial bus. The processing unit 10 acquires image data from the camera 2 via the local network using the first communication unit 12. The first communication unit 12 may be the same device as the second communication unit 13.

[0067] The second communication unit 13 is a communication device that enables communication with external communication equipment via network N in the space where camera 2 is installed. The second communication unit 13 may be a wired LAN network card, a communication device that enables carrier communication via a carrier network, or a communication device that supports wireless networks such as WiFi or Bluetooth (registered trademark). The second communication unit 13 may support encrypted communication such as SSL with server 3. The second communication unit 13 may also be an interface for enabling connection with server 3 via a dedicated line.

[0068] The image processing device 1 may directly accept user input via a user interface connected through the second communication unit 13.

[0069] Figure 3 is a block diagram showing the configuration of Server 3. Server 3 may consist of a single server computer, or it may be configured to distribute processing across multiple server computers. Server 3 comprises a processing unit 30, a storage unit 31, and a communication unit 32.

[0070] The processing unit 30 includes one or more processors such as CPUs, MPUs, GPUs, or NPUs. The processing unit 30 also includes memory, which is a temporary storage medium such as SRAM or DRAM.

[0071] The storage unit 31 is a relatively large-capacity non-temporary storage medium such as a hard disk or flash memory. The storage unit 31 stores the program (program product) and configuration data necessary for the processing unit 30 to execute processing.

[0072] The program product stored in the memory unit 31 includes the server program P3. The server program P3 includes a module that functions as a data server, reading the model group stored in the database 300 and sending it to the image processing device 1. The server program P3 also includes a module that functions as a web server, and can output the results of the processing performed by the server 3 to the client 4 via a web page.

[0073] The program product includes a language model (LM) M3. The language model M3 outputs a response corresponding to the input natural language sentence. The language model M3 is used to output a process selection in response to a user request, separate from the language models stored in the database 300. The language model M3 may be partially or entirely utilized by an external language model provider service via the network N. Details regarding processing using the language model M3 will be described later.

[0074] The server program P3 and language model M3 may be obtained by the processing unit 30 reading the server program P8 and language model M8 stored in a storage medium 8 readable from a computer and storing them in the storage unit 31, or the processing unit 30 may download them from another download server via the communication unit 32 and store them in the storage unit 31.

[0075] The configuration data stored in the memory unit 31 includes data identifying the image processing device 1 to be selected by the process via the server 3. This configuration data also includes, in association with the data identifying the image processing device 1, data and a name for identifying the space in which the image processing device 1 is installed, and the correspondence between this space and the identification data of the camera 2 from which the image processing device 1 can acquire images. Identification data of image processing devices 1 or spaces that the user is permitted to request may be stored as a whitelist in association with the user's account data. This allows the server 3 to identify the target image processing device 1 when the user specifies the name of a space and requests what kind of image processing should be performed on the images captured by the camera 2 installed in that space.

[0076] The database 300 may be built on the storage unit 31 or on an external storage device. Part of the database 300 may include a model provision service used on the Web, which is communicated via the network N, as described above.

[0077] The communication unit 32 is a communication device that enables communication connections with the client 4 and the image processing device 1 via the network N.

[0078] Figure 4 is a block diagram showing the configuration of client 4. Client 4 is a personal computer, smartphone, or tablet device. Client 4 may be used by the administrator of the space where camera 2 is installed, or by the operator of the server management company.

[0079] Client 4 comprises a processing unit 40, a storage unit 41, a communication unit 42, a display unit 43, and an operation unit 44. The processing unit 40 includes one or more processors such as CPUs, MPUs, GPUs, or NPUs. The processing unit 40 also includes memory, which is a temporary storage medium such as SRAM or DRAM.

[0080] The storage unit 41 is a memory of a non-temporary storage medium such as a hard disk or flash memory. The storage unit 41 stores the functions of the data server provided by the server 3, and the client program P4 for the Web server. The client program P4 is, for example, a Web browser program. The client program P4 may also be a program that causes the processing unit 40 to execute the process of displaying the data provided by the server 3 on the screen.

[0081] The communication unit 42 is a communication device that enables communication with the server 3 via the network N. The communication unit 42 may also be a communication device that enables communication with the server 3 via a dedicated line. The communication unit 42 may also be a communication device that enables direct communication with the second communication unit 13 of the image processing device 1 via a wireless communication medium or a USB cable, etc.

[0082] The display unit 43 uses a display such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 43 displays a web page containing text and images based on processing by the client program P4 of the processing unit 40. The display unit 43 may also use a touch panel display.

[0083] The operation unit 44 is a user interface such as a keyboard or pointing device that accepts operations from the user or operator. The operation unit 44 may be a touch panel built into the display of the display unit 43, or it may be physical buttons. The operation unit 44 may be a voice input unit that accepts operations by voice using a voice recognition function. The operation unit 44 can notify the processing unit 40 of operation information from the user or operator.

[0084] In the image processing system 100 configured in this way, the process by which a process is selected for the image processing device 1 and the selected process becomes executable will be described. Figure 5 is a flowchart showing an example of the process selection process in the image processing system 100 of the first embodiment. When a user accesses the server 3 using the client 4 and accesses a web page for configuring the image processing device 1, the server 3 starts the following process.

[0085] The processing unit 30 of server 3 receives data identifying the space where the camera 2 to be processed is installed, as a request from the user (step S301). In step S301, the processing unit 30 receives one or more of the following: the user's account data, the space identification data or name, and the camera 2 identification data. In step S301, the processing unit 30 may also receive a selection from a list of the identification data of the image processing device 1 that is authorized to access the account data used when the client 4 accessed server 3, and the corresponding space identification data or name.

[0086] The processing unit 30 identifies the identification data of the image processing device 1 corresponding to the space identified by the received data (step S302).

[0087] The processing unit 30 receives, on the web page, the settings of the installation environment where the camera 2, which is the target of processing for the identified image processing device 1, is installed, and the user's request regarding the image captured by the camera 2 (step S303). In step S303, the processing unit 30 receives, as the user's request, natural language entered into an input field included in the web page displayed on the client 4. In step S303, the processing unit 30 may also receive the selection of options for a question included in the web page displayed on the client 4. For example, the processing unit 30 may receive multiple selections from the options for a task (person or object to be detected, attribute of the object to be recognized) displayed on the web page. In step S303, the processing unit 30 receives, as the settings of the installation environment, information such as the installation environment of the camera 2 (indoors, outdoors, entrance, exit, corridor, etc.), the size of the object captured by the camera 2 in the image, the camera rate, and the specifications of the image processing device 1.

[0088] The processing unit 30 analyzes the user request received (step S304). In step S304, in the first example, the processing unit 30 combines the natural language sentence (query) received as the user request with an instruction sentence that instructs the natural language sentence to output a task in a predetermined format, inputs it into the language model M3, and obtains the sentence (word group) output from the language model M3. The processing unit 30 may also identify the person or object to be detected, or the attributes of the object to be recognized.

[0089] The processing unit 30 selects a process to be executed by the image processing device 1 based on the analysis results of step S304 (step S305). The processing unit 30 identifies a learning model to be used for each selected process from the database 300 (step S306). In step S305 or step S306, the processing unit 30 may select a process or identify a learning model by referring to the camera 2 installation environment settings received in step S303. The processing unit 30 identifies, for example, that LLM should be used if the processing speed and memory are above a predetermined specification level, and that SLM should be used if the processing speed and memory are above a predetermined specification level, based on the data of computing resources included in the specifications of the image processing device 1.

[0090] The processing unit 30 determines the configuration data to be used by the image processing device 1 from the database 300 according to the received installation environment settings (step S307). In step S307, the processing unit 30 determines the configuration data to be used according to the installation environment of the camera 2, such as indoors, outdoors, entrance, exit, and corridor, the size of the object captured by the camera 2 in the image, the camera rate, and the specifications of the image processing device 1.

[0091] The processing unit 30 transmits the identification data of the process selected in step S305, the learning model identified in step S306, and the configuration data determined in step S307 to the image processing device 1 (step S308). The processing unit 30 makes the process using the transmitted learning model and configuration data executable in the image processing device 1 (step S309). In step S309, the processing unit 30 of the server 3 generates an instance of the image processing program P1 configured to execute the selected process and transmits the generated instance to the image processing device 1. The processing unit 10 of the image processing device 1 stores this instance as the image processing program P1 in the storage unit 11. In other words, the processing unit 30 of the server 3 deploys the image processing program P1 to the image processing device 1.

[0092] As a result, the processing unit 10 of the image processing device 1 executes the selected process on the image acquired from the camera 2 using the image processing program P1.

[0093] The processing procedure shown in Figure 5 has been described as being executed by Server 3. However, the image processing device 1 may receive operations from Client 4 via Network N and execute the processing procedure shown in Figure 5, or the image processing device 1 may directly receive operations from Client 4 via Second Communication Unit 13 and execute the processing procedure shown in Figure 5 using Language Model M3 provided by Server 3.

[0094] The processing procedure shown in Figure 5 will be explained with a specific example. Figure 6 is a diagram showing an example of the process selected by the image processing device 1 in the first embodiment. In the example shown in Figure 6, the user specifies data identifying the space where the target camera 2 is installed in client 4 and inputs natural language text such as, "I want to count the number of elderly people wearing hats and glasses who come to this place every week."

[0095] Server 3 takes the natural language sentence received as a user request and adds instructions such as, "Please extract the tasks necessary to fulfill the attached request according to the following rules: Rules: <Target to detect> target, <Method of tracking the target to detect> unit to track, <Attributes to recognize>" and provides it to language model M3.

[0096] In the example in Figure 6, the language model M3 outputs "<Target detection> Person detection, <Method for tracking the target detection> Person tracking and assignment of person ID, <Attributes to be recognized> Elderly, wearing glasses, wearing a hat". The above information, "<Attributes to be recognized> Elderly, wearing glasses, wearing a hat", corresponds to the information of the target to be recognized by the object recognition task included in the query, which is extracted by the "decomposition means" in the claim. Based on the output from the language model M3, the processing unit 30 of the server 3 selects a person detection process that detects a person from the image using a person detection model (person detector), and a tracking process that assigns the same ID to the same person detected across multiple frame images based on the features of the detected person. For the tracking process, the processing unit 30 selects a face recognition model (face classifier) ​​to identify the same person. The processing unit 30 further selects a learning model (age The processing unit 30 selects an elderly person recognition process that uses a recognition device to determine whether the detected person is elderly or not. The processing unit 30 further selects a glasses-wearing recognition process that uses a glasses-wearing recognition device to determine whether the detected person is wearing glasses or not, and a hat-wearing recognition process that uses a hat-wearing recognition device to determine whether the detected person is wearing a hat or not. The processing unit 30 further selects a process that, for a specified period, in this case every week, provides the results of the elderly person recognition process, glasses-wearing recognition process and hat-wearing recognition process for multiple frame images over a week to a large-scale language model to obtain a summarized result.

[0097] When accepting user requests by accepting selections from options presented on a web page without using the language model M3, the processing unit 30 identifies the task as "person detection / person tracking / whether the person is elderly, whether they are wearing glasses, whether they are wearing a hat." In this case, the processing unit 30 may select a person detection process, a tracking process, an elderly person recognition process, a glasses wearer recognition process, a hat wearer recognition process, and a summarization process corresponding to each task.

[0098] The processing unit 30 identifies the person detection model, face recognition model, age recognition learning model, glasses recognition model, hat recognition model, and summarization language model to be used for each selected process from the database 300. The processing unit 30 sends data identifying the selected process, or the executable file corresponding to the process, the model that can be read from the executable file, and the configuration data to the image processing device 1.

[0099] The image processing device 1 stores in the storage unit 11 a person detection model (person detector) 101, a learning model for face recognition (face recognizer) 102, a learning model for age recognition (age recognizer) 103, a model for glasses recognition (glasses wearer recognizer) 104, a model for hat recognition (hat wearer recognizer) 105, and a language model for summarization 106, which can be used as a group of learning models M1, based on the data that identifies the transmitted process. The processing unit 10 configures the image processing program P1 to cooperate with the group of learning models M1 used by the person detection process, tracking process, elderly recognition process, glasses wearer recognition process, hat wearer recognition process, and summarization process, respectively, by referring to configuration data, and makes it executable.

[0100] Subsequently, the processing unit 10 of the image processing device 1 acquires frame images output from the camera 2 in a time series, assigns identification data to the frame images, and executes a person detection process for each frame image. In the person detection process, the processing unit 10 provides the image to the person detection model 101 and obtains the detection result (coordinate data of the person's region) output from the person detection model 101. The processing unit 10 improves the detection accuracy of the person detection model 101 by referring to the image size and other information included in the configuration data. If no person is detected in the person detection process, the processing unit 10 executes processing for the next frame image.

[0101] When a person is detected in the person detection process, the processing unit 10 provides the frame image and the detection result to the tracking process. In the tracking process, the processing unit 10 obtains the facial features of the detected person from the input frame image using a learning model for face recognition, associates the person ID with the features, and identifies the person ID of the detected person. The processing unit 10 may also refer to the configuration data in the tracking process.

[0102] The processing unit 10 passes the frame image, the detection result, and the person's ID to the elderly recognition process, the glasses-wearing recognition process, and the hat-wearing recognition process. In each of the elderly recognition process, the glasses-wearing recognition process, and the hat-wearing recognition process, the processing unit 10 outputs whether the person identified by the person ID is elderly, whether they are wearing glasses, or whether they are wearing a hat, using the learning model group M1. The image processing program P1 may be configured so that the processing unit 10 executes the elderly recognition process, the glasses-wearing recognition process, and the hat-wearing recognition process in a sequential manner. In this case, the processing unit 10 executes the glasses-wearing recognition process to determine whether the person is wearing glasses only if the detected person is recognized as elderly, and executes the hat-wearing recognition process to determine whether the person is wearing a hat only if the glasses-wearing recognition process determines that the detected person is wearing glasses.

[0103] The processing unit 10 integrates the results of the recognition process performed on each of the multiple frame images and stores, for each frame image, data indicating that a subject has been detected if the person detected in the frame image is elderly, wearing glasses, and wearing a hat, along with the frame image identification data and the person ID. The processing unit 10 may also store the time information in which the frame image was taken. The processing unit 10 may store only the frame images in which a subject was detected in the storage unit 11. Then, in the summarization process, the processing unit 10 aggregates the stored data (frame images in which a subject was determined to have been detected) for a specified period (in this case, one week), provides the aggregated results to the language model M3, and outputs an explanation of the aggregated results for one week as natural language.

[0104] The processing unit 10 of the image processing device 1 integrates the results of the recognition process performed on each of the multiple frame images, and for each frame image, if the person detected in the frame image is elderly, wearing glasses, and wearing a hat, it may send the identification data of that frame image to the server 3. In addition to the identification data of the frame image in which the target was detected, the person ID (feature) of the detected person and the time information of the detection may also be sent to the server 3. The aggregation and summarization process using the language model M3 may be performed on the server 3.

[0105] The user can use client 4 to access the detection results and their summary results stored weekly by the image processing device 1. Client 4 may directly obtain the weekly summary results stored in the storage unit 11 of the image processing device 1 via the second communication unit 13 of the image processing device 1 and display them on a screen based on client program P4, or it may obtain them via server 3 and output the summary results on a web page provided by server 3.

[0106] In the example in Figure 6, the aggregated result outputs the natural language sentence, "In the past week, two elderly people wearing hats and glasses have visited this location." Server 3 may store identification data for the frame images associated with the two individuals' IDs, the individuals' IDs, and time information, and the frame images may be accessible.

[0107] As shown in Figure 6, the user simply inputs their requirements for what they want to do with the images captured by the target camera 2 into the image processing system 100, for example, via a web page, using natural language. The image processing system 100 then automatically selects a process that meets the user's requirements. The user does not need to select a learning model themselves from the various models available for diverse data fields and according to the specifications of the image processing device 1, etc. By providing the image processing system 100 with data on the camera 2's installation environment and information on the specifications of camera 2 and image processing device 1, the image processing system 100 can select a model and process that matches the specifications. This avoids the use of unnecessarily high-precision models that do not match the specifications, and conversely, it avoids selecting a low-precision process that does not meet the user's requirements.

[0108] In the example shown in Figure 6, if there are multiple cameras 2 connected to the image processing device 1, and the processing is performed on images of a space captured by multiple cameras 2 (for example, images of a store), the above tracking process becomes a multi-camera object tracking process. Here, multi-camera object tracking means tracking multiple target objects across multiple cameras (and their captured images) while taking occlusion (when the target object is hidden from the camera) into consideration.

[0109] Figure 7 shows the functional blocks of the image processing system 100 of the first embodiment. This Figure 7 is a diagram to explain that each constituent element (each means) of the claims is described in the first embodiment. The image processing system 100 includes, as functional blocks, an input unit 51 into which the user inputs a query corresponding to the desired image processing; a decomposition unit 53 that decomposes the tasks included in the query input by the input unit 51 into a plurality of known tasks; a process selection unit 54 that selects a process corresponding to each of the plurality of tasks obtained by the decomposition unit 53; and a process execution unit 55 that connects and executes each process selected by the process selection unit 54. The image processing system 100 also includes, as functional blocks, a configuration determination unit 52 that determines a configuration, which is setting information referenced when each process is executed, according to at least one of the installation environment of the camera 2 that is the input source of the image used for image processing, and the size of the object to be image processed that is captured in the image of the camera 2. The input unit 51, configuration determination unit 52, decomposition unit 53, process selection unit 54, and process execution unit 55 described above correspond to the input means, configuration determination means, decomposition means, process selection means, and process execution means in the claims, respectively.

[0110] The input unit 51 described above is mainly implemented by the operation unit 44, processing unit 40, and communication unit 42 of the client 4, and the communication unit 32 and processing unit 30 of the server 3. The processing performed by the input unit 51 is the processing of steps S301 to S303 in Figure 5 (mainly the processing of step S303). The configuration determination unit 52 is implemented by the processing unit 30 of the server 3 and performs the processing of step S307 in Figure 5. The decomposition unit 53 is implemented by the processing unit 30 of the server 3 and performs the processing of step S304 in Figure 5. Through the analysis processing of step S304, the decomposition unit 53, in the example shown in Figure 6, decomposes the tasks included in the input query into several known tasks, such as "<Target detection> Person detection, <Method of tracking the target detection> Person tracking and assignment of person ID, <Attributes to be recognized> Elderly, Wearing glasses, Wearing a hat". The process selection unit 54 is implemented by the processing unit 30 of the server 3 and performs the processing of step S305 in Figure 5. As shown in Figure 6, the processes that can be selected by the process selection unit 54 include processes for each of the multiple frame images used for image processing (person detection process, tracking process, elderly recognition process, glasses wearing recognition process, and hat wearing recognition process in Figure 6), and a process that inputs the execution results of the processes for each of the multiple frame images into a language model for summarization to obtain a combined execution result for the multiple frame images (summarization process in Figure 6). These processes are modularized and can be freely connected.

[0111] Furthermore, the query input by the input unit 51 may include information regarding the specifications of the device (mainly the image processing device 1) used to perform image processing. The decomposition unit 53 may not only decompose the tasks included in the input query into multiple known tasks, but may also decompose information regarding the device specifications included in the query (for example, information such as "execute with a high-performance device (image processing device 1)" or "execute with a device (image processing device 1) having a CPU with processing power of ~"). The device specification information extracted through this decomposition is used by the configuration determination unit 52 to determine the configuration.

[0112] [Second Embodiment] In the second embodiment, the image processing system 100 employs RAG to more appropriately select tasks and processes corresponding to those tasks in response to user requests. The configuration of the image processing system 100 in the second embodiment is the same as that of the image processing system 100 in the first embodiment, except for the processing procedures and data for employing RAG which will be described later. Therefore, common components are denoted by the same reference numerals and detailed descriptions are omitted.

[0113] Figure 8 is a block diagram showing the configuration of the server 3 in the second embodiment. In the second embodiment, the server 3 stores a set of document data, such as a manual, in the database 300 or storage unit 31, which defines the rules for breaking down tasks included in user requests (queries) into a plurality of known tasks. Each piece of document data may be created in advance by an operator, or it may be a record of the correspondence between actual user requests and tasks that produced results satisfactory to the user, stored in a predetermined format.

[0114] The process by which the image processing system 100 of the second embodiment selects a process in response to a user request by referring to document data, and makes the selected process executable, will be described. Figure 9 is a flowchart showing an example of the process selection process in the image processing system 100 of the second embodiment. When a user accesses the server 3 using the client 4 and accesses a web page for configuring the image processing device 1, the server 3 starts the following process.

[0115] Of the processing steps shown in Figure 9, steps that are common to the processing steps shown in Figure 5 of the first embodiment are given the same step numbers and detailed explanations are omitted.

[0116] The processing unit 30 of server 3 receives data to identify the target space (S301), identifies the identification data for image processing device 1 (S302), and on the web page receives the settings for the installation environment where the camera 2 to be processed by the identified image processing device 1 is installed, and the user's request for the images captured by camera 2, in natural language (step S323).

[0117] The processing unit 30 searches for document data useful for the user's request received (step S324). In step S324, the processing unit 30 may extract similar document data that includes the word group contained in the query corresponding to the user's request, or it may use the language model M3 to extract appropriate document data from the document data group stored in the storage unit 31.

[0118] The processing unit 30 adds instructions to the received user request, along with the installation environment settings and the document data retrieved in step S324, and provides this to the language model M3 (step S325). In step S325, the processing unit 30 combines the natural language sentence corresponding to the user request with an instruction sentence that instructs the system to refer to the installation environment settings and the document data and output according to the rules defined in the document data, and inputs this into the language model M3. Details of the rules defined in the document data will be described later.

[0119] The processing unit 30 obtains the sentence (word group) output from the language model M3 (step S326). The processing unit 30 identifies the task corresponding to the sentence output from the language model M3 (step S327). The language model M3, which refers to the document data, decomposes the user's request (query) into known tasks, such as predefined names or identification data like "person detection," "gender recognition," "elderly recognition," and "hat wearing recognition." In step S327, the processing unit 30 should at least decompose the task into whether it is a detection task or a recognition task. In step S327, the processing unit 30 obtains the word group indicating the task output from the language model M3, which has referred to the document data. In step S327, the processing unit 30 may also identify the task by referring to the settings of the camera 2's installation environment. For example, for the image processing device 1 with limited computing resources, the processing unit 30 may select SLM as the language model, or if the camera 2 is installed outdoors, it may identify a task that includes processing to reduce the influence of ambient light.

[0120] The processing unit 30 selects a process corresponding to the task identified in step S327 (S328). In step S328, a corresponding process is associated with a pre-configured task, and the processing unit 30 selects a process based on this association. For example, the "person detection" task may include a "person detection process".

[0121] The processing unit 30 identifies the learning model to be used for each selected process from the database 300 (S306). The processing unit 30 identifies, for example, that LLM should be used if the processing speed and memory are above a predetermined specification level, and that SLM should be used if the processing speed and memory are above a predetermined specification level, based on the data of computing resources included in the specifications of the image processing device 1.

[0122] Subsequently, the processing unit 30 determines the configuration data (S307), as in the first embodiment, transmits the identification data of the selected process, the identified learning model, and the determined configuration data to the image processing device 1 (S308), and executes the process of step S309.

[0123] In the second embodiment, the processing procedure shown in Figure 9 was described as being executed by the server 3. However, the image processing device 1 may receive operations from the client 4 via the network N and execute the processing procedure shown in Figure 9, or the image processing device 1 may directly receive operations from the client 4 via the second communication unit 13 and execute the processing procedure shown in Figure 9 using the language model M3 provided by the server 3.

[0124] The processing procedure shown in Figure 9 will be explained with specific examples. Figure 10 is a diagram showing an example of the process by which a process is selected in the image processing device 1 in the second embodiment. Similar to Figure 6 in the first embodiment, Figure 10 shows the process from when a process is selected in response to a user request until the selected process is executed in the image processing device 1. In Figure 10, as in the first embodiment, the process is shown when the user, using client 4, specifies data identifying the space where the target camera 2 is installed and inputs natural language text such as, "I want to count the number of elderly people wearing hats and glasses who come to this place every week."

[0125] In the second embodiment, the processing unit 30 responds to the aforementioned natural language sentence received as a user request and searches for document data containing, for example, "hat, glasses, elderly" as keywords. The processing unit 30 instructs the language model M3 to refer to the document data and provides it with an instruction statement such as, for example, "Please break down the attached query into tasks, referring to the specified document data."

[0126] The language model M3, which references document data defining decomposition rules, decomposes user-input queries into tasks such as "person detection / elderly recognition / glasses wearer recognition / hat wearer recognition." The processing unit 30 should decompose tasks based at least on whether they are detection tasks or recognition tasks. The processing unit 30 can then directly select the processes corresponding to the decomposed tasks, such as the person detection process, the elderly recognition process, the glasses wearer recognition process, and the hat wearer recognition process.

[0127] The processing unit 30, using the language model M3 which references the document data, decomposes the aforementioned tasks such as "person detection / elderly recognition / glasses wearer recognition / hat wearer recognition" into tasks for each frame, and also decomposes them into a summarization task that aggregates the detection and recognition results across multiple frame images from the word "every week".

[0128] As shown in Figure 10, the language model M3 allows the task to be broken down into steps that facilitate process selection, making it possible to choose the appropriate process.

[0129] The processing unit 30 may further specify in the document data that it identifies specification information regarding the computing resources of the image processing device 1 to be used. This allows the processing unit 30 to refer to the specification information associated with the identification data of the image processing device 1, which has been identified by the language model M3 prior to the user's request, and to identify the task, learning model, and process to be selected from the referenced specification information. If the specifications of the image processing device 1 are low level (relatively poor computing resources), the processing unit 30 can select the lightest possible process or learning model. In this way, the processing unit 30 can refer to the rules specified in the document data, decompose the input user request into the necessary tasks, select an appropriate process, and then connect the selected processes to execute image processing.

[0130] Figure 11 shows another example of the process selected by the image processing device 1 in the second embodiment. Similar to Figure 10, Figure 11 shows the result of decomposing the input user request (query) using the language model M3. In Figure 11, the input request is in natural language form, "I want to count the number of elderly people wearing hats and glasses who come to this place every week."

[0131] In the example in Figure 11, the processing unit 30, similar to Figure 10, uses the language model M3, which references the document data, to decompose the tasks such as "person detection / elderly recognition / glasses wearer recognition / hat wearer recognition" into tasks for each frame, and also decomposes them into a summarization task that aggregates the detection and recognition results across multiple frame images from the phrase "every week". However, in the example in Figure 11, the processing unit 30 clearly decomposes the tasks into those that should be executed sequentially in real time and those that should be executed retrospectively over a block of time, such as "<Real-time processing target> person detection / elderly recognition / glasses wearer recognition / hat wearer recognition, <Offline processing target> summarization," and then selects the process corresponding to each task. This process corresponds to the process in the claim where ("decomposition means") "decomposes the tasks included in the query input by the input means into at least real-time processing tasks and offline processing tasks."

[0132] As a result, since real-time processing and offline processing require different appropriate processes, the image processing system 100 can accurately select the process corresponding to a task by appropriately decomposing it, and then execute the selected processes in sequence. In this case, tasks on the real-time processing side (tasks such as "person detection / elderly recognition / glasses wearer recognition / hat wearer recognition" mentioned above) may be executed on the image processing device 1 (the "edge-side device" in the claims), and tasks on the offline processing side (the "summarization" task) may be executed on the server 3 (the "cloud-side" in the claims). For example, on the real-time processing side, the image processing device 1 (edge-side device) may apply VLM on an image frame-by-frame basis to convert the image into text information (outputting text information which is the processing result of each task on the offline processing side), and the server 3 (cloud-side) may perform processing such as applying LLM to that text information to extract information (for example, the "summarization" process mentioned above).

[0133] In the second embodiment, the user simply inputs their request to the image processing system 100, for example via a web page, in natural language, specifying what they want to do with the images captured by the target camera 2. The image processing system 100 then automatically selects a process that corresponds to the user's request. The user does not need to select a learning model themselves from the various models available for different fields of data, depending on the specifications of the various image processing devices 1, etc.

[0134] In the second embodiment, the decomposition unit 53 (see Figure 7) has a search function that searches for documents useful for decomposing a task included in a query into multiple known tasks, and a language model M3 for task decomposition. By inputting the documents useful for task decomposition found by the search function into the language model M3 for task decomposition along with the input query, the task included in the query is decomposed into multiple known tasks. This process corresponds to steps S324 to S327 in Figure 9.

[0135] [Third Embodiment] In the second embodiment, the processing unit provides the input query to the language model M3 and employs a RAG that references document data so that the query can be broken down into appropriate tasks. In the third embodiment, if the server 3 determines that all the necessary information is not included and that it cannot be broken down into appropriate tasks, it prompts the user to input the necessary information.

[0136] The image processing system 100 of the third embodiment is the same as the image processing system 100 of the first or second embodiment, except that the processing procedure shown below differs from that of the first or second embodiment. Therefore, common components are denoted by the same reference numerals and detailed descriptions are omitted.

[0137] Figures 12 and 13 are flowcharts illustrating an example of the process selection procedure in the image processing system 100 of the third embodiment. When a user accesses the server 3 using the client 4 and accesses a web page for configuring the image processing device 1, the server 3 starts the following process. Of the processing procedures shown in Figures 12 and 13, steps that are common with the processing procedures shown in Figure 5 of the first embodiment and Figure 9 of the second embodiment are given the same step numbers and detailed explanations are omitted.

[0138] The processing unit 30 receives data to identify the target space (S301), identifies the identification data for the image processing device 1 (S302), and on the web page, it receives the settings for the installation environment where the camera 2 to be processed by the identified image processing device 1 is installed, and the user's request regarding the image captured by the camera 2, in natural language (S323).

[0139] The processing unit 30 determines whether all the necessary information is included in the installation environment settings and user requests received in step S323 (step S331). In step S331, the processing unit 30 inputs, for example, the settings and requests received in step S323 and a statement asking whether these settings and requests can be broken down into tasks or whether all the necessary information can be obtained into the language model M3 to obtain a response statement. If the response statement is positive, it determines that all the information is included; otherwise, it determines that not all the information is included. In step S331, the processing unit 30 may also determine whether all the necessary information is included from the settings and requests received in step S323 by comparing them with templates, etc., without using the language model M3.

[0140] If it is determined in step S331 that all items are included (S331: YES), the processing unit 30 searches for document data useful for the user's request received in step S323 (S324). The processing unit 30 then executes the processes in S325-S328 and S306-S309.

[0141] If it is determined in step S332 that not all information is included (S331: NO), the processing unit 30 identifies the necessary information that has not yet been entered (step S332). The processing unit 30 causes the client 4 to receive a message prompting it to enter the information identified in step S332 (step S333). In step S333, the processing unit 30 may use a standard phrase for each of the missing pieces of information to provide the message. The processing unit 30 may also use a response sentence obtained from the language model M3 in step S331 to provide the message. The processing unit 30 may also give the language model M3 an instruction sentence instructing it to create a message prompting the input of the missing information, and have the message created. The processing unit 30 receives the information identified in step S332 (the necessary information that has not yet been entered) from the user input on the web page displayed on the client 4 (step S334).

[0142] The processing unit 30 takes the information received in step S334, adds it to the settings and user requests received in step S323 (step S335), and returns the processing to step S331.

[0143] The processing procedures shown in Figures 12 and 13 will be explained with specific examples. Figure 14 shows an example of the process by which a process is selected in the image processing device 1 in the third embodiment. In Figure 14, similar to Figure 6 of the first embodiment and Figures 10 and 11 of the second embodiment, the process from the selection of a process in response to a user request to the execution of the selected process in the image processing device 1 is shown.

[0144] The example in Figure 14 shows the process when a user, using client 4, specifies data identifying the space where the target camera 2 is installed and inputs natural language text such as, "I want to count the number of elderly people wearing hats and glasses who come to this location." In the third embodiment, the processing unit 30 determines that the natural language text received as a user request does not contain all the necessary information (S331: NO) and identifies that the information is missing the "period" (S332). The processing unit 30 creates the message, "Please enter the period," and notifies client 4 (S333).

[0145] In the third embodiment, the user simply inputs their request to the image processing system 100, for example via a web page, in natural language, specifying what they want to do with the images captured by the target camera 2. The image processing system 100 then automatically selects a process that corresponds to the user's request. The user does not need to select a learning model themselves from the various models available for different fields of data, depending on the specifications of the various image processing devices 1, etc.

[0146] The user can learn from the server 3 whether there is any excess or deficiency of information necessary to reflect their requests in the image processing device 1. Rather than obtaining inaccurate results using incomplete queries, the user can obtain output from the image processing device 1 that reflects their requests by responding to inquiries about the necessary information in advance.

[0147] Figure 15 is a functional block diagram of the image processing system 100 of the third embodiment. In addition to the functional blocks shown in Figure 7, the image processing system 100 of the third embodiment includes a determination unit 61 that determines whether or not all the information necessary for the query input by the input unit 51 is included, and a notification unit 62 that outputs information to prompt the user to input the information that has not yet been input from the necessary information, if the determination unit 61 determines that not all the necessary information is included. The determination unit 61 and the notification unit 62 correspond to the determination means and notification means in the claims, respectively. The determination unit 61 is implemented by the processing unit 30 of the server 3 and performs the processing of step S331 in Figure 12. The notification unit 62 is mainly implemented by the processing unit 30 of the server 3 and the processing unit 40 and display unit 43 of the client 4 and mainly performs the processing of step S333 in Figure 12.

[0148] The embodiments disclosed above are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims, and all modifications within the meaning and scope equivalent to the claims are included. [Explanation of symbols]

[0149] 100 Image Processing Systems 1. Image processing device (edge-side device) 10 Processing Unit 11 Storage section P1 Image Processing Program 2 cameras 3. Server (cloud side) 30 Processing Unit 51 Input section (input means) 52 Configuration Determination Unit (Configuration Determination Means) 53 Disassembly part (disassembly means) 54 Process Selection Unit (Process Selection Means) 55 Process execution unit (process execution means) 61 Judgment unit (judgment means) 62. Information Department (Information Methods) 300 databases P3 Server Program M3 Language Model 4 Clients 43 Display section

Claims

1. A process selection means for selecting a process (hereinafter referred to as "each process") corresponding to each of the multiple tasks that constitute the image processing requested by the user, An image processing system comprising a process execution means that connects and executes each process selected by the process selection means.

2. The system further comprises a configuration determination means that determines a configuration, which is setting information referenced when each process is executed, in accordance with at least one of the installation environment of the camera that is the source of the image input used in the image processing, and the size of the object to be processed that is captured in the image of the camera. The image processing system according to claim 1, characterized in that the process execution means executes each of the processes by referring to the configuration determined by the configuration determination means.

3. The image processing system according to claim 1, wherein the processes selectable by the process selection means include a process corresponding to an object detection task and a process corresponding to an object recognition task, and the detector used in the process corresponding to the object detection task and the recognizer used in the process corresponding to the object recognition task are modular and can be freely connected.

4. The image processing system according to claim 1, characterized in that the process execution means inputs the execution results for each frame of a plurality of frame images used in the image processing into a language model for summarization, thereby obtaining a combined execution result for the plurality of frame images.

5. The image processing system according to claim 1, wherein the processes selectable by the process selection means include a process for each of a plurality of frame images used in the image processing, and a process that inputs the execution results of each of the plurality of frame images into a language model for summarization to obtain a combined execution result for the plurality of frame images, and these processes are modularized and can be freely connected.

6. An input means for the user to input a query corresponding to the desired image processing, An image processing system comprising a decomposition means for decomposing a task included in a query input by the aforementioned input means into a plurality of known tasks.

7. The decomposition means includes a search function that searches for documents useful for decomposing the tasks included in the query into the plurality of known tasks, and a language model for task decomposition, and the image processing system according to claim 6 is characterized in that the tasks included in the query are decomposed into the plurality of known tasks by inputting the documents useful for task decomposition searched by the search function into the language model for task decomposition along with the query input by the input means.

8. The image processing system according to claim 6, characterized in that the decomposition means decomposes the tasks included in the query input by the input means into at least an object detection task and an object recognition task.

9. The image processing system according to claim 6, characterized in that the decomposition means decomposes the query input by the input means into at least specification information of a device used to perform the image processing.

10. The image processing system according to claim 6, characterized in that the decomposition means decomposes the tasks included in the query input by the input means into at least tasks corresponding to a process for each frame and tasks corresponding to a process for multiple frames.

11. The image processing system according to claim 6, characterized in that the decomposition means decomposes the tasks included in the query input by the input means into at least real-time processing tasks and offline processing tasks.

12. The image processing system according to claim 11, characterized in that the real-time processing tasks are performed on an edge device and the offline processing tasks are performed on the cloud.

13. A determination means for determining whether or not all the information necessary for the query entered by the input means is included, The image processing system according to claim 6, further comprising a notification means that, when the determination by the determination means shows that all the necessary information is not included, prompts the user to input the information that has not yet been entered from the necessary information using the input means.

14. An input means for the user to input a query corresponding to the desired image processing, A decomposition means for decomposing a task included in a query input by the input means into a plurality of known tasks, A process selection means for selecting a process corresponding to each of the multiple tasks obtained by the aforementioned decomposition means, An image processing system comprising a process execution means that connects and executes each process selected by the process selection means.

15. The image processing system according to claim 14, characterized in that the decomposition means extracts information of the object to be detected by the object detection task included in the query and information of the object to be recognized by the object recognition task included in the query from the query input by the input means.

16. Computers, A process selection means for selecting a process corresponding to each of the multiple tasks that constitute the image processing requested by the user, An image processing program that functions as a process execution means for chaining together and executing each process selected by the aforementioned process selection means.

17. Computers, An input means for the user to input a query corresponding to the desired image processing, An image processing program that functions as a decomposition means for decomposing a task included in a query input by the aforementioned input means into a plurality of known tasks.

18. Computers, An input means for the user to input a query corresponding to the desired image processing, A decomposition means for decomposing a task included in a query input by the input means into a plurality of known tasks, A process selection means for selecting a process corresponding to each of the multiple tasks obtained by the aforementioned decomposition means, An image processing program that functions as a process execution means for chaining together and executing each process selected by the aforementioned process selection means.

Images