Image processing system and image processing program

Abstract

An image processing system and an image processing program are provided that can appropriately perform image processing desired by a user. [Solution] An image processing system includes a process selection means for selecting a process corresponding to each of a plurality of tasks constituting image processing requested by a user, and a process execution means for connecting and executing the processes selected by the process selection means. The image processing system may also include an input means for a user to input a query corresponding to the desired image processing, and a decomposition means for decomposing a task included in the query input by the input means into a plurality of known tasks.

Description

[Technical Field]

[0001] The present invention relates to an image processing system and an image processing program that realize image processing that meets user requirements. [Background technology]

[0002] Technologies are widely used to detect objects such as people in images captured by a camera and to recognize the identity of the detected object and its attributes, such as its color and type. Pattern recognition technology is also used, but recently, advances have been made in trained models (hereinafter referred to as "trained models") using neural networks (hereinafter referred to as NNs) that are trained to input image data and output detection or recognition results for objects captured in the image data. These trained models are trained for each type of object or for the environment in which the image was taken, and are further subdivided and optimized to enable more accurate detection or recognition depending on the type of object captured in the image.

[0003] In response to this, a foundation model is provided that is trained using a huge amount of data set that can handle objects in any field and also handles language input and output. The foundation model can perform a variety of tasks, such as image generation and natural language conversation, on data from a wide range of fields. By using a foundation model that has been trained broadly to meet the demands of such diverse fields, inference can be performed in a wide range of fields regardless of the analysis target.

[0004] Learning models that are trained for each type of object to be detected or for each environment are suited to a specific purpose or environment and therefore have low versatility. On the other hand, systems that use foundational models are trained using datasets with huge amounts of data and are highly versatile, but may not be able to provide the output accuracy desired by users for specific purposes or environments.

[0005] Patent Document 1 proposes a method for selecting a model through user operation, stating that character recognition can be performed more accurately by using an individual learning model that is trained according to characteristics rather than a common learning model (trained model).Patent Document 1 particularly proposes selecting a model according to the handwriting habits that differ for each user for character recognition. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2024-023055 Summary of the Invention [Problem to be solved by the invention]

[0007] There are also many learning models available for different data types in various fields. Even if it is possible to manually select a learning model, selecting an appropriate model requires a high level of skill and knowledge.

[0008] An object of the present invention is to provide an image processing system and an image processing program that can appropriately perform image processing desired by a user. [Means for solving the problem]

[0021] In an image processing system according to an embodiment of the present disclosure, The image processing system includes an input means for inputting a query corresponding to a desired image processing by a user, a decomposition means for decomposing a task included in the query input by the input means into a plurality of known tasks, a process selection means for selecting a process (hereinafter referred to as "each process") corresponding to each of the plurality of known tasks obtained by the decomposition means, and a process execution means for connecting and executing the processes selected by the process selection means, The decomposition means decomposes a task included in the query into the plurality of known tasks. for The system has a search function for searching documents and a language model for task decomposition, and searches the documents by the search function together with a query input by the input means. Ta sentence The task included in the query is decomposed into the plurality of known tasks by inputting the query into the language model for task decomposition. do .

[0022] In an image processing system according to an embodiment of the present disclosure, when a query input by a user is decomposed into multiple tasks, external useful documents and a language model for task decomposition are used to appropriately decompose the query. This enables the query to be decomposed into tasks that produce highly accurate results, and the optimal processes for each task to be connected and executed. The useful documents are typically data such as a manual that defines rules for decomposing tasks included in the query into multiple known tasks. The image processing system according to this embodiment applies Retrieval Augmented Generation (RAG) technology. The simplest implementation is to prepare expected user queries and examples of task decomposition results as additional documents for RAG.

[0025] In an image processing system according to an embodiment of the present disclosure, The image processing system includes an input means for inputting a query corresponding to a desired image processing by a user, a decomposition means for decomposing a task included in the query input by the input means into a plurality of known tasks, a process selection means for selecting a process (hereinafter referred to as "each process") corresponding to each of the plurality of known tasks obtained by the decomposition means, and a process execution means for connecting and executing the processes selected by the process selection means, The decomposition means uses the query input by the input means at least for executing the image processing. Device Break down into spec information do .

[0026] In an image processing system according to an embodiment of the present disclosure, device specification information useful for selecting an appropriate process for an image to be processed is acquired from a query entered by a user. This allows a device with limited computational resources to select a lighter process rather than a process that places an excessively heavy load on the device, and a device with more computational resources to select a process that increases accuracy even if it places a heavier load on the device.

[0029] In an image processing system according to an embodiment of the present disclosure, The image processing system includes an input means for inputting a query corresponding to a desired image processing by a user, a decomposition means for decomposing a task included in the query input by the input means into a plurality of known tasks, a process selection means for selecting a process (hereinafter referred to as "each process") corresponding to each of the plurality of known tasks obtained by the decomposition means, and a process execution means for connecting and executing the processes selected by the process selection means, The decomposition means decomposes a task included in a query input by the input means into at least a real-time processing task and an offline processing task.

[0030] An image processing system according to an embodiment of the present disclosure decomposes tasks included in a query input by a user into at least real-time processing tasks and offline processing tasks. Because different processes are appropriate for real-time processing tasks and offline processing tasks, appropriate decomposition can produce more accurate output.

[0031] In an image processing system according to an embodiment of the present disclosure, the real-time processing task is executed on an edge device, and the offline processing task is executed on a cloud side.

[0032] In an image processing system according to an embodiment of the present disclosure, offline processing is performed in the cloud, and real-time processing is performed at the edge, thereby separating the processing and enabling edge computing resources to be used for real-time processing.

[0041] An image processing program according to an embodiment of the present disclosure includes a computer including: an input unit for inputting a query corresponding to a desired image processing by a user; a decomposition unit for decomposing a task included in the query input by the input unit into a plurality of known tasks; and a processing unit for processing the plurality of known tasks obtained by the decomposition unit. known The process selection means selects a process corresponding to each task, and the process execution means executes the processes selected by the process selection means in a chain. The decomposition means has a search function for searching for documents for decomposing a task included in the query into the plurality of known tasks, and a language model for task decomposition, and by inputting the document searched by the search function together with the query input by the input means into the language model for task decomposition, the task included in the query is decomposed into the plurality of known tasks. . An image processing program according to one embodiment of the present disclosure causes a computer to function as an input means for inputting a query corresponding to a desired image processing by a user, a decomposition means for decomposing a task included in the query input by the input means into a plurality of known tasks, a process selection means for selecting a process corresponding to each of the plurality of known tasks obtained by the decomposition means, and a process execution means for connecting and executing each process selected by the process selection means, wherein the decomposition means decomposes the query input by the input means into at least specification information of a device to be used to execute the image processing. An image processing program according to one embodiment of the present disclosure causes a computer to function as an input means for a user to input a query corresponding to desired image processing, a decomposition means for decomposing a task included in the query input by the input means into a plurality of known tasks, a process selection means for selecting a process corresponding to each of the plurality of known tasks obtained by the decomposition means, and a process execution means for connecting and executing each process selected by the process selection means, wherein the decomposition means decomposes the task included in the query input by the input means into at least a real-time processing task and an offline processing task. [Effects of the Invention]

[0042] According to the present disclosure, multiple processes can be appropriately selected in response to user requests, and processes using learning models that have been trained with high accuracy and specialized for each field and / or configuration can be linked together, resulting in more accurate image processing results than when using a base model that covers a wide range of fields. [Brief explanation of the drawings]

[0043] [Figure 1] 1 is a schematic diagram of an image processing system according to a first embodiment. [Figure 2] FIG. 1 is a block diagram showing a configuration of an image processing device. [Figure 3] FIG. 2 is a block diagram showing the configuration of a server. [Figure 4] FIG. 2 is a block diagram showing the configuration of a client; [Figure 5] 5 is a flowchart showing an example of a processing procedure for process selection in the image processing system according to the first embodiment. [Figure 6] FIG. 3 is a diagram showing an example of a process selection step in the image processing apparatus according to the first embodiment. [Figure 7] FIG. 1 is a functional block diagram of an image processing system according to a first embodiment. [Figure 8] FIG. 10 is a block diagram showing the configuration of a server according to a second embodiment. [Figure 9] 10 is a flowchart showing an example of a processing procedure for process selection in an image processing system according to a second embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a process selection step in the image processing apparatus according to the second embodiment. [Figure 11] FIG. 10 is a diagram showing another example of a process selection process in the image processing apparatus according to the second embodiment. [Figure 12] 11 is a flowchart showing an example of a processing procedure for process selection in an image processing system according to a third embodiment. [Figure 13] 11 is a flowchart showing an example of a processing procedure for process selection in an image processing system according to a third embodiment. [Figure 14] FIG. 11 is a diagram showing an example of a process selection process in the image processing apparatus according to the third embodiment. [Figure 15] FIG. 10 is a functional block diagram of an image processing system according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0044] The present disclosure will be specifically described with reference to the drawings showing embodiments thereof.

[0045] [First embodiment] 1 is a schematic diagram of an image processing system 100 according to a first embodiment. The image processing system 100 includes a camera 2, an image processing device 1 connected to the camera 2, a server 3 capable of communication with the image processing device 1, and a client 4 capable of communication with the server 3.

[0046] The image processing system 100 of the first embodiment is a system that selects a process to be executed by the image processing device 1 for an image captured by a camera 2 installed in a target space, and causes the image processing device 1 to execute the process. The process may be executed using a trained learning model provided by an external service outside the system, or may be executed using a learning model designed and trained by the image processing system 100. The image processing system 100 receives a user request via a client 4 to determine which task process to execute, and selects the process in accordance with the received request. The image processing system 100 may receive the user's request in natural language, describing what to do using the camera 2, and determine which process to select using a language model, or may receive the request by having the user directly select the process.

[0047] In the image processing system 100, a user request may be received at a user interface connected to the image processing device 1, and the image processing device 1 may perform a function of selecting a process corresponding to a task in accordance with the request. In the image processing system 100, the client 4 may receive a user request and transmit the request to the image processing device 1 via the network N, and the image processing device 1 may perform a function of selecting a process corresponding to a task in accordance with the user. In the following explanation, a configuration in which the process selection function is performed by the server 3 will be described.

[0048] A configuration for realizing such an image processing system 100 will be described in more detail. In Fig. 1, camera 2 uses an image element that corresponds to visible light and / or near-infrared light and outputs image data. Camera 2 outputs image data in time series at a rate of several fps to several tens of fps. Camera 2 sequentially transmits the image data to image processing device 1 via a directly connected communication line or a local network.

[0049] The image processing device 1 is a device that executes a process selected by the image processing system 100 on image data acquired from the camera 2. The image processing device 1 outputs the results obtained by executing the selected process to the client 4. The image processing device 1 may store the results internally and make them readable by the client 4, or may transmit them to the client 4 via the network N. The image processing device 1 may transmit the execution results of the process to the server 3 via the network N, store them in the server 3, and make them readable by the client 4.

[0050] The server 3 stores a group of learning models used in each process to be executed by the image processing device 1 in a database 300. The database 300 also includes a group of learning models provided by external services outside the system. The server 3 reads out from the database 300 a learning model corresponding to a process selected for the image processing device 1 and deploys it to the image processing device 1.

[0051] Database 300 stores, in a manner that allows it to provide detection models that are trained to determine whether a specific person or object is captured in an input image according to the characteristics of the object, such as a model that detects whether a person is captured in the image or a model that detects whether a vehicle is captured in the image. Database 300 also stores, in a manner that allows it to provide recognition models that recognize attributes of detected people or objects. Database 300 stores attribute-specific models that recognize a person's age range as an attribute. Database 300 stores attribute-specific models that recognize attributes such as the type and product number of a detected object as an attribute. Database 300 stores object-specific models that recognize clothing and accessories worn by a detected person. Database 300 may also store models that recognize the color or pattern of a detected object.

[0052] The person detection model, object detection model, and attribute recognition model are each modularized as detectors or recognizers, and may be provided so that one or more detectors or recognizers can be connected in any order.

[0053] The database 300 holds a language model trained to output a group of words with a high probability of occurrence as a response to an input query in natural language. The language models held by the database 300 include a large language model (LLM) that can be used on devices with abundant computing resources, a small language model (SLM) that can be used on devices with limited computing resources, and a medium-sized language model that can be used on devices with medium computing resources. The database 300 holds a vision language model (VLM) that accepts queries in addition to image data, and a multimodal language model that accepts queries in addition to voice data, so that they can be provided.

[0054] Database 300 holds multiple types of configuration data including setting information such as the size of the detection target area or the size of the recognition target area in the image for models that detect people or objects from images or models that recognize the attributes of detected people or objects. Database 300 holds configuration data that can be provided according to the installation environment of camera 2, image size, or resolution.

[0055] In the image processing system 100 of the first embodiment, the server 3 selects the learning model and configuration data stored in the database 300 in this way in response to a user request, and causes the image processing device 1 to execute the selected model.

[0056] The server 3 may store the data transmitted from the image processing device 1 in association with data identifying the target space. The server 3 may aggregate the transmitted data and create data that can be referenced by the client 4.

[0057] The detailed configurations of the image processing device 1 and the server 3 for realizing such an image processing system 100, as well as the details of the processing, will be described below.

[0058] 2 is a block diagram showing the configuration of the image processing device 1. The image processing device 1 uses an edge computer. In the following description, the image processing device 1 is described as a single computer, but it may be configured such that multiple computers share the processing for each process. The image processing device 1 includes a processing unit 10, a storage unit 11, a first communication unit 12, and a second communication unit 13.

[0059] The processing unit 10 includes one or more processors such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), or a neural processing unit (NPU). The processing unit 10 includes a memory that is a temporary storage medium such as a static random access memory (SRAM) or a dynamic random access memory (DRAM). The processing unit 10 includes a timer and can acquire time information at each point in time from data from the timer. The processing unit 10 may be configured as a single piece of hardware (SoC: System On a Chip) that integrates a processor, a memory, a storage unit 11, a first communication unit 12, and a second communication unit 13.

[0060] The processing unit 10 causes the processor to execute image processing based on the image processing program P1 stored in the storage unit 11 and the learning model selected from the database 300 and deployed.

[0061] The storage unit 11 is a relatively large-capacity non-transitory storage medium such as a hard disk, a flash memory, etc. A part of the storage unit 11 may be removable.

[0062] The storage unit 11 stores a program (program product) required for the processing unit 10 to execute processing, the results of processing by the processing unit 10, and reference setting data. The setting data includes identification data of the device itself. The program product includes an OS (Operating System) program, an image processing program P1 that runs on the OS, and a learning model group M1. The learning model group M1 will be described in detail later.

[0063] The image processing program P1 stored in the storage unit 11 may be an image processing program P9 stored in a computer-readable storage medium 9 that has been read by the processing unit 10 and stored in the storage unit 11, or may be a program that has been stored in advance at the time of shipment. The image processing program P1 stored in the storage unit 11 may be an image processing program P9 that has been downloaded by the processing unit 10 from the server 3 or another download server via the second communication unit 13 and stored in the storage unit 11.

[0064] The image processing program P1 stored in the storage unit 11 is configured to cause the computer to execute processes corresponding to a plurality of tasks, and it is possible to select which task process to execute. The image processing program P1 may be configured by acquiring program modules for executing processes corresponding to a plurality of tasks from the server 3 and combining these modules.

[0065] At least a part of the learning model group M1 stored in the memory unit 11 is selected from the database 300. The setting data stored in the memory unit 11 may include configuration data selected from the database 300. The learning model group M1 and configuration data selected from the database 300 and received by the processing unit 10 via the second communication unit may not be stored in the memory unit 11, but may be stored in a temporary storage medium (RAM) built into the processing unit 10.

[0066] The first communication unit 12 is a communication device that realizes communication via a local network in the space where the camera 2 is installed. The first communication unit 12 may be a LAN network card or a CAN communication device. The first communication unit 12 may be a communication device compatible with wireless networks such as Wi-Fi or Bluetooth (registered trademark). The first communication unit 12 may include multiple communication devices compatible with various types of cameras 2. The first communication unit 12 may include an interface such as a USB (Universal Serial Bus) that is connected to the camera 2. The first communication unit 12 can be replaced by an interface that is connected to the camera 2 via a coaxial cable or another serial bus. The processing unit 10 acquires image data from the camera 2 via the local network using the first communication unit 12. The first communication unit 12 may be the same device as the second communication unit 13.

[0067] The second communication unit 13 is a communication device that realizes communication via the network N with communication devices outside the space in which the camera 2 is installed. The second communication unit 13 may be a network card for a wired LAN, a communication device that realizes carrier communication via a carrier network, or a communication device compatible with a wireless network such as WiFi or Bluetooth (registered trademark). The second communication unit 13 may be compatible with encrypted communication such as SSL with the server 3. The second communication unit 13 may be an interface that realizes connection with the server 3 via a dedicated line.

[0068] The image processing device 1 may directly receive user operations via a user interface connected via the second communication unit 13.

[0069] 3 is a block diagram showing the configuration of the server 3. The server 3 may be configured as a single server computer, or may be configured to distribute processing among multiple server computers. The server 3 includes a processing unit 30, a storage unit 31, and a communication unit 32.

[0070] The processing unit 30 includes one or more processors such as a CPU, an MPU, a GPU, an NPU, etc. The processing unit 30 includes a memory that is a temporary storage medium such as an SRAM or a DRAM.

[0071] The storage unit 31 is a relatively large-capacity non-temporary storage medium such as a hard disk, flash memory, etc. The storage unit 31 stores programs (program products) and setting data required for the processing unit 30 to execute processing.

[0072] The program products stored in the storage unit 31 include a server program P3. The server program P3 includes a module that functions as a data server that reads out a group of models stored in the database 300 and transmits them to the image processing device 1. The server program P3 includes a module that functions as a web server, and can output the results of processing executed by the server 3 to the client 4 via a web page.

[0073] The program product includes a language model (LM) M3. The language model M3 outputs a response corresponding to an input natural sentence. The language model M3 is used to provide an output for selecting a process in response to a request from a user, separate from the language models stored in the database 300. The language model M3 may be a model that is used in part or in whole from an external language model providing service via the network N. The processing using the language model M3 will be described in detail below.

[0074] The server program P3 and the language model M3 may be server program P8 and language model M8 stored in a computer-readable storage medium 8 that are read by the processing unit 30 and stored in the memory unit 31, or they may be downloaded by the processing unit 30 from another download server via the communication unit 32 and stored in the memory unit 31.

[0075] The setting data stored in the storage unit 31 includes data identifying the image processing device 1 that is the target of process selection via the server 3. The setting data also includes, in association with the data identifying the image processing device 1, data and names for identifying the space in which the image processing device 1 is installed, and a correspondence between the data and names and identification data of the cameras 2 from which the image processing device 1 can acquire images. Identification data of image processing devices 1 or spaces for which user requests are permitted may be stored as a whitelist in association with the user's account data. This allows the server 3 to identify the target image processing device 1 when a user specifies the name of a space and requests what kind of image processing should be performed on images captured by the cameras 2 installed in that space.

[0076] The database 300 may be constructed in the storage unit 31 or in an external storage device. As described above, part of the database 300 may include a model providing service available on the Web that is connected via the network N for communication.

[0077] The communication unit 32 is a communication device that realizes a communication connection with the client 4 and the image processing device 1 via the network N.

[0078] 4 is a block diagram showing the configuration of the client 4. The client 4 is a personal computer, a smartphone, or a tablet terminal. The client 4 may be used by a manager of the space where the camera 2 is installed, or by an operator of the management company of the server 3.

[0079] The client 4 includes a processing unit 40, a storage unit 41, a communication unit 42, a display unit 43, and an operation unit 44. The processing unit 40 includes one or more processors such as a CPU, an MPU, a GPU, or an NPU. The processing unit 40 includes a memory that is a temporary storage medium such as an SRAM or a DRAM.

[0080] The storage unit 41 is a non-transitory storage medium memory such as a hard disk or flash memory. The storage unit 41 stores the data server functions provided by the server 3 and a client program P4 for the web server. The client program P4 is, for example, a web browser program. The client program P4 may also be a program that causes the processing unit 40 to execute a process for displaying data provided by the server 3 on a screen.

[0081] The communication unit 42 is a communication device that realizes a communication connection with the server 3 via the network N. The communication unit 42 may be a communication device that realizes a communication connection with the server 3 via a dedicated line. The communication unit 42 may be a communication device that realizes a direct communication connection with the second communication unit 13 of the image processing device 1 via a wireless communication medium, a USB cable, or the like.

[0082] The display unit 43 uses a display such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 43 displays a web page including text and images through processing based on the client program P4 of the processing unit 40. The display unit 43 may use a display with a built-in touch panel.

[0083] The operation unit 44 is a user interface such as a keyboard or a pointing device that accepts operations from a user or an operator. The operation unit 44 may be a touch panel built into the display of the display unit 43, or may be physical buttons. The operation unit 44 may be a voice input unit that accepts operations by voice using a voice recognition function. The operation unit 44 can notify the processing unit 40 of operation information by the user or operator.

[0084] The following describes the process in the image processing system 100 configured as above, in which a process is selected for the image processing device 1 and the selected process becomes executable. Fig. 5 is a flowchart showing an example of the process selection procedure in the image processing system 100 of the first embodiment. When a user uses the client 4 to access the server 3 and access a web page for making settings for the image processing device 1, the server 3 starts the following process.

[0085] The processing unit 30 of the server 3 receives, as a user request, data specifying the space in which the camera 2 to be processed is installed (step S301). In step S301, the processing unit 30 receives one or more of the user's account data, the identification data or name of the space, and the identification data of the camera 2. In step S301, the processing unit 30 may receive a selection from a list of the identification data of the image processing device 1 that is permitted to access the account data used when the client 4 accessed the server 3, and the identification data or name of the corresponding space.

[0086] The processing unit 30 identifies the identification data of the image processing device 1 corresponding to the space identified by the received data (step S302).

[0087] The processing unit 30 receives, on the web page, settings of the installation environment in which the camera 2 to be processed by the identified image processing device 1 is installed, and a user's request for an image captured by the camera 2 (step S303). In step S303, the processing unit 30 receives, as the user's request, natural language input into an input field included in the web page displayed on the client 4. In step S303, the processing unit 30 may receive a selection of options for a question included in the web page displayed on the client 4. For example, the processing unit 30 may receive multiple selections from options for a task (person or object to be detected, attribute of a recognition target) displayed on the web page. In step S303, the processing unit 30 receives, as the installation environment settings, information such as the installation environment of the camera 2 (indoor, outdoor, entrance, exit, passageway, etc.), the size of the subject captured by the camera 2 in the image, the camera rate, and the specifications of the image processing device 1.

[0088] The processing unit 30 analyzes the received user request (step S304). In a first example, in step S304, the processing unit 30 combines the natural sentence (query) received as the user request with an instruction statement instructing to output a task corresponding to the natural sentence in a predetermined format, inputs the combined result into the language model M3, and acquires a sentence (group of words) output from the language model M3. The processing unit 30 may identify the person or object to be detected and the attributes of the recognition target.

[0089] Based on the analysis results of step S304, the processing unit 30 selects processes to be executed by the image processing device 1 (step S305). The processing unit 30 identifies a learning model to be used for each selected process from the database 300 (step S306). In step S305 or step S306, the processing unit 30 may select a process or identify a learning model by referring to the settings of the installation environment of the camera 2 received in step S303. From the data on computational resources included in the specifications of the image processing device 1, the processing unit 30 identifies that an LLM should be used if, for example, the processing speed and memory are equal to or higher than a predetermined specification level, and identifies that an SLM should be used if the opposite is true.

[0090] The processing unit 30 determines the configuration data to be used in the image processing device 1 from the database 300 according to the received installation environment settings (step S307). In step S307, the processing unit 30 determines the configuration data to be used according to the installation environment of the camera 2 (indoors, outdoors, entrance, exit, passageway, etc.), the size of the subject captured by the camera 2 in the image, the camera rate, and the specifications of the image processing device 1.

[0091] The processing unit 30 transmits the identification data of the process selected in step S305, the learning model specified in step S306, and the configuration data determined in step S307 to the image processing device 1 (step S308). The processing unit 30 makes the process using the transmitted learning model and configuration data executable on the image processing device 1 (step S309). In step S309, the processing unit 30 of the server 3 generates an instance of the image processing program P1 configured to execute the selected process, and transmits the generated instance to the image processing device 1. The processing unit 10 of the image processing device 1 stores this instance as the image processing program P1 in the storage unit 11. That is, the processing unit 30 of the server 3 deploys the image processing program P1 to the image processing device 1.

[0092] As a result, the processing unit 10 of the image processing device 1 executes the selected process on the image acquired from the camera 2 according to the image processing program P1.

[0093] The processing procedure shown in Fig. 5 has been described as being executed by the server 3. However, the present invention is not limited to this, and the image processing device 1 may accept an operation from the client 4 via the network N and execute the processing procedure shown in Fig. 5, or the image processing device 1 may accept an operation from the client 4 directly via the second communication unit 13 and execute the processing procedure shown in Fig. 5 using the language model M3 provided by the server 3.

[0094] The processing procedure shown in Fig. 5 will be explained using a specific example. Fig. 6 is a diagram showing an example of the steps in which a process is selected in the image processing device 1 in the first embodiment. The example shown in Fig. 6 shows the steps when a user specifies data identifying the space in which the target camera 2 is installed on the client 4 and inputs a natural sentence such as "I would like to count the number of elderly people wearing hats and glasses who come to this location every week."

[0095] The server 3 adds an instruction such as "Please extract the tasks necessary to realize the attached request using the following rules. Rules: <detection target> target, <detection target tracking method> tracking unit, <attributes to be recognized>" to the above natural sentence received as a user request, and provides it to the language model M3.

[0096] In the example of Figure 6, the language model M3 outputs "<detection target> person detection, <detection target tracking method> person tracking and assigning a person ID, <attributes to be recognized> elderly, wearing glasses, wearing a hat." The above information "<attributes to be recognized> elderly, wearing glasses, wearing a hat" corresponds to the "information of the recognition target by the object recognition task included in the query" extracted by the "decomposition means" in the claims. Based on the output from the language model M3, the processing unit 30 of the server 3 selects a person detection process that detects people from images using a person detection model (person detector), and a tracking process that assigns the same ID to the same person detected across multiple frame images from the features of the detected person. For the tracking process, the processing unit 30 selects a model for face identification (face classifier) ​​to identify the same person. The processing unit 30 further selects a learning model for age recognition (age recognition). The processing unit 30 selects an elderly person recognition process that uses a model for eyeglasses recognition (an eyeglasses-wearing recognizer) to recognize whether the detected person is elderly or not. The processing unit 30 further selects an eyeglasses-wearing recognition process that uses a model for hat recognition (a hat-wearing recognizer) to recognize whether the detected person is wearing eyeglasses or not, and a hat-wearing recognition process that uses a model for hat recognition (a hat-wearing recognizer) to recognize whether the detected person is wearing a hat or not. The processing unit 30 further selects a process that provides the results of the elderly person recognition process, the eyeglasses-wearing recognition process, and the hat-wearing recognition process for a plurality of frame images over a specified period, here, one week, to a large-scale language model to obtain a summarized result.

[0097] When a user request is received by receiving a selection from options presented on a web page without using the language model M3, the processing unit 30 identifies the tasks as "person detection / person tracking / elderly or not, wearing glasses or not, wearing a hat or not." In this case, the processing unit 30 may select a person detection process, a tracking process, an elderly person recognition process, a glasses-wearing recognition process, a hat-wearing recognition process, and a summarization process corresponding to each task.

[0098] The processing unit 30 identifies the person detection model, face identification model, age recognition learning model, eyeglasses recognition model, hat recognition model, and summarization language model to be used for each selected process from the database 300. The processing unit 30 transmits data identifying the selected process, or an executable file corresponding to the process, a model that can be read from the executable file, and configuration data to the image processing device 1.

[0099] In the image processing device 1, based on the transmitted process identifying data, a person detection model (person detector) 101, a learning model for face identification (face classifier) ​​102, a learning model for age recognition (age recognizer) 103, a model for eyeglasses recognition (eyeglasses-wearing recognizer) 104, a model for hat recognition (hat-wearing recognizer) 105, and a language model for summarization 106 are stored in the storage unit 11 so as to be usable as a learning model group M1. Based on the process identifying data, the processing unit 10 configures the image processing program P1 to link with the learning model group M1 used by the person detection process, tracking process, elderly person recognition process, eyeglasses-wearing recognition process, hat-wearing recognition process, and summarization process by referring to the configuration data, and makes them executable.

[0100] Thereafter, the processing unit 10 of the image processing device 1 acquires frame images output from the camera 2 in chronological order, assigns identification data to the frame images, and executes a person detection process for each frame image. In the person detection process, the processing unit 10 provides the image to the person detection model 101 and obtains a detection result (coordinate data of the person area) output from the person detection model 101. The processing unit 10 improves the detection accuracy of the person detection model 101 by referring to the image size and the like included in the configuration data. If no person is detected in the person detection process, the processing unit 10 executes processing for the next frame image.

[0101] When a person is detected in the person detection process, the processing unit 10 provides the frame image and the detection result to the tracking process. In the tracking process, the processing unit 10 acquires the facial features of the person detected from the input frame image using a face recognition learning model, associates the features with a person ID, and identifies the person ID of the detected person. In the tracking process, the processing unit 10 may also refer to the configuration data.

[0102] The processing unit 10 passes the frame image, the detection result, and the person's person ID to the elderly person recognition process, the eyeglasses-wearing recognition process, and the hat-wearing recognition process. In each of the elderly person recognition process, the eyeglasses-wearing recognition process, and the hat-wearing recognition process, the processing unit 10 outputs whether the person identified by the person ID is elderly, whether the person is wearing eyeglasses, and whether the person is wearing a hat using the learning model group M1. The image processing program P1 may be configured so that the processing unit 10 executes the elderly person recognition process, the eyeglasses-wearing recognition process, and the hat-wearing recognition process in a permutation. In this case, the processing unit 10 executes the eyeglasses-wearing recognition process to determine whether the person is wearing eyeglasses only if the detected person is recognized as elderly, and executes the hat-wearing recognition process to determine whether the person is wearing a hat only if the eyeglasses-wearing recognition process recognizes that the detected person is wearing eyeglasses.

[0103] The processing unit 10 integrates the results of the recognition process executed for each of the multiple frame images, and for each frame image, if the person detected in the frame image is elderly, wearing glasses, and a hat, stores data indicating that an object has been detected, together with the frame image identification data and the person ID. The processing unit 10 may also store time information when the frame image was captured. The processing unit 10 may store only the frame images in which the object has been detected in the storage unit 11. Then, in the summarization process, the processing unit 10 aggregates the stored data (frame images in which it has been determined that an object has been detected) for a specified period (here, one week), provides the aggregation results to the language model M3, and outputs an explanation of the aggregation results for one week in natural language.

[0104] The processing unit 10 of the image processing device 1 may integrate the results of the recognition process executed for each of the multiple frame images, and for each frame image, if the person detected from the frame image is elderly, wearing glasses and a hat, may transmit identification data of that frame image to the server 3. In addition to the identification data of the frame image in which the target is detected, the person ID (feature amount) of the detected person and detection time information may also be transmitted to the server 3. The process of aggregation and summarization using the language model M3 may be executed by the server 3.

[0105] Using the client 4, the user can refer to the detection results stored for each week in the image processing device 1 and an explanation of the tabulated results. The client 4 may directly acquire the tabulated results for each week stored in the memory unit 11 of the image processing device 1 via the second communication unit 13 of the image processing device 1 and display them on a screen based on the client program P4, or may acquire them via the server 3 and output the tabulated results on a web page provided by the server 3.

[0106] 6, the counting result is output as a natural sentence: "Two elderly people wearing hats and glasses have visited this location in the past week." The server 3 may store identification data, person IDs, and time information of frame images associated with the person IDs of two people, and the frame images may be referenced.

[0107] As shown in Figure 6, a user simply inputs a request into the image processing system 100 in natural language, for example, on a web page, regarding what they want to do with the image captured by the target camera 2, and the image processing system 100 automatically selects a process that meets the user's request. The user does not need to select a learning model from among those prepared according to various specifications (of the image processing device 1, etc.) for data in a wide variety of fields. Data on the installation environment of the camera 2 and information on the specifications of the camera 2 and the image processing device 1 are also passed to the image processing system 100, and the image processing system 100 selects a model and process that meets the specifications. This avoids the use of an unnecessarily high-precision model that does not match the specifications, and conversely, it is possible to avoid selecting a low-precision process, which results in results that do not meet the user's request.

[0108] 6, if a plurality of cameras 2 are connected to the image processing device 1 and processing is performed on images of a space (for example, images of a store) captured by the plurality of cameras 2, the tracking process described above becomes a process for multi-camera object tracking. Here, multi-camera object tracking means tracking multiple tracked objects across (images captured by) multiple cameras while taking into account occlusion (when tracked objects are hidden from the cameras).

[0109] FIG. 7 shows functional blocks of an image processing system 100 according to a first embodiment. This FIG. 7 is a diagram for explaining that the constituent elements (means) of the claims are described in the first embodiment. The image processing system 100 includes, as functional blocks, an input unit 51 through which a user inputs a query corresponding to desired image processing; a decomposition unit 53 that decomposes a task included in the query input by the input unit 51 into a plurality of known tasks; a process selection unit 54 that selects a process corresponding to each of the plurality of tasks obtained by the decomposition unit 53; and a process execution unit 55 that connects and executes the processes selected by the process selection unit 54. The image processing system 100 also includes, as functional blocks, a configuration determination unit 52 that determines a configuration, which is setting information to be referenced when each process is executed, based on at least one of the installation environment of a camera 2 that inputs an image used for image processing and the size of an object to be image-processed that appears in an image captured by the camera 2. The above-mentioned input unit 51, configuration determination unit 52, decomposition unit 53, process selection unit 54, and process execution unit 55 correspond to the input means, configuration determination means, decomposition means, process selection means, and process execution means in the claims, respectively.

[0110] The input unit 51 is mainly realized by the operation unit 44, the processing unit 40, and the communication unit 42 of the client 4, and the communication unit 32 and the processing unit 30 of the server 3. The processing performed by the input unit 51 is the processing of steps S301 to S303 in FIG. 5 (mainly the processing of step S303). The configuration determination unit 52 is realized by the processing unit 30 of the server 3, and performs the processing of step S307 in FIG. 5. The decomposition unit 53 is realized by the processing unit 30 of the server 3, and performs the processing of step S304 in FIG. 5. In the example shown in FIG. 6, the decomposition unit 53 performs the analysis processing of step S304 to decompose the task included in the input query into a plurality of known tasks, such as "<detection target> person detection, <detection target tracking method> person tracking and assignment of person ID, <attributes to be recognized> elderly, wearing glasses, wearing a hat," as described above. The process selection unit 54 is realized by the processing unit 30 of the server 3, and performs the processing of step S305 in Fig. 5. As shown in Fig. 6, the processes that can be selected by the process selection unit 54 include processes for each of the multiple frame images used in image processing (the person detection process, tracking process, elderly person recognition process, eyeglasses wearing recognition process, and hat wearing recognition process in Fig. 6), and a process that collects execution results of the processes for each of the multiple frame images and inputs them into a language model for summarization to obtain a summary of the execution results for the multiple frame images (the summarization process in Fig. 6), and these processes are modularized and can be freely connected.

[0111] Note that the query input by the input unit 51 may include information about the specifications of a device (mainly the image processing device 1) used to execute image processing, and the decomposition unit 53 may not only decompose the task included in the input query into a plurality of known tasks, but also decompose information about the device specifications included in the query (for example, information such as "execute on a high-performance device (image processing device 1)" or "execute on a device (image processing device 1) having a CPU with the processing power of ..."). The device specification information extracted by this decomposition is used by the configuration determination unit 52 to determine the configuration.

[0112] [Second embodiment] In the second embodiment, the image processing system 100 employs RAG to more appropriately select a task corresponding to a user request and a process corresponding to the task. The configuration of the image processing system 100 of the second embodiment is the same as that of the image processing system 100 of the first embodiment, except for the processing procedure and data for employing RAG, which will be described later. Therefore, the same reference numerals are used for the common configuration, and detailed description thereof will be omitted.

[0113] 8 is a block diagram showing the configuration of the server 3 according to the second embodiment. In the second embodiment, the server 3 stores, in the database 300 or the storage unit 31, a set of document data, such as a manual that defines rules for decomposing a task included in a request (query) input by a user into a plurality of known tasks. Each piece of document data may be created in advance by an operator, or may be a record of correspondence between a request actually input by a user and a task that has been able to output a result that satisfies the user, stored in a predetermined format.

[0114] The following describes the process in which the image processing system 100 of the second embodiment refers to document data, selects a process in response to a user request, and makes the selected process executable. Fig. 9 is a flowchart showing an example of the process selection procedure in the image processing system 100 of the second embodiment. When a user uses the client 4 to access the server 3 and access a web page for making settings for the image processing device 1, the server 3 starts the following process.

[0115] Of the processing procedures shown in FIG. 9, the steps common to the processing procedures shown in FIG. 5 of the first embodiment are given the same step numbers and detailed descriptions thereof will be omitted.

[0116] The processing unit 30 of the server 3 receives data identifying the target space (S301), and when it identifies the identification data of the image processing device 1 (S302), it receives, on the web page, in natural language, the settings of the installation environment in which the camera 2 to be processed by the identified image processing device 1 is installed, and the user's request for images to be captured by the camera 2 (step S323).

[0117] The processing unit 30 searches for document data useful for the received user request (step S324). In step S324, the processing unit 30 may extract similar document data that includes a group of words included in the query corresponding to the user request, or may use the language model M3 to extract appropriate document data from the document data group stored in the storage unit 31.

[0118] The processing unit 30 adds an instruction to refer to the installation environment settings and the document data searched in step S324 to the received user request, and provides the result to the language model M3 (step S325). In step S325, the processing unit 30 combines a directive statement that refers to the installation environment settings and the document data and instructs output according to the rules defined in the document data with the natural sentence corresponding to the user request, and inputs the combined statement to the language model M3. The rules defined in the document data will be described in detail later.

[0119] The processing unit 30 acquires a sentence (group of words) output from the language model M3 (step S326). The processing unit 30 identifies a task corresponding to the sentence output from the language model M3 (step S327). The language model M3, which references the document data, breaks down a user request (query) into known tasks, such as predefined names or identification data, such as "person detection," "gender recognition," "elderly recognition," and "hat wear recognition." In step S327, the processing unit 30 may at least break down the task into a detection task or a recognition task. In step S327, the processing unit 30 acquires a group of words indicating the task output from the language model M3 that references the document data. In step S327, the processing unit 30 may also identify the task by referring to the settings of the installation environment of the camera 2. For example, the processing unit 30 may select SLM as the language model for an image processing device 1 with limited computing resources, or may identify a task that includes processing to reduce the influence of ambient light for a camera 2 installed outdoors.

[0120] The processing unit 30 selects a process corresponding to the task identified in step S327 (S328). In step S328, a corresponding process is associated with a preset task, and the processing unit 30 selects a process based on this association. For the task of "person detection," a "person detection process" is defined.

[0121] The processing unit 30 identifies the learning model to be used in each of the selected processes from the database 300 (S306). From the data on the computational resources included in the specifications of the image processing device 1, the processing unit 30 identifies that an LLM is to be used if the processing speed and memory are equal to or higher than a predetermined specification level, and identifies that an SLM is to be used if the opposite is true.

[0122] Thereafter, the processing unit 30 determines the configuration data (S307), as in the first embodiment, transmits the identification data of the selected process, the identified learning model, and the determined configuration data to the image processing device 1 (S308), and executes the processing of step S309.

[0123] In the second embodiment, the processing procedure shown in Fig. 9 has also been described as being executed by the server 3. However, the present invention is not limited to this, and the image processing device 1 may accept an operation from the client 4 via the network N and execute the processing procedure shown in Fig. 9, or the image processing device 1 may accept an operation from the client 4 directly via the second communication unit 13 and execute the processing procedure shown in Fig. 9 using the language model M3 provided by the server 3.

[0124] The processing procedure shown in Fig. 9 will be explained using a specific example. Fig. 10 is a diagram showing an example of the steps for selecting a process in the image processing device 1 in the second embodiment. Like Fig. 6 in the first embodiment, Fig. 10 shows the steps from when a process is selected in response to a user request to when the selected process is executed in the image processing device 1. Like the first embodiment, Fig. 10 also shows the steps when a user specifies data identifying the space in which the target camera 2 is installed on the client 4 and inputs a natural sentence such as "I would like to count the number of elderly people wearing hats and glasses who come to this location every week."

[0125] In the second embodiment, the processing unit 30 searches for document data that corresponds to the above-mentioned natural sentence received as a user request and includes keywords such as "hat, glasses, old age." The processing unit 30 instructs the language model M3 to refer to the document data, and adds an instruction such as "Please decompose the attached query into tasks with reference to the specified document data."

[0126] A query input by a user is decomposed into tasks such as "person detection / elderly person recognition / glasses-wearing recognition / hat-wearing recognition" by a language model M3 that references document data defining decomposition rules. The processing unit 30 may decompose the task based on at least whether it is a detection task or a recognition task. The processing unit 30 can directly select processes corresponding to the decomposed tasks, such as a person detection process, an elderly person recognition process, a glasses-wearing recognition process, and a hat-wearing recognition process.

[0127] Using a language model M3 that references the document data, the processing unit 30 treats the above-mentioned tasks such as "person detection / elderly recognition / glasses wearing recognition / hat wearing recognition" as tasks for each frame, and also decomposes the word "every week" into a summarization task that aggregates the detection and recognition results across multiple frame images.

[0128] As shown in FIG. 10, the language model M3 allows decomposition into tasks for which process selection is easy, making it possible to select an appropriate process.

[0129] The processing unit 30 may further specify in the document data that specification information regarding the computational resources of the image processing device 1 to be used is to be specified. This allows the processing unit 30 to refer to the specification information associated with the identification data of the image processing device 1, which is specified in advance of the user's request using the language model M3, and to specify the task, learning model, and process to be selected from the referenced specification information. When the specifications of the image processing device 1 are low (comparatively poor computational resources), the processing unit 30 can select a process or learning model with the lightest possible processing capacity. In this way, the processing unit 30 refers to the rules specified in the document data, breaks down the input user's request into necessary tasks, selects an appropriate process, and connects the selected processes to execute image processing.

[0130] Fig. 11 is a diagram showing another example of steps in which a process is selected in the image processing device 1 according to the second embodiment. As in Fig. 10, Fig. 11 shows the results of decomposing an input user request (query) using language model M3. Fig. 11 also shows steps when the input request is a natural sentence such as "I want to count the number of elderly people wearing hats and glasses who come to this location every week."

[0131] In the example of Figure 11, similar to Figure 10, the processing unit 30 uses language model M3 with reference to document data to decompose tasks such as the above-mentioned "person detection / elderly recognition / glasses / hat recognition" into tasks for each frame, and also decomposes them into a summarization task that aggregates detection and recognition results across multiple frame images based on the phrase "every week." However, in the example of Figure 11, the processing unit 30 clearly decomposes tasks into those that should be executed sequentially in real time, such as "<real-time processing target> person detection / elderly recognition / glasses / hat recognition, <offline processing target> summarization," or those that should be executed over a period of time after the fact, and then selects a process corresponding to each task. This processing corresponds to the processing (by the "decomposition means") in the claims that "decomposes tasks included in a query input by an input means into at least real-time processing tasks and offline processing tasks."

[0132] As a result, since different processes are appropriate for real-time processing and offline processing, appropriate decomposition allows the image processing system 100 to accurately select processes corresponding to tasks and execute the selected processes in a chain. In this case, the real-time processing tasks (such as the above-mentioned "person detection / elderly recognition / glasses wear recognition / hat wear recognition") may be executed by the image processing device 1 (the "edge device" in the claims), and the offline processing tasks (the "summarization" task) may be executed by the server 3 (the "cloud side" in the claims). For example, on the real-time processing side, the image processing device 1 (the edge device) may apply VLM to each image frame to convert the image into text information (outputting the text information resulting from each offline processing task), and the server 3 (the cloud side) may apply LLM to the text information to extract information (e.g., the above-mentioned "summarization" process).

[0133] In the second embodiment, too, the user simply inputs a request in natural language to the image processing system 100, for example, on a web page, regarding what they want to do with the image captured by the target camera 2, and the image processing system 100 automatically selects a process according to the user's request. For data in a wide variety of fields, the user does not need to select from learning models prepared according to the specifications of various image processing devices 1, etc.

[0134] The decomposition unit 53 (see FIG. 7) in the second embodiment has a search function for searching for documents useful for decomposing a task included in a query into multiple known tasks, and a language model M3 for task decomposition, and by inputting documents useful for task decomposition searched for by the search function together with an input query into the language model M3 for task decomposition, the task included in the query is decomposed into multiple known tasks. This process corresponds to the processes of steps S324 to S327 in FIG. 9.

[0135] [Third embodiment] In the second embodiment, the processing unit provides the input query to the language model M3, and employs RAG that references document data so that the query can be decomposed into appropriate tasks. In the third embodiment, if the server 3 determines that not all necessary information is included and the query cannot be decomposed into appropriate tasks, the server 3 prompts the user to input additional necessary information.

[0136] The image processing system 100 of the third embodiment is similar to the image processing system 100 of the first or second embodiment except that the processing procedure shown below differs from the processing shown in the first or second embodiment. Therefore, the common configurations are given the same symbols and detailed explanations are omitted.

[0137] 12 and 13 are flowcharts showing an example of a processing procedure for process selection in the image processing system 100 of the third embodiment. When a user uses the client 4 to access the server 3 and access a web page for making settings for the image processing device 1, the server 3 starts the following processing. Of the processing procedures shown in Fig. 12 and 13, steps common to the processing procedures shown in Fig. 5 of the first embodiment and Fig. 9 of the second embodiment are assigned the same step numbers and detailed descriptions thereof will be omitted.

[0138] The processing unit 30 receives data identifying the target space (S301), identifies the identification data of the image processing device 1 (S302), and receives, on the web page, the settings of the installation environment in which the camera 2 to be processed by the identified image processing device 1 is installed and the user's request for images to be captured by the camera 2 in natural language (S323).

[0139] The processing unit 30 determines whether all necessary information is included in the installation environment settings and the user request received in step S323 (step S331). In step S331, the processing unit 30 inputs, for example, the settings and request received in step S323 and a statement asking whether the settings and request can be broken down into tasks or whether all necessary information can be obtained, into the language model M3 to obtain a response statement, and determines that all information is included if the response statement is yes, and determines that all information is not included if the response statement is no. In step S331, the processing unit 30 may determine whether all necessary information is included from the settings and request received in step S323 by comparison with a template or the like, without using the language model M3.

[0140] If it is determined in step S331 that all of the document data are included (S331: YES), the processing unit 30 searches for document data that is useful for the user request accepted in step S323 (S324). The processing unit 30 executes the processes of S325 to S328 and S306 to S309.

[0141] If it is determined in step S332 that not all of the information is included (S331: NO), the processing unit 30 identifies necessary information that has not yet been input (step S332). The processing unit 30 notifies the client 4 of a message prompting the client 4 to input the information identified in step S332 (step S333). In step S333, the processing unit 30 may notify the client 4 using a standard phrase for each piece of missing information. The processing unit 30 may notify the client 4 using a response sentence obtained from the language model M3 in step S331. The processing unit 30 may provide the language model M3 with an instruction statement instructing the language model M3 to create a message prompting the client 4 to input the missing information, thereby causing the language model M3 to create the message. The processing unit 30 accepts the information identified in step S332 (necessary information that has not yet been input) as a user input from the web page displayed on the client 4 (step S334).

[0142] The processing unit 30 adds the information received in step S334 to the settings and user request received in step S323 (step S335), and returns the process to step S331.

[0143] The processing procedures shown in Figures 12 and 13 will be explained using specific examples. Figure 14 is a diagram showing an example of the steps for selecting a process in the image processing device 1 in the third embodiment. Like Figure 6 in the first embodiment and Figures 10 and 11 in the second embodiment, Figure 14 shows the steps from selecting a process in response to a user request to executing the selected process in the image processing device 1.

[0144] The example in Fig. 14 shows the process when a user specifies data identifying the space in which the target camera 2 is installed on the client 4 and inputs a natural sentence such as "I would like to count the number of elderly people wearing hats and glasses who come to this location." In the third embodiment, the processing unit 30 determines that the natural sentence received as the user's request does not contain all of the necessary information (S331: NO), and identifies that the "period" is information that has not been input (S332). The processing unit 30 creates a message saying "Please input the period," and notifies the client 4 (S333).

[0145] In the third embodiment, too, the user simply inputs a request to the image processing system 100 in natural language, for example, on a web page, regarding what they want to do with the image captured by the target camera 2, and the image processing system 100 automatically selects a process according to the user's request. For data in a wide variety of fields, the user does not need to select a learning model from those prepared according to the specifications of various image processing devices 1, etc.

[0146] The user can be informed by a notification from the server 3 whether there is sufficient or insufficient information required to reflect his / her request in the image processing device 1. Rather than obtaining a low-accuracy result using an incomplete query, the user can obtain an output from the image processing device 1 that reflects his / her request by answering a query for the necessary information in advance.

[0147] FIG. 15 is a functional block diagram of an image processing system 100 according to a third embodiment. In addition to the functional blocks shown in FIG. 7 , the image processing system 100 according to the third embodiment includes a determination unit 61 that determines whether the query input via the input unit 51 contains all necessary information, and a notification unit 62 that outputs information to prompt the user to input any necessary information that has not yet been input via the input unit 51 when the determination by the determination unit 61 indicates that the query does not contain all necessary information. The determination unit 61 and notification unit 62 correspond to the determination means and notification means in the claims, respectively. The determination unit 61 is implemented by the processing unit 30 of the server 3 and performs the process of step S331 in FIG. 12. The notification unit 62 is implemented mainly by the processing unit 30 of the server 3 and the processing unit 40 and display unit 43 of the client 4 and mainly performs the process of step S333 in FIG. 12.

[0148] The embodiments disclosed above are illustrative in all respects and are not restrictive. The scope of the present invention is defined by the claims, and includes all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0149] 100 Image Processing System 1. Image processing device (edge ​​device) 10 Processing section 11 Storage section P1 Image Processing Program 2 Cameras 3 Server (Cloud side) 30 Processing section 51 Input unit (input means) 52 Configuration determination unit (configuration determination means) 53 Disassembly part (disassembly means) 54 Process selection unit (process selection means) 55 Process execution unit (process execution means) 61 Judgment unit (judgment means) 62 Notification unit (notification means) 300 databases P3 Server Program M3 language model 4 Client 43 Display section

Claims

1. An input means for a user to input a query corresponding to a desired image processing; decomposition means for decomposing a task included in a query input by the input means into a plurality of known tasks; a process selection means for selecting a process (hereinafter referred to as "each process") corresponding to each of the plurality of known tasks obtained by the decomposition means; a process execution means for executing the processes selected by the process selection means in a chain; The decomposition means has a search function that searches for documents for decomposing a task included in the query into the plurality of known tasks, and a language model for task decomposition, and by inputting the documents searched by the search function into the language model for task decomposition together with the query input by the input means, the image processing system decomposes the task included in the query into the plurality of known tasks.

2. An input means for a user to input a query corresponding to a desired image processing; decomposition means for decomposing a task included in a query input by the input means into a plurality of known tasks; a process selection means for selecting a process (hereinafter referred to as "each process") corresponding to each of the plurality of known tasks obtained by the decomposition means; a process execution means for executing the processes selected by the process selection means in a chain; The decomposition means decomposes the query input by the input means into at least specification information of a device to be used to execute the image processing.

3. An input means for a user to input a query corresponding to a desired image processing; decomposition means for decomposing a task included in a query input by the input means into a plurality of known tasks; a process selection means for selecting a process (hereinafter referred to as "each process") corresponding to each of the plurality of known tasks obtained by the decomposition means; a process execution means for executing the processes selected by the process selection means in a chain; The decomposition means decomposes a task included in a query input by the input means into at least a real-time processing task and an offline processing task.

4. The image processing system according to claim 3 , wherein the real-time processing task is executed by an edge device, and the offline processing task is executed on a cloud side.

5. Computer, an input means for a user to input a query corresponding to a desired image processing; decomposition means for decomposing a task included in a query input by the input means into a plurality of known tasks; a process selection means for selecting a process corresponding to each of the plurality of known tasks obtained by the decomposition means; functioning as a process execution means for connecting and executing the processes selected by the process selection means; The decomposition means has a search function that searches for documents for decomposing a task included in the query into the plurality of known tasks, and a language model for task decomposition, and by inputting the documents searched by the search function into the language model for task decomposition together with the query input by the input means, the image processing program decomposes the task included in the query into the plurality of known tasks.

6. A computer, an input means for a user to input a query corresponding to a desired image processing; decomposition means for decomposing a task included in a query input by the input means into a plurality of known tasks; a process selection means for selecting a process corresponding to each of the plurality of known tasks obtained by the decomposition means; functioning as a process execution means for connecting and executing the processes selected by the process selection means; The decomposition means is an image processing program that decomposes the query input by the input means into at least specification information of a device to be used to execute the image processing.

7. A computer, an input means for a user to input a query corresponding to a desired image processing; decomposition means for decomposing a task included in a query input by the input means into a plurality of known tasks; a process selection means for selecting a process corresponding to each of the plurality of known tasks obtained by the decomposition means; functioning as a process execution means for connecting and executing the processes selected by the process selection means; The decomposition means is an image processing program that decomposes a task included in a query input by the input means into at least a real-time processing task and an offline processing task.

Images