Image processing system and image processing program

Abstract

An image processing system and an image processing program are provided that can accurately identify a scene desired by a user. [Solution] The image processing system includes an information extraction means for extracting local information, which is information about a specific area or each object in one or multiple temporally consecutive frame images, and time information corresponding to the frame images, and a scene identification means for identifying a scene desired by a user based on the local information and time information extracted by the information extraction means.

Description

[Technical Field]

[0001] The present invention relates to an image processing system and an image processing program. [Background technology]

[0002] In a fourth embodiment of the specification of Japanese Patent Application No. 2025-001868, the present applicant proposes an image display system in which an edge device outputs text information about frame images that may show a user's desired viewing target, and a cloud uses the text information transmitted from the edge device to select frame images showing the user's desired viewing target according to detailed criteria. In this embodiment, the edge device inputs each frame image into a model such as a VLM (Vision-Language Model) to extract text information from the corresponding frame image. Summary of the Invention [Problem to be solved by the invention]

[0003] As described above, by inputting a frame image into a VLM, it is easy to obtain general text information about the entire frame image (hereinafter referred to as "global information") from the frame image, but it is difficult to obtain text information about the detailed content of the frame image. More specifically, if text information is extracted (obtained) from a frame image without entering any prompt when inputting a frame image into a VLM (or by entering a prompt such as "Please analyze and describe this image"), text information about the detailed objects in the frame image cannot be obtained (the detailed objects are ignored). Therefore, it is difficult to identify (search) the user's desired scene using this text information. On the other hand, if a prompt such as "Please analyze this image in detail" is entered when inputting a frame image into a VLM, text information about the detailed analysis results of the frame image can be obtained, but the perspective from which the frame image is analyzed in detail depends on the frame image, so the obtained text information is not versatile. Therefore, it is difficult to identify (search) the user's desired scene using this text information.

[0004] The present invention is intended to solve the above-mentioned problems, and aims to provide an image processing system and an image processing program that can obtain detailed information about the content of a frame image and use this detailed information to accurately identify the scene desired by the user. [Means for solving the problem]

[0005] In order to solve the above problems, an image processing system according to a first aspect of the present invention includes an information extraction means for extracting, from one or a plurality of temporally consecutive frame images, local information that is information about a predetermined area or each object in the frame image and time information corresponding to the frame image, and a scene identification means for identifying a scene desired by a user based on the local information and time information extracted by the information extraction means. The local information is generated based on information obtained from at least one of the areas obtained by dividing the frame image into a plurality of areas, and the local information is text-format information obtained by inputting an image obtained by dividing the frame image into a plurality of areas by grid lines into a VLM. .

[0006] In this image processing system, it is preferable that the local information and the time information are in text format, and that the scene identification means identifies the scene desired by the user based on the local information and the time information in text format.

[0008] In this image processing system, it is preferable that the scene identification means inputs the local information in text format into an LLM (Large Language Model) and identifies a frame image corresponding to the user-desired scene.

[0009] In order to solve the above-mentioned problems, an image processing system according to a second aspect of the present invention includes an information extraction means for extracting, from one or a plurality of temporally consecutive frame images, global information which is summary information about the entire frame image, local information which is information about a predetermined area or each object in the frame image, and time information corresponding to the frame image, and a scene identification means for identifying a scene desired by a user based on the global information, local information, and time information extracted by the information extraction means. The local information is generated based on information obtained from at least one of the areas obtained by dividing the frame image into a plurality of areas, and the local information is text-format information obtained by inputting an image obtained by dividing the frame image into a plurality of areas by grid lines into a VLM. .

[0010] In this image processing system, it is desirable that the global information, the local information, and the time information are in text format, and that the scene identification means identifies the scene desired by the user based on the global information, the local information, and the time information in text format.

[0011] In this image processing system, Global information is It is desirable that the information be in text format obtained by inputting the frame image into VLM.

[0012] In this image processing system, it is desirable that the scene identification means inputs the global information, the local information, and the time information in text format to an LLM, and identifies a frame image corresponding to the user-desired scene.

[0013] In this image processing system, it is desirable that the scene identification means add together the global information and the local information in text format and input the result to the LLM.

[0017] In this image processing system, it is desirable that the local information be text-format information obtained by inputting into a VLM an image on the frame image that depicts the grid lines and the bounding boxes of each object detected by object detection previously performed on the frame image.

[0018] In this image processing system, the local information may be extracted based on a positional relationship between the grid lines and a bounding box of each of the objects.

[0019] In this image processing system, it is desirable that the system further comprises a scene information input means for inputting scene information, which is information about the scene desired by the user, and that the scene identification means identify a scene corresponding to the scene information input by the scene information input means.

[0020] In this image processing system, it is desirable to further include an image display control means for extracting a frame image corresponding to the user-desired scene identified by the scene identification means from a moving image including this frame image and controlling the display means to display this frame image.

[0021] In this image processing system, the image processing system may include an edge device-side system and a cloud-side system, and the edge device-side system may be equipped with the information extraction means, and the cloud-side system may be equipped with the scene identification means.

[0022] An image processing program according to a third aspect of the present invention causes a computer to function as an information extraction means for extracting, from one or a plurality of temporally consecutive frame images, local information that is information about a predetermined area or each object in the frame image and time information corresponding to the frame image, and as a scene identification means for identifying a scene desired by a user based on the local information and time information extracted by the information extraction means. The local information is generated based on information obtained from at least one of the areas obtained by dividing the frame image into a plurality of areas, and the local information is text-format information obtained by inputting an image obtained by dividing the frame image into a plurality of areas by grid lines into a VLM. . [Effects of the Invention]

[0023] According to the image processing system of the first aspect of the present invention and the image processing program of the third aspect, local information, which is information about a predetermined area or each object in a frame image, and time information corresponding to the frame image are extracted from one or a plurality of temporally consecutive frame images. This makes it possible to obtain detailed information about the content of the frame image (information about a predetermined area or each object in the frame image, rather than general information about the entire frame image), and this detailed information can be used to accurately identify a scene desired by the user.

[0024] Furthermore, according to the image processing system of the second aspect of the present invention, global information, which is general information about the entire frame image, local information, which is information about a predetermined area or each object in the frame image, and time information corresponding to the frame image are extracted from one or a plurality of temporally consecutive frame images. This makes it possible to obtain detailed information about the frame image (not only general information about the entire frame image, but also information about a predetermined area or each object in the frame image), and this detailed information can be used to accurately identify the scene desired by the user. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a schematic diagram of an image processing system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing a configuration of an edge device. [Figure 3] FIG. 2 is a block diagram showing the configuration of a cloud server. [Figure 4] FIG. 2 is a block diagram showing the configuration of a client; [Figure 5] FIG. 1 is a functional block diagram of an image processing system. [Figure 6] 10 is a flowchart illustrating an example of a processing procedure for scene identification in the image processing system. [Figure 7] 10 is a flowchart illustrating an example of a processing procedure for scene identification in the image processing system. [Figure 8] FIG. 2 is an explanatory diagram of global information and local information. [Figure 9] FIG. 2 is an explanatory diagram of global information and local information. [Figure 10] 10 is a flowchart showing an example of a display control procedure in the image processing system 100. [Figure 11] FIG. 10 is an explanatory diagram showing an example of an image displayed on a client. [Figure 12] FIG. 10 is an explanatory diagram showing an example of an image displayed on a client. [Figure 13] FIG. 10 is a functional block diagram of an image processing system according to a modified example. [Figure 14] FIG. 10 is a functional block diagram of an image processing system according to a second embodiment. [Figure 15] 10 is a flowchart illustrating an example of a processing procedure performed by an information extraction unit according to the second embodiment. [Figure 16] FIG. 10 is an explanatory diagram of local information in the second embodiment. [Figure 17] FIG. 10 is a functional block diagram of an image processing system according to a third embodiment. [Figure 18] 11 is a flowchart showing an example of a processing procedure of a secondary scene identification unit in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0026] The present disclosure will be specifically described with reference to the drawings showing embodiments thereof.

[0027] [First embodiment] 1 is a schematic diagram of an image processing system 100 according to a first embodiment. The image processing system 100 is installed in a specific space inside or outside a store, which is designated as a target space. The image processing system 100 includes a camera 2 installed in the target space, an edge device 1 connected to the camera 2, a cloud server 3 capable of communicating with the edge device 1, and a client 4 capable of communicating with the cloud server 3. A plurality of edge devices 1 and cameras 2 may be installed in the same space.

[0028] Camera 2 uses an image element that is responsive to visible light and / or near-infrared light and outputs image information. Camera 2 outputs image information in time series at a rate of several fps to several tens of fps. Camera 2 may be installed so as to look down from above the target space, such as the ceiling or shelves of a store, which is the target space. Camera 2 may be a type that is attached to the ceiling and has a field of view that covers the entire 360-degree space. Camera 2 may also be installed on a pole or the like so as to look down in an outdoor area, which is the target space.

[0029] The camera 2 and the edge device 1 can be connected to each other via a local network LN, which may be wireless or wired. The local network LN is installed in a facility that includes the target space. The local network LN may be a wired LAN or a wireless network such as WiFi or Bluetooth (registered trademark). The camera 2 sequentially transmits image information to the edge device 1 via the local network LN.

[0030] The edge device 1 can be connected to the cloud server 3 via a network N. The network N is a wired or wireless communication network that may include a public communication network, a dedicated line, or a carrier network. The client 4 can be connected to the cloud server 3 and the edge device 1 via the network N.

[0031] The cloud server 3 stores in a database 310 a group of learning models used in processes selectively executed by the edge device 1 and the cloud server 3. The database 310 may also include a group of learning models provided by an external service outside the system. The cloud server 3 reads out from the database 310 a learning model corresponding to an object to be detected by the processes executed by the edge device 1 and the cloud server 3, and deploys it to the edge device 1 and the cloud server 3.

[0032] The image processing system 100 of the first embodiment extracts feature amounts from an image captured by a camera 2 installed in a target space, detects an object such as a person or object to be detected from the image based on the feature amounts, recognizes attributes of the detected object, and outputs the recognition result. The image processing system 100 is a system that can automatically display on a client 4 a scene desired by the user that includes an object desired by the user.

[0033] The image processing system 100 of the first embodiment is a system that allows a user to select which attribute of a person or object to use as a detected object for determining whether or not the scene is the user's desired scene. The image processing system 100 of the first embodiment reduces the amount of communication between the edge device 1 and the cloud server 3, and further reduces the processing load on the edge device 1 by sharing processing between the edge device 1 and the cloud server 3.

[0034] The configurations of the edge device 1, cloud server 3, and client 4 for realizing such an image processing system 100, as well as the processing executed by each of them, will be described in detail below.

[0035] 2 is a block diagram showing the configuration of the edge device 1. The edge device 1 is a box-shaped device that can be installed in a target space together with a camera 2. The edge device 1 includes a processing unit 10, a storage unit 11, a first communication unit 12, and a second communication unit 13.

[0036] The processing unit 10 includes one or more processors such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), or a neutral processing unit (NPU). The processing unit 10 includes a memory that is a temporary storage medium such as a static random access memory (SRAM) or a dynamic random access memory (DRAM). The processing unit 10 includes a timer and can acquire time information at each point in time from data from the timer. The processing unit 10 may be configured as a single piece of hardware (SoC: System On a Chip) that integrates a processor, a memory, a storage unit 11, a first communication unit 12, and a second communication unit 13. The specifications of the processing unit 10 may be the same or different between edge devices 1.

[0037] The processing unit 10 reads the first image processing program P1 stored in the storage unit 11 into the memory and executes it, thereby causing the processor to execute various processes described below and function as the edge device 1 of the present disclosure.

[0038] The storage unit 11 is a relatively large-capacity non-transitory storage medium such as a hard disk, a flash memory, etc. A part of the storage unit 11 may be removable.

[0039] The storage unit 11 stores a program (program product) required for the processing unit 10 to execute processing, the results of processing by the processing unit 10, and reference setting information. The setting information includes identification data of the edge device 1, identification data of the connected camera 2, etc. The setting information includes a criterion setting for distinguishing between global information of an image, which will be described later, and local information. The criterion setting, for example, specifies an area within the image that corresponds to the local information. The area may be a specific area within the image set by the client 4, or may be a frame image divided into multiple areas. The criterion setting may include whether or not to use a detected bounding box.

[0040] The setting information stored in the storage unit 11 includes information referenced to identify a scene desired by the user. The information for identifying a scene includes, for example, text indicating an object desired by the user, such as "human." The text may also be "dog," "dangerous animal," or text indicating attributes, such as "adult," "wearing a hat / cap," or "wearing glasses." The text indicating the object desired by the user is set based on scene information input by the user using the client 4 via the cloud server 3 (see "scene information input unit 305" in FIG. 5).

[0041] The program product stored in the storage unit 11 includes an OS (Operation System) program, a first image processing program P1, and a learning model group M1. The learning model group M1 will be described in detail below. The first image processing program P1 or the learning model group M1 stored in the storage unit 11 may be the first image processing program P1 and the learning model group M9 stored in a computer-readable non-transitory storage medium 9 that the processing unit 10 reads and stores in the storage unit 11, or may be pre-stored at the time of shipment. The first image processing program P1 or the learning model group M1 stored in the storage unit 11 may be downloaded by the processing unit 10 from the database 310 of the cloud server 3 or from another download server via the second communication unit 13.

[0042] The storage unit 11 stores image information obtainable from the connected camera 2 in chronological order in association with time information. The image information may be frame images or moving images. The storage unit 11 stores only frame images in which a user's desired object (viewing target) is detected as a result of processing by the processing unit 10, which will be described later, or image information that is a moving image including previous and next images. The storage unit 11 may continue to store data by overwriting it according to the size of its capacity in accordance with a FIFO (First In First Out) method. The storage unit 11 may also store text information describing the detected object, text information indicating the attributes of the detected object, etc. in association with time information.

[0043] The first communication unit 12 is a communication device that realizes communication via the local network LN. The first communication unit 12 may be a LAN network card or a CAN communication device. The first communication unit 12 may be a communication device compatible with wireless networks such as WiFi or Bluetooth (registered trademark). The first communication unit 12 may include multiple communication devices compatible with various types of cameras 2. The first communication unit 12 may include an interface such as a USB (Universal Serial Bus) that is connected to the camera 2. The first communication unit 12 can be replaced by an interface that is connected to the camera 2 via a coaxial cable or another serial bus. The processing unit 10 acquires image information from the camera 2 via the local network LN using the first communication unit 12. The first communication unit 12 may be the same device as the second communication unit 13, which will be described later.

[0044] The second communication unit 13 is a communication device that realizes communication via an external network N. The second communication unit 13 may be a network card for a wired LAN, or may be a communication device that realizes carrier communication via a carrier network. The second communication unit 13 may be a communication device that supports a wireless network such as WiFi or Bluetooth (registered trademark). The second communication unit 13 may support secure communication with the cloud server 3 using SSL (Secure Socket Layer) or the like. The second communication unit 13 may be an interface that realizes a communication connection with the cloud server 3 via a dedicated line.

[0045] 3 is a block diagram showing the configuration of the cloud server 3. The cloud server 3 may be configured as a single server computer, or may be configured to distribute processing among multiple server computers. The cloud server 3 includes a processing unit 30, a storage unit 31, and a communication unit 32.

[0046] The processing unit 30 includes one or more processors such as a CPU, an MPU, a GPU, an NPU, etc. The processing unit 30 includes a memory that is a temporary storage medium such as an SRAM or a DRAM.

[0047] The storage unit 31 is a relatively large-capacity non-temporary storage medium such as a hard disk, flash memory, etc. The storage unit 31 stores programs (program products) and setting information required for the processing unit 30 to execute the processes described below.

[0048] The program products stored in the storage unit 31 include a server program P31. The server program P31 includes a module that functions as a data server that reads out a group of models stored in the database 310 and transmits them to the edge device 1. The server program P31 includes a module that functions as a web server, and can output the results of processing executed by the cloud server 3 to the client 4 via a web page.

[0049] The program product stored in the storage unit 31 includes a second image processing program P32 that runs on the cloud server 3, and a learning model group M3. The second image processing program P32 is a program that causes a processor to execute processing linked to processing based on the second image processing program P32 in the edge device 1. The learning model group M3 is selected from and stored in the database 310. The second image processing program P32 and the learning model group M3 will be described in detail below.

[0050] The server program P31, the second image processing program P32, or the learning model group M3 stored in the storage unit 31 may be the server program P31 and the second image processing program P32 stored in a computer-readable non-transitory storage medium 8 that the processing unit 30 reads and stores in the storage unit 31. The learning model group M3 may also be the learning model group M8 stored in the non-transitory storage medium 8 that the processing unit 30 reads and stores in the storage unit 31. The server program P31 and the second image processing program P32 stored in the storage unit 31 may be downloaded by the processing unit 30 from another download server via the communication unit 32.

[0051] The setting information stored in the storage unit 31 includes data for identifying the edge device 1 and a correspondence relationship between the data and the name for identifying the space in which the edge device 1 is installed. Since the cloud server 3 executes processing simultaneously and in parallel for multiple target spaces, the storage unit 31 may store, as a whitelist, identification data of the edge device 1 or target space for which the user is permitted to change settings, etc., in association with the user's account data. This allows the cloud server 3 to identify the target edge device 1 when the user specifies the name of the target space and specifies which object is to be detected in an image captured by the camera 2 installed in the target space.

[0052] The setting information stored in the storage unit 31 includes information to be referenced for processing to identify a user-desired scene, which will be described later. The information for identifying a scene on the cloud server 3 side is information for identifying a scene more precisely than that on the edge device 1 side. For example, settings such as "a scene with multiple people walking together," "a scene with a person staying in the same place for more than five minutes," and "a scene with a dangerous animal approaching a person" are set.

[0053] The storage unit 31 stores image information including a scene captured by the camera 2 and identified by the scene identification process. The image information may be a frame image or a moving image including previous and next images.

[0054] The database 310 may be built in the storage unit 31 or in an external storage device. As described above, part of the database 310 may include a model providing service used on the Web, which is connected to the database 310 via the network N. The database 310 holds a group of learning models used in processes executed by the edge device 1 and the cloud server 3. The group of learning models includes, for example, detection learning models such as a person detector, a head detector, a face detector, an animal detector, and a specific device detector. The group of learning models also includes recognition learning models that recognize attributes, such as an age recognizer, an eyeglasses-wearing recognizer, a hat-wearing recognizer, a face orientation recognizer, or a posture recognizer. The group of learning models includes large and small language models (LLMs, SLMs), vision-language models (VLMs), etc.

[0055] The communication unit 32 is a communication device that realizes a communication connection with the client 4 and the edge device 1 via the network N.

[0056] 4 is a block diagram showing the configuration of the client 4. The client 4 is a personal computer, a smartphone, or a tablet terminal. The client 4 may be used by the administrator of the space where the camera 2 is installed, or by the operator of the management company of the cloud server 3.

[0057] The client 4 includes a processing unit 40, a storage unit 41, a communication unit 42, a display unit 43, and an operation unit 44. The processing unit 40 includes one or more processors such as a CPU, an MPU, a GPU, or an NPU. The processing unit 40 includes a memory that is a temporary storage medium such as an SRAM or a DRAM.

[0058] The storage unit 41 is a memory of a non-temporary storage medium such as a hard disk or flash memory. The storage unit 41 stores the functions of the data server provided by the cloud server 3 and a client program P4 for the web server. The client program P4 is, for example, a web browser program. The client program P4 is a program that causes the processing unit 40 to execute a process of displaying various data, including images, provided by the cloud server 3 on a screen.

[0059] The communication unit 42 is a communication device that realizes a communication connection with the cloud server 3 via the network N. The communication unit 42 may be a communication device that realizes a communication connection with the cloud server 3 via a dedicated line. The communication unit 42 may be a communication device that realizes a direct communication connection with the second communication unit 13 of the edge device 1 via a wireless communication medium, a USB cable, or the like.

[0060] The display unit 43 uses a display such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 43 displays a web page including text and images through processing based on the client program P4 of the processing unit 40. The display unit 43 may use a display with a built-in touch panel.

[0061] The operation unit 44 is a user interface such as a keyboard or a pointing device that accepts operations from the user. The operation unit 44 may be a touch panel built into the display of the display unit 43, or may be physical buttons. The operation unit 44 may be a voice input unit that accepts operations by voice using a voice recognition function. The operation unit 44 can notify the processing unit 40 of operation information by the user.

[0062] In the image processing system 100 configured as described above, the edge device 1 and the cloud server 3 share the processing of images captured by the camera 2 in each space, and display image information of the user's desired scene on the client 4 so that the user can view it. In the image processing system 100 of the first embodiment, the edge device 1 executes processing to identify scenes that may contain the user's desired object. In the image processing system 100, the cloud server 3 secondarily executes processing to identify scenes that definitely contain the object from frame images corresponding to the scenes primarily identified on the edge device 1 side. In the present disclosure, as a result of the primary scene identification on the edge device 1 side, text information describing what is shown in the frame images is extracted, and the text information is sent to the cloud server 3 instead of image information of the identified scene, thereby taking personal information into consideration and reducing communication volume.

[0063] Fig. 5 is a functional block diagram of the image processing system 100. In the image processing system 100, the functions shown in Fig. 5 are shared and performed based on a first image processing program P1 in the edge device 1 and a second image processing program P32 in the cloud server 3. In the image processing system 100, the processing unit 10 in the edge device 1 functions as an information extraction unit 101, a primary scene identification unit 102, and an edge device-side transmission unit 103, and the processing unit 30 in the cloud server 3 functions as a cloud-side storage unit 301, a secondary scene identification unit 302, a selection operation unit 303, an image display control unit 304, and a scene information input unit 305.

[0064] In the first embodiment, the information extraction unit 101 receives frame images acquired by the camera 2 via the first communication unit 12. The information extraction unit 101 extracts visually obtainable information from the acquired frame images in text format. The information extraction unit 101 uses a language model M11 to extract text-format information representing objects, such as objects or people, captured in the frame images and their attributes. In other words, the language model M11 includes the functions of an object detection model and an object recognition model that recognizes attributes. The information extraction unit 101 may use the language model M11 to obtain information inferred from the attributes, positions, etc. of objects captured in the frame images, described in text format. As the language model M11, a multimodal-compatible model (e.g., a Vision-Language Model (VLM)) that can input images is selected from the database 310 and stored in the storage unit 11. Furthermore, a verbalization process using the language model M11 is deployed in the edge device 1.

[0065] The information extraction unit 101 instructs a language model M11 such as a VLM to write sentences or words that describe people or objects that appear in the entire frame image, and obtains text-format information output from the language model M11 as global information that is general information about the entire frame image.The information extraction unit 101 instructs a language model M11 such as a VLM to write sentences or words that describe people or objects that appear in a specific area of ​​the frame image, and obtains the text-format information output from the language model M11 as local information that is information about a specific area within the frame image.

[0066] The information extraction unit 101 extracts, in text format, summary information such as "three people are walking," "multiple trees with autumn leaves are visible," and "a specific animal is visible in the shade of a tree" as global information for the frame image. The information extraction unit 101 extracts, in text format, local information for a specific area or object within the frame image, information such as "one person (aged 50 or older but younger than 70) is walking in the specific area, and a specific animal is in the shade of a tree beside the path they are walking on," and "two people (01 is wearing a hat, and 02 is wearing glasses) are walking."

[0067] The information extraction unit 101 extracts global information and local information for the frame image as well as temporal information for the target frame image. Note that this example mainly shows an example in which the information extraction unit 101 extracts global information and local information for this frame image and temporal information corresponding to this frame image from this frame image, but is not limited to this, and the information extraction unit 101 may also extract global information and local information and temporal information corresponding to this frame image from a plurality of temporally consecutive frame images (i.e., a moving image).

[0068] The primary scene identification unit 102 determines whether a target frame image is likely to show a user-desired object based on the global information or local information extracted by the information extraction unit 101. Like the secondary scene identification unit 302 described below, the primary scene identification unit 102 may include an LLM. That is, the primary scene identification unit 102 may input at least one of the global information and the local information to the LLM along with text instructing the LLM to analyze whether the target frame image is likely to show the user-desired object, and determine whether the corresponding frame image is likely to show the user-desired object based on the result of the LLM output. If the target frame image is likely to show the object to be viewed, the primary scene identification unit 102 identifies the target frame image as an image that is likely to be part of the user-desired scene. The primary scene identification unit 102 stores frame images corresponding to the identified scene in the storage unit 11 in association with the time information extracted by the information extraction unit 101. The primary scene identification unit 102 saves a predetermined time of video including frame images corresponding to the identified scene in the storage unit 11. When the primary scene identification unit 102 selects frame images that are consecutive in time series (temporally), it may save them as a continuous video. The primary scene identification unit 102 discards frame images that it did not identify. When the processing unit 10 determines, as the primary scene identification unit 102, from the output from the information extraction unit 101 that the object desired by the user is not shown, it terminates processing for the frame image in question, does not perform any further processing for the frame in question, and performs processing for the next frame image.

[0069] The primary scene identification unit 102 stores the global information and local information extracted from the target frame image by the information extraction unit 101 in the storage unit 11 together with the frame image and time information of the identified scene.

[0070] In the first embodiment, the edge device-side transmitting unit 103 transmits global information and local information in text format corresponding to the frame image identified by the primary scene identifying unit 102, and time information corresponding to the frame image, to the cloud server 3 via the second communication unit 13. The edge device-side transmitting unit 103 transmits the global information and local information in text format and the corresponding time information in association with the identification data of the edge device 1. The time information is text information indicating the time corresponding to the frame image, the elapsed time or count from a specific time, etc.

[0071] The cloud-side storage unit 301 receives global information describing an outline of the identified frame image, local information describing information about a specific area within the frame image, corresponding time information, and identification data of the edge device 1, which have been transmitted from the edge device 1 by the edge device-side transmission unit 103. The cloud-side storage unit 301 stores the received information in the memory unit 31. No image information is transmitted from the edge device 1 to the cloud server 3.

[0072] The secondary scene identification unit 302 uses the global information, local information, and time information in text format stored in the storage unit 31 by the cloud-side storage unit 301 to identify a scene in which an object desired by the user appears according to detailed criteria, and stores the time information of the identified scene in the storage unit 31. The detailed criteria include criteria for attributes used to determine whether an object is the user's desired object for a frame image in which the desired object is detected by the edge-side primary scene identification unit 102. For example, if the primary scene identification unit 102 identifies a scene in which a person appears, the detailed criteria are criteria for the person's attributes. For example, if the primary scene identification unit 102 identifies a scene in which a person appears, the secondary scene identification unit 302 uses criteria that allow it to identify a scene in which a person wearing specific clothing or a person with a specific facial orientation appears.

[0073] Other examples may be used as long as the identification criteria in the primary scene identification unit 102 and the identification criteria in the secondary scene identification unit 302 are more detailed than the former. The secondary scene identification unit 302 performs identification so as to increase the probability that what is identified by the primary scene identification unit 102 of the edge device 1 is a scene showing an object desired by the user. The detailed criteria in the secondary scene identification unit 302 are, for example, criteria that allow the secondary scene identification unit 302 to determine that a person with desired attributes is shown in the scene when the primary scene identification unit 102 identifies that the target frame image shows an object of approximately the same size as a person and that the scene shows a person.

[0074] The secondary scene identification unit 302 may input either the global information or the local information corresponding to the frame image identified by the primary scene identification unit 102 to the LLM along with an instruction to analyze whether or not the object desired by the user is displayed, and may identify whether or not the corresponding frame image corresponds to the scene desired by the user based on the result of whether or not the object is displayed output from the LLM.

[0075] The secondary scene identification unit 302 may delete all of the time information, global information, and local information stored by the cloud-side storage unit 301 except for the time information, global information, and local information corresponding to the scene identified by the secondary scene identification unit 302.

[0076] The selection operation unit 303 controls the operation unit 44 of the client 4 to accept a selection operation of a scene to be displayed on the display unit 43 of the client 4 connected to the cloud server 3, from among the scenes narrowed down by the processing of the secondary scene identification unit 302. The processing unit 30 accepts the selection operation on a web page displayed on the client 4, based on a module for the web server included in the server program P31.

[0077] The image display control unit 304 controls the display unit 43 of the client 4 connected to the cloud server 3 to display the scenes narrowed down by the processing of the secondary scene identification unit 302. In other words, the image display control unit 304 controls the display unit 43 of the client 4 to extract a frame image corresponding to the user-desired scene identified by the secondary scene identification unit 302 from the video stored in the storage unit 11 of the edge device 1 and display it on the display unit 43 of the client 4. The processing unit 30 may redirect a frame image corresponding to time information specified by the client 4 to the client 4, with the edge device 1 as a host. When images or videos are sent directly from the edge device 1, it is possible to avoid storing data in the cloud server 3.

[0078] The scene information input unit 305 has a function of accepting input of scene information for setting a detection target object, which serves as a basis for scene identification by the primary scene identification unit 102 and the secondary scene identification unit 302. Before the processing described below is started, the processing unit 30 accepts input of scene information, which describes the type of scene a user wants to view in the target space, via the client 4, by the scene information input unit 305, in association with identification data of the target space. The scene information may be, for example, keywords for the detection target object, such as "human" or "dog," as described above, or attributes, such as "adult," "wearing a hat / cap," or "wearing glasses." The processing unit 30 distributes, using the function of the scene information input unit 305, setting information to be referenced by the primary scene identification unit 102 of the edge device 1 and setting information to be referenced by the secondary scene identification unit 302 of the cloud server 3, according to the accepted scene information.

[0079] The procedure for each function of the image processing system 100 shown in Fig. 5 will be described with reference to a flowchart. Fig. 6 and Fig. 7 are flowcharts showing an example of a processing procedure for scene identification in the image processing system 100. Before the image processing system 100 starts operating, the scene information input unit 305 accepts information input for identifying a desired scene for each target space, and when operation starts, the following processing is executed.

[0080] The processing unit 10 of the edge device 1 acquires a frame image from the camera 2 (step S111).

[0081] The processing unit 10, using the function of the information extraction unit 101, inputs the acquired frame image to the language model M11 together with instructions for acquiring global information for the acquired frame image (step S112). In step S112, the processing unit 10 creates a sentence such as "Analyze what is shown in this image" as an "instruction" for acquiring global information, and inputs the sentence together with the frame image to the language model M11. The processing unit 10 may create the sentence of the "instruction" to be given to the language model M11, or may read the sentence of the "instruction" from a fixed phrase stored in the storage unit 11.

[0082] The processing unit 10 acquires, as global information, text information that is a sentence or word output from the language model M11 in response to the input at step S112 (step S113).

[0083] The processing unit 10 extracts a specific area from the frame image acquired in step S111 along with instructions for acquiring local information, and inputs these to the language model M11 (step S114). In step S114, the processing unit 10 creates image data from which the specific area set in the setting data has been extracted, and a sentence such as "Analyze what is shown in the specified part of this image," as an "instruction" for acquiring local information, and inputs these to the language model M11 together with the frame image. In step S114, the sentence of the "instruction" provided to the language model M11 may also be one that has been read. In step S114, the processing unit 10 may specify a range using coordinates in the "instruction" rather than extracting an area from the frame image.

[0084] The processing unit 10 acquires, as local information, text information that is a sentence or word output from the language model M11 in response to the input at step S114 (step S115).

[0085] The processing unit 10 determines whether or not all of the local information for the frame image acquired in step S111 has been acquired (step S116). If it is determined that all of the local information has not been acquired (S116: NO), the processing unit 10 returns to step S114 to execute processing for other areas in the frame image.

[0086] If the processing unit 10 determines that all the local information has been acquired (S116: YES), the processing unit 10 ends the processing as the information extraction unit 101.

[0087] Using the function of the primary scene identification unit 102, the processing unit 10 determines, based on the extracted global information and local information, whether or not the frame image acquired in step S111 is likely to include the object desired by the user (step S121).

[0088] In step S121, the processing unit 10 determines whether or not the text in the global information and the local information matches a part of the text set in the setting information.

[0089] If the processing unit 10 determines that there is a possibility that the object desired by the user is captured in the frame image (S121: YES), it stores the global information, the local information, and the time information indicating the time when the frame image was captured in the storage unit 31 (step S122). The primary scene identification unit 102 of the processing unit 10 ends the processing on the frame image acquired in step S111.

[0090] In step S121, if it is determined that there is no possibility that the object desired by the user is captured in the frame image (S121: NO), the processing unit 10 ends the processing for the acquired frame image. In this case (S121: NO), the acquired frame image does not need to be saved in the storage unit 11.

[0091] When a frame image corresponding to a scene identified by the primary scene identification unit 102 is stored in the storage unit 11, the processing unit 10 of the edge device 1 transmits the global information and local information corresponding to the frame image and the corresponding time information to the cloud server 3 by the function of the edge device-side transmission unit 103 (step S131). In step S131, the processing unit 10 also transmits identification data of the edge device 1. This allows the cloud server 3 corresponding to multiple target spaces to identify in which target space the frame image was captured.

[0092] On the cloud server 3 side, the processing unit 30 receives the identification data of the edge device 1 together with the global information, local information, and time information transmitted from the edge device 1 through the function of the cloud-side storage unit 301 (step S311). The processing unit 30 associates the received global information, local information, and time information with the identification data of the edge device 1 received together with this information, and stores (stores) them in the memory unit 31 (step S312), thereby completing the receiving and storing process.

[0093] Each time global information and local information are saved in the storage unit 31, the processing unit 30, using the function of the secondary scene identification unit 302, determines based on the global information and local information whether the corresponding frame image shows a scene desired by the user, using detailed criteria (step S321). In step S321, the processing unit 30 may make the determination using not only the global information and local information of the target, but also global information and local information corresponding to a chronologically previous or subsequent frame image.

[0094] When the processing unit 30 determines based on the detailed criteria that the object desired by the user is captured in the frame image corresponding to the target global information and local information (S321: YES), the processing unit 30 stores the corresponding time information in the storage unit 31 (step S322). When the processing unit 30 stores time information that is chronologically (temporally) continuous in step S322, the processing unit 30 may also store information on the period. The processing unit 30 stores the global information or local information that was the basis for determining that the object is captured in the storage unit 31 in association with the time information (step S323). However, the processing of step S323 means storing text for search, and is therefore unnecessary when the text-based search described in FIG. 10 is not accepted.

[0095] If the processing unit 30 determines that the frame image corresponding to the target global information and local information does not show the object desired by the user (S321: NO), it deletes the global information and local information stored in the memory unit 31 by the cloud-side storage unit 301 from the memory unit 31 (step S324).

[0096] 6 and 7, the frame image stored on the edge device 1 side corresponding to the time information stored in the storage unit 31 becomes the frame image included in the scene identified by the secondary scene identification unit 302. The cloud server 3 stores the time information corresponding to the identified scene, as well as the global information and the local information.

[0097] Figures 8 and 9 are explanatory diagrams of global information and local information. Figures 8 and 9 are schematic diagrams showing an example of a frame image obtained from a camera 2 installed in an outdoor passageway of a facility. Dashed lines in the frame image indicate grid lines that divide the image into multiple areas.

[0098] FIG. 8 shows global information that provides an overview of the entire frame image. As shown in FIG. 8, the global information is output from the language model M11 when the entire frame image and an instruction to "analyze what is shown in this image" are input to the language model M11. As an analysis result of the entire image, the global information is a text representation of the analysis result, "Many people are walking along the tree-lined path," as shown in the example of FIG. 8.

[0099] FIG. 9 shows an example of local information in which specific areas within a frame image are designated by dividing each area into four vertically and horizontally by grid lines. In the example of FIG. 9, local information is output from the language model M11 in response to an instruction to "analyze what is shown in the specified part of this image" after specifying a specific area of ​​the frame image. In the example of FIG. 9, local information obtained by specifying the upper left portion indicates clearer information about depth and positional relationships, such as "A person is walking in the shade of a tree." Local information obtained by specifying the upper right portion indicates information that "Many trees with autumn leaves are shown," indicating that the upper right of the entire frame image is an area that shows trees, not people. Local information obtained by specifying the lower left portion indicates clear numerical information, such as "Three people are shown. A snake is shown in the tree in front of two trees. The snake is close to one of the people." Furthermore, information about small dangerous animals not mentioned in the global information and their positional relationships with the detected person are detailed. Local information obtained by specifying the lower right portion indicates detailed numerical information, such as "One person is walking. There are three trees by the roadside."

[0100] As shown in Figure 9, by using the local information as the analysis result for each area within the frame image, detailed information that is not included in the result of analyzing the entire frame image (global information), such as clarifying the positional relationships of objects, can be obtained in text format. From this information, the secondary scene identification unit 302 on the cloud server 3 side can improve the accuracy of identifying a scene in which a user-desired object appears, without leaking any information, in the processing performed later. For example, if a dangerous animal is the object to be detected, even if it may not be detectable from the global information, the local information can identify the area within the frame image and detect whether or not the dangerous animal is present.

[0101] As shown in Figure 9, the primary scene identification unit 102 may not specify a specific area (top left, top right, bottom left, bottom right) and have the language model M11 analyze it, but may instead provide the language model M11 with data for each of the images obtained by dividing the frame image into four parts, along with instructions for analyzing these images.

[0102] As shown in Figures 8 and 9, the edge device 1 first analyzes the image using LLM and VLM, and then determines whether the user's desired object is captured in the image using text information. This reduces the amount of data to be processed and the amount of data to be transmitted to the cloud server 3. However, when converted to text, the global information analyzed for the entire image may objectively and accurately describe the overall image, but it may not necessarily focus on notable people or objects, etc., according to the purpose of installing the camera 2. If the camera is installed for security purposes, such as detecting danger to people in and around the facility, the manager of the target space can select and discard information while viewing the entire image, understanding the purpose. However, it is difficult for a manager to constantly monitor the image for long periods of time to detect danger, which can lead to human attention deficits or misjudgments. Therefore, it is necessary to automatically and accurately identify scenes that require attention. As shown in Figures 8 and 9, the edge device 1 of the image processing system 100 can use text information to reduce the amount of data used for specific processing or the amount of data sent to the cloud server 3.In addition, by dividing the information into global and local information and using the language model M11 to convert the events shown in each frame into text, it is possible to more accurately identify noteworthy scenes.

[0103] Furthermore, as described above, the processing of scene identification is distributed between the edge device 1 and the cloud server 3, thereby reducing the processing load on the edge device 1, which has relatively few computing resources. After primary scene identification is performed by the edge device 1, text information is transmitted, which reduces security risks and consideration for personal information compared to transmitting captured frame images to the cloud server 3, while also reducing communication volume.

[0104] Based on the time information stored in the storage unit 31 of the cloud server 3 by processing by the secondary scene identification unit 302, a scene showing an object desired by the user can be displayed on the display unit 43 of the client 4. Fig. 10 is a flowchart showing an example of a display control procedure in the image processing system 100. When a user (for example, an operator who centrally manages a chain store) uses the client 4 to access the cloud server 3, the processing unit 30 of the cloud server 3 functions as a selection operation unit 303 and an image display control unit 304 and starts the following processing.

[0105] The processing unit 30 of the cloud server 3 receives data specifying the target space from the client 4 as the selection operation unit 303 (step S331). In step S331, the processing unit 30 accepts the user's account data, the identification data or name of the target space, etc. In step S331, the processing unit 30 may accept a selection from a list of identification data of the edge device 1 that is permitted to access the account data used when the client 4 accessed the cloud server 3, and the corresponding identification data or name of the target space.

[0106] The processing unit 30 identifies the identification data of the edge device 1 corresponding to the target space identified by the received data (step S332), and reads out the global information, local information, and time information associated with the identification data of the identified edge device 1 from the storage unit 31 (step S333). In step S333, the processing unit 30 may read out consecutive pieces of time information together.

[0107] The processing unit 30, as the selection operation unit 303, receives a search word corresponding to an object (viewing target) desired by the user from the client 4 (step S334). The processing unit 30, as the image display control unit 304, extracts global information or local information that matches the text corresponding to the search word from the global information and local information read in step S333 (step S335). A match between the global information or local information and the text of the search word may include a case where the global information or local information contains the search word, a case where the global information or local information is a synonym for the search word, etc.

[0108] The processing unit 30 specifies time information corresponding to the extracted global information or local information, and transmits an acquisition request for thumbnail images corresponding to the frame image corresponding to the specified time information to the edge device 1 identified by the associated identification data (step S336). The processing unit 30 displays a list of thumbnail images transmitted in response to the acquisition request on the display unit 43 of the client 4 (step S337), and ends the process.

[0109] Of the processing steps shown in FIG. 10, the processing of steps S334-S335 relating to the search may be omitted, and a list of thumbnail images corresponding to the time information read out in step S333 may be displayed in step S337.

[0110] 11 and 12 are explanatory diagrams showing an example of image display on the client 4. In FIGS. 11 and 12, an example of a screen provided by the web server function of the cloud server 3 in response to access by an operation on the client 4, displayed by a web browser included in the client program P4, is shown. A screen 430 shown in FIG. 11 displays a list 431 of target spaces that the user operating the client 4 is permitted to view. The list 431 includes an interface 432 with a link to an image display screen that displays images saved for each target space. In the example of FIG. 11, a list 431 of "Store (X Chain Store A)," "Around the Store (X Chain Store A)," "Store (X Chain Store B)," and "Around the Store (X Chain Store B)" is displayed.

[0111] Fig. 12 is a diagram showing an example of the image display screen 433. The image display screen 433 shown in Fig. 12 is displayed when the user selects the interface 432 for "Around the store (X chain store A)" from the list 431 shown in Fig. 11 using the operation unit 44.

[0112] The image display screen 433 includes an area 434 showing a list of thumbnails of scenes stored as a result of being identified by the primary scene identification unit 102 and the secondary scene identification unit 302, and an area 435 showing a list of images obtained by extracting areas corresponding to the local information on which the scenes were identified. When an image included in area 435 is selected on the image display screen 433, a thumbnail of the original scene corresponding to that image is highlighted in area 434. Each image in area 434 displays text indicating time information. When a selection operation is performed on each image in area 434 using the operation unit 44, the client 4 can obtain a video including the corresponding scene from the image information stored in the edge device 1 via the cloud server 3 and display it on the screen.

[0113] In the image processing system 100 of the first embodiment, instead of transmitting and storing frame images and videos from the edge device 1 to the cloud server 3, only text information such as time information and local information is transmitted from the edge device 1 to the cloud server 3 and stored therein, thereby reducing communication volume. Furthermore, the configuration of the image processing system 100 of the first embodiment allows images of objects, particularly customers, captured in each space to be displayed on the client 4 as desired by the user without being stored on the cloud server 3. This allows the user to view frame images or videos that are determined to show a specific object with a higher degree of accuracy, while taking personal information into consideration.

[0114] As described above, according to the image processing system 100 of the first embodiment, global information, which is general information about the entire frame image, local information, which is information about each area in the frame image, and time information corresponding to the frame image are extracted from one or a plurality of temporally consecutive frame images. This makes it possible to obtain detailed information about the frame image (not only general information about the entire frame image, but also information about each area in the frame image), and this detailed information can be used to accurately identify a scene desired by the user.

[0115] [Variations] In the first embodiment, the edge device 1 and the cloud server 3 are configured with functions separated into a primary scene identification unit 102 and a secondary scene identification unit 302. However, this is not limiting, and scenes may be identified collectively as a scene identification unit on the cloud server 3 side. Fig. 13 is a functional block diagram of an image processing system 100 in a modified example.

[0116] In the modified example, the processing unit 10 of the edge device 1 functions as an information extraction unit 101 and an edge device-side transmission unit 103. The processing unit 10 transmits the global information and local information extracted by the information extraction unit 101 for each frame image from the edge device-side transmission unit 103 to the cloud server 3, regardless of whether or not the frame image from which this information was extracted is likely to show an object desired by the user.

[0117] In this modification, the processing unit 30 of the cloud server 3 functions as a cloud-side storage unit 301, a scene identification unit 306, a selection operation unit 303, an image display control unit 304, and a scene information input unit 305. In this modification, the scene identification unit 306 combines the functions of the primary scene identification unit 102 and the secondary scene identification unit 302 of the first embodiment, and a detailed description thereof will be omitted. The scene identification unit 306 uses the global information, local information, and time information of each frame image stored in the storage unit 31 to identify a scene desired by the user, and stores in the storage unit 31 only the time information corresponding to the identified scene.

[0118] Even in this modification, only text information is transmitted from the edge device 1 to the cloud server 3, which contributes to consideration of personal information and reduction of communication volume. Furthermore, detailed content information of the frame image (not only general information about the entire frame image but also information about each area in the frame image) can be obtained, and this detailed content information can be used to accurately identify the scene desired by the user.

[0119] [Second embodiment] In the first embodiment, as shown in FIG. 9, an image is divided into four areas, and local information is obtained for each area. In the second embodiment, an object that should appear in the user's desired scene is first detected, and local information is obtained based on the bounding box of the object. The configuration of the image processing system 100 of the second embodiment is the same as that of the image processing system 100 of the first embodiment, except for the detailed processing content when obtaining the local information. Therefore, in the image processing system 100 of the second embodiment shown below, components that are common to the image processing system 100 of the first embodiment are assigned the same reference numerals, and detailed description thereof will be omitted.

[0120] FIG. 14 is a functional block diagram of an image processing system 100 according to the second embodiment. In the second embodiment, the information extraction unit 101 uses not only the language model M11 but also the detection learning model M12. In the second embodiment, the processing unit 10, as the information extraction unit 101, first extracts an area in which the object may appear using the detection learning model M12 for the user-desired object for the acquired frame image. The detection learning model M12 is selected from a person detector, a head detector, a face detector, an animal detector, a specific device detector, etc. provided in the database 310 according to the user-desired object (person, animal, vehicle, device, etc.) set for each target space, and is stored in the storage unit 11. Furthermore, an object detection process using the detection learning model M12 is deployed in the edge device 1. The information extraction unit 101 obtains detection results (coordinate data and accuracy of the area in which the object appears) output from the detection learning model M12. In the second embodiment, if the object desired by the user is not detected using the detection learning model M12, the information extraction unit 101 discards the target frame image without performing any further processing on it.

[0121] In the second embodiment, as in the first embodiment, the information extraction unit 101 instructs the language model M11 to output sentences or words describing people or objects appearing in the entire frame image, and obtains the text-format information output from the language model M11 as global information, which is general information about the entire frame image. In the second embodiment, the information extraction unit 101 uses the detection results obtained from the detection learning model M12 to obtain local information in text format, information inferred from the attributes, location, etc., of the user-desired object determined from the frame image. Specifically, the information extraction unit 101 specifies a bounding box (a range within the frame image) indicating the area in which the object indicated by the detection results appears, and instructs the language model M11 to output sentences or words describing the user-desired object appearing in the specified range, and obtains the text-format information output from the language model M11 as local information about the target object.

[0122] In the second embodiment, the information extraction unit 101 also extracts summaries in text format as global information for the frame image, such as "Three people are walking," "Multiple trees with autumn leaves are visible," and "A specific animal is visible in the shade of a tree."

[0123] When a "person" is specified as a user-desired object as local information about a specific area or object in a frame image, the information extraction unit 101 extracts information in text format about the detected "person," such as "The detected person (aged 50 or older, under 70) is walking toward the front. There are three trees nearby. There is a specific animal in the shade of the trees," "The detected person (aged 30 or older, under 50, wearing a hat) is walking toward the back. There is another person (wearing glasses) next to the person," "The detected person (aged 30 or older, under 50, wearing glasses) is walking toward the back. There is another person (wearing a hat) next to the person," and "The detected person (aged 20 or older, under 30) is walking toward the back. There is a dog next to the person." When there are multiple people, each person can be identified by being assigned an ID, and when the same person appears across multiple frame images using face recognition, the same ID can be assigned and followed.

[0124] As described above, the information extraction unit 101 of the second embodiment acquires local information by specifying a bounding box. Alternatively, the information extraction unit 101 may perform object detection on a frame image using the detection learning model M12 in advance, and then input an image of the frame image, in which grid lines dividing the frame image and bounding boxes of each object detected by the object detection are drawn, into a language model M11 such as a VLM, to output local information in text format. This allows local information to be extracted based on the positional relationship between the grid lines and the bounding boxes of each object. This allows, for example, a clear recognition of the area within a frame image, such as "A person (aged 50 or older but under 70) detected in the upper left area divided by the grid lines is walking toward the front." Note that, in the second embodiment, the information extraction unit 101 may also extract global information, local information, and time information corresponding to a plurality of temporally consecutive frame images (i.e., a video).

[0125] In the second embodiment, the primary scene identification unit 102, as in the first embodiment, identifies a scene in which a user-desired object appears in a target frame image based on global information or local information extracted by the information extraction unit 101. Because the detection learning model M12 is used, the primary scene identification unit 102 can identify a scene based on attributes or the movement (walking, standing still, etc.) of the detection target object in slightly more detail than the detection criteria used by the information extraction unit 101 in the first embodiment. The processing unit 10, using the function of the primary scene identification unit 102, stores frame images corresponding to the identified scene in the storage unit 11 in association with the time information extracted by the information extraction unit 101. The primary scene identification unit 102 saves a predetermined amount of video including a frame image corresponding to the identified scene in the storage unit 11. When the processing unit 10 determines, based on the output from the information extraction unit 101, that the user-desired object does not appear in the target frame image, the primary scene identification unit 102 terminates processing of the target frame image, does not perform subsequent processing on the target frame, and performs processing on the next frame image.

[0126] In the second embodiment, other functions, that is, the functions of the edge device side transmitting unit 103, the cloud side storing unit 301, and the secondary scene identifying unit 302 are the same as those in the first embodiment, and detailed description thereof will be omitted.

[0127] 15 is a flowchart showing an example of a processing procedure by the information extraction unit 101 of the second embodiment. The processing of the primary scene identification unit 102 of the processing unit 10 of the edge device 1 of the second embodiment and the edge device side transmission unit 103 is the same as the processing procedure shown in FIG. 6 of the first embodiment, and therefore detailed description thereof will be omitted.

[0128] The processing unit 10 of the edge device 1 of the second embodiment acquires a frame image from the camera 2 by the information extraction unit 101 (step S151).

[0129] The processing unit 10 inputs the acquired frame image to the language model M11 together with an instruction for acquiring global information for the acquired frame image (step S152). In step S152, the processing unit 10 generates a sentence such as "Analyze what is shown in this image" as an "instruction" for acquiring global information, and inputs this together with the frame image to the language model M11.

[0130] The processing unit 10 acquires, as global information, text information that is a sentence or word output from the language model M11 in response to the input at step S152 (step S153).

[0131] The processing unit 10 of the second embodiment inputs the acquired frame image to the detection learning model M12 (step S154) and acquires the detection result (step S155). The processing unit 10 determines whether or not the object desired by the user has been detected in the acquired frame image using the detection learning model M12, by the function of the information extraction unit 101 (step S156).

[0132] If the processing unit 10 determines that the object desired by the user has not been detected (S156: NO), the processing unit 10 ends the processing for the acquired frame image. The acquired frame image and global information do not need to be saved in the storage unit 11.

[0133] In step S156, if the processing unit 10 determines that an object desired by the user has been detected in the acquired frame image (S156: YES), it inputs the frame image to the language model M11, specifying the area (bounding box) in which the object indicated by the detection result acquired in step S155 appears, together with an instruction for acquiring local information (step S157). In step S157, the processing unit 10 specifies the area in which the object appears as an "instruction" for acquiring local information, and generates a sentence for the frame image such as "Analyze what is shown in the specified part of this image," and inputs this sentence together with the frame image to the language model M11.

[0134] The processing unit 10 acquires, as local information, text information that is a sentence or word output from the language model M11 in response to the input at step S157 (step S158).

[0135] The processing unit 10 determines whether or not all of the local information of each object detected as the user's desired object has been acquired based on the detection result acquired in step S155 (step S159). If it is determined that all of the local information has not been acquired (S159: NO), the processing unit 10 returns to step S157 to process other objects in the frame image.

[0136] If it is determined that all the local information has been acquired (S159: YES), the processing as the information extraction unit 101 ends.

[0137] FIG. 16 is an explanatory diagram of local information in the second embodiment. FIG. 16 shows an example for the frame image shown in FIG. 8 of the first embodiment. In FIG. 16, the bounding boxes of detected objects (people) are indicated by dashed lines. Five people (objects) with ID01 to ID05 have been detected in the frame image shown in FIG. 16. The local information shown is an example of local information obtained by specifying areas corresponding to the detected objects with ID01 to ID05. In the example of FIG. 16, the local information is output from the language model M11 in response to an instruction, for example, to "analyze what is captured around the person with ID01" for the objects detected in the frame image.

[0138] In the example of Figure 16, local information obtained by specifying a person (object) with ID01 indicates more detailed information about the person's orientation and movement, such as "Person (ID01) is walking along the tree-lined path toward the front." Local information obtained by specifying a person with ID02 indicates clearer information about the person's orientation, movement, and positional relationship with other people, such as "Person (ID02) is walking along the tree-lined path toward the back. Person (ID03) is walking next to him. Person (ID04) is walking in the background." Local information obtained by specifying a person with ID03 indicates detailed information about small dangerous animals not included in the global information, as well as their positional relationship with the detected person, such as "Person (ID03) is walking along the tree-lined path toward the back. Person (ID02) is walking next to him. Person (ID04) is walking in the background. There is a tree nearby, and a snake in the shade of the tree."

[0139] Local information obtained by specifying person ID04 provides clear information about the person's direction or movement and their relative position relative to other people, such as "Person (ID04) is walking toward the front of the tree-lined path. Two other people (ID02, ID03) are walking in front of him." Local information obtained by specifying person ID05 provides clearer information focusing on the detected person's movement and their relative position relative to trees and other people, such as "Person (ID05) is walking toward the back of the tree-lined path. He is in the shade of the trees. Three other people (ID04, ID03, ID02) are visible in the foreground."

[0140] 16, the image processing system 100 of the second embodiment uses text information analyzed using a language model M11 based on objects detected in a frame image as local information. Detailed information that clarifies positional relationships and the like, which is not included in the results of analyzing the entire frame image, can be obtained in text format.

[0141] The image processing system 100 of the second embodiment also reduces communication traffic by transmitting only text information, such as time information and local information, from the edge device 1 to the cloud server 3 for storage, rather than transmitting and storing frame images and videos from the edge device 1 to the cloud server 3. The configuration of the image processing system 100 of the second embodiment allows the client 4 to display the images the user desires, without storing images of objects captured in each space, particularly customer images, on the cloud server 3. The image processing system 100 of the second embodiment further enables more accurate scene identification using verbalized local information focusing on the area of ​​the frame image where the user's desired object is detected. In this way, the second embodiment also allows the user to view frame images or videos that are determined to contain a specific object with a high degree of accuracy, while taking personal information into consideration.

[0142] [Third embodiment] In the third embodiment, a secondary scene identification unit 302 on the cloud server 3 side identifies a scene desired by a user using LLM from global information, local information, and time information stored in a storage unit 31. The configuration of an image processing system 100 of the third embodiment is the same as the configuration of the image processing system 100 of the first or second embodiment, except for the processing content of the secondary scene identification unit 302. Therefore, in the image processing system 100 of the third embodiment described below, components common to the image processing system 100 of the first or second embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0143] FIG. 17 is a functional block diagram of an image processing system 100 according to a third embodiment. In the third embodiment, a secondary scene identification unit 302 on the cloud server 3 side identifies a scene using a language model M31. Specifically, the secondary scene identification unit 302 uses a language model M31 (LLM) to identify time information of a frame image showing a user-desired object from global information or local information in text format for each frame image stored in the storage unit 31 by the cloud-side storage unit 301. The secondary scene identification unit 302 may identify a scene from either the global information or the local information, or may use a combination of global information and local information corresponding to the same time information. The secondary scene identification unit 302 provides the LLM with the global information and / or local information for each stored time information (frame image) and a text-format instruction for searching for the user-desired scene, and identifies the scene using a response from the LLM.

[0144] In the third embodiment, the edge device 1 executes the processing procedure shown in Fig. 6 as in the first embodiment, or the processing procedure shown in Fig. 15 of the second embodiment. In the third embodiment, the cloud-side storage unit 301 of the cloud server 3 also executes the processing procedures shown in Fig. 6 and Fig. 7. In contrast, the secondary scene identification unit 302 executes the following processing procedures.

[0145] 18 is a flowchart showing an example of the processing procedure of the secondary scene identification unit 302 in the third embodiment. The processing unit 30 of the cloud server 3 executes the following processing in parallel with the processing procedure by the edge device 1, using the function of the secondary scene identification unit 302.

[0146] The processing unit 30 executes the following process each time global information and local information are stored in the storage unit 31. The processing unit 30 may execute the following process each time frame images for a predetermined time period, such as 10 minutes, 30 minutes, 1 hour, 12 hours, or 24 hours, are saved in the edge device 1, or may execute the following process each time processing is performed on a frame image in the edge device 1.

[0147] The processing unit 30 uses the function of the secondary scene identification unit 302 to read out the global information and local information associated with the time information stored in the storage unit 31 (step S341). The processing unit 30 creates an instruction statement for the LLM (language model M31) to search for a scene in which an object desired by the user appears (step S342), and inputs the instruction statement for searching for a scene based on the read global information and local information into the LLM (language model M31) (step S343).

[0148] In step S342, the processing unit 30 generates a command in natural language, such as, "Based on the attached text information, find a scene in which a person is in danger and provide time information to identify the scene." The processing unit 30 may also perform a search targeting scenes spanning a certain period of time (plurality of consecutive frame images) rather than individual frame images, such as, "Based on the attached text information, find a scene in which a person is in the same place for five minutes or more and provide time information to identify the scene." As described above, when performing processing to determine whether a plurality of frame images (stored in the storage unit 31) correspond to a user-desired scene, it is desirable that the secondary scene identification unit 302 inputs global information, local information, and time information in text format for the read plurality of frame images together into the LLM to identify frame images corresponding to the user-desired scene (output time information of the frame images corresponding to the user-desired scene). As described above, when multiple frame images are subjected to a collective process of determining whether or not they correspond to a scene desired by the user, the secondary scene identification unit 302 may input only local information and time information in text format about these frame images into the LLM to identify the frame image that corresponds to the scene desired by the user.

[0149] The processing unit 30 acquires a response from the LLM (language model M31) (step S344). The processing unit 30 uses the function of the secondary scene identification unit 302 to identify the time information of the scene desired by the user from the acquired response (step S345). The processing unit 30 stores the identified time information in the storage unit 31 (step S346) and ends the scene identification process. The processing unit 30 may keep the global information and local information corresponding to the time information identified in step S345 stored for search purposes, or may delete them. The processing unit 30 may delete from the storage unit 31 time information other than the time information identified in step S345, as well as the global information and local information corresponding thereto.

[0150] In the image processing system 100 of the third embodiment, the function of the secondary scene identification unit 302 that performs the above-mentioned processing (particularly, inputting global information, local information, and time information in text format for multiple consecutive frame images together into the LLM to identify frame images that correspond to the user's desired scene) makes it possible to use the LLM to search for frame images that correspond to global information and / or local information using synonyms, or frame images that correspond to scenes that evoke those words as context, rather than searching for global information and local information that include words that indicate the conditions for the user's desired scene that have been set. This increases the likelihood that the user's desired scene will be accurately identified.

[0151] In the image processing system 100 of the third embodiment, only text information, which is time information, local information, and global information, is transmitted from the edge device 1 to the cloud server 3, and frame images or videos are not stored in the cloud server 3, allowing for consideration of personal information. Because text format information is transmitted from the edge device 1 to the cloud server 3, the amount of communication traffic can be significantly reduced compared to when image information is transmitted. While it is highly accurate to identify a scene containing a user-desired object from a group of frame images using the cloud server 3, which has abundant computing resources, analysis using LLM in the image processing system 100 of the third embodiment allows for accurate identification of the user-desired scene while prioritizing consideration of personal information and reduction of communication traffic.

[0152] The embodiments disclosed above are illustrative in all respects and are not restrictive. The scope of the present invention is defined by the claims, and includes all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0153] 100 Image Processing System 1. Edge Devices 10 Processing section 101 Information extraction part 102 Primary Scene Identification Unit 11 Storage section P1 First image processing program 3. Cloud Server 30 Processing section 302 Secondary Scene Identification Unit 304 Image display control unit 305 Scene information input section 306 Scene Identification Unit 31 Storage section P32 2nd image processing program

Claims

1. an information extraction means for extracting, from one or a plurality of temporally consecutive frame images, local information which is information about a predetermined area or each object in the frame image, and time information corresponding to the frame image; a scene specifying means for specifying a scene desired by a user based on the local information and time information extracted by the information extracting means, the local information is created based on information obtained from at least one of a plurality of areas obtained by dividing the frame image into a plurality of areas, The local information is text-format information obtained by dividing the frame image into a plurality of areas by grid lines and inputting the divided images into a VLM (Vision-Language Model).

2. the local information and the time information are in text format, 2. The image processing system according to claim 1, wherein the scene identification means identifies a scene desired by a user based on the local information in text format and the time information.

3. 3. The image processing system according to claim 2, wherein the scene identification means inputs the local information in text format into an LLM (Large Language Model) to identify a frame image corresponding to the scene desired by the user.

4. an information extraction means for extracting, from one or a plurality of temporally consecutive frame images, global information which is general information about the entire frame image, local information which is information about a predetermined area or each object in the frame image, and time information corresponding to the frame image; a scene identification means for identifying a scene desired by a user based on the global information, local information, and time information extracted by the information extraction means, the local information is created based on information obtained from at least one of a plurality of areas obtained by dividing the frame image into a plurality of areas, The local information is text-format information obtained by inputting an image obtained by dividing the frame image into a plurality of areas by grid lines into a VLM.

5. the global information, the local information, and the time information are in text format; 5. The image processing system according to claim 4, wherein the scene identification means identifies a scene desired by a user based on the global information, the local information, and the time information in text format.

6. 6. The image processing system according to claim 5, wherein the global information is information in a text format obtained by inputting the frame image into a VLM.

7. 6. The image processing system according to claim 5, wherein the scene identification means inputs the global information, the local information, and the time information in text format into an LLM to identify a frame image corresponding to the scene desired by the user.

8. 7. The image processing system according to claim 6, wherein the scene identification means inputs the global information, the local information, and the time information in text format into an LLM to identify a frame image corresponding to the scene desired by the user.

9. 9. The image processing system according to claim 7, wherein the scene identification means adds together the global information and the local information in text format and inputs the sum into the LLM.

10. 2. The image processing system according to claim 1, wherein the local information is text-format information obtained by inputting into a VLM an image on the frame image that depicts the grid lines and the bounding boxes of each object detected by object detection previously performed on the frame image.

11. The image processing system according to claim 10 , wherein the local information is extracted based on a positional relationship between the grid lines and a bounding box of each of the objects.

12. further comprising a scene information input means for inputting scene information which is information about the scene desired by the user, 12. The image processing system according to claim 1, wherein the scene identification means identifies a scene corresponding to the scene information input by the scene information input means.

13. The image processing system according to any one of claims 1 to 8, 10 and 11, further comprising an image display control means for extracting a frame image corresponding to a user-desired scene identified by the scene identification means from a moving image including this frame image and controlling the display means to display the frame image.

14. the image processing system includes an edge device-side system and a cloud-side system, the edge device-side system includes the information extraction means, The image processing system according to claim 1 , wherein the cloud-side system includes the scene identification means.

15. Computer, an information extraction means for extracting, from one or a plurality of temporally consecutive frame images, local information which is information about a predetermined area or each object in the frame image, and time information corresponding to the frame image; and functioning as a scene identification means for identifying a scene desired by a user based on the local information and time information extracted by the information extraction means; the local information is created based on information obtained from at least one of a plurality of areas obtained by dividing the frame image into a plurality of areas, The local information is an image processing program in the form of text information obtained by inputting an image obtained by dividing the frame image into a plurality of areas by grid lines into a VLM.

Images