Image processing system and image processing program

Abstract

This invention provides an image processing system and image processing program capable of accurately identifying user-requested scenes. [Solution] The image processing system includes an information extraction means for extracting local information, which is information about a predetermined area or each object in a frame image, from one or a plurality of temporally consecutive frame images, and a scene identification means for identifying a scene desired by the user based on the local information extracted by the information extraction means. The local information is created by dividing the frame image into a plurality of areas and obtaining information from at least one of the divided areas, and is text-formatted information obtained by inputting an image obtained by dividing the frame image into a plurality of areas with grid lines into the VLM.

Description

Technical Field

[0003]

[0001] The present invention relates to an image processing system and an image processing program.

Background Art

[0002] In the fourth embodiment of the specification of Japanese Patent Application No. 2025-001868, the applicant has proposed an image display system in which, on the edge device side, text information regarding a frame image in which a browsing target desired by a user may be reflected is output, and on the cloud side, using the above text information transmitted from the edge device side, a frame image in which a browsing target desired by the user is reflected is selected according to detailed criteria. In this embodiment, on the edge device side, each frame image is input into a model such as a VLM (Vision-Language Model) to extract text information from the corresponding frame image.

Summary of the Invention

Problems to be Solved by the Invention

[0003] As described above, by inputting a frame image into VLM, it is easy to obtain summary text information about the entire frame image (hereinafter referred to as "global information") from the frame image, but it is difficult to obtain detailed text information about the content of the frame image. More specifically, when inputting a frame image into VLM without entering any prompt (or entering a prompt such as "Analyze and describe this image"), if text information is extracted from the frame image, it is not possible to obtain text information about the finer objects in the frame image (the finer objects are ignored). Therefore, it is difficult to identify (search) the scene desired by the user using this text information. On the other hand, when inputting a frame image into VLM with the prompt "Analyze this image in detail", it is possible to obtain text information of the detailed analysis results of the frame image, but the aspects from which the frame image is analyzed in detail depend on the frame image, so the obtained text information is not generalizable. For this reason, it is difficult to identify (search) the scene desired by the user using this text information.

[0004] The present invention aims to solve the above problems and to provide an image processing system and image processing program that can obtain detailed information about the contents of a frame image and use this detailed information to accurately identify a scene desired by the user. [Means for solving the problem]

[0005] To solve the above problems, the first aspect of the present invention provides an image processing system that includes an information extraction means for extracting local information, which is information about a predetermined area or each object in a frame image, from one or a plurality of temporally consecutive frame images, and a scene identification means for identifying a user-desired scene based on the local information extracted by the information extraction means, wherein the local information is created by dividing the frame image into a plurality of areas and obtaining information from at least one of the divided areas, and the local information is text-format information obtained by inputting an image obtained by dividing the frame image into a plurality of areas with grid lines into a VLM. This is text-formatted information obtained by inputting an image into VLM in which the grid lines and the bounding boxes of each object detected by object detection performed on the frame image in advance are drawn on the frame image. .

[0006] In this image processing system, the local information is preferably in text format, and the scene identification means preferably identifies a user-desired scene based on the local information in text format.

[0007] In this image processing system, it is desirable that the scene identification means inputs the local information in text format into an LLM (Large Language Model) to identify the frame image corresponding to the user-requested scene.

[0008] To solve the above problems, an image processing system according to a second aspect of the present invention comprises: an information extraction means for extracting global information, which is summary information about the entire frame image, and local information, which is information about a predetermined area or each object in the frame image, from one or a plurality of temporally consecutive frame images; and a scene identification means for identifying a user-desired scene based on the global information and local information extracted by the information extraction means, wherein the local information is created by dividing the frame image into a plurality of areas and obtaining information from at least one of the divided areas, and the local information is text-format information obtained by inputting an image obtained by dividing the frame image into a plurality of areas with grid lines into a VLM. This is text-formatted information obtained by inputting an image into VLM in which the grid lines and the bounding boxes of each object detected by object detection performed on the frame image in advance are drawn on the frame image. .

[0009] In this image processing system, the global information and the local information are in text format, and it is desirable that the scene identification means identify the scene desired by the user based on the global information and the local information in text format.

[0010] In this image processing system, it is desirable that the global information is text-formatted information obtained by inputting the frame image into the VLM.

[0011] In this image processing system, it is desirable that the scene identification means inputs the global information and local information in text format into the LLM to identify the frame image corresponding to the user-requested scene.

[0012] In this image processing system, it is desirable that the scene identification means add the global information in text format and the local information together and input them into the LLM.

[0013] In this image processing system, it is desirable that the local information is text-formatted information obtained by inputting an image into the VLM that has been drawn on the frame image, in which the grid lines and the bounding boxes of each object detected by object detection performed on the frame image in advance are drawn.

[0014] In this image processing system, the local information may be extracted based on the positional relationship between the grid lines and the bounding boxes of each object.

[0015] In this image processing system, it is desirable to further include a scene information input means for inputting scene information, which is information about the user-desired scene, and the scene identification means for identifying the scene corresponding to the scene information input by the scene information input means.

[0016] In this image processing system, it is desirable to further include an image display control means that extracts a frame image corresponding to a user-desired scene identified by the scene identification means from a video containing this frame image and controls it to display on a display means.

[0017] In this image processing system, the image processing system may include an edge device system and a cloud system, wherein the edge device system is equipped with the information extraction means and the cloud system is equipped with the scene identification means.

[0018] An image processing program according to a third aspect of the present invention causes a computer to function as an information extraction means for extracting local information, which is information about a predetermined area or each object in a frame image, from one or a plurality of temporally consecutive frame images, and as a scene identification means for identifying a user-desired scene based on the local information extracted by the information extraction means, wherein the local information is created by dividing the frame image into a plurality of areas and obtaining information from at least one of the divided areas, and the local information is text-formatted information obtained by inputting an image obtained by dividing the frame image into a plurality of areas with grid lines into a VLM. This is text-formatted information obtained by inputting an image into VLM in which the grid lines and the bounding boxes of each object detected by object detection performed on the frame image in advance are drawn on the frame image. . [Effects of the Invention]

[0019] According to the first aspect of the present invention, an image processing system, and according to the third aspect, an image processing program, both of which are information about a predetermined area or each object in a frame image, are extracted from one or a plurality of temporally consecutive frame images. This allows for obtaining detailed information about the contents of a frame image (not general information about the entire frame image, but information about a predetermined area or each object in the frame image), and this detailed information can be used to accurately identify a scene desired by the user.

[0020] Also, according to the image processing system according to the second aspect of the present invention, global information, which is overview information about the entire frame image, and local information, which is information about a predetermined area or each object in the frame image, are extracted from one or a plurality of temporally consecutive frame images. As a result, information on the detailed content of the frame image (not only overview information about the entire frame image but also information about a predetermined area or each object in the frame image) can be obtained, and thus the scene desired by the user can be accurately specified using this information on the detailed content.

Brief Description of the Drawings

[0021] [Figure 1] It is a schematic diagram of the image processing system of the first embodiment. [Figure 2] It is a block diagram showing the configuration of the edge device. [Figure 3] It is a block diagram showing the configuration of the cloud server. [Figure 4] It is a block diagram showing the configuration of the client. [Figure 5] It is a functional block diagram of the image processing system. [Figure 6] It is a flowchart showing an example of the scene specification processing procedure in the image processing system. [Figure 7] It is a flowchart showing an example of the scene specification processing procedure in the image processing system. [Figure 8] It is an explanatory diagram of global information and local information. [Figure 9] It is an explanatory diagram of global information and local information. [Figure 10] It is a flowchart showing an example of the display control procedure in the image processing system 100. [Figure 11] It is an explanatory diagram showing an example of the display of an image on the client. [Figure 12] It is an explanatory diagram showing an example of the display of an image on the client. [Figure 13] This is a functional block diagram of the image processing system in a modified example. [Figure 14] This is a functional block diagram of the image processing system in the second embodiment. [Figure 15] This flowchart shows an example of the processing procedure by the information extraction unit of the second embodiment. [Figure 16] This is an explanatory diagram of local information in the second embodiment. [Figure 17] This is a functional block diagram of the image processing system in the third embodiment. [Figure 18] This flowchart shows an example of the processing procedure for the secondary scene identification unit in the third embodiment. [Modes for carrying out the invention]

[0022] This disclosure will be described in detail with reference to drawings illustrating embodiments thereof.

[0023] [First Embodiment] Figure 1 is a schematic diagram of the image processing system 100 of the first embodiment. The image processing system 100 is installed in a specific space inside or outside a store as the target space. The image processing system 100 includes a camera 2 installed in the target space, an edge device 1 connected to the camera 2, a cloud server 3 capable of communicating with the edge device 1, and a client 4 capable of communicating with the cloud server 3. Multiple edge devices 1 and cameras 2 may be installed in the same space.

[0024] Camera 2 outputs image information using an image sensor that supports visible light and / or near-infrared light. Camera 2 outputs image information in a time series at a rate of several fps to tens of fps. Camera 2 may be installed so as to look down from above the target space, such as on the ceiling or shelves of the target space, which is the store. Camera 2 may be a ceiling-mounted type that captures a 360-degree field of view. Camera 2 may also be installed on a support column or similar structure so as to look down in an outdoor area which is the target space.

[0025] Camera 2 and edge device 1 can communicate via a local network (LN), which can be wireless or wired. The local network (LN) is installed in the facility containing the target space. The local network (LN) may be a wired LAN or a wireless network such as Wi-Fi or Bluetooth®. Camera 2 sequentially transmits image information to edge device 1 via the local network (LN).

[0026] Edge device 1 can communicate with cloud server 3 via network N. Network N is a wired or wireless communication network that may include a public communication network, a dedicated line, or a carrier network. Client 4 can communicate with cloud server 3 and edge device 1 via network N.

[0027] Cloud server 3 stores a set of learning models in database 310 that are used in processes selectively executed on edge device 1 and cloud server 3. Database 310 may also include a set of learning models provided by external services outside the system. Cloud server 3 reads learning models from database 310 that correspond to the objects detected by the processes executed on edge device 1 and cloud server 3, and deploys them to edge device 1 and cloud server 3.

[0028] The image processing system 100 of the first embodiment extracts feature quantities from an image captured by a camera 2 installed in a target space, detects objects such as people or objects to be detected from the image based on the feature quantities, recognizes the attributes of the detected objects, and outputs the recognition result. The image processing system 100 is a system that enables the user to automatically display a user-desired scene containing the object desired by the user on the client 4.

[0029] The image processing system 100 of the first embodiment is a system that allows the user to select which attributes of person or object should be used as the detection object for determining whether or not the scene is one desired by the user. The image processing system 100 of the first embodiment reduces the amount of communication between the edge device 1 and the cloud server 3, and further reduces the processing load on the edge device 1 by sharing the processing between the edge device 1 and the cloud server 3.

[0030] The following describes the configuration of the edge device 1, cloud server 3, and client 4 for realizing such an image processing system 100, as well as the details of the processing performed by each.

[0031] Figure 2 is a block diagram showing the configuration of edge device 1. Edge device 1 is a box-type device that can be installed in the target space together with camera 2. Edge device 1 comprises a processing unit 10, a storage unit 11, a first communication unit 12, and a second communication unit 13.

[0032] The processing unit 10 includes one or more processors such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), or NPU (Neutral Processing Unit). The processing unit 10 includes memory, which is a temporary storage medium such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory). The processing unit 10 includes a timer and can obtain time information at each point in time from data from the timer. The processing unit 10 may be configured as a single hardware (SoC: System On a Chip) integrating the processor, memory, and furthermore, the storage unit 11, the first communication unit 12, and the second communication unit 13. The specifications of the processing unit 10 may be the same or different for each edge device 1.

[0033] The processing unit 10 reads the first image processing program P1 stored in the storage unit 11 into memory and executes it, thereby causing the processor to perform various processes described later, and to function as the edge device 1 of this disclosure.

[0034] The storage unit 11 is a relatively large-capacity non-temporary storage medium such as a hard disk or flash memory. A portion of the storage unit 11 may be removable.

[0035] The storage unit 11 stores the program (program product) necessary for the processing unit 10 to execute processing, the results of the processing by the processing unit 10, and reference setting information. The setting information includes identification data for the edge device 1, identification data for the connected camera 2, etc. The setting information includes setting a criterion for distinguishing between global information and local information of the image, which will be described later. Setting the criterion, for example, defines an area in the image that corresponds to the local information. The area may be a specific area in the image set by the client 4, or it may be a frame image divided into multiple areas. Setting the criterion may also include whether or not to use the detected bounding box.

[0036] The setting information stored in the memory unit 11 includes information referenced to identify a user-desired scene. The information for identifying a scene includes, for example, text indicating a user-desired object such as "human". The text may also be "dog", "dangerous animal", or text indicating attributes such as "adult", "wearing a hat / cap", or "wearing glasses". The text indicating the user-desired object is set based on scene information entered by the user via the cloud server 3 (see "Scene Information Input Unit 305" in Figure 5) using the client 4.

[0037] The program product stored in the memory unit 11 includes an OS (Operation System) program, a first image processing program P1, and a group of learning models M1. Details of the group of learning models M1 will be described later. The first image processing program P1 or the group of learning models M1 stored in the memory unit 11 may be obtained by the processing unit 10 reading the first image processing program P1 and the group of learning models M9 stored in a non-temporary storage medium 9 that can be read from a computer and storing them in the memory unit 11, or they may be stored in advance at the time of shipment. The first image processing program P1 or the group of learning models M1 stored in the memory unit 11 may also be obtained by the processing unit 10 via the second communication unit 13 from the database 310 of the cloud server 3, or from another download server.

[0038] The storage unit 11 stores image information acquired from the connected camera 2 in chronological order, associated with time information. The image information may be frame images or video. The storage unit 11 stores only image information that is a frame image or video including preceding and succeeding images in which the user's desired object (viewing target) is detected as a result of the processing of the processing unit 10 described later. The storage unit 11 may continue to store data by overwriting it according to the FIFO (First In First Out) method, within a range corresponding to the size of its capacity. The storage unit 11 may also store text information describing the detected object, text information indicating the attributes of the detected object, etc., associated with time information.

[0039] The first communication unit 12 is a communication device that enables communication via the local network LN. The first communication unit 12 may be a LAN network card or a CAN communication device. The first communication unit 12 may be a communication device that supports wireless networks such as WiFi or Bluetooth®. The first communication unit 12 may include multiple communication devices that support various types of cameras 2. The first communication unit 12 may include an interface such as USB (Universal Serial Bus) that connects to the camera 2. The first communication unit 12 can be replaced with an interface that connects to the camera 2 via a coaxial cable or other serial bus. The processing unit 10 acquires image information from the camera 2 via the local network LN using the first communication unit 12. The first communication unit 12 may be the same device as the second communication unit 13 described later.

[0040] The second communication unit 13 is a communication device that enables communication via an external network N. The second communication unit 13 may be a wired LAN network card or a communication device that enables carrier communication via a carrier network. The second communication unit 13 may also be a communication device that supports wireless networks such as WiFi or Bluetooth (registered trademark). The second communication unit 13 may support secure communication such as SSL (Secure Socket Layer) with the cloud server 3. The second communication unit 13 may also be an interface for enabling communication connection with the cloud server 3 via a dedicated line.

[0041] Figure 3 is a block diagram showing the configuration of the cloud server 3. The cloud server 3 may consist of a single server computer, or it may be configured to distribute processing across multiple server computers. The cloud server 3 includes a processing unit 30, a storage unit 31, and a communication unit 32.

[0042] The processing unit 30 includes one or more processors such as CPUs, MPUs, GPUs, or NPUs. The processing unit 30 also includes memory, which is a temporary storage medium such as SRAM or DRAM.

[0043] The storage unit 31 is a relatively large-capacity non-temporary storage medium such as a hard disk or flash memory. The storage unit 31 stores the program (program product) and configuration information necessary for the processing unit 30 to perform the processing described later.

[0044] The program product stored in the memory unit 31 includes the server program P31. The server program P31 includes a module that functions as a data server, reading the model group stored in the database 310 and sending it to the edge device 1. The server program P31 also includes a module that functions as a web server, and can output the results of the processing performed by the cloud server 3 to the client 4 as a web page.

[0045] The program products stored in the memory unit 31 include the second image processing program P32, which runs on the cloud server 3, and the learning model group M3. The second image processing program P32 is a program that causes the processor to execute processing in conjunction with the processing based on the second image processing program P32 on the edge device 1. The learning model group M3 is selected from the database 310 and stored. The second image processing program P32 and the learning model group M3 will be described in detail later.

[0046] The server program P31, the second image processing program P32, or the learning model group M3 stored in the memory unit 31 may be obtained by the processing unit 30 reading the server program P31 and the second image processing program P32 stored in the non-temporary storage medium 8 readable from the computer and storing them in the memory unit 31. Similarly, the learning model group M3 may be obtained by the processing unit 30 reading the learning model group M8 stored in the non-temporary storage medium 8 and storing it in the memory unit 31. The server program P31 and the second image processing program P32 stored in the memory unit 31 may also be obtained by the processing unit 30 downloading them from another download server via the communication unit 32.

[0047] The configuration information stored in the memory unit 31 includes a correspondence between data identifying the edge device 1 and data and a name for identifying the space where the edge device 1 is installed. Since the cloud server 3 performs processing on multiple target spaces simultaneously, it may also store identification data of edge devices 1 or target spaces that users are permitted to change settings for, etc., as a whitelist in the memory unit 31, associated with the user's account data. This allows the cloud server 3 to identify the target edge device 1 when a user specifies the name of a target space and which objects to detect in the images captured by the camera 2 installed in that target space.

[0048] The configuration information stored in the memory unit 31 includes information referenced for the process of identifying the user-requested scene, which will be described later. The information used to identify the scene on the cloud server 3 side is even more specific than the information used on the edge device 1 side. For example, settings such as "a scene with multiple people walking together," "a scene with a person in the same place for more than 5 minutes," and "a scene with a dangerous animal approaching a person" may be configured.

[0049] The memory unit 31 stores image information that includes scenes captured by the camera 2 and identified by the scene identification process. The image information may be frame images or moving images that include preceding and succeeding images.

[0050] The database 310 may be built on the storage unit 31 or on an external storage device. Part of the database 310 may include a model provision service used on the Web, which is communicated via the network N, as described above. The database 310 holds a set of learning models used in processes executed on the edge device 1 and the cloud server 3. The set of learning models includes, for example, learning models for detection such as a person detector, a head detector, a face detector, an animal detector, and a specific device detector. The set of learning models also includes learning models for recognition that recognize attributes such as an age recognition device, a glasses wearer recognition device, a hat wearer recognition device, a face orientation recognition device, or a posture recognition device. The set of learning models includes large and small language models (LLM: Large Language Model, SLM: Small Language Model), and VLM (Vision-Language Model), etc.

[0051] The communication unit 32 is a communication device that enables communication connections with the client 4 and the edge device 1 via the network N.

[0052] Figure 4 is a block diagram showing the configuration of client 4. Client 4 is a personal computer, smartphone, or tablet device. Client 4 may be used by the administrator of the space where camera 2 is installed, or by the operator of the cloud server 3 management company.

[0053] Client 4 comprises a processing unit 40, a storage unit 41, a communication unit 42, a display unit 43, and an operation unit 44. The processing unit 40 includes one or more processors such as CPUs, MPUs, GPUs, or NPUs. The processing unit 40 also includes memory, which is a temporary storage medium such as SRAM or DRAM.

[0054] The storage unit 41 is a memory of a non-temporary storage medium such as a hard disk or flash memory. The storage unit 41 stores the functions of the data server provided by the cloud server 3, and the client program P4 for the web server. The client program P4 is, for example, a web browser program. The client program P4 is a program that causes the processing unit 40 to execute the process of displaying various data, including images, provided by the cloud server 3 on the screen.

[0055] The communication unit 42 is a communication device that enables communication with the cloud server 3 via the network N. The communication unit 42 may also be a communication device that enables communication with the cloud server 3 via a dedicated line. The communication unit 42 may also be a communication device that enables direct communication with the second communication unit 13 of the edge device 1 via a wireless communication medium or a USB cable, etc.

[0056] The display unit 43 uses a display such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 43 displays a web page containing text and images based on processing by the client program P4 of the processing unit 40. The display unit 43 may also use a touch panel display.

[0057] The operation unit 44 is a user interface such as a keyboard or pointing device that accepts user input. The operation unit 44 may be a touch panel built into the display of the display unit 43, or it may be physical buttons. The operation unit 44 may be a voice input unit that accepts voice input through a voice recognition function. The operation unit 44 can notify the processing unit 40 of user operation information.

[0058] In the image processing system 100 configured in this way, the edge device 1 and the cloud server 3 share the processing of images captured by the camera 2 in each space, and display the image information of the user-requested scene on the client 4 so that the user can view it. In the image processing system 100 of the first embodiment, the edge device 1 performs a process to identify scenes that may contain the user-requested object. In the image processing system 100, the cloud server 3 secondarily performs a process to identify scenes that definitely contain the object from the frame images corresponding to the scenes initially identified by the edge device 1. In this disclosure, as a result of the initial scene identification by the edge device 1, text information describing what is shown in the frame image is extracted, and the text information, rather than the image information of the identified scene, is sent to the cloud server 3, thereby considering personal information and reducing the amount of communication.

[0059] Figure 5 is a functional block diagram of the image processing system 100. The image processing system 100 performs the functions shown in Figure 5, based on the first image processing program P1 of the edge device 1 and the second image processing program P32 of the cloud server 3. In the image processing system 100, the processing unit 10 of the edge device 1 functions as an information extraction unit 101, a primary scene identification unit 102, and an edge device side transmission unit 103, while the processing unit 30 of the cloud server 3 functions as a cloud side storage unit 301, a secondary scene identification unit 302, a selection operation unit 303, an image display control unit 304, and a scene information input unit 305.

[0060] In the first embodiment, the information extraction unit 101 receives frame images obtained by the camera 2 via the first communication unit 12. The information extraction unit 101 extracts visually obtained information from the acquired frame images in text format. The information extraction unit 101 uses a language model M11 to extract text-format information representing objects such as objects or people depicted in the frame images and their attributes. In other words, the language model M11 includes the functions of an object detection model and a recognition model that recognizes attributes. The information extraction unit 101 may also use the language model M11 to obtain information in text format that describes information inferred from the attributes, arrangement, etc., of objects depicted in the frame images. As the language model M11, a multimodal model capable of inputting images (for example, a VLM (Vision-Language Model)) is selected from the database 310 and stored in the storage unit 11. Furthermore, a language processing process using the language model M11 is deployed on the edge device 1.

[0061] The information extraction unit 101 instructs the language model M11, such as VLM, to write sentences or words describing people or objects that appear in the entire frame image, and obtains the text-format information output from the language model M11 as global information, which is summary information about the entire frame image. The information extraction unit 101 also instructs the language model M11, such as VLM, to write sentences or words describing people or objects that appear in a specific area of ​​the frame image, and obtains the text-format information output from the language model M11 as local information, which is information about a specific area within the frame image.

[0062] The information extraction unit 101 extracts summary information in text format as global information for the frame image, such as "Three people are walking," "Multiple trees with autumn leaves are visible," and "A specific animal is visible in the shade of a tree." The information extraction unit 101 also extracts local information in text format for a specific area or object within the frame image, such as "One person (50 years old or older, under 70 years old) is walking in a specific area, and a specific animal is in the shade of a tree beside the path they are walking on," and "Two people (01 is wearing a hat, 02 is wearing glasses) are walking."

[0063] The information extraction unit 101 extracts global information and local information for the frame image, along with the time information of the target frame image. In this example, the information extraction unit 101 mainly shows an example in which it extracts global information, local information for a single frame image, and the time information corresponding to that frame image from that single frame image. However, it is not limited to this, and the information extraction unit 101 may also extract global information, local information, and the time information corresponding to that frame image from multiple temporally consecutive frame images (i.e., a video).

[0064] The primary scene identification unit 102 determines whether or not the target frame image may contain an object desired by the user, based on the global or local information extracted by the information extraction unit 101. Similar to the secondary scene identification unit 302 described later, the primary scene identification unit 102 may include an LLM. That is, the primary scene identification unit 102 may input at least one of the global information and local information to the LLM along with text instructing it to analyze whether or not the object desired by the user may contain it, and then determine whether or not the corresponding frame image may contain the object desired by the user from the result of whether or not it may contain the object output from the LLM. If the primary scene identification unit 102 determines that the target frame image may contain the object to be viewed, it identifies the target frame image as an image that may be part of the scene desired by the user. The primary scene identification unit 102 stores the frame image corresponding to the identified scene in the storage unit 11, in association with the time information extracted by the information extraction unit 101. The primary scene identification unit 102 stores a predetermined amount of video before and after the identified scene, including the frame image corresponding to the identified scene, in the storage unit 11. The primary scene identification unit 102 may save the frame images that are consecutive in time as a single video if it has selected them. The primary scene identification unit 102 discards any frame images that it has not identified. If the processing unit 10, acting as the primary scene identification unit 102, determines from the output of the information extraction unit 101 that the object desired by the user is not visible, it terminates processing on the target frame image, does not perform any further processing on the target frame, and proceeds to process the next frame image.

[0065] The primary scene identification unit 102 stores the frame image and time information of the identified scene, along with the global and local information extracted from the target frame image by the information extraction unit 101, in the storage unit 11.

[0066] In the first embodiment, the edge device side transmission unit 103 transmits global and local information in text format corresponding to the frame image identified by the primary scene identification unit 102, and time information corresponding to the frame image, to the cloud server 3 via the second communication unit 13. The edge device side transmission unit 103 transmits the global and local information in text format, as well as the corresponding time information, in association with the identification data of the edge device 1. The time information is text information indicating the time corresponding to the frame image, the elapsed time from a specific time, or a count, etc.

[0067] The cloud-side storage unit 301 receives global information describing the outline of the identified frame image, local information describing information of a specific area within the frame image, corresponding time information, and identification data of the edge device 1, all transmitted from the edge device 1 by the edge device-side transmission unit 103. The cloud-side storage unit 301 stores the received information in the storage unit 31. No image information is transmitted from the edge device 1 to the cloud server 3.

[0068] The secondary scene identification unit 302 uses the global information, local information, and time information in text format stored in the storage unit 31 by the cloud-side storage unit 301 to identify a scene containing an object desired by the user according to detailed criteria, and stores the time information of the identified scene in the storage unit 31. The detailed criteria include criteria for attributes used to determine that an object is the object desired by the user in a frame image in which the desired object was detected by the edge-side primary scene identification unit 102. For example, if the primary scene identification unit 102 identifies a scene containing a person, the detailed criteria would be criteria for the attributes of the person. For example, if the primary scene identification unit 102 identifies a scene containing a person, the criteria for the secondary scene identification unit 302 would be such that it can identify a scene containing a person wearing specific clothing or a person with a specific facial orientation.

[0069] Other examples are acceptable as long as the criteria for the secondary scene identification unit 302 are more detailed than the criteria for the primary scene identification unit 102. The secondary scene identification unit 302 identifies the scene identified by the primary scene identification unit 102 of the edge device 1 in a way that increases the likelihood that the scene contains the object desired by the user. For example, if the primary scene identification unit 102 identifies the target frame image as containing an object of roughly the same size as a person and therefore a scene containing a person, the secondary scene identification unit 302 can determine that the scene contains a person with the desired attributes.

[0070] The secondary scene identification unit 302 may input either the global information or the local information corresponding to the frame image identified by the primary scene identification unit 102 to the LLM, along with an instruction to analyze whether or not a user-desired object is depicted, and then determine whether or not the corresponding frame image corresponds to a user-desired scene based on the result of whether or not it is depicted output from the LLM.

[0071] The secondary scene identification unit 302 may delete time information, global information, and local information stored by the cloud-side storage unit 301, except for the time information, global information, and local information corresponding to the scene identified by the secondary scene identification unit 302.

[0072] The selection operation unit 303 controls the operation unit 44 of the client 4 to accept the selection operation for a scene to be displayed on the display unit 43 of the client 4 connected to the cloud server 3, from among the scenes narrowed down by the processing of the secondary scene identification unit 302. The processing unit 30 accepts the selection operation on the web page displayed on the client 4 based on the Web server module included in the server program P31.

[0073] The image display control unit 304 controls the display unit 43 of the client 4 connected to the cloud server 3 to display the scenes narrowed down by the processing of the secondary scene identification unit 302. That is, the image display control unit 304 controls the display unit 43 of the client 4 to display the frame image corresponding to the user-desired scene identified by the secondary scene identification unit 302, which is extracted from the video stored in the memory unit 11 of the edge device 1. The processing unit 30 may redirect the frame image corresponding to the time information specified by the client 4 to the client 4, with the edge device 1 as the host. When images or video are sent directly from the edge device 1, it is possible to avoid storing data on the cloud server 3.

[0074] The scene information input unit 305 is a function that accepts input of scene information for setting the object to be detected, which serves as the basis for scene identification by the primary scene identification unit 102 and the secondary scene identification unit 302. The processing unit 30 accepts, via the client 4, in advance of the processing described later, the scene information input of the scene information input unit 305, which corresponds to the identification data of the target space, and indicates what kind of scene the user wants to view in the target space. The scene information may be, for example, keywords for the object to be detected, such as "person (human)" or "dog (dog)," as described above, or attributes such as "adult (adult)," "wearing a hat / cap (wearing a hat / cap)," or "wearing glasses (wearing glasses)." Based on the function of the scene information input unit 305, the processing unit 30 distributes the setting information to be referenced by the primary scene identification unit 102 of the edge device 1 and the setting information to be referenced by the secondary scene identification unit 302 of the cloud server 3 according to the received scene information.

[0075] The procedures for each function of the image processing system 100 shown in Figure 5 will be explained with reference to the flowchart. Figures 6 and 7 are flowcharts of an example of the scene identification processing procedure in the image processing system 100. Before the image processing system 100 starts operation, the scene information input unit 305 accepts information input for each target space to identify the desired scene, and when operation starts, the following processes are executed.

[0076] The processing unit 10 of the edge device 1 acquires a frame image from the camera 2 (step S111).

[0077] The processing unit 10, using the functions of the information extraction unit 101, inputs the acquired frame image to the language model M11 along with instructions for acquiring global information (step S112). In step S112, the processing unit 10 creates a sentence such as "Analyze what is depicted in this image" as an "instruction" for acquiring global information, and inputs it to the language model M11 along with the frame image. The processing unit 10 may create the sentence for the "instruction" to give to the language model M11, or it may read the sentence for the "instruction" from the standard sentences stored in the storage unit 11.

[0078] The processing unit 10 acquires text information, which is a sentence or word output from the language model M11 in response to the input in step S112, as global information (step S113).

[0079] The processing unit 10 extracts a specific area from the frame image acquired in step S111, along with instructions for acquiring local information, and inputs it to the language model M11 (step S114). In step S114, the processing unit 10 creates image data of a specific area set in the configuration data and a sentence such as "Analyze what is shown in the specified part of this image" as "instructions" for acquiring local information, and inputs them to the language model M11 along with the frame image. In step S114, the sentence of "instructions" given to the language model M11 may also be read from a source. In step S114, the processing unit 10 may specify the range using coordinates in the "instructions" instead of extracting an area from the frame image.

[0080] The processing unit 10 acquires text information, which is a sentence or word output from the language model M11 in response to the input in step S114, as local information (step S115).

[0081] The processing unit 10 determines whether all local information for the frame image acquired in step S111 has been acquired (step S116). If it is determined that not all information has been acquired (S116: NO), the processing unit 10 returns to step S114 to perform processing on other areas within the frame image.

[0082] If the processing unit 10 determines that all local information has been acquired (S116: YES), it terminates its processing as the information extraction unit 101.

[0083] The processing unit 10, using the functions of the primary scene identification unit 102, determines, based on the extracted global information and local information, whether or not the frame image acquired in step S111 may contain an object desired by the user (step S121).

[0084] In step S121, the processing unit 10 determines whether the text of the global information and local information matches a part of the text set in the configuration information.

[0085] If the processing unit 10 determines that the frame image may contain an object desired by the user (S121: YES), it stores global information, local information, and time information indicating the time the frame image was taken in the storage unit 31 (step S122). The primary scene identification unit 102 of the processing unit 10 terminates processing of the frame image acquired in step S111.

[0086] If, in step S121, it is determined that the frame image does not contain the object desired by the user (S121:NO), the processing unit 10 terminates processing of the acquired frame image. In this case (S121:NO), the acquired frame image does not need to be saved in the storage unit 11.

[0087] The processing unit 10 of the edge device 1, when a frame image corresponding to a scene identified by the primary scene identification unit 102 is stored in the storage unit 11 by the function of the edge device side transmission unit 103, transmits the global information and local information corresponding to that frame image, along with the corresponding time information, to the cloud server 3 (step S131). In step S131, the processing unit 10 also transmits the identification data of the edge device 1. This allows the cloud server 3, which supports multiple target spaces, to identify which target space the frame image was taken in.

[0088] On the cloud server 3 side, the processing unit 30 receives global information, local information, and time information transmitted from the edge device 1, along with the identification data of the edge device 1, through the functions of the cloud-side storage unit 301 (step S311). The processing unit 30 associates the received global information, local information, and time information with the identification data of the edge device 1 received together with this information, and stores (stores) them in the storage unit 31 (step S312), thus ending the reception and storage process.

[0089] Each time global information and local information are stored in the storage unit 31 by the function of the secondary scene identification unit 302, the processing unit 30 determines, based on the global information and local information, whether or not the corresponding frame image contains the scene desired by the user, using detailed criteria (step S321). In step S321, the processing unit 30 may make the determination using not only the target global information and local information, but also global information and local information corresponding to frame images that are chronologically earlier or later.

[0090] If the processing unit 30 determines, based on detailed criteria, that the object desired by the user is visible in the frame image corresponding to the global and local information of the target (S321: YES), it stores the corresponding time information in the storage unit 31 (step S322). If the processing unit 30 stores time information that is continuous in time (chronologically) in step S322, it may also store information about the period. The processing unit 30 stores the global or local information that formed the basis for the determination that the object is visible in the storage unit 31, associating it with the time information (step S323). However, the processing in step S323 means storing text for searching, so it is unnecessary if text-based searching as described in Figure 10 is not accepted.

[0091] If the processing unit 30 determines that the frame image corresponding to the target global information and local information does not contain the object desired by the user (S321: NO), it deletes the global information and local information stored in the storage unit 31 from the storage unit 31 by the cloud-side storage unit 301 (step S324).

[0092] According to the processing procedure shown in Figures 6 and 7, the frame images stored on the edge device 1 that correspond to the time information stored in the memory unit 31 become the frame images included in the scene identified by the secondary scene identification unit 302. The cloud server 3 stores the time information corresponding to the identified scene, as well as global and local information.

[0093] Figures 8 and 9 are explanatory diagrams of global and local information. Figures 8 and 9 schematically show an example of a frame image obtained from camera 2 installed on an outdoor walkway of the facility. In the frame image, dashed lines indicate grid lines that divide the image into multiple areas.

[0094] Figure 8 shows global information that provides an overview of the entire frame image. As shown in Figure 8, global information is output by language model M11 when the instruction "Analyze what is depicted in this image" is input to language model M11 along with the entire frame image. As an analysis result for the entire image, the global information will be expressed in text as "Many people are walking along a tree-lined street," as shown in the example in Figure 8.

[0095] Figure 9 shows an example of local information, where each of the four areas divided vertically and horizontally by grid lines is specified as a specific area within the frame image. In the example in Figure 9, local information is output from the language model M11 in response to the instruction "Analyze what is shown in the specified part of this image," which specifies a particular area of ​​the frame image. In the example in Figure 9, the local information obtained by specifying the upper left area shows clearer information about depth and positional relationships, such as "A person is walking in the shade of a tree." The local information obtained by specifying the upper right area shows that "Many trees with autumn leaves are visible," indicating that the upper right of the entire frame image is an area where trees, not people, are visible. The local information obtained by specifying the lower left area shows clear numerical information, such as "Three people are visible. A snake is visible in the tree in front of two other trees. The snake and one person are close together," and further details the positional relationship between the detected person and information about a small dangerous animal that was not mentioned in the global information. The local information obtained by specifying the lower right area shows clear numerical and detailed information, such as "One person is walking. There are three trees by the roadside."

[0096] As shown in Figure 9, by using local information as the analysis result for each area within the frame image, detailed information that clarifies the positional relationships of objects, etc., which was not included in the results of analyzing the entire frame image (global information), can be obtained in text format. From this information, the secondary scene identification unit 302 on the cloud server 3 can improve the accuracy of identifying scenes containing the user's desired objects without losing any information. For example, if a dangerous animal is the object to be detected, it may not be detectable from the global information, but the local information can identify the area within the frame image and detect whether or not a dangerous animal is present.

[0097] As shown in Figure 9, the primary scene identification unit 102 may not specify a particular area (upper left, upper right, lower left, lower right) and have the language model M11 analyze it, but rather may provide the language model M11 with the data of each of the four divided images of the frame image, along with instruction sentences that instruct the analysis of these images, and have the language model M11 perform the analysis.

[0098] As shown in Figures 8 and 9, the edge device 1 initially analyzes the image using LLM and VLM, and then uses text information to determine whether or not the user-desired object is visible in the image. This reduces the amount of data targeted for specific processing and the amount of data transmitted to the cloud server 3. However, when converted to text, the global information analyzed for the entire image, even if it objectively and accurately describes the overall image, may not necessarily focus on the person or object of interest relevant to the purpose of camera 2's installation. When the camera is installed for security purposes, such as detecting danger to people in and around the facility, the administrator of the target space can select and filter information while viewing the entire image, understanding its purpose. However, it is difficult for administrators to constantly monitor the image for extended periods to detect danger, and this can lead to human attention lapses or misjudgments. Therefore, it is necessary to be able to automatically and accurately identify scenes of interest. As shown in Figures 8 and 9, the edge device 1 of the image processing system 100 can reduce the amount of data used for specific processing or the amount of data sent to the cloud server 3 by using text information. Furthermore, by separating global information and local information and using the language model M11 to convert the events depicted in each frame into text, it is possible to more accurately identify scenes of interest.

[0099] Furthermore, as mentioned above, by distributing the scene identification process between edge device 1 and cloud server 3, the processing load on edge device 1, which has relatively few computing resources, can be reduced. Since text information is transmitted after the initial scene identification is performed on edge device 1, this reduces the amount of data transmitted while mitigating security risks and protecting personal information, compared to sending the captured frame images to cloud server 3.

[0100] Based on the time information stored in the storage unit 31 of the cloud server 3 by processing by the secondary scene identification unit 302, the display unit 43 of the client 4 can display a scene in which the user's desired object is shown. Figure 10 is a flowchart showing an example of the display control procedure in the image processing system 100. When a user (for example, an operator who centrally manages a chain store) accesses the cloud server 3 using the client 4, the processing unit 30 of the cloud server 3, as the selection operation unit 303 and the image display control unit 304, starts the following processing.

[0101] The processing unit 30 of the cloud server 3, acting as a selection operation unit 303, receives data from the client 4 to identify the target space (step S331). In step S331, the processing unit 30 accepts the user's account data, the identification data or name of the target space, etc. In step S331, the processing unit 30 may accept a selection from a list of the identification data of the edge device 1 that is authorized to access the account data used when the client 4 accessed the cloud server 3, and the corresponding identification data or name of the target space.

[0102] The processing unit 30 identifies the identification data of the edge device 1 corresponding to the target space identified by the received data (step S332), and reads the global information, local information, and time information associated with the identified edge device 1's identification data from the storage unit 31 (step S333). In step S333, the processing unit 30 may read consecutive time information together.

[0103] The processing unit 30, acting as a selection operation unit 303, receives a search word from the client 4 that corresponds to the object (viewing target) desired by the user (step S334). The processing unit 30, acting as an image display control unit 304, extracts global information or local information that matches the text corresponding to the search word from the global information and local information read in step S333 (step S335). A match between the global information or local information and the text of the search word may include cases where the global information or local information contains the search word, or where the global information or local information is a synonym for the search word.

[0104] The processing unit 30 specifies time information corresponding to the extracted global or local information and sends a request to acquire a thumbnail image corresponding to the frame image corresponding to the specified time information to the edge device 1 identified by the associated identification data (step S336). The processing unit 30 displays a list of thumbnail images sent in response to the acquisition request on the display unit 43 of the client 4 (step S337) and terminates the process.

[0105] In the processing procedure shown in Figure 10, steps S334-S335 related to searching may be omitted, and a list of thumbnail images corresponding to the time information read in step S333 may be displayed in step S337.

[0106] Figures 11 and 12 are explanatory diagrams showing examples of image display on client 4. Figures 11 and 12 show examples of screens provided by the Web server function of cloud server 3, displayed by the web browser included in client program P4, in response to access by operations on client 4. In the screen 430 shown in Figure 11, a list 431 of target spaces that the user operating client 4 is permitted to view is displayed. The list 431 includes an interface 432 with links to image display screens that display images saved for each target space. In the example in Figure 11, the list includes "Store (X Chain A Store)", "Store Surroundings (X Chain A Store)", "Store (X Chain B Store)", List 431 of "Store Surroundings (X Chain Store B)" is displayed.

[0107] Figure 12 shows an example of the image display screen 433. The image display screen 433 shown in Figure 12 is displayed when the interface 432 for "Outside the store (X chain A store)" is selected by the user using the operation unit 44 from the list 431 shown in Figure 11.

[0108] The image display screen 433 includes an area 434 that shows a list of thumbnails of scenes saved after being identified by the primary scene identification unit 102 and the secondary scene identification unit 302, and an area 435 that shows a list of images extracted from areas corresponding to the local information that formed the basis for identifying the scenes. When an image in area 435 is selected on the image display screen 433, the thumbnail of the original scene corresponding to that image is highlighted in area 434. Each image in area 434 has text indicating time information. When an operation operation is performed on each image in area 434 by the operation unit 44, the client 4 can retrieve the video containing the corresponding scene from the image information stored on the edge device 1 via the cloud server 3 and display it on the screen.

[0109] In the image processing system 100 of the first embodiment, instead of sending frame images and videos from the edge device 1 to the cloud server 3 for storage, only text information such as time information and local information is sent from the edge device 1 to the cloud server 3 for storage, thereby reducing the amount of communication. Furthermore, with the configuration of the image processing system 100 of the first embodiment, images of subjects captured in each space, especially customers, are not stored on the cloud server 3, and the user can display the images they need from the client 4. This makes it possible for users to view frame images or videos that can be judged with a higher degree of certainty to contain specific objects, while taking into consideration personal information.

[0110] As described above, the image processing system 100 of the first embodiment extracts global information, which is an overview of the entire frame image, local information, which is information about each area in the frame image, and time information corresponding to the frame image from one or a plurality of temporally consecutive frame images. This makes it possible to obtain detailed information about the content of the frame image (not only overview information about the entire frame image, but also information about each area in the frame image), and this detailed information can be used to accurately identify the scene desired by the user.

[0111] [Differentiation] In the first embodiment, the functions were divided between the edge device 1 and the cloud server 3 into a primary scene identification unit 102 and a secondary scene identification unit 302. However, the system is not limited to this configuration, and the cloud server 3 may also be used to identify scenes collectively as a single scene identification unit. Figure 13 is a functional block diagram of the image processing system 100 in a modified example.

[0112] In the modified example, the processing unit 10 of the edge device 1 functions as an information extraction unit 101 and an edge device-side transmission unit 103. The processing unit 10 transmits the global information and local information extracted by the information extraction unit 101 for each frame image to the cloud server 3 via the edge device-side transmission unit 103, regardless of whether or not the frame image from which this information was extracted may contain an object desired by the user.

[0113] In the modified example, the processing unit 30 of the cloud server 3 functions as a cloud-side storage unit 301, a scene identification unit 306, a selection operation unit 303, an image display control unit 304, and a scene information input unit 305. In the modified example, the scene identification unit 306 combines the functions of the primary scene identification unit 102 and the secondary scene identification unit 302 of the first embodiment, and a detailed explanation is omitted. The scene identification unit 306 uses the global information, local information, and time information of each frame image stored in the storage unit 31 to identify the scene desired by the user, and stores only the time information corresponding to the identified scene in the storage unit 31.

[0114] Even in the modified configuration, only text information is transmitted from edge device 1 to cloud server 3, contributing to consideration for personal information and reduction of communication volume. Furthermore, since detailed information about the frame image (not just an overview of the entire frame image, but also information about each area within the frame image) can be obtained, this detailed information can be used to accurately identify the scene desired by the user.

[0115] [Second Embodiment] In the first embodiment, as shown in Figure 9, local information is obtained for each of the four divided areas of the image. In the second embodiment, objects that should appear in the user-desired scene are detected first, and local information is obtained based on their bounding boxes. The configuration of the image processing system 100 in the second embodiment is the same as that of the image processing system 100 in the first embodiment, except for the detailed processing content when obtaining local information. Therefore, in the image processing system 100 of the second embodiment described below, components that are common with the image processing system 100 of the first embodiment are denoted by the same reference numerals, and detailed explanations are omitted.

[0116] Figure 14 is a functional block diagram of the image processing system 100 in the second embodiment. In the second embodiment, the information extraction unit 101 uses not only the language model M11 but also the detection learning model M12. In the second embodiment, the processing unit 10, as the information extraction unit 101, first uses the detection learning model M12 for user-desired objects to extract areas where objects may be present in the acquired frame image. The detection learning model M12 is selected from the person detector, head detector, face detector, animal detector, and specific equipment detector provided in the database 310 according to the user-desired object (person, animal, vehicle, or equipment, etc.) set for each target space, and stored in the storage unit 11. Furthermore, the object detection process using the detection learning model M12 is deployed to the edge device 1. The information extraction unit 101 obtains the detection result (coordinate data and accuracy of the area where the object is present) output from the detection learning model M12. In the second embodiment, if the information extraction unit 101 does not detect the object desired by the user using the detection learning model M12, it discards the target frame image without performing any further processing on it.

[0117] In the second embodiment, as in the first embodiment, the information extraction unit 101 instructs the language model M11 to output a sentence or word describing a person or object that is visible in the entire frame image, and obtains the text-format information output from the language model M11 as global information, which is summary information about the entire frame image. In the second embodiment, the information extraction unit 101 uses the detection results obtained from the detection learning model M12 to obtain local information in text format, which is information inferred from the attributes, placement, etc., of the frame image of the object desired by the user. Specifically, the information extraction unit 101 specifies a bounding box (range within the frame image) that indicates the area in which the object indicated by the detection result is visible, instructs the language model M11 to output a sentence or word describing the object desired by the user that is visible in the specified range, and obtains the text-format information output from the language model M11 as local information about the target object.

[0118] In the second embodiment as well, the information extraction unit 101 extracts summary information in text format as global information for the frame image, such as "Three people are walking," "Multiple trees with autumn leaves are visible," and "A specific animal is visible in the shade of a tree."

[0119] The information extraction unit 101 extracts local information about a specific area or object within the frame image. If "person" is specified as the object desired by the user, it extracts information in text format about the detected "person," such as: "The detected person (50 years old or older, under 70 years old) is walking towards the foreground. There are three trees nearby. A specific animal is in the shade of the trees.", "The detected person (30 years old or older, under 50 years old, wearing a hat) is walking towards the background. Another person (wearing glasses) is next to the person.", "The detected person (30 years old or older, under 50 years old, wearing glasses) is walking towards the background. Another person (wearing a hat) is next to the person.", and "The detected person (20 years old or older, under 30 years old) is walking towards the background. There is a dog next to the person." If there are multiple people, each is assigned an ID for identification, and if the same person appears across multiple frame images due to face recognition, the same ID is assigned to track them.

[0120] In the second embodiment, the information extraction unit 101 obtains local information by specifying a bounding box, as described above. However, the information extraction unit 101 may also perform object detection on the frame image in advance using a detection learning model M12, and input an image on the frame image in which grid lines dividing the frame image and bounding boxes of each object detected by the object detection are drawn, into a language model M11 such as VLM, to output local information in text format. In this way, local information can be extracted based on the positional relationship between the grid lines and the bounding boxes of each object. This makes it possible to clearly recognize areas within the frame image, for example, that "the person (50 years old or older, under 70 years old) detected in the upper left area divided by the grid lines is walking toward the foreground." In the second embodiment, the information extraction unit 101 may also extract global information, local information, and time information corresponding to the frame image from a plurality of temporally consecutive frame images (i.e., a video).

[0121] In the second embodiment, the primary scene identification unit 102, similar to the first embodiment, identifies a scene in the target frame image in which the user-desired object is visible, based on global or local information extracted by the information extraction unit 101. Because the detection learning model M12 is used, the primary scene identification unit 102 can identify a scene in slightly more detail than the detection criteria of the information extraction unit 101 in the first embodiment, based on attributes or the movement of the object to be detected (walking, standing still, etc.). The processing unit 10 stores the frame image corresponding to the identified scene in the storage unit 11, in association with the time information extracted by the information extraction unit 101, based on the functions of the primary scene identification unit 102. The primary scene identification unit 102 stores a predetermined amount of video before and after the identified scene, including the frame image corresponding to the identified scene, in the storage unit 11. If the processing unit 10, as the primary scene identification unit 102, determines from the output of the information extraction unit 101 that the user-desired object is not visible, it terminates processing on the target frame image, does not execute further processing on the target frame, and executes processing on the next frame image.

[0122] In the second embodiment, the functions of the other functions, namely the edge device side transmission unit 103, the cloud side storage unit 301, and the secondary scene identification unit 302, are the same as in the first embodiment, and a detailed explanation will be omitted.

[0123] Figure 15 is a flowchart showing an example of the processing procedure by the information extraction unit 101 of the second embodiment. The processing of the primary scene identification unit 102 and the edge device side transmission unit 103 of the processing unit 10 of the edge device 1 of the second embodiment is the same as the processing procedure shown in Figure 6 of the first embodiment, so a detailed explanation is omitted.

[0124] In the second embodiment, the processing unit 10 of the edge device 1 acquires a frame image from the camera 2 using the information extraction unit 101 (step S151).

[0125] The processing unit 10 inputs the acquired frame image to the language model M11 along with instructions for acquiring global information (step S152). In step S152, the processing unit 10 generates a sentence such as "Analyze what is depicted in this image" as an "instruction" for acquiring global information, and inputs it to the language model M11 along with the frame image.

[0126] The processing unit 10 acquires text information, which is a sentence or word output from the language model M11 in response to the input in step S152, as global information (step S153).

[0127] In the second embodiment, the processing unit 10 inputs the acquired frame image to the detection learning model M12 (step S154) and obtains the detection result (step S155). The processing unit 10 uses the functions of the information extraction unit 101 to determine whether or not a user-desired object has been detected in the acquired frame image using the detection learning model M12 (step S156).

[0128] If the processing unit 10 determines that the object desired by the user has not been detected (S156: NO), it terminates processing of the acquired frame image. The acquired frame image and global information do not need to be stored in the storage unit 11.

[0129] In step S156, if the processing unit 10 determines that a user-desired object has been detected in the acquired frame image (S156:YES), it inputs the frame image to the language model M11, specifying the region (bounding box) in which the object indicated by the detection result acquired in step S155 is depicted, along with instructions to acquire local information (step S157). In step S157, the processing unit 10 specifies the region in which the object is depicted as an "instruction" to acquire local information, generates text such as "Analyze what is depicted in the specified part of this image" for the frame image, and inputs it to the language model M11 along with the frame image.

[0130] The processing unit 10 acquires text information, which is a sentence or word output from the language model M11 in response to the input in step S157, as local information (step S158).

[0131] The processing unit 10 determines, based on the detection result obtained in step S155, whether or not it has obtained all of the local information for each object detected as a user-desired object (step S159). If it is determined that not all of the information has been obtained (S159: NO), the processing unit 10 returns to step S157 to perform processing on the other objects in the frame image.

[0132] If it is determined that all local information has been acquired (S159:YES), the information extraction unit 101 terminates its processing.

[0133] Figure 16 is an explanatory diagram of local information in the second embodiment. Figure 16 shows an example for the frame image shown in Figure 8 of the first embodiment. In Figure 16, the bounding boxes of the detected objects (people) are shown by dashed lines. In the frame image shown in Figure 16, five people (objects) from ID01 to ID05 are detected. The local information shows an example of local information obtained by specifying the area corresponding to each of the detected objects from ID01 to ID05. In the example in Figure 16, the local information is output from the language model M11 in response to an instruction such as "analyze what is visible around person ID01" for the objects detected in the frame image.

[0134] In the example in Figure 16, the local information obtained by specifying person (object) ID01 shows more detailed information such as the person's orientation and movement, stating, "Person (ID01) is walking down a tree-lined path towards the foreground." The local information obtained by specifying person ID02 shows information that clarifies the orientation, movement, and positional relationship with other people, stating, "Person (ID02) is walking down a tree-lined path towards the background. Person (ID03) is walking next to them. Person (ID04) is walking further back." The local information obtained by specifying person ID03 shows detailed information about a small dangerous animal that was not mentioned in the global information, and its positional relationship with the detected person, stating, "Person (ID03) is walking down a tree-lined path towards the background. Person (ID02) is walking next to them. Person (ID04) is walking further back. There is a tree nearby, and a snake is in the shade of the tree."

[0135] The local information obtained by specifying person ID04 shows that "Person (ID04) is walking towards the foreground on a tree-lined street. Two other people (ID02, ID03) are walking in the foreground." This clearly indicates the direction or movement of the person and their positional relationship with other people. The local information obtained by specifying person ID05 shows that "Person (ID05) is walking towards the background on a tree-lined street. They are in the shadow of a tree. Three other people (ID04, ID03, ID02) are visible in the foreground." This provides clearer information about the detected person's movement and their positional relationship with trees and other people.

[0136] As shown in Figure 16, the image processing system 100 of the second embodiment uses text information analyzed using the language model M11 based on objects detected in the frame image as local information. Detailed information, such as clarified positional relationships, that was not included in the results of analyzing the entire frame image can be obtained in text format.

[0137] In the image processing system 100 of the second embodiment, instead of sending frame images and videos from the edge device 1 to the cloud server 3 for storage, the amount of communication can be reduced by sending only text information such as time information and local information from the edge device 1 to the cloud server 3 for storage. With the configuration of the image processing system 100 of the second embodiment, images of subjects captured in each space, especially customers, can be stored on the cloud server 3, and the user can display the images they need from the client 4. Furthermore, in the image processing system 100 of the second embodiment, the scene can be identified more accurately by using local information that has been verbalized by focusing on the range in the frame image where the object desired by the user is detected. In this way, even in the second embodiment, it is possible to view frame images or videos that can be judged with a higher degree of certainty to contain a specific object, while taking into consideration personal information.

[0138] [Third Embodiment] In the third embodiment, the secondary scene identification unit 302 on the cloud server 3 side identifies the user-desired scene using LLM from the global information, local information, and time information stored in the storage unit 31. The configuration of the image processing system 100 in the third embodiment is the same as that of the image processing system 100 in the first or second embodiment, except for the processing content of the secondary scene identification unit 302. Therefore, among the image processing system 100 of the third embodiment shown below, components that are common with the image processing system 100 of the first or second embodiment are denoted by the same reference numerals and detailed descriptions are omitted.

[0139] Figure 17 is a functional block diagram of the image processing system 100 in the third embodiment. In the third embodiment, the secondary scene identification unit 302 on the cloud server 3 side identifies a scene using the language model M31. Specifically, the secondary scene identification unit 302 uses the language model M31 (LLM) to identify the time information of a frame image containing an object desired by the user, from the text-format global information or local information for each frame image stored in the storage unit 31 by the cloud-side storage unit 301. The secondary scene identification unit 302 may identify a scene from the global information or local information separately, or it may use a combination of global information and local information corresponding to the same time information. The secondary scene identification unit 302 provides the LLM with the global information and / or local information for each stored time information (frame image) and a text-format instruction to find the scene desired by the user, and identifies the scene using the response from the LLM.

[0140] In the third embodiment, the edge device 1 executes the processing procedure shown in Figure 6, or the processing procedure shown in Figure 15 of the second embodiment, similar to the first embodiment. In the third embodiment, the cloud-side storage unit 301 of the cloud server 3 also executes the processing procedures shown in Figures 6 and 7. In contrast, the secondary scene identification unit 302 executes the following processing procedure.

[0141] Figure 18 is a flowchart showing an example of the processing procedure of the secondary scene identification unit 302 in the third embodiment. The processing unit 30 of the cloud server 3 performs the following processing in parallel with the processing procedure of the edge device 1, based on the function of the secondary scene identification unit 302.

[0142] The processing unit 30 executes the following process each time global information and local information are stored in the storage unit 31. The processing unit 30 may also execute the following process each time frame images for a predetermined time, such as 10 minutes, 30 minutes, 1 hour, 12 hours, or 24 hours, are stored in the edge device 1, or it may execute the following process each time processing is performed on the frame images in the edge device 1.

[0143] The processing unit 30 reads global information and local information associated with time information stored in the storage unit 31 using the function of the secondary scene identification unit 302 (step S341). The processing unit 30 creates an instruction statement for the LLM (Language Model M31) to find a scene in which the user-desired object is shown (step S342), and inputs the instruction statement to the LLM (Language Model M31) to search for the scene, using the read global information and local information as the target (step S343).

[0144] In step S342, the processing unit 30 creates a natural language instruction, for example, "Based on the attached text information, find a scene in which a person is in a dangerous situation and tell me the time information to identify the scene." The processing unit 30 may also perform a search targeting a scene (a series of consecutive frame images) that spans a certain period of time, rather than individual frame images, such as "Based on the attached text information, find a scene in which a person is in the same place for more than 5 minutes and tell me the time information to identify the scene." As described above, when performing a process to determine whether or not multiple frame images (stored in the memory unit 31) correspond to a scene desired by the user, it is desirable that the secondary scene identification unit 302 inputs the global information, local information, and time information in text format for the multiple frame images read out into the LLM and identifies the frame image corresponding to the scene desired by the user (outputs the time information of the frame image corresponding to the scene desired by the user). Furthermore, as described above, when the process of identifying whether or not multiple frame images correspond to a scene desired by the user is performed collectively, the secondary scene identification unit 302 may input only the local information and time information in text format for these frame images to the LLM to identify the frame image corresponding to the scene desired by the user.

[0145] The processing unit 30 obtains a response from the LLM (Language Model M31) (step S344). From the obtained response, the processing unit 30 uses the function of the secondary scene identification unit 302 to identify the time information of the scene desired by the user (step S345). The processing unit 30 stores the identified time information in the storage unit 31 (step S346) and terminates the scene identification process. The processing unit 30 may keep the global and local information corresponding to the time information identified in step S345 for retrieval purposes, or it may delete it. The processing unit 30 may delete time information other than the time information identified in step S345, as well as the corresponding global and local information, from the storage unit 31.

[0146] In the image processing system 100 of the third embodiment, the function of the secondary scene identification unit 302 that performs the above-described processing (in particular, inputting global information, local information, and time information in text format for multiple consecutive frame images into the LLM to identify the frame image corresponding to the user-desired scene) allows the system to search using the LLM for frame images corresponding to global information and / or local information using synonyms, or frame images corresponding to scenes that evoke those words in context, rather than searching for global information and local information that contain words indicating the conditions of the user-desired scene that have been set. This increases the likelihood of accurately identifying the user-desired scene.

[0147] In the image processing system 100 of the third embodiment, only text information, which includes time information, local information, and global information, is transmitted from the edge device 1 to the cloud server 3, and no frame images or videos are stored on the cloud server 3, thus respecting personal information. Since information in text format is transmitted from the edge device 1 to the cloud server 3, the amount of communication can be significantly reduced compared to transmitting image information. While it is highly accurate to identify scenes containing user-desired objects from a group of frame images on the cloud server 3, which has abundant computing resources, analysis using LLM in the image processing system 100 of the third embodiment allows for accurate identification of user-desired scenes while prioritizing respect for personal information and reducing communication volume.

[0148] The embodiments disclosed above are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims, and all modifications within the meaning and scope equivalent to the claims are included. [Explanation of Symbols]

[0149] 100 Image Processing Systems 1. Edge devices 10 Processing Unit 101 Information extraction part 102 Primary Scene Identification Section 11 Storage section P1 First Image Processing Program 3. Cloud Server 30 Processing Unit 302 Secondary Scene Identification Section 304 Image display control unit 305 Scene Information Input Section 306 Scene Identification Section 31 Storage section P32 Second stroke processing program

Claims

1. Information extraction means for extracting local information, which is information about a predetermined area or each object in a frame image, from one or a plurality of temporally consecutive frame images, The system includes a scene identification means that identifies a scene desired by the user based on the local information extracted by the information extraction means, The local information is created by dividing the frame image into multiple areas and based on information obtained from at least one of the divided areas. The aforementioned local information is, This is text-formatted information obtained by inputting an image obtained by dividing the aforementioned frame image into multiple areas using grid lines into a VLM (Vision-Language Model). An image processing system that obtains text-formatted information by inputting an image into a VLM (Visual Language Machine) in which the grid lines and the bounding boxes of each object detected by object detection performed on the frame image in advance are drawn on the frame image.

2. The aforementioned local information is in text format, The image processing system according to claim 1, wherein the scene identification means identifies a user-desired scene based on the local information in text format.

3. The image processing system according to claim 2, wherein the scene identification means inputs the local information in text format into an LLM (Large Language Model) to identify a frame image corresponding to the user-requested scene.

4. Information extraction means for extracting global information, which is summary information about the entire frame image, and local information, which is information about a predetermined area or each object in the frame image, from one or a series of temporally consecutive frame images. The system includes a scene identification means that identifies a scene desired by the user based on the global information and local information extracted by the aforementioned information extraction means, The local information is created by dividing the frame image into multiple areas and based on information obtained from at least one of the divided areas. The aforementioned local information is, This is text-formatted information obtained by inputting an image obtained by dividing the aforementioned frame image into multiple areas using grid lines into a VLM. An image processing system that obtains text-formatted information by inputting an image into a VLM (Visual Language Machine) in which the grid lines and the bounding boxes of each object detected by object detection performed on the frame image in advance are drawn on the frame image.

5. The global information and the local information are in text format. The image processing system according to claim 4, wherein the scene identification means identifies a user-desired scene based on the global information in text format and the local information.

6. The image processing system according to claim 5, wherein the global information is text-formatted information obtained by inputting the frame image into the VLM.

7. The image processing system according to claim 5, wherein the scene identification means inputs the global information and the local information in text format into the LLM to identify a frame image corresponding to the user-requested scene.

8. The image processing system according to claim 6, wherein the scene identification means inputs the global information and the local information in text format into the LLM to identify a frame image corresponding to the user-requested scene.

9. The image processing system according to claim 7 or 8, wherein the scene identification means adds the global information in text format and the local information and inputs the result into the LLM.

10. The image processing system according to claim 1, wherein the local information is extracted based on the positional relationship between the grid lines and the bounding boxes of each object.

11. The system further includes a scene information input means for inputting scene information, which is information about the user-desired scene, The image processing system according to any one of claims 1 to 8 and 10, wherein the scene identification means identifies a scene corresponding to the scene information input by the scene information input means.

12. The image processing system according to any one of claims 1 to 8 and 10, further comprising an image display control means for extracting a frame image corresponding to a user-desired scene identified by the scene identification means from a video containing the frame image and controlling it to be displayed on a display means.

13. The aforementioned image processing system includes a system on the edge device side and a system on the cloud side. The system on the edge device side includes the information extraction means, The cloud-side system is an image processing system according to any one of claims 1 to 8 and 10, comprising the scene identification means.

14. Computers, Information extraction means for extracting local information, which is information about a predetermined area or each object in a frame image, from one or a plurality of temporally consecutive frame images, and Based on the local information extracted by the aforementioned information extraction means, it functions as a scene identification means to identify a scene desired by the user. The local information is created by dividing the frame image into multiple areas and based on information obtained from at least one of the divided areas. The aforementioned local information is, This is text-formatted information obtained by inputting an image obtained by dividing the aforementioned frame image into multiple areas using grid lines into a VLM. An image processing program, which is text-formatted information obtained by inputting an image into a VLM (Visual Language Machine) in which the grid lines and the bounding boxes of each object detected by object detection performed on the frame image in advance are drawn on the frame image.

Images