Image processing apparatus, image processing method, and program

Abstract

To appropriately create a thumbnail image representing an imaging position of an imaging apparatus.SOLUTION: An image processing apparatus has: first acquisition means that acquires an image created by imaging a first imaging position as a thumbnail image representing the first imaging position; second acquisition means that acquires environmental information that is information obtained from a picked-up image obtained by imaging a second imaging position different from the first imaging position, or information obtained from settings when the picked-up image was imaged; and processing means that performs processing of the thumbnail image on the basis of at least the environmental information acquired by the second acquisition means.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] There are systems that continuously broadcast fixed-point video captured by a network camera to an external location for extended periods of time. In such systems, in the event of an emergency or special event, it is necessary to control the pan, tilt, and zoom of the network camera to move it to a preset position (hereinafter referred to as "preset operation"). In such cases, the person who registers the preset position may be different from the person who issues the preset operation, so a clear thumbnail image is required that makes it easy to imagine the video captured at the destination preset position. Patent Document 1 discloses performing a preset operation at a specific time and updating the thumbnail image. Patent Document 2 discloses switching thumbnail images according to the time of day. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-164885 [Patent Document 2] Japanese Patent Application Laid-Open No. 2008-22260 Summary of the Invention [Problem to be solved by the invention]

[0004] If time passes after a thumbnail image is captured, the surrounding brightness, the number of people and vehicles, and other factors may change, and the thumbnail image may no longer reflect the current imaging environment. The method disclosed in Patent Document 1 requires a preset operation to update the thumbnail image, making it impossible to continuously deliver fixed-point video. The method disclosed in Patent Document 2 does not necessarily switch to a thumbnail image that is appropriate for the current imaging environment.

[0005] An object of the present invention is to appropriately generate thumbnail images that represent the imaging positions of an imaging device. [Means for solving the problem]

[0006] The image processing device according to the present invention is characterized by having a first acquisition means that acquires an image generated by capturing an image at a first imaging position as a thumbnail image representing the first imaging position, a second acquisition means that acquires environmental information that is information obtained from an image captured at a second imaging position different from the first imaging position or information obtained from settings when the image was captured, and a processing means that performs processing to process the thumbnail image based on at least the environmental information acquired by the second acquisition means. [Effects of the Invention]

[0007] According to the present invention, it is possible to appropriately generate thumbnail images that represent the imaging positions of an imaging device. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of an imaging system. [Figure 2] FIG. 2 illustrates an example of the hardware configuration of each device. [Figure 3] FIG. 2 is a diagram illustrating an example of the functional configuration of each device. [Figure 4] 10 is a flowchart illustrating a process executed by the imaging device. [Figure 5] 10 is a flowchart illustrating a process executed by the image processing device. [Figure 6] 10 is a flowchart showing thumbnail image processing processing. [Figure 7] FIG. 10 is a diagram showing an example of a screen displayed on an external terminal. [Figure 8] FIG. 10 is a diagram showing an example of a screen displayed on an external terminal. [Figure 9] 10 is a flowchart showing thumbnail image processing processing. [Figure 10]FIG. 10 is a diagram showing an example of a screen displayed on an external terminal. [Figure 11] FIG. 2 is a diagram illustrating an example of a functional configuration of an image processing apparatus. [Figure 12] 10 is a flowchart showing thumbnail image processing processing. [Figure 13] FIG. 10 is a diagram showing an example of a screen displayed on an external terminal. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention.

[0010] [Embodiment 1] 1 shows an example of the overall configuration of an imaging system 100 according to this embodiment. The imaging system 100 includes an imaging device 110, an image processing device 120, and an external terminal 130. The imaging device 110, the image processing device 120, and the external terminal 130 are connected via a network 140. The imaging device 110, the image processing device 120, and the external terminal 130 transmit and receive data to and from each other via the network 140.

[0011] The imaging device 110 is a network camera capable of changing the imaging direction and zoom magnification by changing the pan position, tilt position, and zoom position (hereinafter referred to as PTZ). The imaging device 110 delivers an image captured at a specified PTZ position as a video to an external device via the network 140. The PTZ position refers to the PT position and zoom lens position of the imaging device 110. Note that the imaging device 110 may be capable of changing at least one of the pan position (pan angle), tilt position (tilt angle), and zoom position (zoom magnification). The PTZ position is an example of an imaging position (imaging direction). The imaging position here does not correspond to the position where the imaging device 110 is installed, but corresponds to the position of the subject imaged by the imaging device 110. In other words, the imaging position here can also be said to be the imaging direction and imaging angle of view of the imaging device 110.

[0012] A PTZ position is registered by a user operation. An image captured at a registered PTZ position is associated with that PTZ position as a thumbnail image. Note that this thumbnail image may be the captured image itself, or may be a reduced-size version of the captured image. In either case, the thumbnail image referred to here is a thumbnail image based on an image captured by the imaging device 110 before processing, which will be described later, is performed. Furthermore, imaging parameters and device information at the time of capturing the image are associated with the thumbnail image. The imaging parameters include, for example, setting values ​​such as white balance, exposure, and color temperature. Device information includes, for example, time information, date and time information, and operating time. The imaging device 110 transmits various information including the thumbnail image to the image processing device 120.

[0013] The image processing device 120 is an information processing device such as a personal computer, a server device, a tablet device, or a mobile terminal. The image processing device 120 receives information including thumbnail images and information about the current imaging environment from the imaging device 110 via the network 140. The image processing device 120 then processes the thumbnail images based on changes in imaging parameters and device information since the thumbnail images were captured, and then transmits the processed thumbnail images to the imaging device 110.

[0014] The external terminal 130 includes a display unit 226 (FIG. 2(c)) that presents various information to the user, and an input unit 225 (FIG. 2(c)) that accepts user operations. The external terminal 130 displays, on the display unit 226, video delivered from the imaging device 110 and thumbnail images linked to PTZ positions registered in the imaging device 110. The external terminal 130 generates a signal corresponding to the user operation accepted by the input unit 225 as operation information and transmits it to the imaging device 110. In response to the operation information received from the external terminal 130, the imaging device 110 changes the imaging direction and zoom magnification, adjusts camera settings such as image quality, registers a PTZ position, selects a PTZ position, and so on.

[0015] Network 140 is a network such as the Internet or an IP network. Network 140 is configured, for example, by multiple routers, switches, and cables conforming to a communication standard such as Ethernet (registered trademark). Network 140 may also be configured by a wired local area network (LAN), a wireless LAN, or a wide area network (WAN).

[0016] In the present embodiment, the imaging device 110, the image processing device 120, and the external terminal 130 are described as being separate entities, but this is not limiting. For example, the imaging device 110 and the image processing device 120 may be configured as an integrated unit, or the imaging device 110 and the external terminal 130 may be configured as an integrated unit. Furthermore, the imaging device 110, the image processing device 120, and the external terminal 130 may all be configured as an integrated unit. Furthermore, in the present embodiment, the imaging device 110 is described as being one unit, but two or more units may be used.

[0017] The hardware configuration of each device included in the imaging system 100 will be described with reference to Fig. 2. Fig. 2(a) shows an example of the hardware configuration of an imaging device 110 according to this embodiment. The imaging device 110 includes an imaging unit 201 , an image processing unit 202 , a driving unit 203 , a control unit 204 , a storage unit 205 , a communication unit 206 , and an input unit 207 .

[0018] The imaging unit 201 is composed of a lens barrel equipped with imaging lenses including a focus lens, a zoom lens, etc., an image sensor, and a mechanical drive system and circuits that drive them, and focuses light toward an imaging element (not shown). The imaging element is composed of a two-dimensional image sensor such as a CCD or CMOS, and photoelectrically converts the light focused on the light receiving surface and outputs the resulting electrical signal to the image processing unit 202.

[0019] The image processing unit 202 generates image data by performing image processing such as noise removal or gamma correction on the electrical signals converted by the imaging unit 201, and outputs the generated image data as a captured image to the control unit 204. The image processing unit 202 also outputs imaging parameters such as exposure settings and device information such as date and time information to the control unit 204 along with the captured image. The image processing unit 202 can also process control commands received from the control unit 204. For example, the image processing unit 202 adjusts the image quality in accordance with a control command received from the control unit 204 that instructs adjustment of the image quality.

[0020] The driving unit 203 performs an operation to change the PTZ. The driving unit 203 is composed of a mechanical driving system (not shown), a driving motor, and a sensor for identifying the PTZ position. The driving unit 203 drives a driving system that moves the optical axis of the imaging device 110 in the pan direction and tilt direction, and drives a driving system that moves the zoom lens of the imaging unit 201 in the optical axis direction, in accordance with a PTZ driving command obtained from the control unit 204. Furthermore, the driving unit 203 identifies the PTZ position using a sensor for identifying the PTZ position, and outputs the identified PTZ position to the control unit 204. In this embodiment, a photointerrupter (hereinafter, PI) is used as the sensor for identifying the PTZ position, but the present invention is not limited to this.

[0021] The PTZ position is identified by the following method. First, the driving unit 203 calculates the PT position from the PT driving amount and the PI, and identifies the Z position from the driving amount of the zoom lens. The control unit 204 stores the identified PTZ position in the storage unit 205. In this embodiment, the driving unit 203 identifies the PTZ position using the PI, but the PTZ position may be identified by other methods.

[0022] In addition, in this embodiment, the imaging device 110 has a PT function, but it may be PT driven using an external device such as an external camera platform. In this case, the driving unit 203 outputs a PT drive command to the external device, and the output PT drive command is stored in the storage unit 205 as a PT position.

[0023] The control unit 204 is a CPU (Central Processing Unit) and controls each unit of the imaging device 110. The control unit 204 acquires various control commands according to operation information received from the external terminal 130 and input information input from the input unit 207. When the control unit 204 acquires a PTZ control command, the control unit 204 generates a PTZ drive command and outputs it to the drive unit 203, thereby controlling the drive unit 203. The control unit 204 also stores the PTZ position acquired from the drive unit 203, information about the lens attached to the imaging unit 201, and the like in the storage unit 205.

[0024] The control unit 204 also controls the display of a UI screen for various settings and various instructions. This UI screen is displayed on the display unit 226 (FIG. 2(c)) of the external terminal 130. An image being captured at the current PTZ position is displayed on this UI screen. In response to a user request, slider bars for instructing changes to the capturing direction and zoom magnification, camera settings such as the AutoFocus function and image quality, and buttons for instructing registration of the current PTZ position are also displayed. The control unit 204 controls the display of this UI screen and acquires various control commands in accordance with user operation information received from the external terminal 130 via the communication unit 206. Note that this UI screen may be displayed on the display unit 216 (FIG. 2(b)) of the image processing device 120. In this case, the control unit 204 acquires various control commands in accordance with the operator's operation information received from the image processing device 120.

[0025] The storage unit 205 has a ROM (non-volatile memory), a RAM (volatile memory), and an HDD (hard disk drive). The RAM functions as a work area for the control unit 204 and a temporary storage area for data. The ROM and HDD store various data such as various programs, various setting information, and images captured by the imaging unit 201. The storage unit 205 saves PTZ positions registered by the user as preset information.

[0026] As shown in Table 1 below, the preset information is data in which a preset ID for identifying the preset information is associated with a PTZ position, a thumbnail image as an image captured at that PTZ position, and environmental feature information at the time the thumbnail image was captured. In this embodiment, the environmental feature information is information obtained from settings such as device information and imaging parameters output from the image processing unit 202. Note that the environmental feature information is not particularly limited and may be internal system information (for example, lens information) of the imaging device 110 or information obtained by image analysis of the captured image output from the image processing unit 202.

[0027] [Table 1]

[0028] The communication unit 206 is a communication interface for communicating with external devices connected to the network 140. The control unit 204 receives user operation information from the image processing device 120 or the external terminal 130 via the network 140 using the communication unit 206, and acquires various control commands in accordance with the received operation information. Furthermore, the control unit 204 uses the communication unit 206 to transmit preset information read from the storage unit 205 and information on the current imaging environment (for example, captured image, device information, and imaging parameters output from the image processing unit 202) to the image processing device 120. The control unit 204 also uses the communication unit 206 to receive processed thumbnail images from the image processing device 120.

[0029] The input unit 207 is an input device for inputting input information in response to an operation by the person taking the image. The input unit 207 may accept input wirelessly from an infrared remote control or the like, or may accept input from a hardware switch of the image capturing device 110. The input information accepted by the input unit 207 is also analyzed by the control unit 204 and acquired as various control commands, in the same way as when the input information is received using the communication unit 206.

[0030] 2(b) shows an example of the hardware configuration of the image processing device 120 according to this embodiment. The image processing device 120 includes a CPU 211, a RAM 212, a ROM 213, a HDD 214, an input unit 215, a display unit 216, and a communication unit 217. These units are connected via a bus 218.

[0031] The CPU 211 controls the entire image processing device 120. The RAM 212 functions as a work area for the CPU 211 and a temporary data storage area. The ROM 213 stores various programs and various setting information. The HDD 214 stores various data such as preset information received from the imaging device 110 and information related to the imaging environment. The CPU 211 uses the RAM 212 as work memory to execute various programs stored in the ROM 213, etc. The input unit 215 is a device for inputting operation information from an operator. The display unit 216 is a display device such as an LCD monitor or an organic EL panel, and displays a screen that presents the results of various processes to the operator of the image processing device 120. The communication unit 217 is a communication interface for communicating with external devices connected to the network 140.

[0032] 2(c) shows an example of the hardware configuration of the external terminal 130 according to this embodiment. The image processing device 120 has a CPU 221, a RAM 222, a ROM 223, a HDD 224, an input unit 225, a display unit 226, and a communication unit 227. These units are connected via a bus 228.

[0033] The CPU 221 controls the entire external terminal 130. The RAM 222 functions as a work area for the CPU 221 and a temporary data storage area. The ROM 223 and HDD 224 store various programs and various data. The CPU 221 uses the RAM 222 as a work memory to execute various programs stored in the ROM 223, HDD 224, etc. The input unit 225 is an input device that accepts user operations on a UI screen displayed on the display unit 226. The display unit 226 is a display device such as an LCD monitor or an organic EL panel, and displays a UI screen provided by the imaging device 110. The communication unit 227 is a communication interface for communicating with external devices connected to the network 140.

[0034] FIG. 7 shows a thumbnail image display screen as an example of a UI screen displayed on the display unit 226. In the example shown in FIG. 7, a thumbnail image display screen 700 displays a video 701 being captured by the imaging device 110 at the current PTZ position, and thumbnail images 702 and 703 associated with the PTZ positions of two registered preset information. Note that image 704 indicates that no thumbnail image exists because no preset information has been registered. When the user selects thumbnail images 702 and 703, the external terminal 130 determines that the PTZ position corresponding to the thumbnail image has been selected, and transmits information for identifying the selected PTZ position (e.g., a preset ID) to the imaging device 110 as operation information. The imaging device 110 performs an operation to change the PTZ based on the selected PTZ position in accordance with the received operation information. As a result, a video at the PTZ position corresponding to the selected thumbnail image is distributed.

[0035] Next, the functional configurations of the imaging device 110 and the image processing device 120 will be described with reference to Fig. 3. Fig. 3(a) shows an example of the functional configuration of the imaging device 110 according to this embodiment. The control unit 204 executes a program stored in the storage unit 205 or the like, thereby realizing the functions of each unit shown in Fig. 3(a). The preset management unit 311 comprehensively performs processing related to preset information. Specifically, in response to an instruction command instructing registration of preset information, the preset management unit 311 stores in the storage unit 205 preset information that is a set of the current PTZ position, a thumbnail image representing the PTZ position, and environmental feature information at the time the thumbnail image was captured. The preset management unit 311 also updates the thumbnail image and the environmental feature information. Furthermore, when the communication unit 206 receives an edited thumbnail image from the image processing device 120, the preset management unit 311 updates the thumbnail image displayed on the display unit 226 of the external terminal 130 with the received edited thumbnail image.

[0036] The environmental feature extraction unit 312 extracts environmental feature information from the device information and imaging parameters output from the image processing unit 202, information obtained by image analysis of the captured image output from the image processing unit 202, and system information (e.g., lens information) within the imaging device 110. The environmental feature information is provided to the preset management unit 311 and the environmental information analysis unit 313. For example, the environmental feature extraction unit 312 extracts, as environmental feature information, exposure settings, color temperature, color space (RGB, HSV), the type and amount of objects reflected in the image, and lens information used. Note that these are merely examples and are not limiting. For example, the environmental feature extraction unit 312 may extract feature quantities within an image as environmental feature information using a classification algorithm trained by machine learning to identify objects such as people and vehicles, weather, changes in water depth, and the like. In other words, the presence or absence of people or vehicles, the number of people, and the like may be used as environmental feature information. Information such as whether it is raining or not may also be used as environmental feature information. In any case, the environmental characteristic information may be information obtained from a captured image or information obtained from the settings when the captured image was captured. The environmental information analysis unit 313 compares the current environmental feature information extracted by the environmental feature extraction unit 312 with the environmental feature information at the time the thumbnail image was captured, and detects any change of a predetermined value or more. If a change of the environmental feature information of a predetermined value or more is detected, the environmental information analysis unit 313 instructs the preset management unit 311 to update the thumbnail image and environmental feature information.

[0037] 3(b) shows an example of the functional configuration of the image processing device 120 according to this embodiment. The CPU 211 uses the RAM 212 as a work memory and executes a program stored in the ROM 213 or the like, thereby realizing the functions of each unit shown in FIG. 3(b). The imaging device control unit 321 exchanges various types of information with the imaging device 110 using the communication unit 217 in order to update thumbnail images linked to PTZ positions registered in the imaging device 110. Specifically, the imaging device control unit 321 acquires preset information managed by the imaging device 110 and information related to the current imaging environment of the imaging device 110. The information related to the current imaging environment of the imaging device 110 includes an image being captured at the current PTZ position, device information, imaging parameters, internal system information, etc. Furthermore, the imaging device control unit 321 transmits to the imaging device 110, together with the thumbnail image processed by the thumbnail image processing unit 324, an instruction command instructing the imaging device 110 to update the thumbnail image to the processed thumbnail image.

[0038] The environmental feature extraction unit 322 extracts environmental feature information from information about the current imaging environment of the imaging device 110 acquired from the imaging device 110, for comparison with environmental feature information included in the preset information. The environmental feature extraction unit 322 may extract environmental feature information by performing image analysis on each of the thumbnail images included in the preset information and the captured image being captured at the current PTZ position. The image analysis here may use a classification algorithm that has been trained by machine learning to identify objects such as people and vehicles, weather, changes in water depth, and the like. The environmental information analysis unit 323 monitors the environmental feature information extracted by the environmental feature extraction unit 322 and detects a change in the environmental feature information that is equal to or greater than a predetermined value. When a change in the environmental feature information that is equal to or greater than the predetermined value is detected, the environmental information analysis unit 323 instructs the thumbnail image processing unit 324 to process the thumbnail image, or requests the imaging device control unit 321 to instruct replacement with the thumbnail image at the time of registration (thumbnail image before processing).

[0039] The thumbnail image processing unit 324 processes the thumbnail image included in the preset information based on the difference between the environmental feature information included in the preset information and the current environmental feature information extracted by the environmental feature extraction unit 322. For example, if the brightness has fluctuated to be brighter, the brightness of the thumbnail image is corrected.

[0040] 4 is a flowchart showing the process of saving thumbnail images by the imaging device 110 according to this embodiment. The process of this flowchart is realized by the control unit 204 executing a program stored in the storage unit 205 or the like. In the following description of the flowchart, each process (step) is denoted by adding an S to the beginning, and the process (step) is notated in detail.

[0041] First, the imaging device 110 displays the video being captured at the current PTZ position on the display unit 226 of the external terminal 130. Then, in response to a request from the external terminal 130, the imaging device 110 displays, on the display unit 226 of the external terminal 130, a slider bar for instructing a change in the imaging direction and zoom magnification, and a button for instructing registration of the current PTZ position. When the slider bar for instructing a change in the imaging direction and zoom magnification is operated via the input unit 225, the external terminal 130 transmits operation information corresponding to the operation content to the imaging device 110. The imaging device 110 performs an operation to change the PTZ in accordance with the received operation information. This changes the imaging direction and zoom magnification of the displayed video. Furthermore, when a button for instructing registration of preset information at the current PTZ position is pressed via the input unit 225, the external terminal 130 transmits an instruction command to the image processing device 120 instructing registration of the preset information.

[0042] In S400, when the preset management unit 311 receives user operation information from the external terminal 130 via the communication unit 206, it determines whether or not the user has performed a preset registration operation based on the received operation information. Here, the preset management unit 311 determines whether or not an instruction command instructing registration of preset information has been received from the external terminal 130. If the preset management unit 311 detects that a preset registration operation has been performed, the process proceeds to S402, and if the preset management unit 311 does not detect that a preset registration operation has been performed, the process proceeds to S401.

[0043] In S401, the preset management unit 311 determines whether or not the end flag of this flowchart is set. If the end flag is set, the processing of this flowchart ends, and if the end flag is not set, the processing returns to S400. This end flag is set by an external interrupt or the like. In S402, the preset management unit 311 saves the current PTZ position identified by the driving unit 203 in the storage unit 205 and sets it as preset information. At the same time, the preset management unit 311 saves the captured image output from the image processing unit 202 as a thumbnail image, including it in the preset information.

[0044] In S403, the environment feature extraction unit 312 extracts environment feature information from the device information, imaging parameters, and the like output from the image processing unit 202 simultaneously with the thumbnail image (captured image) saved in S402. The environment feature extraction unit 312 may also extract environment feature information from information obtained by image analysis of the thumbnail image (captured image) saved in S502, and from internal system information of the imaging device 110. The preset management unit 311 then saves the extracted environment feature information by including it in the preset information. Note that this step may be skipped, and the image processing device 120 may execute processing to extract environment feature information from the thumbnail image. In S404, the preset management unit 311 determines whether or not the end flag of this flowchart is set. If the end flag is set, the process of this flowchart ends, and if the end flag is not set, the process proceeds to S405.

[0045] In S405, the preset management unit 311 determines whether the imaging direction and zoom magnification of the imaging device 110 have been changed from the imaging direction and zoom magnification when the preset information was registered in S402. For example, the preset management unit 311 determines whether operation information representing a user's operation instructing a change in the imaging direction and zoom magnification has been received from the external terminal 130. If the imaging direction and zoom magnification have been changed, the process proceeds to S406, and if the imaging direction and zoom magnification have not been changed, the process proceeds to S407.

[0046] In S406, when the preset management unit 311 confirms that the PTZ drive has stopped and that the user's operation to change the imaging direction and zoom magnification has ended, the environment feature extraction unit 312 extracts current environment feature information and saves it in the storage unit 205. The extraction of environment feature information here is the same process as in S403. The environment feature information saved in this step is used in the process of S504 in FIG. 5. The process of this flowchart then ends. The preset management unit 311 then transmits the preset information saved in S402 and S403 and the environment feature information extracted in this step to the image processing device 120.

[0047] In S407, the environmental feature extraction unit 312 extracts current environmental feature information. The environmental information analysis unit 313 then compares the current environmental feature information with the environmental feature information saved in S403 and detects a change equal to or greater than a threshold. For example, in the case of brightness, where the minimum value is 0 and the maximum value is 1, a change of 0.1 or greater is detected. In the case of exposure setting, a change of EV equal to or greater than 1 is detected. In the case of color temperature, a change of 1000 K or greater is detected. In the case of a change in the number of people, an increase or decrease of 100 people or greater is detected based on the results of image analysis. In the case of a change in the number of vehicles, an increase or decrease of 10 vehicles or greater is detected based on the results of image analysis. Note that these are merely examples and are not limiting. The threshold may also be changeable by user operation. If the environmental information analysis unit 313 detects a change equal to or greater than the threshold, the process returns to S402 because a thumbnail image needs to be recaptured. If the change is less than the threshold, the process proceeds to S404.

[0048] 5 is a flowchart showing the process of updating thumbnail images stored in the imaging device 110 by the image processing device 120 according to this embodiment. The process of this flowchart is realized by the CPU 211 using the RAM 212 as a work memory and executing a program stored in the ROM 213 or the like. The process of this flowchart is started when it is confirmed in S406 that the user's operation to change the imaging direction and zoom magnification has ended after preset information including thumbnail images has been registered in S402 of FIG. In other words, the process is started on the condition that imaging is being performed at a PTZ position different from the PTZ position at the time of capturing the thumbnail images.

[0049] In S500, the imaging device control unit 321 determines whether one or more pieces of preset information including thumbnail images have been acquired from the imaging device 110. If one or more pieces of preset information have been acquired, the process proceeds to S501, and if no preset information has been acquired, the process of this flowchart ends. When acquiring the preset information from the imaging device 110, the imaging device control unit 321 also acquires the environmental feature information extracted in S406 of FIG. 4. In S501, the imaging device control unit 321 determines whether or not the end flag of this flowchart is set. If the end flag is set, the processing of this flowchart ends, and if the end flag is not set, the processing proceeds to S502. This end flag is set by an external interrupt or the like.

[0050] In S502, the environment feature extraction unit 322 acquires information about the current imaging environment from the imaging device 110, and extracts, from the acquired information, environment feature information for the imaging device 110 currently capturing images. The extraction of the environment feature information here is the same process as in S403. In S503, the environmental information analysis unit 323 compares the current environmental feature information extracted in S502 with environmental feature information extracted in the past, and detects a change equal to or greater than a threshold. The process for detecting a change equal to or greater than the threshold here may be the same as or different from S407. If the environmental information analysis unit 323 detects a change equal to or greater than the threshold, the thumbnail image needs to be changed, so the process proceeds to S504. If the change is less than the threshold, the thumbnail image does not need to be changed, so the process returns to S501.

[0051] In S504, the environmental information analysis unit 323 compares the current environmental feature information extracted in S502 with the environmental feature information acquired in S500 at the time the thumbnail image was captured, and detects whether a change greater than or equal to a threshold has occurred. The environmental feature information extracted in S406 of FIG. 4 may be acquired from the image capture device 110 as environmental feature information for the image capture device 110 currently capturing images, and used as the target for comparison with the environmental feature information acquired at the time the thumbnail image was captured. The process for detecting a change greater than or equal to the threshold may be the same as or different from S503. If the environmental information analysis unit 323 detects a change greater than or equal to the threshold, the process proceeds to S505 to reflect the environmental change in the thumbnail image at the time of registration. If the change is less than the threshold, the process proceeds to S508 to use the thumbnail image at the time of registration as is. It may also be determined that the thumbnail image should be edited if a specific object (such as a person or vehicle) appears in the current environmental feature information, or if the number of specific objects exceeds a threshold. In other words, it may be possible to determine whether to process the thumbnail image based only on the current environmental feature information.

[0052] In S505, the thumbnail image processing unit 324 performs processing of the thumbnail image acquired in S500. The contents of this processing will be described in detail with reference to FIG. In S506, the imaging device control unit 321 transmits the thumbnail image processed in S505 to the imaging device 110. When the imaging device 110 receives the processed thumbnail image from the image processing device 120, it updates the thumbnail image displayed on the external terminal 130 with the processed thumbnail image. The imaging device control unit 321 may generate processing information indicating the processing content and transmit the generated processing information to the imaging device 110. In this case, the imaging device 110 may process the thumbnail image stored in the storage unit 205 based on the received processing information. Here, too, it may be determined whether to process the thumbnail image based only on the current environmental feature information. If a specific object (person or vehicle) appears in the current environmental feature information or the number of specific objects exceeds a threshold, the thumbnail image may be processed to reflect this. Similarly, for other environmental feature information, comparison is not necessarily required if the current environmental feature information is always reflected. However, performing comparison reduces the frequency of processing thumbnail images and reduces the processing load.

[0053] In S507, the environmental information analysis unit 323 stores the environmental feature information extracted in S502 in the storage unit 205 for comparison in S503. The environmental feature information stored in the storage unit 205 is updated whenever a change equal to or greater than a threshold is detected between the environmental feature information and newly extracted environmental feature information. Thereafter, the process returns to S501. In S508, the imaging device control unit 321 transmits a control command to the imaging device 110 to instruct the imaging device 110 to replace the thumbnail image displayed on the external terminal 130 with the thumbnail image at the time of registration (thumbnail image before processing). When the imaging device 110 receives the control command to instruct the replacement with the thumbnail image at the time of registration from the image processing device 120, the imaging device 110 updates the thumbnail image displayed on the external terminal 130 to the thumbnail image at the time of registration (thumbnail image before processing). Thereafter, the process proceeds to S507.

[0054] 6 is a flowchart showing the thumbnail image processing according to this embodiment. The processing of this flowchart is the processing executed in S505 of FIG. In S600, the thumbnail image processing unit 324 calculates the difference between the current environmental feature information extracted in S502 and the environmental feature information at the time of capturing the thumbnail image acquired in S500. Note that in this step, the environmental feature extraction unit 322 may extract environmental feature information by performing image analysis on the image being captured at the current PTZ position and the thumbnail image acquired in S500, and the thumbnail image processing unit 324 may calculate the difference therebetween.

[0055] In S601, the thumbnail image processing unit 324 processes the thumbnail image acquired in S500 based on the difference in the environmental feature information calculated in S600. For example, if there is a change of +0.2 in brightness, where the minimum value is 0 and the maximum value is 1, the image brightness of the original thumbnail image is corrected to +0.2. Similarly, for variations in EV, color temperature, RGB, etc., the original thumbnail image is processed to reflect the amount of variation. Furthermore, if a difference in environmental features due to a lens change is detected, the difference is calculated by, for example, comparing it with the lens information table in the imaging system 100 to calculate the difference in the lens focal length, F-number, and image sensor, and then calculating the change in the angle of view. The thumbnail image is then processed, for example, by cropping it to match the change in the angle of view. Note that the processing method based on the environmental feature information and its variation described here is merely an example and is not limited to this. The processing of this flowchart then ends.

[0056] 7 and 8, an example in which thumbnail images are updated on the thumbnail image display screen 700 displayed on the external terminal 130 will be described. The thumbnail image display screen 700 in FIG. 7 illustrates a case in which image capture is performed during the day. FIG. 8 illustrates a case in which the thumbnail image display screen 700 in FIG. 7 is updated with a processed thumbnail image according to the flowchart in FIG. 6. In the thumbnail image display screen 700 in FIG. 8, an image 801 being captured by the imaging device 110 at the current PTZ position is shown to be darker than the image 701 due to the setting of the sun. Images 802 and 803 are images that reflect the change in environmental information, i.e., the darkening of the image compared to the thumbnail images 702 and 703. This allows thumbnail images that reflect the current imaging environment to be presented to the user. Therefore, when selecting a registered PTZ position, the user can easily imagine the image that will be delivered after the selection.

[0057] In this embodiment, the thumbnail image captured when the imaging position (imaging direction) was registered can be processed by reflecting the imaging environment of the imaging device currently capturing images at the current imaging position (an imaging position different from the registered imaging position). In other words, an appropriate thumbnail image can be generated by processing the thumbnail image representing the second imaging position to match the current imaging environment while still capturing fixed-point images at the first imaging position.

[0058] 5, a series of processes (S502 to S508) for updating thumbnail images is repeated until an end flag is set when the process for acquiring preset information is completed, but the series of processes may be executed at a predetermined timing. For example, the series of processes may be executed when an instruction to update thumbnail images is given by the user.

[0059] In the second modification, changes in the environmental extraction information are quantified and thumbnail images are updated in accordance with the fluctuations in the numerical values. However, the thumbnail images may be corrected in accordance with the predicted results of a predetermined change in the thumbnail images (for example, becoming darker), such as a change in time information from morning to night. Furthermore, the image processing device 120 may acquire weather changes from an external source via the network 140 and correct the brightness, blur, etc. of the thumbnail images in accordance with the weather changes. In other words, the image processing device 120 may acquire environmental information about the imaging device currently capturing images from a device other than the imaging device 110.

[0060] As a third modification, when updating a thumbnail image, the imaging device 110 or the image processing device 120 may inquire of the user as to whether it is OK to update the thumbnail image, and update the thumbnail image if permission is confirmed. Furthermore, when inquiring as to whether it is OK to update the thumbnail image, the processed thumbnail image may be presented on the display unit 226 of the external terminal 130.

[0061] [Embodiment 2] In the second embodiment, a method for generating text information to be added to thumbnail images will be described in the case where problems arise when processing thumbnail images by simply reflecting the differences from the current imaging environment in the thumbnail images. Note that the same components as those in the first embodiment will be omitted from detailed description by using the same reference numerals as those in the first embodiment, and the description will focus on the differences from the first embodiment.

[0062] Fig. 9 is a flowchart showing thumbnail image processing according to this embodiment. The processing in this flowchart is executed in S505 in Fig. 5. In Fig. 9, the processing in S900 to S901 is executed after S600 in the flowchart in Fig. 6. In S900, the thumbnail image processing unit 324 determines whether a problem will arise if the difference content of the environmental feature information calculated in S600 is reflected in the thumbnail image acquired in S500. If it is determined that a problem will arise, it is determined that the processing conditions are not met and the processing proceeds to S701, but if it is determined that no problem will arise, it is determined that the processing conditions are met and the processing proceeds to S601 to generate an image in which the difference content is reflected in the thumbnail image.

[0063] For example, if a switch from a long-focal-length lens to a short-focal-length lens is detected, a thumbnail image with a wide-angle image should be displayed. However, because it is not possible to process a thumbnail image into a wide-angle image that reflects the actual scenery, this case is determined to be problematic. Furthermore, sudden lighting fluctuations or the placement of an object in front of the camera can significantly change the brightness of an image. In such cases, reflecting the difference content may result in a large amount of overexposure or underexposure in the thumbnail image, which is determined to be problematic. For example, a change in color temperature of 5000 K or more may be determined to be problematic. In this way, a change in a specific feature value greater than or equal to a threshold value may be determined to be problematic. Alternatively, the thumbnail image processing unit 324 may process a thumbnail image based on the difference content, check each pixel in the processed image, and determine that a problem exists if the number of pixels with a pixel value of 200 or greater is greater than or equal to a threshold value (e.g., 1,000,000 or more). In this way, whether a problem exists may be determined based on whether the image processed based on the difference content is appropriate. Furthermore, the thumbnail image processing unit 324 may determine that a problem has occurred if the difference content represents a change in the state of the subject, such as an increase or decrease in the number of people or vehicles, etc. Note that the determination methods described here are merely examples and are not limited to these.

[0064] In S901, the thumbnail image processing unit 324 generates text information related to the difference in the environmental feature information calculated in S600, and processes the thumbnail image by superimposing the generated text information on the thumbnail image. For example, if a difference such as a lens change is detected, a message indicating that the lens change has occurred is superimposed on the thumbnail image. For example, if an increase or decrease in the number of people or vehicles is detected, a message indicating that the number of objects has increased or decreased is superimposed. For example, if the image has become brighter due to a lighting change, a message indicating that the environment has become brighter is superimposed. The processing of this flowchart then ends.

[0065] FIG. 10 illustrates a case where thumbnail images are updated on the thumbnail image display screen 700 of FIG. 7 according to the flowchart of FIG. 9. On the thumbnail image display screen 700 of FIG. 10, an image 1001 being captured by the imaging device 110 at the current PTZ position indicates an increase in traffic volume compared to the state of the image 701. Because this increase in traffic volume cannot be addressed by image correction alone, it is determined in S900 that a problem will occur. Images 1002 and 1003 are images in which text information 1012 and 1013 indicating the increase in traffic volume is superimposed on the thumbnail images 702 and 703. This allows thumbnail images with added text information indicating the differences from the current imaging environment to be presented to the user. Therefore, when selecting a registered PTZ position, the user can recognize that the image delivered after the selection may be an image showing increased traffic volume.

[0066] Note that the text information added to the thumbnail image is not limited to text information regarding the difference from the current imaging environment, but may also be a warning message indicating that there may be a difference from the current imaging environment. For example, if the thumbnail image processing unit 324 predicts that the image will be in a bright environment and will have overexposed highlights, it may generate a warning message indicating the predicted content, such as "possibility of overexposed highlights," and process the thumbnail image by superimposing the generated warning message.

[0067] In this embodiment, when the difference in the environment extraction information cannot be reflected as is, text information indicating the difference from the current imaging environment can be generated and added to the thumbnail image. In other words, by adding information corresponding to the current imaging environment to the thumbnail image representing the second imaging position while capturing fixed-point images at the first imaging position, an appropriate thumbnail image can be generated.

[0068] [Embodiment 3] In the third embodiment, a method for processing thumbnail images using a generation AI will be described when a change occurs in the state of an object in an image, such as an increase in the number of people or vehicles, and the thumbnail image cannot be processed by image correction alone. Note that for components similar to those in the first embodiment, detailed explanations will be omitted by using the same reference numerals as in the first embodiment, and the explanation will focus on the differences from the first embodiment.

[0069] Fig. 11 shows an example of the functional configuration of an image processing device 120 according to this embodiment, which differs from the functional configuration diagram of Fig. 3(b) in that a prompt generation unit 1101 is added. The prompt generation unit 1101 generates a prompt required for AI image generation based on the difference between the environmental feature information included in the preset information and the current environmental feature information extracted by the environmental feature extraction unit 322.

[0070] Fig. 12 is a flowchart showing thumbnail image processing according to this embodiment. The processing in this flowchart is the processing executed in S505 in Fig. 5. In Fig. 12, the processing in S1200 to S1202 is executed after S600 in the flowchart in Fig. 6. In S1200, the thumbnail image processing unit 324 determines whether a problem will arise if the difference content of the environmental feature information calculated in S600 is reflected in the thumbnail image acquired in S500. If it is determined that a problem will arise, the process proceeds to S1201, and if it is determined that no problem will arise, the process proceeds to S601 to generate an image in which the difference content is reflected in the thumbnail image. The process of this step is the same as S900 in FIG. 9.

[0071] In S1201, the prompt generation unit 1101 generates a prompt according to the difference content and generates an object on the image using the prompt. For example, if an increase of 100 or more people is confirmed in the current captured image, the prompt generation unit 1101 obtains a prompt corresponding to the target changed environmental feature from an internal table. The internal table is stored in advance in the HDD 214 or the like. In this case, a prompt such as "over 100 people" or an abstracted "large group, people" or "many people" is selected. On the other hand, if a decrease in the number of people is confirmed, the prompt generation unit 1101 selects a prompt such as "a few people" or "small group, people."

[0072] In S1202, the thumbnail image processing unit 324 generates a new thumbnail image using the prompt generated in S1201 and the thumbnail image acquired in S500. For example, if an increase of 100 or more people is confirmed in the current captured image, the thumbnail image processing unit 324 analyzes the thumbnail image and the current captured image to determine an area suitable for where people are located. For example, if the image is of a stadium, it would be seats, and if the image is of a road, it would be a sidewalk. The thumbnail image processing unit 324 then applies the image obtained by inputting the prompt into the generation AI to the determined area in the thumbnail image. The processing of this flowchart then ends.

[0073] Fig. 13 shows a case where thumbnail images are updated on the thumbnail image display screen 700 of Fig. 7 according to the flowchart of Fig. 12. In the thumbnail image display screen 700 of Fig. 13, an image 1001 being captured by the imaging device 110 at the current PTZ position shows an increase in the number of vehicles compared to the state of the image 701, as in Fig. 10. Because this increase in the number of vehicles cannot be handled by processing using image correction alone, it is determined in S1200 that a problem will occur. Then, in S1201, the prompt generation unit 1101 generates a prompt based on the details of the increase in the number of vehicles. In the example of Fig. 13, it is assumed that a prompt such as "car, on load, driving" has been generated.

[0074] Thereafter, in S1202, the thumbnail image processing unit 324 analyzes the thumbnail image and the current captured image (image of video 1001) to determine an area suitable for applying a prompt. In the example of FIG. 13, since the increase in the number of vehicles is calculated as a difference, the area corresponding to the road is estimated by image analysis, and an image in which cars are present on the road is generated by applying the image obtained by the prompt. Image 1302 is the image generated in S1202. On the other hand, thumbnail image 703 is displayed as it was when the preset was registered, because an area suitable for applying a prompt could not be detected by image analysis.

[0075] In this embodiment, a prompt is determined based on changes in the imaging environment and applied to the thumbnail image, thereby generating a new thumbnail image that incorporates the changes in the imaging environment. The method for generating a new thumbnail image is not limited to the above. For example, the thumbnail image may be used as a source image and as a base for processing with the prompt. Furthermore, while the prompts used are stored in an internal table in this embodiment, the prompt content may be automatically generated based on the results of image analysis. Furthermore, the imaging environment to which the prompts are applied is not limited to the number of people. For example, the prompts may also be based on changes in weather or traffic volume, changes in the position of the sun or day / night due to time fluctuations, the appearance of a new building, or the like.

[0076] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0077] The disclosure of each of the above-described embodiments includes the following configurations, methods, and programs. (Configuration 1) a first acquisition means for acquiring an image generated by capturing an image at a first imaging position as a thumbnail image representing the first imaging position; a second acquisition means for acquiring environmental information, which is information obtained from a captured image captured at a second imaging position different from the first imaging position, or information obtained from settings when the captured image was captured; a processing means for processing the thumbnail image based on at least the environmental information acquired by the second acquisition means; 1. An image processing device comprising: (Configuration 2) 2. The image processing device according to configuration 1, wherein the processing means performs processing to process the thumbnail image when detecting a change in the environmental information that is equal to or greater than a threshold value. (Configuration 3) a third acquisition means for acquiring environmental information relating to the first imaging position; The processing means when a change equal to or greater than a threshold value in the environmental information acquired by the second acquisition means is detected and a difference between the environmental information acquired by the third acquisition means and the environmental information acquired by the second acquisition means exceeds a threshold value, processing the thumbnail image; The image processing device according to configuration 2, characterized in that when a change equal to or greater than a threshold value in the environmental information acquired by the second acquisition means is detected and the difference between the environmental information acquired by the third acquisition means and the environmental information acquired by the second acquisition means is less than the threshold value, performs processing to return the thumbnail image to its pre-processing state. (Configuration 4) 4. The image processing device according to any one of configurations 1 to 3, wherein the environmental information is imaging parameters set in the imaging device. (Configuration 5) 5. The image processing device according to any one of configurations 1 to 4, wherein the environmental information is equipment information of an imaging device or information about a lens used in the imaging device. (Configuration 6) 6. The image processing device according to any one of configurations 1 to 5, wherein the environmental information is a result obtained by image analysis of an image captured by an imaging device. (Configuration 7) The image processing device according to any one of configurations 1 to 6, characterized in that the processing of the thumbnail image is processing for applying at least one of an increase or decrease in the number or amount of objects in the image, a change in brightness, and a change in weather to the thumbnail image. (Configuration 8) a third acquisition means for acquiring environmental information related to the first imaging position; The image processing device according to any one of configurations 1 to 7, wherein the processing of the thumbnail image is processing of generating character information representing a difference between the environmental information acquired by the third acquisition means and the environmental information acquired by the second acquisition means, to be added to the thumbnail image. (Configuration 9) a third acquisition means for acquiring environmental information related to the first imaging position; The image processing device according to any one of configurations 1 to 8, characterized in that the process of processing the thumbnail image is a process of generating a prompt based on a difference between the environmental information acquired by the third acquisition means and the environmental information acquired by the second acquisition means, and applying an image obtained by inputting the generated prompt into a generation AI to the thumbnail image. (Configuration 10) The image processing device according to any one of configurations 1 to 9, wherein the processing means processes the thumbnail image if a predetermined condition for processing the thumbnail image is satisfied, and does not process the thumbnail image if the predetermined condition is not satisfied. (Configuration 11) 11. The image processing device according to any one of configurations 1 to 10, wherein the imaging position includes at least one of a pan position, a tilt position, and a zoom position of the imaging device. (method) a first acquisition step of acquiring an image generated by capturing an image at a first imaging position as a thumbnail image representing the first imaging position; a second acquisition step of acquiring environmental information, which is information obtained from a captured image captured at a second imaging position different from the first imaging position, or information obtained from settings when the captured image was captured; a processing step of processing the thumbnail image based on at least the environmental information acquired in the second acquisition step; An image processing method comprising: (program) The computer of the image processing device, a first acquisition means for acquiring an image generated by capturing an image at a first imaging position as a thumbnail image representing the first imaging position; a second acquisition means for acquiring environmental information, which is information obtained from a captured image captured at a second imaging position different from the first imaging position, or information obtained from settings when the captured image was captured; a processing means for processing the thumbnail image based on at least the environmental information acquired by the second acquisition means; A program characterized by functioning as [Explanation of symbols]

[0078] 100: Imaging system, 110: Imaging device, 120: Image processing device, 130: External terminal

Claims

1. a first acquisition means for acquiring an image generated by capturing an image of a first imaging position as a thumbnail image representing the first imaging position; a second acquisition means for acquiring environmental information, which is information obtained from a captured image captured at a second imaging position different from the first imaging position, or information obtained from settings when the captured image was captured; a processing means for processing the thumbnail image based on at least the environmental information acquired by the second acquisition means; 1. An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the processing means performs processing to process the thumbnail image when a change in the environmental information that is equal to or greater than a threshold value is detected.

3. a third acquisition means for acquiring environmental information relating to the first imaging position; The processing means when a change equal to or greater than a threshold value in the environmental information acquired by the second acquisition means is detected and a difference between the environmental information acquired by the third acquisition means and the environmental information acquired by the second acquisition means exceeds a threshold value, processing the thumbnail image; 3. The image processing device according to claim 2, further comprising: a processing unit configured to perform processing to restore the thumbnail image to its pre-processing state when a change in the environmental information acquired by the second acquisition means is detected that is greater than or equal to a threshold value, and when a difference between the environmental information acquired by the third acquisition means and the environmental information acquired by the second acquisition means is less than the threshold value.

4. 2. The image processing apparatus according to claim 1, wherein the environmental information is image capture parameters set in the image capture apparatus.

5. 2. The image processing apparatus according to claim 1, wherein the environmental information is equipment information of an image capturing apparatus or information of a lens used in the image capturing apparatus.

6. 2. The image processing apparatus according to claim 1, wherein the environmental information is a result obtained by analyzing an image captured by an imaging device.

7. 2. The image processing device according to claim 1, wherein the processing of the thumbnail image is processing for applying at least one of an increase or decrease in the number or amount of objects in the image, a change in brightness, and a change in weather to the thumbnail image.

8. a third acquisition means for acquiring environmental information relating to the first imaging position; 2. The image processing device according to claim 1, wherein the processing of the thumbnail image is processing of generating character information representing a difference between the environmental information acquired by the third acquisition means and the environmental information acquired by the second acquisition means, to be added to the thumbnail image.

9. a third acquisition means for acquiring environmental information relating to the first imaging position; The image processing device described in claim 1, characterized in that the process of processing the thumbnail image is a process of generating a prompt based on the difference between the environmental information acquired by the third acquisition means and the environmental information acquired by the second acquisition means, and applying an image obtained by inputting the generated prompt into a generation AI to the thumbnail image.

10. 2. The image processing device according to claim 1, wherein the processing means processes the thumbnail image if a predetermined condition for processing the thumbnail image is satisfied, and does not process the thumbnail image if the predetermined condition is not satisfied.

11. 2. The image processing device according to claim 1, wherein the image capturing position includes at least one of a pan position, a tilt position, and a zoom position of the image capturing device.

12. a first acquisition step of acquiring an image generated by capturing an image at a first imaging position as a thumbnail image representing the first imaging position; a second acquisition step of acquiring environmental information, which is information obtained from a captured image captured at a second imaging position different from the first imaging position, or information obtained from settings when the captured image was captured; a processing step of processing the thumbnail image based on at least the environmental information acquired in the second acquisition step; An image processing method comprising:

13. The computer of the image processing device, a first acquisition means for acquiring an image generated by capturing an image of a first imaging position as a thumbnail image representing the first imaging position; a second acquisition means for acquiring environmental information, which is information obtained from a captured image captured at a second imaging position different from the first imaging position, or information obtained from settings when the captured image was captured; a processing means for processing the thumbnail image based on at least the environmental information acquired by the second acquisition means; A program characterized by functioning as

Images