Imaging device, method for controlling imaging device, and program

Abstract

To enable each user who uses an imaging device to perform photographing at desired timing.SOLUTION: An imaging devices includes: reception means configured to receive an imaging instruction by a user; imaging means configured to perform imaging in accordance with the imaging instruction; allocation means configured to allocate control time of the imaging device to each user based on the imaging instruction and the number of users who use the imaging device; and control means configured to control the imaging means so as to perform imaging based on the imaging instruction from the user to whom the control time is allocated.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to an imaging device, a control method for an imaging device, and a program. [Background technology]

[0002] It is common to control cameras installed in remote locations via a network using a mobile device, PC, or other device to take pictures. It is conceivable that an imaging device capable of implementing such a photography method could be used simultaneously by multiple remote users via the network. In this case, there is a demand for technology to control the imaging device in a way that satisfies the needs of each remote user, who wants to take pictures with the camera settings and shutter release timing they desire.

[0003] An imaging device designed to meet these needs is disclosed in Patent Document 1. The imaging device disclosed in Patent Document 1 sets priorities for operations such as release, access to images, and image transfer, as well as for multiple users who operate the device, and restricts operations and users with low priorities. For example, the priorities for operations are set as follows: (high priority) release > access to images > image transfer (low priority). When a release operation request is received from user A, access by user B for image confirmation is restricted, and the imaging device's resources are allocated to user A's release operation, allowing user A to take a photo at the timing desired. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-98914 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the technology disclosed in Patent Document 1, when multiple users of an imaging device simultaneously request a release operation, the release operation of a user with a lower priority is restricted, and the user with a lower priority is unable to take a picture at the timing they desire. An object of the present invention is to enable each user of an imaging device to take a picture at the timing they desire. [Means for solving the problem]

[0006] The imaging device of the present invention is characterized by having an accepting means for accepting a shooting instruction from a user, an imaging means for taking a picture in accordance with the shooting instruction, an allocating means for allocating control time of the imaging device to each of the users based on the shooting instruction and the number of users who will use the imaging device, and a control means for controlling the imaging means so that the imaging means takes a picture during the allocated control time based on the shooting instruction from the user to whom the control time is allocated. [Effects of the Invention]

[0007] According to the present invention, each user of the imaging device can take a photograph at a timing desired by the user. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a system to which an imaging device is applied. [Figure 2] FIG. 1 illustrates an example of the configuration of an imaging device. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of a processing unit within the camera. [Figure 4] FIG. 2 is a diagram illustrating an example of a terminal device operated by a user. [Figure 5] 10 is a flowchart illustrating an example of processing performed by the imaging device. [Figure 6] 10 is a flowchart illustrating an example of a photographing sequence execution process. [Figure 7] 10 is a flowchart illustrating an example of a photographing process. [Figure 8]10 is a flowchart illustrating an example of a user allocation time setting process. [Figure 9] FIG. 10 is a diagram illustrating an example of camera control time allocation. [Figure 10] FIG. 10 is a diagram illustrating an example of camera control time allocation. [Figure 11] FIG. 1 is a diagram illustrating image synthesis in an imaging device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. 1 is a diagram illustrating an example of the configuration of a system to which an image capturing device according to this embodiment is applied. A camera 101, which is an example of an image capturing device according to this embodiment, has a camera body 102 to which a removable interchangeable lens 103 can be attached.

[0010] For example, at the venue of an event where a subject is to be photographed, the site cameraman S111 can select an interchangeable lens 103 that will fit the subject PSA112 and the subject PSB113 simultaneously within the angle of view, and attach the interchangeable lens 103 to the camera 101. Furthermore, the site cameraman S111 can determine the location to place the camera 101 and the orientation of the camera 101 in order to fit the subject PSA112 and the subject PSB113 simultaneously within the angle of view.

[0011] The camera 101 is capable of communicating via a wireless LAN (Local Area Network) and is connected to a communication network 105 such as the Internet via a wireless LAN access point 104. A terminal device 122 operated by a user A 121 is connected to the communication network 105 such as the Internet via a wireless LAN access point 123. Similarly, terminal devices 125 and 128 operated by users B 124 and C 127, respectively, are connected to the communication network 105 such as the Internet via wireless LAN access points 126 and 129. Hereinafter, the "terminal device" may also be simply referred to as a "terminal." A server 131 is connected to the communication network 105 such as the Internet, and performs numerical calculations, data storage, etc.

[0012] User A 121 can use camera 101 to photograph subjects 112 and 113 by operating a camera control application installed on terminal 122. Similarly to user A 121, user B 124 can use camera 101 to photograph subjects 112 and 113 by operating terminal 125, and user C 127 can use camera 101 to photograph subjects 112 and 113 by operating terminal 128. Users A 121, B 124, and C 127 each set their desired shutter speed, aperture, focus position, and ISO sensitivity in camera 101 at their desired timing to take a photograph. Here, ISO sensitivity is a value defined by the ISO 12232 standard that indicates the degree to which an electrical signal acquired by photoelectrically converting an optical image in an image sensor is amplified.

[0013] The communication path used to access the camera 101 from a terminal operated by each user may include a wired LAN, Bluetooth, etc. Alternatively, the camera 101, the terminals 122, 125, 128, and the server 131 may be connected to a LAN only so as to be able to communicate with each other, without being connected to a communication network 105 such as the Internet.

[0014] FIG. 2 is a diagram illustrating an exemplary configuration of the camera 101 according to this embodiment. In FIG. 2, the in-camera processing unit 220 performs various processes and controls related to the camera body 102 and the interchangeable lens 103. Processing units (not shown) arranged within the camera body 102 and the interchangeable lens 103 communicate with each other to perform processing, and these processing units are collectively referred to as the in-camera processing unit 220. For example, the in-camera processing unit 220 inputs and outputs data to and from the image sensor 211, amplifier unit 212, communication unit 213, display unit 214, and operation unit 215. The in-camera processing unit 220 performs calculations based on the input data and stores the calculation results. The calculations performed by the in-camera processing unit 220 include image processing such as image synthesis and conversion, acceptance and execution of each user's shooting instructions, and calculation of each user's camera control timing. Furthermore, for example, the in-camera processing unit 220 outputs control signals based on the calculation results to the actuators, motors, lens unit, and other components of the camera 101. The in-camera processing unit 220 is an example of a receiving means, an allocation means, a control means, and a synthesis means.

[0015] The interchangeable lens 103 has an imaging optical system including a field lens 201, a zoom lens 202 that changes magnification, an electronic diaphragm unit 203 that adjusts the amount of light, and a variable focal length lens 204 that adjusts focus. The zoom lens 202 is held by a lens holding frame (not shown) and driven in the optical axis direction by a motor controlled by a control signal output from an in-camera processing unit 220. A D / A converter and a drive circuit for operating the motor are not shown in FIG. 2. The variable focal length lens 204, like the zoom lens 202, is held by a lens holding frame (not shown) and adjusts its focus position in response to a control signal output from the in-camera processing unit 220. The electronic diaphragm unit 203 has an electronic diaphragm 205. The electronic diaphragm 205 is, for example, a transparent liquid crystal display having a light-shielding surface electrically formed on the surface.

[0016] The camera body 102 includes an image sensor 211, an amplifier 212, a communication unit 213, a display unit 214, an operation unit 215, and an in-camera processing unit 220. The image sensor 211 is configured with a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) element that converts an optical image into an electrical signal. The image sensor 211 photoelectrically converts a subject image formed by the imaging optical system of the interchangeable lens 103 and outputs an electrical signal (analog signal). The electrical signal output by the image sensor 211 is amplified (hereinafter also referred to as gain) in the amplifier 212 in accordance with a control signal output by the in-camera processing unit 220. The electrical signal amplified in the amplifier 212 is converted into a digital signal (hereinafter also referred to as video signal) by an A / D converter (not shown) provided in the amplifier 212, and then input to the in-camera processing unit 220. Here, the gain is an analog gain applied before the electrical signal is input to the A / D converter, but digital gain may also be applied to the video signal. The video signal is processed by the in-camera processing unit 220 etc., and the processed results are displayed on the display unit 214 or transmitted from the communication unit 213 to the terminals 122, 125, 128 of the respective users and displayed thereon.

[0017] The in-camera processing unit 220 can communicate with the terminals 122, 125, 128, and server 131 connected to a communication network such as the Internet via the communication unit 213. The in-camera processing unit 220 can transmit a video signal from the communication unit 213 to a terminal operated by each user, and cause each user's terminal to perform image processing such as composition and conversion of the video signal. Alternatively, the in-camera processing unit 220 can transmit a video signal from the communication unit 213 to the server 131, cause the server 131 to perform image processing such as composition and conversion of the video signal, and receive the processing results from the server 131.

[0018] The display unit 214 is, for example, a display, and displays images, information, etc. The display unit 214 displays, for example, a live view image based on a video signal so that the site cameraman S111 can check the composition, focus, etc. The operation unit 215 is an operation unit for performing various operations related to the camera 101. The operation unit 215 has, for example, an input device for inputting shooting instructions. The operation unit 215 has physical operation buttons, a touch panel, etc. as an input device for accepting user operations. Note that a touch panel is, for example, an input device configured as a flat input unit that outputs coordinate information corresponding to the position where a fingertip or the like touches the input unit. When a touch panel is used as the operation unit 215 and a display is used as the display unit 214, the operation unit 215 and the display unit 214 can be configured as an integrated unit. For example, the touch panel can be configured so that the light transmittance does not interfere with the display and is attached to the upper layer of the display surface of the display, and the input coordinates on the touch panel can be associated with the display coordinates on the display. Furthermore, by operating the operation unit 215, the video signal can be stored in the in-camera processing unit 220 at any timing.

[0019] In this embodiment, image synthesis performed by the in-camera processing unit 220 is performed using, for example, a neural network. A neural network performs calculations using a hierarchically connected structure in which an input layer, multiple intermediate layers that extract features contained in data input from the previous layer, and an output layer are connected. Image synthesis using a neural network is determined by the layer structure and parameters of each layer. Information that determines these neural networks is called a neural network model. The neural network model may be a model trained using Generative Adversarial Networks (GAN), which is one of the learning methods.

[0020] 3 is a diagram showing an example configuration of the in-camera processing unit 220. The in-camera processing unit 220 has a CPU 301, memory 302, neural network processing unit 303, image processing unit 304, data storage unit 305, input unit 306, control unit 307, communication control unit 308, and display control unit 309. The CPU 301, memory 302, neural network processing unit 303, image processing unit 304, data storage unit 305, input unit 306, control unit 307, communication control unit 308, and display control unit 309 are connected to each other via a system bus 310 so as to be able to communicate with each other.

[0021] A CPU (Central Processing Unit) 301 controls a neural network processing unit 303, an image processing unit 304, a control unit 307, a communication control unit 308, a display control unit 309, and the like in accordance with a program stored in a memory 302. For example, the CPU 301 executes a control command related to the magnification change operation of the zoom lens 202 in response to the operation of a zoom switch (not shown) included in the operation unit 215, and changes the position of the zoom lens 202 in accordance with a control signal. The memory 302 is, for example, a RAM (Random Access Memory), and is used as a work memory for the CPU 301, the neural network processing unit 303, and the like. The memory 302 stores programs read from a data storage unit 305, video signals, data received via the communication control unit 308, neural network models, and the like. For example, the CPU 301 executes processing in accordance with a program stored in the memory 302, thereby realizing functions such as a receiving unit, an allocating unit, a control unit, and a combining unit.

[0022] The neural network processing unit 303 is configured, for example, with one or more GPUs, DSPs, FPGAs, or a board equipped with hardware for inference calculation processing. The neural network processing unit 303 performs calculation processing for image synthesis, for example. GPU stands for Graphics Processing Unit, DSP stands for Digital Signal Processor, and FPGA stands for Field Programmable Gate Array. A GPU is a computing device that can perform efficient calculations by processing data in parallel. Therefore, it is effective to use a GPU to perform calculations for inference using a trained inference device.

[0023] In this embodiment, a neural network processing unit 303 is used in addition to the CPU 301 for image synthesis. Specifically, in the in-camera processing unit 220, the CPU 301 reads out a neural network model stored in the data storage unit 305 and sets the read data in the neural network processing unit 303. Next, the CPU 301 starts the neural network processing unit 303 and starts image synthesis. Note that the image synthesis calculations may be performed by only the CPU 301 or the neural network processing unit 303.

[0024] The image processing unit 304 performs image processing such as noise reduction, gamma correction, grayscale conversion, and brightness conversion on the image captured by the camera 101. The data storage unit 305 is, for example, a hard disc drive (HDD), a solid state drive (SSD), a read only memory (ROM), or a magneto-optical (MO) disk. The data storage unit 305 stores various programs for operating the CPU 301, data specific to the camera body and lens, neural network models, and the like.

[0025] The input unit 306 receives input such as data output by a sensor (not shown) arranged in the camera 101, lens information of the interchangeable lens 103, and operation information from the operation unit 215, and supplies these to the CPU 301. The control unit 307 outputs control signals for controlling motors, actuators, and the like incorporated in the camera 101 in accordance with control by the CPU 301. The communication control unit 308 controls data communication and the like via the communication network 105 and the like in accordance with control by the CPU 301. The communication control unit 308 controls, for example, data communication via the communication network 105 between each functional unit of the in-camera processing unit 220 and the terminals 122, 125, 128, and the like. The display control unit 309 controls the display of information such as a live view image of the camera 101 and selection information and values ​​related to camera settings.

[0026] Fig. 4 is a diagram illustrating an example of a terminal device that communicates with the camera 101 in this embodiment. Fig. 4 shows a user interface for controlling the camera 101, which is provided by an application installed on the terminal 128 operated by the user C 127 shown in Fig. 1. The user interface of the terminal 128 includes a live view display area 401, a site information display section 402, a camera setting section 403, and control buttons 404, 405, and 406.

[0027] The live view display area 401 displays a live view image of the camera 101 connected via a communication network 105 such as the Internet. The live view display area 401 also includes a touch panel, which is an operation panel, and allows the user C127 to specify an area on the live view image on which they want to focus by tapping with their finger. For example, when subjects 112 and 113 are displayed as a live view image in the live view display area 401, the user C127 can tap the position of the face of the subject 113 to display a dashed rectangle 420 indicating the specified focus area. The site information display section 402 displays information about the current shooting status of the camera 101. For example, the model name of the camera 101, the model name of the interchangeable lens 103, the name of the shooting event, labels indicating the shooting position and angle, the number of simultaneous users, the name of the on-site photographer, etc. are displayed.

[0028] The camera setting unit 403 is used to specify camera settings such as the aperture value (F-number) and shutter speed of the camera 101. For example, the aperture value can be specified by swiping drum rolls 408 and 409 to move the value and entering the desired value in the F-number setting unit 407. Similarly to the aperture value, the shutter speed can be specified by swiping drum rolls 411 and 412 to enter the desired value in the shutter speed setting unit 410. Various functions can be assigned to the control buttons 404, 405, and 406. For example, pressing the control button 404 may start a live view display, and pressing the control button 404 again may stop the live view display. For example, pressing the control button 404 may call and set a favorite preset, and pressing the control button 405 may perform a release operation. These functions are implemented by hardware resources (not shown) equipped in the terminal 128.

[0029] The user interfaces of terminal 122 operated by user A 121 and terminal 125 operated by user B 124 may be configured similarly to the user interface of terminal 128. However, the user interfaces may be different as long as they have the functions described above.

[0030] The operation of the camera 101 in this embodiment will be described with reference to Figures 5 to 8. The processing in each flowchart described below is executed by the CPU 301 of the in-camera processing unit 220 controlling each unit in accordance with input signals and programs.

[0031] FIG. 5 is a flowchart illustrating an example of processing by the camera 101 in this embodiment. FIG. 5 shows the operation of the camera 101 from start to finish. For example, the start and finish of the operation of the camera 101 are controlled based on operations performed by the site cameraman S111 on the camera 101, etc. When the camera 101 starts operating, the camera 101 is connected to terminals 122, 125, and 128 via the communication network 105, enabling each of user A 121, user B 124, and user C 127 of the camera 101 to take photographs. In addition, when the camera 101 starts operating, the processing of the flowcharts relating to the photographing sequence execution processing shown in FIGS. 6(A) and 6(B) is started and executed.

[0032] 5, in step S501, CPU 301 acquires user information to identify the user who uses camera 101. CPU 301 acquires IP addresses as user information from all terminals operated by connected users via communication network 105, for example. Note that the user information is not limited to the IP addresses of the terminals, and may be the user's name or the like as long as it can identify the user who uses camera 101.

[0033] In step S502, CPU 301 determines whether or not a shooting instruction has been received from the user using camera 101, and waits for the shooting instruction from the user. If CPU 301 determines that a shooting instruction has been received (YES), processing proceeds to step S503. Here, shooting instructions include an instruction to end camera 101, an instruction to start live view display, an instruction to stop live view display, an instruction to set camera settings (aperture value (f-number), shutter speed, focus position, gain value (or ISO value)), and an instruction to start shooting (an instruction to request a release operation). The instruction to start live view display and the instruction to start shooting are each accompanied by a camera setting instruction.

[0034] In step S503, the CPU 301 acquires a shooting instruction from the terminal operated by the user. In step S504, CPU 301 adds user information corresponding to the user who issued the photographing instruction to the photographing instruction acquired in step S503. For example, CPU 301 adds the IP address of the terminal that issued the photographing instruction to the photographing instruction acquired in step S503.

[0035] In step S505, CPU 301 determines whether the shooting instruction acquired in step S503 is an instruction issued by site cameraman S111 to shut down camera 101. If CPU 301 determines that the shooting instruction is an instruction to shut down camera 101 (YES), the process proceeds to step S506. On the other hand, if CPU 301 determines that the shooting instruction is not an instruction to shut down camera 101 (NO), the process proceeds to step S507.

[0036] In step S506, the CPU 301 issues a command to terminate the camera 101, and terminates the operation of the camera 101 after completing the processes related to the shooting sequence execution process for all users.

[0037] In step S507, CPU 301 executes a user allocation time setting process to set a user allocation of camera control time based on the acquired shooting instruction content and user information. After camera control time has been allocated to all users in the user allocation time setting process, the process proceeds to step S502 and waits for a shooting instruction from a user again. The user allocation time setting process in step S507 will be described in detail later.

[0038] 6(A) and 6(B) are flowcharts illustrating an example of a photographing sequence execution process that is started when the camera 101 starts operating and is executed for each user. In the photographing sequence execution process, the camera 101 repeatedly performs a photographing process and transmits video information acquired in the photographing process in response to a photographing instruction from the user.

[0039] 6A, in step S601, CPU 301 acquires user information to identify the user who uses camera 101. CPU 301 acquires IP addresses as user information from all terminals operated by connected users via communication network 105, for example. Note that the user information is not limited to the IP addresses of the terminals, and may be the user's name or the like as long as it can identify the user who uses camera 101.

[0040] In step S602, based on the user information acquired in step S601, the CPU 301 executes the photographing sequence execution process shown in Fig. 6B for each of all users who use the camera 101. That is, in the example shown in the present embodiment, the photographing sequence execution process for user A 121, the photographing sequence execution process for user B 124, and the photographing sequence execution process for user C 127 are executed.

[0041] Fig. 6(B) is a flowchart illustrating the photographing sequence execution process in step S602 of Fig. 6(A). In the following, the user in the photographing sequence execution process will be described as user X. In this example, the label X that identifies the user is either A, B, or C.

[0042] When the shooting sequence execution process for user X is started, CPU 301 starts the first repetitive process from step S611 onwards. Here, the execution timing of the series of processes from steps S612 to S614 that are repeated in the first repetitive process is determined based on the initial value for the first repetitive process and the information set by the user allocated time setting process in step S507 of FIG.

[0043] In step S612, CPU 301 determines whether or not an end command for camera 101 was issued in step S506 of Fig. 5. If CPU 301 determines that an end command for camera 101 was issued (YES), the photographing sequence execution process ends. On the other hand, if CPU 301 determines that an end command for camera 101 was not issued (NO), the process proceeds to step S613.

[0044] In step S613, the CPU 301 executes the photographing process shown in Fig. 7. After executing the photographing process shown in Fig. 7, the process proceeds to step S614. In step S614, the CPU 301 transmits the video information acquired in the shooting process to the terminal operated by the user X.

[0045] FIG. 7 is a flowchart showing an example of the photographing process in step S613 of FIG. 6(B). When the photographing process in the photographing sequence execution process for user X is started, CPU 301 starts the second repetitive process from step S701 onwards. Here, the execution timing of the series of processes from steps S702 to S707 that are repeated in the second repetitive process is determined based on the initial value for the second repetitive process and the information set by the user allocated time setting process in step S507 of Fig. 5.

[0046] In step S702, the CPU 301 performs aperture setting in the camera 101. In aperture setting, the CPU 301 controls the in-camera processing unit 220 to send a control signal corresponding to the aperture value and the like included in the camera setting instruction, which is a shooting instruction, to the electronic aperture unit 203. The electronic aperture 205 is formed in response to this control signal.

[0047] In step S703, the CPU 301 sets the focus position of the camera 101. In setting the focus position, the CPU 301 controls the in-camera processing unit 220 to send a control signal corresponding to focus position information and the like included in the camera setting instruction, which is a shooting instruction, to the variable focal length lens 204. In response to this control signal, an actuator (not shown) provided in the variable focal length lens 204 deforms the lens, thereby adjusting the focus to a position desired by the user.

[0048] In step S704, the CPU 301 sets the gain in the camera 101. In the gain setting, the CPU 301 controls the in-camera processing unit 220 to transmit a control signal corresponding to gain information and the like included in the camera setting instruction, which is a shooting instruction, to the amplifier unit 212. In response to this control signal, the amplifier unit 212 is controlled to amplify the electrical signal acquired by the image sensor 211 by the set gain and output the amplified signal.

[0049] Here, the order in which the aperture setting in step S702, the focus position setting in step S703, and the gain setting in step S704 are performed is not limited to this, and the execution order is random. Also, the aperture setting, focus position setting, and gain setting in steps S702 to S704 are collectively referred to as the camera setting process below. It is assumed that the time required to perform this camera setting process is constant.

[0050] Next, in step S705, the CPU 301 controls the shutter to open and perform exposure, whereby the optical image of the subject formed by the imaging optical system of the interchangeable lens 103 is converted into an electrical signal by the imaging element 211. In step S706, the CPU 301 controls, if necessary, to amplify the electrical signal output by the image sensor 211. For example, the amplifier 212 amplifies the electrical signal according to the gain amplification factor set in step S704. In step S707, the CPU 301 stores the digital signal (video signal) output as electronic data via the A / D converter provided in the amplifier 212 in the data storage unit 305 as a pre-combine image. In this embodiment, the time required for the processing in step S706 (gain) and the processing in step S707 (storing the pre-combine image) is assumed to be constant.

[0051] Next, in step S708, the CPU 301 extracts the pre-combine image stored in the data storage unit 305 in step S707 at the timing set by the user allocated time setting process in step S507 of FIG. In step S709, the CPU 301 composites the images (pre-composite images) extracted in step S708 using the neural network processing unit 303 or the like. In step S710, the CPU 301 stores the image (composite image) obtained as a result of the image composition in step S709 in the data storage unit 305. After storing the composite image in step S710, the process proceeds to step S614 in FIG. 6B. In this embodiment, the time required for the process in step S708 (extraction), the process in step S709 (image synthesis), and the process in step S710 (storage of the synthesized image) is assumed to be constant.

[0052] Next, the user allotted time setting process in step S507 of Fig. 5 will be described with reference to Fig. 8. Fig. 8 is a flowchart illustrating an example of the user allotted time setting process in step S507 of Fig. 5.

[0053] 8, in step S801, the CPU 301 acquires status information of the camera 101. The status information of the camera 101 includes the number of users using the camera 101 and values ​​indicating the timing of the shooting instructions, camera settings, and repetitive processing executed by each of the users. In step S802, the CPU 301 obtains the total number of users based on the status information obtained in step S801.

[0054] In step S803, CPU 301 calculates the time for which the camera is controlled for each user (camera control time for each user) based on the status information acquired in step S801 and the total number of users acquired in step S802. CPU 301 also calculates the amplification factor of the video information acquired by the imaging process. Here, the time for which the camera is controlled is shown as a table of the number of repetitions of the first repetitive process shown in FIG. 6B and the second repetitive process shown in FIG. 7, the start times of the repetitive processes, and the exposure times in the process in step S705 in FIG. 7.

[0055] When allocating camera control time to each user in step S803, the CPU 301 allocates the camera control time to each user, for example, as follows: If one or more users using the camera 101 are performing only image capture or only live view display, the CPU 301 allocates the camera control time equally to each user using the camera 101. For example, if three users, user A, user B, and user C, are performing only image capture or only live view display, the CPU 301 allocates the camera control time to each user so that the ratio of camera control time to each user is 1:1:1. Furthermore, if image capture and live view display are performed simultaneously, the CPU 301 allocates twice as much camera control time to the user performing image capture as the camera control time to the user performing live view display. For example, if image capture is performed for user A and live view display is performed for users B and C, the CPU 301 allocates the camera control time to users A, B, and C so that the ratio of camera control time to each user is 2:1:1. For example, if images are captured for users A and B and live view display is performed for user C, CPU 301 allocates camera control time so that the ratio of camera control time for users A, B, and C is 2:2:1. Note that, although twice as much camera control time is allocated to the user performing image capture as to the user performing live view display, this is not limitative. The ratio of camera control time allocated to the user performing live view display and the user performing image capture can be set arbitrarily.

[0056] In step S804, the CPU 301 sets the result of the calculation in step S803, that is, the camera control time allocated to each user, in the first and second repetitive processes.

[0057] For example, assume that live view display is to be performed on the terminal 122 operated by user A 121. At this time, for example, the CPU 301 sets the start time of the first repeat process in the shooting sequence execution process for user A to be every 1 msec from the time when the start of live view display is instructed. Furthermore, the CPU 301 sets the second repeat process in the shooting sequence execution process for user A to not repeat and to set the exposure time in step S705 to 300 μsec. Then, in response to detailed shooting instructions for the live view display from user A 121, gain processing and image processing (not shown) are performed in step S706 of FIG. 7. The computational load may be reduced by setting NOP (No Operation) in the process (image composition) in step S709 of FIG. 7. With these settings, the terminal 122 operated by user A 121 can receive live view images at 1000 fps (frames per second) and an exposure time of (3 / 10000) seconds, allowing user A 121 to view the live view images.

[0058] As another example, when releasing the shutter at a shutter speed of (1 / 200) seconds in response to an operation by user A 121, CPU 301 sets the first repeat process in the shooting sequence execution process for user A to not repeat. Furthermore, CPU 301 sets the second repeat process in the shooting sequence execution process for user A to start every 1 msec from the time Tr when the release (shooting start) is instructed, up to Tr + 5 msec. Furthermore, it sets the exposure time in step S705 to 300 μsec. With these settings, camera 101 captures five images for user A with an exposure of (3 / 10000) seconds every 1 msec during the time 1 / 200 seconds after the release (shooting start) is instructed. By amplifying and then combining the images acquired in these five captures, an image captured at a shutter speed of (1 / 200) seconds can be obtained.

[0059] In these examples, if another user issues a shooting instruction during shooting in response to an operation by user A121, CPU 301 executes the user allocation time setting process in step S507 of Fig. 5 to recalculate the time to be allocated to all users. Then, based on the calculation result, CPU 301 updates the table of the number of repetitions of the first repetitive process and the second repetitive process, the start time, and the exposure time in the process in step S705 of Fig. 7.

[0060] 9 and 10 are diagrams illustrating an example of camera control time allocation in this embodiment. Fig. 9 is a timing chart showing an example in which live view display and shooting (operating the release button to release the shutter) are performed simultaneously for user A 121, user B 124, and user C 127. In Fig. 9, events related to live view display and shooting for each user are shown by times T901 to T909 on a time axis 910.

[0061] 9, events for user A 121 are plotted on user A's event axis 920. Similarly, events for user B 124 are plotted on user B's event axis 930, and events for user C 127 are plotted on user C's event axis 940.

[0062] For user A121, at time T901, live view display is started (921) based on a shooting instruction from user A121. Thereafter, based on the shooting instruction from user A121, shooting is started (the shutter is released) at time T904 (922) and shooting ends at time T909 (923). From time T909 onwards, live view display is performed. It is assumed that the shutter speed 924 set by user A121 is 1 / 1000 seconds.

[0063] Similarly, for user B 124, live view display starts at time T902 based on a shooting instruction from user B 124 (931). Thereafter, based on the shooting instruction from user B 124, shooting starts at time T905 (932) and ends at time T908 (933). It is assumed that the shutter speed 934 set by user B 124 is (1 / 2000) seconds.

[0064] Furthermore, for user C 127, live view display starts (941) at time T903 based on a shooting instruction from user C 127. Thereafter, based on the shooting instruction from user C 127, shooting starts (942) at time T906 and ends (943) at time T907. It is assumed that the shutter speed 944 set by user C 127 is (1 / 8000) seconds.

[0065] At this time, the camera 101 performs exposure with camera settings for each user by allocating camera control times, which are obtained by dividing one frame time, to user A 121, user B 124, and user C 127. In FIG. 9, each period indicated by arrows 951, 952, 953, 954, 955, and 956 is one frame time. Here, one frame time is the reciprocal of the frame rate of the video (unit: fps). The camera 101 captures video at a predetermined frame rate, and in response to each user's shooting instruction, transmits the video as a live view image, or synthesizes multiple frame images to generate a still image. Note that one frame time is defined as follows: <1> In addition to using the inverse of the video frame rate, <2> The fastest shutter speed that each user can set may be divided by N times the number of users (N=1, 2, 3, . . .). In this embodiment, the definition of one frame time is <1> The case where <2> The same can be done in the case of

[0066] When performing live view display and shooting as described above, the CPU 301 of the camera 101 allocates user A's camera control time 961 to the first frame time 951, which begins at time T901 when only user A 121 is performing live view display. In Figure 9, the length of the side parallel to the time axis of the rectangular frame indicating the camera control time represents the time for which the camera is controlled, and the assigned user name is shown inside the rectangular frame. This allocation of camera control time is repeated for each frame until time T902, when the next event occurs.

[0067] Furthermore, the CPU 301 equally allocates the camera control time 962 for user A and the camera control time 963 for user B to the first frame time 952, which starts at time T902 when user A 121 and user B 124 simultaneously perform live view display. This allocation of camera control time is repeated for each frame until time T903, when the next event occurs.

[0068] Similarly, CPU 301 divides the first frame time 953, which begins at time T903 when user A 121, user B 124, and user C 127 simultaneously perform live view display, equally among the three users, allocating camera control time to each. Here, the allocation ratio is set so that if all three users are only capturing images or only performing live view display, the camera control time is allocated equally to the three users. Furthermore, if capturing images and live view display are performed simultaneously, among the three users, the camera control time allocated to the photographer is set to twice that of the live view display user.

[0069] That is, between times T904 and T905 when user A 121 is shooting and user B 124 and user C 127 are displaying live view, CPU 301 allocates camera control time to each user in each frame as follows: CPU 301 allocates camera control time that is (½), (¼), and (¼) of one frame time to user A 121, user B 124, and user C 127, respectively, in each frame.

[0070] Between times T905 and T906, when user A121 and user B124 are shooting and user C127 is displaying live view, CPU 301 allocates camera control time to each user for each frame as follows: For each frame, CPU 301 allocates camera control time that is (2 / 5), (2 / 5), and (1 / 5) of one frame time to user A121, user B124, and user C127, respectively.

[0071] Furthermore, during times T906 to T907 when user A 121, user B 124, and user C 127 are taking pictures, CPU 301 divides one frame time equally among the number of users (3) and allocates camera control time to each of them.

[0072] The allocation rate for allocating one frame time to each user shown in this example is merely an example, and other values ​​may be used. Furthermore, if a user's shooting instruction is received before the end of a frame, shooting based on the shooting instruction may be performed starting with the next frame after the current frame is completed, or the current frame may be interrupted and shooting based on the shooting instruction may be immediately started. The timing for allocating camera control time to a user may be determined by a machine learning model. For example, the allocation timing may be determined based on the predicted movement of the subject the user wants to focus on and the image quality standard desired by the user. The machine learning model may be trained to determine the camera control time to be allocated to a user based on the predicted value of the subject's movement and the quality standard for the composite image. Here, the quality standard for the composite image may be, for example, the tolerance for differences in noise, blur, color, brightness, and flicker compared to an image captured with 100% of the camera control time allocated.

[0073] Fig. 10 is a timing chart for explaining the camera control time allocated to users. With reference to Fig. 10, the camera control time 961 allocated only to user A 121 in Fig. 9 and the camera control times 962 and 963 allocated to user A 121 and user B 124, respectively, will be described in detail. Fig. 10 shows the timing at which the processes of steps S702 to S707 are executed in the second repetitive process included in the shooting process shown in Fig. 7. In Fig. 10, the periods indicated by arrows 1021, 1022, and 1023 are one frame time, which is a fixed time. The periods indicated by arrows 1024 and 1025 are (1 / 2) frame times, which are (1 / 2) of one frame time.

[0074] User A's camera control time 1041 is the sum of the execution time of the camera setting process (camera setting 1042) in steps S702 to S704 in Fig. 7 and the exposure time in the process (exposure 1043) in step S705. Camera control time 1041 corresponds to, for example, user A's camera control time 961 shown in Fig. 9. From time t1001 to t1002, the camera setting process (aperture setting, focus position setting, and gain setting) in steps S702 to S704 is performed, and from time t1002 to t1003, the process (exposure) in step S705 is performed. When exposure ends at time t1003, the process (gain 1044) in step S706 is performed until time t1004, and then the process (storing pre-composite image 1045) in step S707 is performed until time t1005. The processing related to the gain 1044 and the storage 1045 of the pre-composite image is performed after the signal is read from the image sensor 211, and can therefore be performed in parallel with the camera setting 1052 and exposure 1053 performed during the next camera control time 1051 for user A. During the next camera control time 1051, when the exposure ends at time t1006, the processing in steps S706 and S707 (gain 1054 and storage 1045 of the pre-composite image) is performed by time t1007. Here, the execution time of each of the camera setting processing in steps S702 to S704 of FIG. 7, the gain processing in step S706, and the storage processing of the pre-composite image in step S707 can each be a constant time, regardless of the user or camera setting values. Furthermore, if the execution time of the camera setting processing is defined as Tconf=t1002-t1001, the exposure time is camera control time (=t1003-t1001)-Tconf.

[0075] Next, a case will be described in which user A's camera control time and user B's camera control time are allocated within one frame time 1023. In the example shown in Fig. 10, user A's camera control time 1061 is allocated to (1 / 2) frame time 1024, which starts at time t1010 and ends at time t1011. user B's camera control time 1071 is allocated to (1 / 2) frame time 1025, which starts at time t1011 and ends at time t1012. user A's camera control time 1061 and user B's camera control time 1071 correspond to user A's camera control time 962 and user B's camera control time 963 shown in Fig. 9, for example.

[0076] From time t1010 to t1011, the camera setting process (camera setting 1062) in steps S702 to S704 and the process (exposure 1063) in step S705 are performed in the photographing process for user A 121. When exposure is completed at time t1011, the processes (gain 1064 and storage of pre-composite image 1065) in steps S706 and S707 in the photographing process for user A 121 are then performed. The processes related to gain 1064 and storage of pre-composite image 1065 are performed after signals are read out from the image sensor 211, and therefore can be performed in parallel with the camera setting 1072 and exposure 1073 performed at the next camera control time 1071 for user B. At the next camera control time 1071, when exposure is completed at time t1012 in the photographing process for user B 124, the processes (gain 1074 and storage of pre-composite image 1075) in steps S706 and S707 are performed.

[0077] Similarly, in one frame time of the next frame onwards, the camera control time of user A is allocated to the first half of the frame time, and the camera control time of user B is allocated to the second half of the frame time.

[0078] Fig. 11 is a diagram for explaining image composition in this embodiment. In Fig. 11, the same components as those shown in Fig. 9 are assigned the same reference numerals. Fig. 11 shows the process of generating a composite image by combining images taken at a shutter speed of 934 (1 / 2000 seconds) based on a shooting instruction from user B124.

[0079] As described in FIG. 9, during the period from time T905 to time T906 when users A 121 and B 124 are photographing and user C 127 is displaying a live view, user B's camera control time 1101 is 2 / 5 of one frame time. The exposure time during the camera control time is the camera control time minus the time for camera setting processing. For ease of explanation, the time for camera setting processing is set to 0 seconds, and the exposure time during the camera control time is set to the same time as the camera control time. In other words, the exposure time during camera control time 1101 is set to 2 / 5 of one frame time. Therefore, the gain of the signal of the image exposed and acquired during camera control time 1101 is set to 5 / 2 (= the reciprocal of the actual exposure time) in the gain setting in step S704 of FIG. 7 so that it is the amount of charge when exposed for one frame time. 7 in accordance with the set gain, the result is output as gained pre-combine image data 1111, which is then stored in step S705 in the data storage unit 305. Until time T906, the second iterative process is executed for each of the subsequent frames, and from time T905 to T906, the gained pre-combine image data is stored for each frame.

[0080] Similarly, between times T906 and T907 when users A 121, B 124, and C 127 are photographing, user B's camera control time 1102 is 1 / 3 of one frame time. The signal of the image exposed and captured during camera control time 1102, which is included in the first frame time, is amplified by 10 dB (=3 / 1), and the resulting gain-processed pre-combination image data 1112 is stored in the data storage unit 305. Until time T907, a second iterative process is performed on each of the subsequent frames, and gain-processed pre-combination image data is stored for each frame between times T906 and T907.

[0081] Furthermore, between times T907 and T908, when users A 121 and B 124 are photographing and user C 127 is performing live view display, user B's camera control time 1103 is (2 / 5) of one frame time, just like times T905 and T906. Therefore, the signal of the image acquired by exposure during camera control time 1103 is amplified with a gain of (5 / 2), and the resulting gain-processed pre-combination image data 1113 is stored in the data storage unit 305. Until time T908, the second repeat process is executed for each of the subsequent frames, and gain-processed pre-combination image data is stored for each frame between times T907 and T908.

[0082] In this way, from time T905 to time T908, image data with the same amount of charge as when only user B 124 is exposed at the shutter speed (1 / 2000 seconds) set by user B 124 is stored for each frame. After capturing of user B 124 is completed at time T908, the CPU 301 extracts the gain-processed pre-combination image data from time T905 to time T908 and combines them using the neural network processing unit 303 or the like to generate a single combined image 1121. At this time, the gain-processed pre-combination image data acquired from time T905 to time T906 is combined by machine learning model A (not shown) to output intermediate image data A. The gain-processed pre-combination image data acquired from time T906 to time T907 is combined by machine learning model B (not shown) to output intermediate image data B. The gain-processed pre-combination image data acquired from time T907 to time T908 is combined by machine learning model A (not shown) to output intermediate image data C. Finally, intermediate image data A, intermediate image data B, and intermediate image data C are synthesized by machine learning model C to output synthesized image data 1121. Since the machine learning models (A, B) created for each gain value reduce the amount of data required to construct the model compared to a single machine learning model corresponding to multiple gain values, machine learning models A and B are switched between and used according to the gain value.

[0083] According to this embodiment, one frame time is allocated to each user based on the number of users simultaneously performing shooting or live view display and their shooting instructions, and camera control time is allocated to each user. Next, for each frame, camera settings corresponding to the user and exposure and gain processing are performed within the camera control time allocated to each user, and one frame's worth of video information is acquired. Finally, by combining the video information acquired for the frame corresponding to the shutter speed desired by each user, a desired captured image according to the shooting instructions can be obtained. In other words, one or more users using one camera 101 can all shoot at their desired shooting timing. Furthermore, each user can shoot with their desired camera settings.

[0084] (Another embodiment of the present invention) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0085] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gateway (FPGA), a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0086] It should be noted that the above-described embodiments are merely examples of specific embodiments for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be carried out in various forms without departing from its technical concept or main features.

[0087] The disclosure of this embodiment includes the following configurations, methods, etc. (Configuration 1) a receiving means for receiving a photographing instruction from a user; an imaging means for taking an image in response to the imaging instruction; an allocation means for allocating control time of the imaging device to each of the users based on the photography instruction and the number of users who use the imaging device; An imaging apparatus comprising: a control means for controlling said imaging means so that an image is taken during said allocated control time in accordance with said imaging instruction from a user to whom said control time is allocated. (Configuration 2) The imaging device according to configuration 1, wherein the control time allocated by the allocation means varies depending on the operation performed in response to the shooting instruction. (Configuration 3) 3. The imaging device according to configuration 2, wherein the operations performed in response to the shooting instruction include shooting and live view display. (Configuration 4) The imaging device according to configuration 3, wherein the control time allocated when the operation performed in response to the shooting instruction is shooting is longer than the control time allocated when the operation performed in response to the shooting instruction is live view display. (Configuration 5) 5. The imaging device according to any one of configurations 1 to 4, further comprising an amplifier that amplifies a signal of an image captured by the imaging device based on the allocated control time. (Configuration 6) 6. The imaging device according to configuration 5, further comprising a synthesizing means for synthesizing the image signals for each frame amplified by the amplifying means to generate one synthesized image. (Configuration 7) 7. The imaging device according to any one of configurations 1 to 6, wherein the allocation means allocates the control time to each user who uses the imaging device for each frame. (Configuration 8) 8. The imaging device according to configuration 7, wherein the allocating means determines the control time to be allocated based on the duration of one frame of a moving image. (Configuration 9) 8. The imaging device according to configuration 7, wherein the allocating means determines the control time to be allocated based on the fastest shutter speed among shutter speeds set by a user of the imaging device. (Configuration 10) The imaging device described in any one of configurations 1 to 9, characterized in that when the receiving means receives the shooting instruction, the allocation means updates the allocation of the control time to each user who uses the imaging device. (Method 1) A control method for an imaging device having an imaging means for taking an image, comprising: a reception step of receiving a photographing instruction from a user; an allocating step of allocating control time of the imaging device to each of the users based on the shooting instruction and the number of users who use the imaging device; a control step of controlling the imaging means so that an image is captured during the allocated control time based on the imaging instruction from the user to whom the control time is allocated. (Program 1) A computer of an imaging device having an imaging means for taking an image, a reception step for receiving a photographing instruction from a user; an allocating step of allocating control time of the imaging device to each of the users based on the shooting instruction and the number of users who use the imaging device; a control step of controlling the imaging means so that photography is performed during the allocated control time based on the photography instruction from the user to whom the control time is allocated. [Explanation of symbols]

[0088] 101: Imaging device (camera) 102: Camera body 103: Interchangeable lens 105: Communication network 122, 125, 128: Terminal device 211: Imaging element 212: Amplification unit 213: Communication unit 214: Display unit 215: Operation unit 220: Camera internal processing unit 301: CPU 302: Memory 303: Neural network processing unit 304: Image processing unit 305: Data storage unit 306: Input unit 307: Control unit 308 Communication control unit 309: Display control unit

Claims

1. a receiving means for receiving a photographing instruction from a user; an imaging means for taking an image in response to the imaging instruction; an allocation means for allocating control time of the imaging device to each of the users based on the photography instruction and the number of users who use the imaging device; An imaging apparatus comprising: a control means for controlling said imaging means so that an image is taken during said allocated control time in accordance with said imaging instruction from a user to whom said control time is allocated.

2. 2. The imaging apparatus according to claim 1, wherein the control time allocated by the allocation means varies depending on the operation performed in response to the photography instruction.

3. 3. The imaging apparatus according to claim 2, wherein the operations performed in response to the shooting instruction include shooting and live view display.

4. 4. The imaging device according to claim 3, wherein the control time allocated when the operation performed in response to the shooting instruction is shooting is longer than the control time allocated when the operation performed in response to the shooting instruction is live view display.

5. 2. The imaging apparatus according to claim 1, further comprising an amplifier that amplifies a signal of an image captured by said imaging means based on said allocated control time.

6. 6. The imaging apparatus according to claim 5, further comprising a synthesizing unit for synthesizing the image signals for each frame amplified by the amplifying unit to generate one synthesized image.

7. 2. The image pickup apparatus according to claim 1, wherein the allocating means allocates the control time to each user who uses the image pickup apparatus for each frame.

8. 8. The imaging device according to claim 7, wherein the allocating means determines the control time to be allocated based on the duration of one frame of a moving image.

9. 8. The image capture device according to claim 7, wherein the allocating unit determines the control time to be allocated based on the fastest shutter speed among shutter speeds set by a user of the image capture device.

10. 2. The imaging device according to claim 1, wherein the allocation unit updates the allocation of the control time to each user who uses the imaging device when the acceptance unit accepts the shooting instruction.

11. A control method for an imaging device having an imaging means for taking an image, comprising: a reception step of receiving a photographing instruction from a user; an allocating step of allocating control time of the imaging device to each of the users based on the shooting instruction and the number of users who use the imaging device; a control step of controlling the imaging means so that an image is captured during the allocated control time based on the imaging instruction from the user to whom the control time is allocated.

12. A computer of an imaging device having an imaging means for taking an image, a reception step for receiving a photographing instruction from a user; an allocating step of allocating control time of the imaging device to each of the users based on the shooting instruction and the number of users who use the imaging device; a control step of controlling the imaging means so that photography is performed during the allocated control time based on the photography instruction from the user to whom the control time is allocated.

Images