Imaging system

Abstract

Provided are an imaging system that enables cost reduction and functional enhancement of an imaging terminal, an imaging terminal, and a server. This imaging system is composed of an imaging terminal (100-1) and a server (200). The imaging terminal (100-1) comprises an imaging unit including an image sensor and which outputs imaging data, a first communication unit for transmitting imaging data output from the imaging unit to the server (200), a first image processing engine (122) for performing image processing on the imaging data output from the imaging unit, and a memory for storing an image subjected to image processing by the first image processing engine (122). The server (200) comprises a second communication unit for receiving imaging data transmitted from the imaging terminal (100-1), and a second image processing engine (210) for performing image processing on the received imaging data to generate an image for recording, the second image processing engine (210) being different from the first image processing engine (122) of the imaging terminal (100-1).

Description

Technical Field

[0001] The present invention relates to an imaging system, an imaging terminal, and a server, and particularly to a technique for performing image processing on unprocessed data acquired by an imaging terminal on a server on a network.

Background Art

[0002] The digital camera system described in Patent Document 1 is composed of a digital camera (a digital camera integrated mobile phone) and a server center that communicates with the digital camera through the Internet.

[0003] The digital camera has an imaging unit, a temporary storage unit that stores RAW data output from the imaging unit in response to a shutter release means, and a communication means that wirelessly transmits the RAW data of the temporary storage unit without compression. The server center has a communication means for receiving RAW data, a compression means for compressing the received RAW data, and a storage means for storing the compressed compression signal. Further, the server center has an interpolation means for interpolating the received RAW data to create a digital image signal for each color, and a white balance adjustment means.

[0004] With this digital camera system, the server center is responsible for image processing and the like for RAW data, so that a high-performance digital camera can be provided at a low cost.

[0005] On the other hand, the digital camera system described in Patent Document 1 has a problem that the usability of the camera deteriorates because image processing for RAW data is always performed on the server center.

[0006] Patent Document 2 proposes an electronic device that solves the above problems.

[0007] The electronic device described in Patent Document 2 comprises a processing unit for processing imaging signals captured by an imaging unit, a communication unit capable of transmitting imaging signals (RAW data) captured by the imaging unit to a server, and a determination unit that determines whether or not to transmit RAW data to the server according to the imaging settings of the imaging unit.

[0008] The decision unit decides not to send RAW data to the server if wireless communication with the server is not possible and the temperature of the image sensor is below a specified value, and to send RAW data to the server if wireless communication with the server is possible and the video is to be recorded on a recording medium in the electronic device, or if the temperature of the image sensor exceeds a specified value.

[0009] The first ASIC (application-specific integrated circuit) within the electronic device performs various image processing on the RAW data if it does not send RAW data to the server. On the other hand, when the server receives RAW data from the electronic device, the server's second ASIC performs image processing on the received RAW data to generate a video for recording (which also serves as the image for live view).

[0010] Specifically, the electronic device described in Patent Document 2 determines whether to send RAW data to the server based on the communication status with the server, the heat generation status of the image sensor, etc., and performs image processing on the RAW data by the first ASIC in the device or by the second ASIC in the server according to that decision. The second ASIC in the server is equivalent to the first ASIC in the electronic device. [Prior art documents] [Patent Documents]

[0011] [Patent Document 1] Japanese Patent Publication No. 2003-87618 [Patent Document 2] Japanese Patent Publication No. 2019-54536 [Overview of the project] [Problems that the invention aims to solve]

[0012] One embodiment of the technology described herein provides an imaging system, an imaging terminal, and a server that enable cost reduction and enhanced functionality of imaging terminals. [Means for solving the problem]

[0013] The invention according to the first aspect is an imaging system comprising at least one imaging terminal and a server, wherein the imaging terminal includes an imaging unit that includes an image sensor and outputs imaging data, a first communication unit that transmits the imaging data output from the imaging unit to the server, a first image processing engine that processes the imaging data output from the imaging unit, and a memory that stores the image processed by the first image processing engine, and the server comprises a second communication unit that receives imaging data transmitted from the first communication unit of the imaging terminal, and a second image processing engine that processes the received imaging data and generates an image for recording, the second image processing engine being different from the first image processing engine of the imaging terminal.

[0014] In the imaging system according to a second aspect of the present invention, it is preferable that the second image processing engine generates a live view image based on continuously received imaging data.

[0015] In the imaging system according to the third aspect of the present invention, it is preferable that the server records the image for recording generated by the second image processing engine in the image recording unit.

[0016] In the imaging system according to the fourth aspect of the present invention, it is preferable that the second image processing engine has technical specifications that exceed the technical specifications of the first image processing engine.

[0017] In the imaging system according to the fifth aspect of the present invention, it is preferable that the first image processing engine and the second image processing engine have different image processing performance.

[0018] In the imaging system according to the sixth aspect of the present invention, it is preferable that the first image processing engine has less information amount per pixel capable of performing image processing with respect to the second image processing engine.

[0019] In the imaging system according to the seventh aspect of the present invention, it is preferable that the first image processing engine has a different number of bits of harmony or processing bits with respect to the second image processing engine.

[0020] In the imaging system according to the eighth aspect of the present invention, it is preferable that the first image processing engine has only some of the functions capable of performing image processing with respect to the second image processing engine.

[0021] In the imaging system according to the ninth aspect of the present invention, it is preferable that the first image processing engine has a smaller number of pixels of the image capable of performing image processing with respect to the second image processing engine.

[0022] In the imaging system according to the tenth aspect of the present invention, it is preferable that the first image processing engine has an arithmetic element with a small thermal design power with respect to the second image processing engine.

[0023] In the imaging system according to the eleventh aspect of the present invention, it is preferable that the first image processing engine has an arithmetic element with a smaller number of transistors with respect to the second image processing engine.

[0024] In the imaging system according to the twelfth aspect of the present invention, it is preferable that the first image processing engine has a smaller number of processor cores with respect to the second image processing engine.

[0025] In the imaging system according to the thirteenth aspect of the present invention, it is preferable that the first image processing engine has an arithmetic element with a low operating clock frequency with respect to the second image processing engine.

[0026] In the imaging system according to the 14th aspect of the present invention, it is preferable that the first image processing engine has arithmetic elements with a lower rated operating current value than the second image processing engine.

[0027] In the imaging system according to the 15th aspect of the present invention, it is preferable that the first image processing engine has a smaller cache memory capacity than the second image processing engine.

[0028] In the imaging system according to the 16th aspect of the present invention, it is preferable that the first image processing engine has a configuration with a smaller number of executable instructions than the second image processing engine.

[0029] In the imaging system according to the 17th aspect of the present invention, it is preferable that the first image processing engine has a configuration with a smaller number of arithmetic units for executing arithmetic instructions than the second image processing engine.

[0030] In the imaging system according to the 18th aspect of the present invention, it is preferable that the first image processing engine has a built-in graphics function and the second image processing engine has an extended graphics function.

[0031] In the imaging system according to the 19th aspect of the present invention, when the imaging terminal receives a still image shooting instruction by a user operation, it transmits shooting instruction information from the first communication unit to the server. When the second image processing engine receives the shooting instruction information via the second communication unit, it is preferable that the second image processing engine performs image processing on the imaging data corresponding to the shooting instruction information among the continuous imaging data and generates a still image for recording.

[0032] In the imaging system according to the 20th aspect of the present invention, when the imaging terminal receives a video recording instruction or a recording end instruction by a user operation, it transmits recording instruction information or recording end instruction information from the first communication unit to the server. When the second image processing engine receives the recording instruction information or the recording end instruction information via the second communication unit, it is preferable that the second image processing engine performs image processing on the imaging data from when it receives the recording instruction information until it receives the recording end instruction information among the continuous imaging data and generates a video for recording.

[0033] In the imaging system according to the 21st aspect of the present invention, it is preferable that the imaging terminal transmits terminal information indicating the imaging terminal to the server when communication with the server begins, the server receives the terminal information and acquires RAW development information corresponding to the received terminal information, and the second image processing engine performs image processing to develop the imaging data in RAW format based on the acquired RAW development information.

[0034] In the imaging system according to the 22nd aspect of the present invention, it is preferable that the server transmits a live view image generated by the second image processing engine to the imaging terminal via the second communication unit, and when the imaging terminal receives the live view image from the server via the first communication unit, it displays the live view image on the display of the imaging terminal.

[0035] In the imaging system according to the 23rd aspect of the present invention, the first image processing engine is preferably operated during periods when communication between the imaging terminal and the server is unavailable.

[0036] In the imaging system according to the 24th aspect of the present invention, it is preferable that the first image processing engine generates a live view image based on imaging data continuously output from the image sensor during periods when communication between the imaging terminal and the server is unavailable, and the imaging terminal displays the live view image generated by the first image processing engine on a display during periods when communication between the imaging terminal and the server is unavailable.

[0037] In the imaging system according to the 25th aspect of the present invention, it is preferable that the imaging terminal communicates with a server, receives an image from the server that is specified by user operation from among the images recorded in the image recording unit, and displays the received image on a display or saves it to memory.

[0038] The invention according to the 26th aspect is an imaging system comprising at least one imaging terminal and a server, wherein the imaging terminal includes an image sensor and comprises an imaging unit that outputs imaging data and a first communication unit that transmits the imaging data output from the imaging unit to the server, the server comprises a second communication unit that receives imaging data transmitted from the first communication unit of the imaging terminal and an image processing engine that processes the received imaging data and generates an image for recording, the data corresponding to one pixel of the image sensor of the imaging data has the maximum number of gradations converted by the analog-to-digital conversion circuit, the image processing engine generates a live view image based on the continuously received imaging data, the server transmits the generated live view image to the imaging terminal via the second communication unit, and when the imaging terminal receives the live view image from the server via the first communication unit, it displays the live view image on the display of the imaging terminal.

[0039] The invention relating to the 27th aspect is an imaging terminal that constitutes an imaging system described in any of the first to 25 aspects.

[0040] The invention relating to the 28th aspect is a server that constitutes an imaging system described in any of the 1st to 25th aspects. [Brief explanation of the drawing]

[0041] [Figure 1] Figure 1 shows the system configuration of the imaging system according to the present invention. [Figure 2] Figure 2 is a block diagram showing an embodiment of the imaging terminal that constitutes the imaging system. [Figure 3] Figure 3 is a block diagram showing an embodiment of the server that constitutes the imaging system. [Figure 4] Figure 4 is a block diagram showing an embodiment of the primary image processing circuit of the front-end LSI. [Figure 5] Figure 5 is a block diagram showing an embodiment of the secondary image processing circuit of the front-end LSI. [Figure 6]Figure 6 is a schematic diagram showing a first embodiment of the imaging system according to the present invention. [Figure 7] Figure 7 is a schematic diagram showing a second embodiment of the imaging system according to the present invention, and illustrates the case when communication between the imaging terminal and the server is normal. [Figure 8] Figure 8 is a schematic diagram showing a second embodiment of the imaging system according to the present invention, illustrating a case where communication between the imaging terminal and the server is impossible. [Figure 9] Figure 9 is a schematic diagram showing a third embodiment of the imaging system according to the present invention. [Figure 10] Figure 10 is a schematic diagram showing a fourth embodiment of the imaging system according to the present invention. [Figure 11] Figure 11 is a flowchart illustrating the operation of the imaging terminal in the imaging system, specifically the operation in shooting mode. [Figure 12] Figure 12 is a flowchart showing the operation of the imaging terminal in the imaging system, and is a flowchart showing the operation in playback mode. [Figure 13] Figure 13 is a flowchart showing the operation of the server in the imaging system. [Modes for carrying out the invention]

[0042] Preferred embodiments of the imaging system, imaging terminal, and server according to the present invention will be described below with reference to the attached drawings.

[0043] [Overview of the imaging system] Figure 1 shows the system configuration of the imaging system according to the present invention.

[0044] As shown in Figure 1, the imaging system 1 consists of at least one imaging terminal 100 and a server 200.

[0045] The imaging terminal 100 includes a digital camera 100a with communication capabilities, a smartphone 100b with a built-in camera, and various other camera-equipped gadgets (drones, rooftop fixed cameras, etc.).

[0046] The imaging terminal 100 is constantly connected to the server 200 via the wireless access point 310 and network 300 while powered on. During shooting, it transmits continuously output imaging data from the imaging unit to the server 200. During playback, it receives the user-requested image (playback image) from the images stored on the server 200 and displays the playback image on the LCD (Liquid Crystal Display). The imaging data output from the imaging unit is data before image processing and is hereinafter referred to as "RAW data".

[0047] Furthermore, the imaging terminal 100 receives live view images generated from continuous RAW data from the server 200 during shooting, and displays the received live view images on the EVF (Electronic View Finder) or LCD.

[0048] Furthermore, the imaging terminal 100 is equipped with an image processing engine (first image processing engine) that enables offline shooting in case communication with the server 200 is impossible or difficult (including cases where communication is unstable).

[0049] The method for identifying when communication between the imaging terminal 100 and the server 200 is impossible or difficult may be any of the following: detecting the communication environment such as radio wave strength, checking the communication status such as data transmission error checks, detecting input operations by the user that indicate communication is difficult, or a combination thereof.

[0050] Furthermore, image processing can be divided into primary image processing, which is performed as a preliminary step before RAW development, such as correcting defects and correcting sensitivity errors for each pixel, and secondary image processing, which is performed during or after RAW development. However, according to one embodiment of the present invention, under normal operating conditions, the imaging data transmitted to the network 300 by the imaging terminal 100 of the imaging system 1 is transmitted in the state of either "unprocessed output data from an image sensor with no primary image processing performed at all" or "data with only a portion of the primary image processing performed." As a result, when the image processing engine (second image processing engine) on the server 200 on the network is available, the advantages of the second image processing engine on the network can be utilized, such as performing primary image processing with higher functionality and performance on the second image processing engine on the network, increasing the processing speed to increase the continuous shooting speed, or improving image quality by applying more complex and high-precision image processing.

[0051] The image processing performed as primary image processing includes defect correction and correction of sensitivity errors for each pixel, as well as tuning processes that aim to obtain desirable image quality by processing color and brightness on a pixel-by-pixel or area-by-area basis. However, the details of these processes are not publicly disclosed by the manufacturer as proprietary know-how. Here, this refers to all processes that improve the image quality of digital data before RAW development.

[0052] The server 200 is equipped with a second image processing engine and an image recording unit that are different from the first image processing engine of the imaging terminal 100, and performs bidirectional communication with one or more imaging terminals 100 via the network 300 and wireless access point 310 to exchange necessary data.

[0053] Server 200 receives continuous RAW data from the imaging terminal 100, generates live view images and recording images from the continuous RAW data using a second image processing engine, transmits the generated live view images to the imaging terminal 100, and stores the recording images in the image recording unit of Server 200. Note that the image recording unit may be provided by a physically separate server (for example, a data server) from Server 200, which has the second image processing engine.

[0054] In this example, "RAW data" is not limited to still image data in RAW file format, but refers to image data before RAW development. Therefore, "continuous RAW data" can be achieved by continuously transmitting data contained in RAW file format, but more broadly, it refers to continuous output data from the image sensor. Thus, "continuous RAW data" includes streaming data that is not formatted into files on a still image basis, as well as A / D (Analog-to-Digital) converted data sequences.

[0055] In the most typical implementation, the "continuous RAW data" transmitted as continuous data is input to the second image processing engine as a data sequence that has not been formatted into individual still image files. The first image processing engine then performs image processing and RAW development, converts it into a video data format for live view display, and transmits it as live view output. For recording purposes, the second image processing engine converts it into a file format set in advance by the user, and then outputs it as a still image or video file.

[0056] The sequential data may be sequential data that has only had its bitrate converted without any primary image processing, such as 16-bit video RAW data.

[0057] Furthermore, the server 200 transmits the image stored in the image recording unit as a reproduced image to the imaging terminal 100 in response to a request from the imaging terminal 100.

[0058] By processing the image sensor output of the imaging terminal 100 using the server 200's second image processing engine, not only can the imaging terminal 100 be made smaller, lighter, and less expensive, but power consumption will also decrease, allowing for longer use. Development will also become easier, making it easier to create variations of the imaging terminal 100. The latest updates can be reflected in the second image processing engine without having to replace the imaging terminal 100 (this can also be done using machine learning). The relationship between the input (RAW data) and output of the second image processing engine can be obtained over the network, providing a vast amount of development data (including training data used for machine learning) for developing a better image processing engine. By time-sharing the image processing engine on the network, waste can be eliminated, making it possible to use a low-cost, high-performance image processing engine. Peripheral circuits of the image processing engine (especially memory for continuous shooting, which is used infrequently) can be shared, so if sharing is done efficiently, the limit on the number of continuous shots can be effectively eliminated. Many other benefits can be expected.

[0059] <Imaging terminal> Figure 2 is a block diagram showing an embodiment of the imaging terminal that constitutes the imaging system.

[0060] The imaging terminal shown in Figure 2 is a digital camera 100a with communication capabilities.

[0061] As shown in Figure 2, the digital camera 100a includes an imaging unit 101, a sensor driver 108, a first processor 110, a memory 112, an operation unit 114, a display control unit 116, an LCD 118, an EVF 120, a first image processing engine 122, and a first communication unit 124.

[0062] The imaging unit 101 includes a photographic lens 102, an image sensor 104, and an AFE (Analog Front End) 106.

[0063] The shooting lens 102 may be a lens integrated with the camera body, or it may be a detachable interchangeable lens that can be attached to the camera body.

[0064] The image sensor 104 can be configured as a CMOS (Complementary Metal-Oxide Semiconductor) type color image sensor. However, the image sensor 104 is not limited to a CMOS type; a CCD (Charge Coupled Device) type image sensor may also be used.

[0065] The image sensor 104 consists of multiple pixels composed of photoelectric conversion elements (photodiodes) arranged two-dimensionally in the x-direction (horizontal direction) and y-direction (vertical direction). On these pixels, red (R), green (G), and blue (B) color filters are arranged in a periodic arrangement (e.g., Bayer array, X-Trans®, etc.), and a microlens is placed on each photodiode.

[0066] The optical image of the subject formed on the light-receiving surface of the image sensor 104 by the photographic lens 102 is converted into an electrical signal by the image sensor 104. An electric charge corresponding to the amount of incident light is accumulated in each pixel of the image sensor 104, and an electrical signal corresponding to the amount of charge (signal charge) accumulated in each pixel is read out from the image sensor 104 as an image signal.

[0067] The AFE106 performs various analog signal processing operations on the analog image signal output from the image sensor 104. The AFE106 includes a correlated double sampling circuit, an AGC (Automatic Gain Control) circuit, and an analog-to-digital conversion circuit (A / D conversion circuit) (all not shown in the diagram). The correlated double sampling circuit performs correlated double sampling on the analog signal from the image sensor 104 to remove noise caused by the reset of the signal charge. The AGC circuit amplifies the analog signal from which noise has been removed by the correlated double sampling circuit, ensuring that the signal level of the analog signal falls within an appropriate range. The A / D conversion circuit converts the image signal, whose gain has been adjusted by the AGC circuit, into a digital signal.

[0068] The digital signal read from the image sensor 104 and output from the AFE 106 is the B, G, and R pixel data (B, G, R data) corresponding to the color filter array of the image sensor 104, and is hereinafter referred to as "RAW data". The RAW data is a mosaic image data in which the B, G, and R data are arranged sequentially according to the color filter array.

[0069] Furthermore, if the image sensor 104 is a CMOS type image sensor, the AFE 106 is often built into the image sensor 104.

[0070] Furthermore, in this specification, analog signal processing by AFE106 is not referred to as image processing; rather, signal processing on digital signals after they have been converted to digital signals by the A / D conversion circuit is defined as image processing. Therefore, the RAW data output from the image sensor 104 (AFE106) is data before image processing.

[0071] The sensor driver 108 controls the reading of image signals from the image sensor 104 according to the commands of the first processor 110. The sensor driver 108 also has an electronic shutter function that, based on an electronic shutter control signal from the first processor 110, discharges (resets) the charge accumulated in each pixel of the image sensor 104 and starts exposure.

[0072] The first processor 110 consists of a CPU (Central Processing Unit) and the like, and controls each part in accordance with user operations using the operation unit 114, and performs various processes.

[0073] Furthermore, the first processor 110 performs AF (Auto Focus) control and AE (Automatic Exposure) control.

[0074] When performing AF control, the first processor 110 calculates the values ​​necessary for AF control based on the digital image signal. In the case of so-called contrast AF, for example, it calculates the integrated value (focus evaluation value) of the high-frequency components of the G data within a predetermined AF area. The first processor 110 moves the focus lens included in the photographic lens 102 to the position where the focus evaluation value is maximum (i.e., the position where the contrast is maximum) during AF control. Note that AF is not limited to contrast AF; for example, if the image sensor 104 includes pixels for phase-detection, it may also perform phase-detection AF, which detects the amount of defocus based on the pixel data of the phase-detection pixels and moves the focus lens so that this amount of defocus becomes zero.

[0075] When performing AE control, the first processor 110 detects the brightness of the subject (subject luminance) and calculates a numerical value (exposure value (EV value)) necessary for AE control corresponding to the subject luminance. Based on the calculated EV value, the first processor 110 determines the F-number, shutter speed, and ISO sensitivity from a predetermined program diagram and can perform AE control.

[0076] It goes without saying that AF control and AE control are performed automatically when auto mode is set via the control unit 114, and that AF control and AE control are not performed when manual mode is set.

[0077] Memory 112 includes flash memory, ROM (Read-only Memory), RAM (Random Access Memory), etc. Flash memory and ROM are non-volatile memories that store various programs, parameters, etc., including firmware.

[0078] The RAM functions as a workspace for processing by the first processor 110 and also temporarily stores firmware and other data stored in non-volatile memory. The first processor 110 may also have a portion of the memory 112 (RAM) built into it.

[0079] The control unit 114 includes a power switch, shutter button, MENU / OK key, directional keys, play button, etc.

[0080] The LCD118 is located on the back of the camera body and functions as a display that shows the live view image in shooting mode, plays back and displays previously taken images in playback mode, and also displays various menu screens. The EVF120 can also display the same information as the LCD118. When shooting mode, if you bring your eye close to the EVF120, the display will automatically switch to the EVF120 via an eye sensor (not shown), and when you move your eye away, it will switch back to the LCD118.

[0081] The MENU / OK key on the control unit 114 is an operation key that combines the functions of a menu button for issuing a command to display a menu on the LCD 118 screen, and an OK button for issuing a command to confirm and execute the selected content.

[0082] The directional pad is an input unit that allows input in four directions: up, down, left, and right. It functions as a button for selecting items from the menu screen and for selecting various settings from each menu. The up and down keys of the directional pad also function as zoom switches during shooting or playback zoom switches in playback mode, while the left and right keys function as frame-by-frame (forward and reverse) buttons in playback mode. The playback button switches to playback mode, which displays the captured images on the LCD118.

[0083] The first image processing engine 122 operates when communication with the server 200 is not possible. The first image processing engine 122 performs image processing such as RAW development on the RAW data continuously read from the imaging unit 101 to generate a live view image and an image for recording. Details of the first image processing engine 122 will be described later. Alternatively, the first processor 110 may perform the image processing functions of the first image processing engine 122, or a part of the image processing functions of the first image processing engine 122.

[0084] Here, the RAW data read from the imaging unit 101 (image sensor 104) is continuous data read out at a set frame rate (e.g., 30fps, 60fps, etc.). The B, G, and R data in the RAW data each have gradations corresponding to the maximum number of bits (14 bits in this example) of the A / D conversion circuit in the AFE 106.

[0085] The first communication unit 124 is constantly connected to the server 200 via the wireless access point 310 and network 300 while the digital camera 100a is running, and performs bidirectional communication with the server 200. When shooting, the first communication unit 124 transmits the image data (RAW data) before image processing to the server 200 in the form of continuous data with consecutive frames, and receives the live view image corresponding to the continuous data (RAW data with consecutive frames) generated by the server 200 through image processing such as RAW development processing. When playback, the first communication unit 124 transmits information about the image to be played back as specified by the user, and receives the image specified by the user from the server 200. The first communication unit 124 also transmits the shutter release signal (shooting instruction information), terminal information indicating the digital camera 100a, etc. to the server 200, but the details thereof will be described later. Furthermore, while full-duplex wireless communication that can simultaneously transmit and receive is preferred for the first communication unit 124, it may also be possible to transmit and receive simultaneously using two or more wireless lines. Furthermore, the first communication unit 124 may have a separate transmitting and receiving configuration (for example, optical light for uplink and radio waves for downlink).

[0086] <server> Figure 3 is a block diagram showing an embodiment of the server that constitutes the imaging system.

[0087] The server 200 shown in Figure 3 performs bidirectional communication with one or more imaging terminals 100 via the network 300 and wireless access point 310, and is responsible for image processing and image recording functions for the imaging terminals 100.

[0088] The server 200 shown in Figure 3 comprises a second communication unit 202, a second image processing engine 210, a second processor 250, and an image recording unit 260.

[0089] The second communication unit 202 communicates bidirectionally with the imaging terminal 100 via the network 300 and wireless access point 310, receives continuous data from the imaging terminal 100, and transmits a live view image corresponding to the continuous data generated by the second image processing engine 210 to the imaging terminal 100. Furthermore, during playback on the imaging terminal 100, the second communication unit 202 receives information regarding the image to be played back as specified by user operation and transmits the corresponding image to the imaging terminal 100. In addition, the second communication unit 202 receives a shutter release signal, terminal information indicating the imaging terminal 100, etc., from the imaging terminal 100. In this example, the imaging terminal 100 is a digital camera 100a with communication capabilities, but it is preferable to have multiple second communication units 202 so that communication with multiple imaging terminals 100 can be performed without time lag.

[0090] The second image processing engine 210 is a different type of image processing engine from the first image processing engine of the imaging terminal 100 (in this example, the first image processing engine 122 of the digital camera 100a with communication function), and has technical specifications that exceed the technical specifications of the first image processing engine 122.

[0091] Conversely, the technical specifications of the first image processing engine 122 are lower than those of the second image processing engine 210.

[0092] Specifically, when comparing the first image processing engine 122 and the second image processing engine 210, at least one of the following image processing performance characteristics differs.

[0093] (1) The first image processing engine 122 has less information per pixel that can be processed compared to the second image processing engine 210.

[0094] (2) The first image processing engine 122 differs from the second image processing engine 210 in the number of bits for gradation or the number of processing bits. For example, the number of bits for gradation of data processed by the first image processing engine 122 can be 8 bits, and the number of bits for gradation of data processed by the second image processing engine 210 can be 14 bits. Also, the image processing by the first image processing engine 122 is image processing for data having 8 bits for gradation, and 2 Image processing engine 210 Image processing by this method may include image processing of data having 14-bit grayscale.

[0095] (3) The first image processing engine 122 has only some of the functions that enable it to perform image processing on the second image processing engine 210. For example, it does not have an image compression function.

[0096] (4) The first image processing engine 122 has fewer image pixels that it can process compared to the second image processing engine 210.

[0097] (5) The first image processing engine 122 has computing elements with lower thermal design power than the second image processing engine 210. That is, the first image processing engine 122 consumes less power and generates less heat than the second image processing engine 210.

[0098] (6) The first image processing engine 122 has fewer processing elements than the second image processing engine 210.

[0099] (7) The first image processing engine 122 has fewer processor cores than the second image processing engine 210.

[0100] (8) The first image processing engine 122 has arithmetic elements with a lower operating clock frequency than the second image processing engine 210.

[0101] (9) The first image processing engine 122 has arithmetic elements with a lower rated operating current value than the second image processing engine 210.

[0102] (10) The first image processing engine 122 has a smaller cache memory capacity than the second image processing engine 210.

[0103] (11) The first image processing engine 122 has a configuration that allows it to execute fewer instructions than the second image processing engine 210.

[0104] (12) The first image processing engine 122 has fewer processing units for executing processing instructions compared to the second image processing engine 210.

[0105] (13) The first image processing engine 122 has an integrated graphics function, and the second image processing engine has an extended graphics function.

[0106] Technical differences in specifications include, but are not limited to, image processing performance, a wider programmable area and greater flexibility in internal processing (which is different from the superiority or inferiority of the image processing performance itself), different process rules even if the architecture and circuit size are the same, resulting in different power consumption, differences in thermal management functions and performance such as temperature sensing capabilities, and differences in chip size resulting in space saving.

[0107] Now, the second image processing engine 210 in this example comprises a front-end LSI (Large-Scale Integration) 239 and its internal primary memory 240, and the front-end LSI 239 includes a primary image processing circuit 220 and a secondary image processing circuit 230.

[0108] The primary image processing circuit 220 of the front-end LSI 239 receives sequential RAW data transmitted from the imaging terminal 100 and received by the second communication unit 202, performs primary image processing on the RAW data of each frame that is input sequentially, generates RAW data that can be used for RAW data recording, and outputs it to the secondary image processing circuit 230 and the primary memory 240.

[0109] Figure 4 is a block diagram showing an embodiment of the primary image processing circuit of the front-end LSI.

[0110] The primary image processing circuit 220 shown in Figure 4 includes an offset processing circuit 221, a pixel defect correction circuit 222, a color tone correction circuit 223, and an individual difference correction circuit 224.

[0111] The RAW data received by the second communication unit 202 is added to the offset processing circuit 221 of the primary image processing circuit 220. The B, G, and R data in this RAW data each have 14 bits of grayscale value.

[0112] The offset processing circuit 221 applies an offset to the input RAW data. The offset processing circuit 221 is a processing unit that corrects the dark current component of the sensor output of the image sensor 104. It calculates the average value of the pixel data corresponding to multiple light-shielded pixels (optical black pixels) of the image sensor 104 and subtracts the calculated average value from the input RAW data. The offset-processed RAW data is output to the pixel defect correction circuit 222.

[0113] The pixel defect correction circuit 222 is a correction circuit that corrects the inherent pixel defects (scratches) of the image sensor 104.

[0114] The server 200 stores various correction information and information necessary for RAW development (RAW development information) in memory (not shown) according to the terminal information of the imaging terminal 100 (for example, product name + serial number). When communication with the imaging terminal 100 begins, the server 200 receives terminal information from the imaging terminal 100 and can read and use the correction information and RAW development information corresponding to that imaging terminal 100 from memory. The correction information and RAW development information for each imaging terminal 100 can be obtained in advance from the imaging terminal 100, or downloaded from the manufacturer's server of the imaging terminal 100 based on the terminal information and stored in memory. The RAW development information also includes information such as the color filter array of the image sensor, the number of pixels, and the effective pixel area.

[0115] The pixel defect correction circuit 222 acquires information about defective pixels (location information of defective pixels) of the image sensor 104 based on terminal information from the imaging terminal 100. When R, G, or B data corresponding to the defective pixel is input, the circuit interpolates the data of the same color in the vicinity of the defective pixel's location and outputs this interpolated pixel data in place of the data corresponding to the defective pixel. If the image sensor 104 includes pixels for phase difference detection, the position data of those phase difference detection pixels is interpolated in the same manner and output in place of the phase difference detection pixel data.

[0116] The color correction circuit 223 performs color correction to correct the spectral characteristics of the B, G, and R output data of the image sensor 104. It performs matrix operations on the three input R, G, and B data and the three outputs B, G, and R. By providing matrix coefficients for a 3x3 matrix, it is possible to obtain color-corrected B, G, and R data from the three input R, G, and B color data. At this time, a linear matrix transformation is performed using matrix coefficients that enhance color reproducibility, corresponding to the terminal identification information of the imaging terminal 100.

[0117] The individual difference correction circuit 224 is responsible for correcting individual differences in the image sensor 104. In the case of an imaging terminal 100 of the same product, in order to achieve the same color reproduction and image quality as a reference imaging terminal, it uses adjustment values ​​according to the spectral sensitivity characteristics of each individual image sensor 104, multiplies the B, G, and R data by these adjustment values, and outputs B, G, and R data with the same color reproduction and image quality regardless of individual differences in the image sensor 104.

[0118] The RAW data processed by the primary image processing circuit 220 is output to the secondary image processing circuit 230 and the primary memory 240. The content and order of image processing performed on the RAW data by the primary image processing circuit 220 are not limited to the above embodiment. Furthermore, the RAW data processed by the primary image processing circuit 220 is data before RAW development processing, and the B, G, and R data have 14 bits of grayscale.

[0119] Figure 5 is a block diagram showing an embodiment of the secondary image processing circuit of the front-end LSI.

[0120] The secondary image processing circuit 230 shown in Figure 5 includes a white balance (WB) correction circuit 231, a gamma correction circuit 232, a demosaicing circuit 233, an edge enhancement circuit 235, a color difference matrix circuit 236, and a compression circuit 237.

[0121] The RAW data processed by the primary image processing circuit 220 is added to the white balance correction circuit 231 of the secondary image processing circuit 230.

[0122] The white balance correction circuit 231 automatically detects the type of light source (such as "sunny," "cloudy," "shade," "incandescent," or "fluorescent") and the shooting scene from the B, G, and R data of the input RAW data. It then amplifies the B, G, and R data using the pre-set white balance gain for each B, G, and R data according to the type of light source, and performs white balance correction. The white balance corrected B, G, and R data is output to the gamma correction circuit 232.

[0123] The gamma correction circuit 232 has, for example, a gamma correction table for each of the B, G, and R data, and performs gradation correction (gamma correction) on the input B, G, and R data according to the gamma characteristics of the gamma correction table, each having a corresponding gamma characteristic, so that the midtones of the linear data are enlarged. In addition, the gamma correction circuit 232 in this example converts 14-bit B, G, and R data to 8-bit B, G, and R data. The 8-bit B, G, and R data gamma-corrected by the gamma correction circuit 232 is output to the demosaicing circuit 233.

[0124] The demosaicing circuit 233 interpolates the missing color data at each pixel position of the image sensor 104 from the sequential B, G, and R color data, which are mosaic image data corresponding to the color filter array of the image sensor 104, and outputs the B, G, and R data for each pixel position simultaneously. In other words, the demosaicing circuit 233 converts the sequential B, G, and R data into simultaneous B, G, and R data and outputs the simultaneous B, G, and R data to the YC conversion circuit 234.

[0125] The YC conversion circuit 234 converts simultaneous B, G, and R data into luminance data (Y) and chrominance data (Cr, Cb), outputs the luminance data (Y) to the edge enhancement circuit 235, and outputs the chrominance data (Cr, Cb) to the chrominance matrix circuit 236. The edge enhancement circuit 235 performs processing to enhance the edges (parts with large luminance changes) of the luminance data (Y). The chrominance matrix circuit 236 performs matrix calculations between the input chrominance data (Cr, Cb) and 2x2 color correction matrix coefficients to perform color correction to achieve good color reproduction.

[0126] In this way, the secondary image processing circuit 230 generates displayable image data from the RAW data by performing various image processing (RAW development processing) on ​​the RAW data.

[0127] The compression circuit 237 compresses the luminance data (Y) output from the edge enhancement circuit 235 and the color difference data (Cr, Cb) output from the color difference matrix circuit 236. For still images, it compresses them in format such as JPEG (Joint Photographic Coding Experts Group), and for videos, it compresses them in format such as H.264. The compressed image data is stored in the primary memory 240.

[0128] Furthermore, the compression circuit 237 receives the RAW data processed by the primary image processing circuit 220, and after compressing the RAW data, the compression circuit 237 outputs it to the primary memory 240. It is preferable to use lossless compression for the RAW data. Alternatively, as shown in Figure 3, the RAW data may be output directly from the primary image processing circuit 220 to the primary memory 240 and stored in the primary memory 240 without compression.

[0129] Returning to Figure 3, the second image processing engine 210 receives continuous RAW data from the imaging terminal 100 (in this example, the digital camera 100a) and transmits the live view image corresponding to the continuous data generated by the second image processing engine 210 to the digital camera 100a. Specifically, the image for each frame of each image processing corresponding to the continuous data, consisting of luminance data (Y) output from the edge enhancement circuit 235 of the secondary image processing circuit 230 shown in Figure 4 and color difference data (Cr,Cb) output from the color difference matrix circuit 236, or B, G, R data converted from luminance data (Y) and color difference data (Cr,Cb), is output as a live view image to the second communication unit 202 and transmitted from the second communication unit 202 to the digital camera 100a via the network 300 and wireless access point 310.

[0130] Furthermore, after continuous data is transmitted from the digital camera 100a, the server 200 processes the image to generate a live view image, and the time until that live view image is transmitted back to the digital camera 100a is largely comprised of the time required for data transmission and reception. However, in the 5G (5th Generation) mobile communication system, ultra-low latency can be achieved, reducing the delay time from data transmission to reception to approximately 1 millisecond. This delay is so short that humans cannot perceive it.

[0131] Therefore, the digital camera 100a can display live view images on the LCD 118 or EVF 120 in near real-time. Furthermore, from 2027 onwards, an even faster next-generation 6G mobile communication system will be introduced. When such communication speeds are achieved, communication speeds comparable to the current bus transfer speed within the imaging terminal will be realized, further eliminating the time lag between the imaging terminal and the server on the network, allowing for even more real-time display of live view images.

[0132] Furthermore, in the digital camera 100a, when the shutter button is operated (when a user inputs a still image capture command), the imaging terminal 100 performs AF control and AE control for still image recording, transmits RAW data for still image recording as one frame of continuous data, and transmits a shutter release signal (shooting command information) by interrupting the continuous data.

[0133] Meanwhile, the second processor 250 is constantly waiting for a shutter release signal to be transmitted from the digital camera 100a by the user's operation of the shutter button. Upon receiving the shutter release signal via the second communication unit 202, the second processor 250 obtains the image data processed by the second image processing engine 210 from the second image processing engine 210, which is the RAW data for still image recording identified by the shutter release signal, and records this image data in the image recording unit 260.

[0134] The second processor 250 records the still image file, which has RAW data for still image recording processed by the second image processing engine 210 and compressed into JPEG format, to the image recording unit 260. In this case, it is preferable that the second processor 250 records the image file to an image folder (a folder in the image recording unit 260) associated with terminal information or user information of the digital camera 100a.

[0135] Furthermore, the second processor 250 records the RAW data for still image recording, which is a RAW file of RAW data processed by the primary image processing circuit 220 of the second image processing engine 210, and which has lossless compressed or uncompressed RAW data held in the primary memory 240, into a folder in the image recording unit 260 associated with the terminal information or user information of the imaging terminal 100.

[0136] Furthermore, the second processor 250 may record at least one of the above-mentioned still image file and RAW file to the image recording unit 260.

[0137] Furthermore, when the digital camera 100a is set to bracket shooting mode or continuous shooting mode, it captures multiple still images in succession with a single press of the shutter button. In this case, the server 200 records the processed still image files and / or RAW data to the image recording unit 260.

[0138] Furthermore, if the digital camera 100a is set to video mode, when it receives a video recording instruction or recording end instruction by user operation of the shutter button, it transmits the recording instruction information or recording end instruction information from the first communication unit 124 to the server 200. The server 200 (second image processing engine 210) performs image processing for video recording on the continuous data received during the period from receiving the video recording instruction information to receiving the recording end instruction information, stores the video file containing the video data generated by the image processing in the primary memory 240, and the second processor 250 records the video file stored in the primary memory 240 to the image recording unit 260.

[0139] On the other hand, when the imaging terminal 100 is started in playback mode, or when it is switched from shooting mode to playback mode, it remains constantly connected to the server 200. When the user performs a playback operation according to the display contents of the operation unit 114 and LCD 118, the first communication unit 124 outputs a playback request corresponding to that operation to the server 200.

[0140] When the server 200 (second processor 250) receives a playback request via the second communication unit 202, it reads the image file corresponding to the playback request from the image recording unit 260, decompresses the compressed image data within the image file, and transmits the playback image from the second communication unit 202 to the imaging terminal 100. If the image file corresponding to the playback request is a RAW file, the server 200 processes the RAW data of the RAW file through RAW development before transmitting the playback image from the second communication unit 202 to the imaging terminal 100.

[0141] For example, when the imaging terminal 100 is started in playback mode, or when it switches from shooting mode to playback mode, it sends a request to play back the latest captured image. When the server 200 receives the request to play back the latest captured image from the imaging terminal 100, it reads the corresponding image file from the image recording unit 260, decompresses the compressed image data in the image file, and sends it back to the imaging terminal 100 from the second communication unit 202 as a played-back image.

[0142] Upon receiving the playback image, the imaging terminal 100 displays the playback image on the LCD 118 via the display control unit 116. Subsequently, when the user performs playback operations such as frame-forwarding or frame-backwarding using the operation unit 114, the imaging terminal 100 sends a playback request corresponding to that playback operation to the server 200. The server 200 reads the image file corresponding to the received playback request from the image recording unit 260 and sends the playback image to the imaging terminal 100 in the same manner as described above.

[0143] Furthermore, when the imaging terminal 100 requests a list of thumbnail images of the user's image files recorded in the image recording unit 260, the server 200 sends a list of thumbnail images corresponding to the request to the imaging terminal 100. Upon receiving the list of thumbnail images, the imaging terminal 100 can then display the list of thumbnail images on the LCD 118.

[0144] The user can select a desired thumbnail image by operating the control unit 114 while viewing a list of thumbnail images displayed on the LCD 118. The imaging terminal 100 can then send a playback request (a playback request including the image file name) to the server 200 for the image corresponding to the selected thumbnail image. When the server 200 receives a playback request with the image file name from the imaging terminal 100, it reads the image file corresponding to the image file name from the image recording unit 260 and sends the playback image to the imaging terminal 100 in the same manner as described above. As a result, the playback image corresponding to the thumbnail image selected by the user is displayed on the LCD 118 of the imaging terminal 100.

[0145] Furthermore, the user can identify the image to be printed using a playback image or thumbnail image, and have the imaging terminal 100 send a print request (a print request with the image file name) for the identified image to the server 200. When the server 200 receives a print request with the image file name from the imaging terminal 100, it reads the image file corresponding to the image file name from the image recording unit 260 and sends it to a print server (not shown) to provide the print service.

[0146] The above example described the process of playing still images, but the same procedure can be used to play videos.

[0147] Furthermore, users can use utility software (retouching software) on server 200 to perform image editing (correction, enhancement, processing), such as adding filter effects, while viewing the edited image (playback image) in real time via the network.

[0148] [Imaging System] <First embodiment of the imaging system> Figure 6 is a schematic diagram showing a first embodiment of the imaging system according to the present invention.

[0149] In Figure 6, solid arrows indicate the data flow when communication between the imaging terminal 100-1 and the server 200 is normal, while dotted arrows indicate the data flow when communication between the imaging terminal 100-1 and the server 200 is impossible (including cases where communication is unstable). Also, in Figure 6, the communication units of the imaging terminal 100-1 and the server 200 are omitted.

[0150] The following describes the case where communication between the imaging terminal 100-1 and the server 200 is normal.

[0151] In this case, the imaging terminal 100-1 is constantly connected to the server 200 via the wireless access point 310 and network 300 (Figure 1) during startup, and in shooting mode, it transmits the RAW data before image processing to the server 200 in the form of continuous data. The front-end LSI 239 of the server 200's second image processing engine 210 generates a live view image from the continuous data consisting of the received RAW data, and the server 200 transmits the live view image generated by the front-end LSI 239 to the imaging terminal 100-1.

[0152] The imaging terminal 100-1, upon receiving the live view image, displays the live view image on the LCD 118 or EVF 120.

[0153] This allows the user to determine the composition and other settings while viewing the live view image, and then press the shutter button 115 to take a picture of the desired subject. When the shutter button 115 is pressed (when the imaging terminal 100-1 receives a still image capture command from the user), it transmits RAW data for still image recording with 14 bits of gradation, read from the image sensor 104, as one frame of continuous data, and also transmits a shutter release signal to the server 200, interrupting the continuous data.

[0154] The server 200's second image processing engine 210 (front-end LSI 239) performs image processing on the RAW data for still image recording received in response to the shutter release signal. Specifically, the front-end LSI 239 processes the received RAW data for still image recording, stores a still image file having JPEG-compressed image data (8-bit gradation) with 8-bit gradation in the primary memory 240, and / or stores a RAW file having lossless compressed or uncompressed RAW data (14-bit gradation), which has been image-processed by the primary image processing circuit 220 (Figure 3), in the primary memory 240.

[0155] The second processor 250 of the server 200 records still image files with 8-bit grayscale and / or RAW files with 14-bit grayscale stored in the primary memory 240 to the image recording unit 260.

[0156] On the other hand, when the imaging terminal 100-1 is started in playback mode, or when it is switched from shooting mode to playback mode, if the user performs a playback operation according to the display contents of the operation unit 114 and LCD 118, the first communication unit 124 outputs a playback request corresponding to the operation to the server 200.

[0157] When server 200 receives a playback request from imaging terminal 100-1, it reads the image file corresponding to the playback request from image recording unit 260, decompresses the compressed image data within the image file, and transmits the playback image to imaging terminal 100 via second communication unit 202. If the image file corresponding to the playback request is a RAW file, the server 200 processes the RAW data of the RAW file through RAW development before transmitting the playback image to imaging terminal 100.

[0158] Upon receiving the playback image, the imaging terminal 100-1 displays the playback image on the LCD 118 via the display control unit 116.

[0159] Next, we will explain the case where communication between the imaging terminal 100-1 and the server 200 is impossible (offline).

[0160] When offline, the first image processing engine 122 of the imaging terminal 100-1 operates, enabling the imaging terminal 100-1 to take images independently.

[0161] RAW data read from the image sensor 104 at a predetermined frame rate (e.g., 30fps, 60fps) is output to the first image processing engine 122.

[0162] As mentioned above, the first image processing engine 122 has different technical specifications from the second image processing engine 210 of the server 200. Compared to the second image processing engine 210, it has limited functionality and performance, and its circuit size is smaller and it is less expensive.

[0163] The first image processing engine 122 converts continuous data consisting of 14-bit grayscale RAW data from the image sensor 104 into, for example, 8-bit grayscale RAW data, and then performs image processing such as RAW development on the 8-bit grayscale RAW data to generate a live view image. This live view image may have lower image quality (resolution, display frame rate) compared to the live view image generated by the second image processing engine 210 when communication is normal.

[0164] Subsequently, when the shutter button 115 is pressed, the first image processing engine 122 performs image processing on the RAW data for still image recording read from the image sensor 104 when the shutter button 115 is pressed, generates image data for still image recording, and records it in internal memory (not shown).

[0165] Furthermore, once communication between the imaging terminal 100-1 and the server 200 is established, the imaging terminal 100-1 transmits the image data recorded in the internal memory of the first image processing engine 122 to the server 200 prior to transmitting continuous data. The second image processing engine 210 performs image processing on the received image data, such as noise reduction processing to improve image quality. The second processor 250 records the image file, which is the image data processed by the second image processing engine 210 and has been converted into a file, into the image recording unit 260.

[0166] The first image processing engine 122 of the imaging terminal 100-1 is an auxiliary image processing engine used as an emergency backup when communication between the imaging terminal 100-1 and the server 200 is impossible. Therefore, its circuit size is significantly smaller, and it can only handle data with 8-bit grayscale, limiting its functions to the bare minimum necessary. The primary image processing part, such as noise reduction processing for image quality improvement, does not need to be state-of-the-art. As a result, the circuit size can be reduced to 1 / 10 to several-tenths the cost of the second image processing engine 210 of the server 200. Furthermore, the frequency of use of the first image processing engine 122 in the imaging terminal 100-1 is almost negligible in areas with well-developed communication networks, and since it is only for emergencies, performance disadvantages are unlikely to occur. To ensure that shooting is not completely impossible in the event of a temporary communication interruption, a minimum amount of image data for recording is recorded. This allows for shooting operations, display (live view image), and image recording, so that shutter opportunities are not missed. If continuous shooting is a priority, a minimum amount of memory for continuous shooting may be provided. In this case, the function will be limited to 5 frames per second or so compared to the usual 100 frames per second, but continuous shooting will still be possible.

[0167] Furthermore, when the imaging terminal 100-1 is offline, it can only play back and display images recorded in the internal memory of the first image processing engine 122, and cannot play back and display images recorded in the image recording unit 260.

[0168] <Second embodiment of the imaging system> Figures 7 and 8 are schematic diagrams showing a second embodiment of the imaging system according to the present invention, respectively. Figure 7 shows the case where communication between the imaging terminal and the server is normal, and Figure 8 shows the case where communication between the imaging terminal and the server is impossible (offline).

[0169] In addition, in Figures 7 and 8, parts common to the first embodiment of the imaging system shown in Figure 6 are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0170] The second embodiment of the imaging system shown in Figures 7 and 8 differs from the imaging terminal 100-1 of the first embodiment of the imaging system shown in Figure 6 in that the imaging terminal 100-2 is different, and in particular the first image processing engine 130 is different.

[0171] The first image processing engine 130 of the imaging terminal 100-2 operates when offline. Therefore, when communication between the imaging terminal 100-2 and the server 200 is normal, the second embodiment of the imaging system shown in Figures 7 and 8 operates in the same manner as the first embodiment of the imaging system shown in Figure 6.

[0172] The following describes the case where the imaging terminal 100-2 is offline.

[0173] The first image processing engine 130 of the imaging terminal 100-2, which operates when offline, comprises a front-end LSI 132 and a primary memory 134, which is internal memory.

[0174] The front-end LSI132 has lower technical specifications, limited functionality and performance, and is smaller in circuit size and cheaper compared to the front-end LSI239 of the server 200's second image processing engine 210.

[0175] The front-end LSI 132 converts continuous data consisting of 14-bit grayscale RAW data from the image sensor 104 into, for example, 8-bit grayscale RAW data, and then performs image processing such as RAW development on the 8-bit grayscale RAW data to generate a live view image. This live view image may have lower image quality (resolution, display frame rate) compared to the live view image generated by the front-end LSI 239 of the second image processing engine 210 when communication is normal.

[0176] The live view image generated by the front-end LSI132 is output to and displayed on the LCD118 or EVF120, as indicated by the solid arrows in Figure 8.

[0177] Subsequently, when the shutter button 115 is pressed, the front-end LSI 132 performs image processing, including RAW development and JPEG compression, on the RAW data for still image recording read from the image sensor 104 when the shutter button 115 is pressed, and records an image file containing the JPEG-compressed image data in the primary memory 134.

[0178] Furthermore, the image data for still image recording generated by the front-end LSI 132 has lower image quality (at least one of the following, such as image size, resolution, and color reproduction) compared to the image data for still image recording generated by the front-end LSI 239 of the server 200. This is because the first image processing engine 130, which has the front-end LSI 132, is less expensive than the second image processing engine 210, which has the front-end LSI 239, because it does not include image processing such as noise reduction, defect correction, and image quality enhancement.

[0179] Meanwhile, once communication between the imaging terminal 100-2 and the server 200 is established, the imaging terminal 100-2 sends the image file recorded in the primary memory 134, which is the internal memory of the first image processing engine 130, to the server 200 prior to transmitting continuous data.

[0180] When the server 200 (second processor 250) receives an image file from the imaging terminal 100-2, it records the received image file in the image recording unit 260. Alternatively, similar to the imaging system of the first embodiment shown in Figure 6, the image data of the received image file may be reprocessed by the second image processing engine 210 before being recorded in the image recording unit 260.

[0181] <Third embodiment of the imaging system> Figure 9 is a schematic diagram showing a third embodiment of the imaging system according to the present invention.

[0182] In Figure 9, parts common to the first embodiment of the imaging system shown in Figure 6 are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0183] The third embodiment of the imaging system shown in Figure 9 differs from the imaging terminal 100-1 of the first embodiment of the imaging system shown in Figure 6 in that the imaging terminal 100-3 is different, and in particular the first image processing engine 140 is different.

[0184] The first image processing engine 140 of the imaging terminal 100-3 operates when offline. Therefore, when communication between the imaging terminal 100-3 and the server 200 is normal, the third embodiment of the imaging system shown in Figure 9 operates in the same manner as the first embodiment of the imaging system shown in Figure 6.

[0185] The following describes the case where the imaging terminal 100-3 is offline.

[0186] The first image processing engine 140 of the imaging terminal 100-3, which operates when offline, comprises a memory controller 142, a primary memory 144 which is internal memory, and a live view engine 146.

[0187] This first image processing engine 140 provides minimal image processing capabilities for offline use.

[0188] The live view engine 146 converts continuous data consisting of 14-bit grayscale RAW data from the image sensor 104 into, for example, 8-bit grayscale RAW data, and then applies image processing such as RAW development processing to the 8-bit grayscale RAW data to generate a live view image. The live view engine 146 only needs to generate a live view image with the minimum image quality necessary for framing, and can omit image processing such as noise reduction, defect correction, and image enhancement, and may have lower image quality (resolution, display frame rate) compared to the live view image generated by the front-end LSI 239 of the second image processing engine 210.

[0189] When offline, the live view image generated by the live view engine 146 is output to and displayed on the LCD 118 or EVF 120, as indicated by the dotted arrow in Figure 9.

[0190] Subsequently, when the shutter button 115 is pressed, the memory controller 142 temporarily records the RAW data for one frame with 14-bit gradation for still image recording, read from the image sensor 104 at the time the shutter button 115 is pressed, into the primary memory 144. If bracket shooting mode or continuous shooting mode is set, the memory controller 142 temporarily records the RAW data for multiple frames for still image recording, read consecutively from the image sensor 104, into the primary memory 144 with a single shutter release operation of the shutter button 115. In addition, the live view engine 146 can continuously generate live view images regardless of the shutter release operation of the shutter button 115.

[0191] Meanwhile, once communication between the imaging terminal 100-3 and the server 200 is established, the imaging terminal 100-3 transmits to the server 200, prior to transmitting continuous data, RAW data for still image recording that has 14 bits of grayscale and is recorded in the primary memory 144 of the first image processing engine 140.

[0192] When the server 200 (front-end LSI 239 of the second image processing engine 210) receives RAW data for still image recording taken while offline from the imaging terminal 100-2, it performs image processing for still image recording on the received RAW data and stores it in the primary memory 240.

[0193] The second processor 250 records still image files with 8-bit grayscale stored in the primary memory 240, and / or RAW files processed with primary image processing and having 14-bit grayscale, to the image recording unit 260.

[0194] Furthermore, when offline, the image processing performed by the front-end LSI239 on RAW data for still image recording can be the same as the image processing performed on RAW data for still image recording received in response to the shutter release signal when communication between the imaging terminal 100-3 and the server 200 is normal.

[0195] <Fourth embodiment of the imaging system> Figure 10 is a schematic diagram showing a fourth embodiment of the imaging system according to the present invention.

[0196] In Figure 10, parts common to the first embodiment of the imaging system shown in Figure 6 are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0197] The fourth embodiment of the imaging system shown in Figure 10 differs from the imaging terminal 100-1 of the first embodiment of the imaging system shown in Figure 6 in that the imaging terminal 100-4 is different, and in particular the first image processing engine 150 is different.

[0198] The first image processing engine 150 of the imaging terminal 100-4 operates when offline. Therefore, when communication between the imaging terminal 100-4 and the server 200 is normal, the fourth embodiment of the imaging system shown in Figure 10 operates in the same manner as the first embodiment of the imaging system shown in the sixth embodiment.

[0199] The following describes the case when the imaging terminal 100-4 is offline.

[0200] The first image processing engine 150 of the imaging terminal 100-4, which operates when offline, comprises a front-end LSI 152 and a primary memory 154 which is internal memory.

[0201] The first image processing engine 150 (front-end LSI 152) converts continuous data consisting of 14-bit grayscale RAW data from the image sensor 104 into, for example, 8-bit grayscale RAW data, and then applies image processing such as RAW development processing to the 8-bit grayscale RAW data to generate a live view image. This live view engine 146 only needs to be able to generate a live view image with the minimum image quality necessary for framing.

[0202] When offline, the live view image generated by the live view engine 146 is output to and displayed on the LCD 118 or EVF 120, as indicated by the dotted arrow in Figure 10.

[0203] Subsequently, when the shutter button 115 is pressed, the front-end LSI 152 processes the RAW data for one frame, which has 14 bits of grayscale and is read from the image sensor 104 at the time the shutter button 115 is pressed, for still image recording.

[0204] The front-end LSI 152 applies image processing, including RAW development and JPEG compression, to the input RAW data, records the JPEG-compressed image data file in the primary memory 134, and / or reduces the amount of data (in this example, the image size) of the input RAW data, and records the reduced-data-size RAW data in the primary memory 134.

[0205] The first image processing engine 150, including the front-end LSI 152, is an auxiliary image processing engine used only in emergencies. Therefore, its circuit size is significantly smaller, and it is limited to the minimum necessary functions that can only handle images of a certain size (for example, 4K image size) or smaller. The primary image processing parts, such as noise reduction processing for improving image quality, do not need to be state-of-the-art. As a result, its circuit size can be made to be one-tenth to several-tenths the size of the second image processing engine 210 on the network, making it very inexpensive.

[0206] Furthermore, the first image processing engine 150 of the imaging terminal 100-4 is rarely used in areas with well-developed communication networks, and if used solely for purposes such as uploading to SNS (Social Networking Service) as a precaution, there is little performance disadvantage. In addition, shooting operations, display (display of live view image), and image recording can be performed, so shutter opportunities are not missed, and this can be solved with minimal additional circuitry (first image processing engine). Even when prioritizing continuous shooting, only a minimum amount of continuous shooting memory is needed, and in this case, although the function is limited to 5 frames per second compared to the usual 100 frames per second, continuous shooting is still possible.

[0207] The front-end LSI 152 performs image processing on the input RAW data, including RAW development and JPEG compression, although JPEG compression may be omitted. Furthermore, when recording RAW data with reduced data size in the primary memory 134, the front-end LSI 152 can reduce the data size by, for example, reducing the image size from 2K to less than 14 bits (e.g., 8 bits) of RAW data with 14 bits of grayscale.

[0208] Meanwhile, when communication between the imaging terminal 100-4 and the server 200 becomes normal, the imaging terminal 100-4 transmits the data recorded in the primary memory 144 of the first image processing engine 140 to the server 200 prior to transmitting continuous data.

[0209] When the server 200 (front-end LSI 239 of the second image processing engine 210) receives data for still image recording taken while offline from the imaging terminal 100-4, it performs image processing for still image recording on the received data and stores it in the primary memory 240.

[0210] The second processor 250 records the image stored in the primary memory 240 to the image recording unit 260.

[0211] [How the imaging system works] Figures 11 and 12 are flowcharts showing the operation of the imaging terminal in the imaging system, and Figure 13 is a flowchart showing the operation of the server in the imaging system.

[0212] Furthermore, the flowcharts shown in Figures 11 and 12 illustrate the operation procedure of the imaging terminal 100-2 in the second embodiment of the imaging system shown in Figures 7 and 8. The imaging terminal 100-2 is the digital camera 100a with communication function shown in Figures 1 and 2.

[0213] <Operation of the imaging terminal> In Figure 11, when the power switch of the imaging terminal 100-2 (digital camera 100a with communication function) is turned ON, it first performs network connection processing to establish a permanent connection with the server 200 (step S10).

[0214] Next, the digital camera 100a (first processor 110) determines whether the power switch is turned OFF or not (step S12). If the power is OFF ("Yes"), the operation of the digital camera 100a ends, and if the power is ON ("No"), the process proceeds to step S14.

[0215] In step S14, it is determined whether the operating mode of the digital camera 100a is shooting mode or playback mode (step S14). If it is determined to be shooting mode, the process proceeds to step S16 and the shooting operation is performed. If it is determined to be playback mode, the process proceeds to the image playback process shown in Figure 12.

[0216] In step S16, the imaging unit 101 (image sensor 104) captures images at a predetermined frame rate, and the raw data before image processing is read out from the image sensor 104 as continuous data at a predetermined frame rate.

[0217] Next, the digital camera 100a (first processor 110) determines whether the network connection status established in step S10 is stable (step S18). If it is determined that the network connection is stable, the digital camera 100a transmits the continuous data read from the image sensor 104 to the server 200 via the first communication unit 124 (step S20).

[0218] As described later, the server 200 applies image processing such as RAW development processing to the continuous data received from the digital camera 100a to generate a live view image, and then transmits the generated live view image to the digital camera 100a.

[0219] The digital camera 100a receives the live view image transmitted from the server 200 via the first communication unit 124 (step S22), and displays the received live view image on the LCD 118 or EVF 120 (step S24). Here, the delay between the transmission of continuous data and the reception of the live view image is so short that a human cannot perceive the delay, so the live view image can be displayed on the LCD 118 or EVF 120 in real time.

[0220] Next, the digital camera 100a (first processor 110) determines whether or not the user has pressed the shutter button 115 to release the shutter (step S26). If it is determined in step S26 that no shutter release operation has been performed ("No"), the process proceeds to step S12. As a result, the processing from step S12 to step S26 is repeated, and the live view image is displayed.

[0221] In step S26, if it is determined that a shutter release operation has occurred ("Yes"), the process proceeds to step S28. In step S25, the digital camera 100a interrupts the continuous data with a shutter release signal and transmits it to the server 200 via the first communication unit 124, after which the process proceeds to step S12. The operation of the server 200 upon receiving the shutter release signal will be described later.

[0222] On the other hand, in step S18, if it is determined that the network connection is unstable (including communication failure), the digital camera 100a activates the first image processing engine 130 and outputs the continuous data read from the image sensor 104 to the first image processing engine 130. The first image processing engine 130 performs image processing such as RAW development on the input continuous data to generate a live view image (step S30). The first image processing engine 130 outputs the generated live view image to the LCD 118 or EVF 120 for display (step S32).

[0223] Next, the digital camera 100a (first processor 110) determines whether or not the user has pressed the shutter button 115 to release the shutter (step S32). If it is determined in step S32 that no shutter release operation has been performed ("No"), the process proceeds to step S18. As a result, the processing from step S18 to step S34 is repeated, and the live view image is displayed.

[0224] If it is determined in step S34 that a shutter release operation has occurred ("Yes"), the system proceeds to step S36. In step S36, the first image processing engine 130 performs RAW development processing and JPEG compression on the 14-bit gradation RAW data for still image recording corresponding to the shutter release signal, and saves (records) the JPEG-compressed image data file in the primary memory 134. After that, the digital camera 100a proceeds to step S12.

[0225] On the other hand, if the playback mode is detected in step S14, the system proceeds to step S50 shown in Figure 12. In step S50, the digital camera 100a sends a mode change instruction to the playback mode to the server 200 via the first communication unit 124. It goes without saying that the transmission of continuous data from the digital camera 100a stops.

[0226] As will be described later, when the server 200 receives a mode change instruction to playback mode, it sends a list of thumbnail images to the digital camera 100a that sent the mode change instruction, and the digital camera 100a receives the list of thumbnail images via the first communication unit 124 (step S52).

[0227] When the digital camera 100a receives a list of thumbnail images from the server 200, it displays the received list of thumbnail images on the LCD 118 (step S54).

[0228] The user can select a desired thumbnail image to be played back by operating the control unit 114 while viewing a list of thumbnail images displayed on the LCD 118. The digital camera 100a determines whether or not it has received a playback image selection instruction input from the user via the control unit 114 (step S56).

[0229] In step S56, when it is determined that a playback image selection instruction has been received, the digital camera 100a sends a playback request (playback request with image file name) for the image corresponding to the selected thumbnail image to the server 200 via the first communication unit 124 (step S58). When the server 200 receives the playback request with image file name from the digital camera 100a, it reads the playback image corresponding to the image file name from the image recording unit 260 and sends it to the digital camera 100a. The digital camera 100a receives the playback image corresponding to the playback request from the server 200 via the first communication unit 124 (step S60).

[0230] The digital camera 100a displays the playback image received from the server 200 on the LCD 118 (step S62), and then proceeds to step S12 in Figure 11.

[0231] If playback mode is maintained, the digital camera 100a can display the playback image corresponding to the playback request on the LCD 118. The digital camera 100a can also display the playback image on an external display (not shown) connected via an interface.

[0232] <Server operation> In Figure 13, when a network connection is established between the server 200 and the digital camera 100a (step S102), the digital camera 100a transmits terminal information indicating itself to the server 200 when communication with the server 200 begins, and the server 200 receives the terminal information of the digital camera 100a via the second communication unit 202 and acquires the terminal information (step S102).

[0233] Next, the server 200 determines whether the operating mode of the network-connected digital camera 100a is shooting mode or playback mode (step S104). If it is determined to be shooting mode, the process proceeds to step S106; if it is determined to be playback mode, the process proceeds to step S120.

[0234] In step S106, the server 200 receives RAW data at a predetermined frame rate as continuous data from the digital camera 100a. The server 200's second image processing engine 210 performs image processing, such as RAW development processing, on the RAW data at the predetermined frame rate to generate a live view image (step S108). The server 200 transmits the live view image generated by the second image processing engine 210 to the digital camera 100a via the second communication unit 202 (step S110).

[0235] Next, the server 200 determines whether or not it has received a shutter release signal from the digital camera 100a (step S112). If it has received a shutter release signal ("Yes"), the second image processing engine 210 processes the RAW data corresponding to the shutter release signal (shooting instruction information) from the continuous RAW data to generate an image (still image) for recording (step S144).

[0236] Furthermore, when the second image processing engine 210 performs image processing, including RAW development processing, on RAW data, it obtains the information necessary for RAW development of the RAW data captured by the digital camera 100a (color filter array of the image sensor 104, number of pixels, pixel defect information, and other parameters) from the manufacturer's server of the digital camera 100a, or reads and uses the corresponding information from the information previously obtained from the manufacturer's server and stored in the recording unit, based on the terminal information acquired in step S102.

[0237] The second processor 250 records the image for recording, generated by the second image processing engine 210, into the image recording unit 260 (step S116).

[0238] Next, the server 200 determines whether the network connection with the digital camera 100a has been terminated (step S118). If the network connection has not been terminated, the server proceeds to step S104. If the network connection has not been terminated, the server 200 terminates processing for the digital camera 100a.

[0239] On the other hand, in step S104, if it is determined that the operating mode of the digital camera 100a is playback mode, the server 200 generates a list of thumbnail images of images taken by the digital camera 100a and recorded in the image recording unit 260, and sends the list of thumbnail images to the digital camera 100a (step S120). Furthermore, the server 200 can identify the image folder in the image recording unit 260 corresponding to the digital camera 100a based on the terminal information acquired in step S102, and create a list of thumbnail images corresponding to the digital camera 100a based on the image files stored in the identified image folder.

[0240] Next, the server 200 determines whether or not it has received a playback request (a playback request with an image file name) from the digital camera 100a (step S122). If it has received a playback request, it reads the playback image corresponding to that playback request from the image recording unit 260 and transmits the read playback image to the digital camera 100a (step S124).

[0241] [others] If the imaging terminal 100 is a smartphone 100b with a built-in camera, the control unit mainly consists of the smartphone 100b's screen, the on-screen touch panel, and a GUI (Graphical User Interface) controller, and the shutter release operation can be performed by touching the shutter button icon.

[0242] Furthermore, in this embodiment, AF control and AE control are performed by the imaging terminal alone. However, a server that receives continuous data from the imaging terminal may generate control information necessary for AF control and AE control based on that continuous data, and transmit the generated control information to the imaging terminal to perform AF control and AE control.

[0243] The server is not limited to a single physical server; different servers may perform different tasks depending on the task, or multiple servers may collaborate to perform the same task.

[0244] Furthermore, in this embodiment, the hardware structure of processing units that perform various processes, such as the first processor 110 and first image processing engine 122 of the digital camera 100a, and the second processor 250 and second image processing engine 210 of the server 200, is a variety of processors as shown below. These various processors include CPUs (Central Processing Units), which are general-purpose processors that execute software (programs) and function as various processing units; Programmable Logic Devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing; and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which have a circuit configuration specifically designed to perform a particular process.

[0245] A single processing unit may be composed of one of these various processors, or it may be composed of two or more processors of the same or different type (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor. Examples of composing multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software are combined to form a single processor, and this processor functions as multiple processing units, as is typical of computers such as client and server systems. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as is typical of System-on-a-Chip (SoC) systems. Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned various processors.

[0246] Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit composed of circuit elements such as semiconductor devices.

[0247] Furthermore, it goes without saying that the present invention is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the invention. [Explanation of Symbols]

[0248] 1. Imaging System 100 imaging terminals 100-1 Imaging terminal 100-2 Imaging terminal 100-3 Imaging terminal 100-4 Imaging terminal 100a Digital Camera 100b Smartphone 101 Imaging Unit 102 Shooting Lens 104 Image Sensor 108 Sensor Drivers 110 First Processor 112 memory 114 Operation section 115 Shutter button 116 Display Control Unit 118 LCD 122 First Image Processing Engine 124 First Communications Department 130 First Image Processing Engine 132 Front-end LSI 134 Primary Memory 140 First Image Processing Engine 142 Memory Controllers 144 Primary Memory 146 Live View Engine 150 First Image Processing Engine 152 Front-end LSI 154 Primary Memory 200 servers 202 Second Communications Department 210 Second Image Processing Engine 220 Primary Image Processing Circuit 221 Offset Processing Circuit 222 Pixel Defect Correction Circuit 223 Color correction circuit 224 Individual Difference Correction Circuit 230 Secondary Image Processing Circuit 231 WB correction circuit 232 Gamma Correction Circuit 233 Demosaicing Circuit 234 YC conversion circuit 235 Contour Enhancement Circuit 236 Chromatic Matrix Circuit 237 Compression Circuit 239 Front-end LSI 240 Primary Memory 250 Second Processor 260 Image Recording Unit 300 Networks 310 Wireless Access Points Steps S10-S36, S50-S62, S100-S124

Claims

1. An imaging system comprising at least one imaging terminal and a server, The aforementioned imaging terminal is The imaging unit includes an image sensor and outputs imaging data, A first communication unit transmits the imaging data output from the imaging unit to the server, A first image processing engine that processes the image data output from the imaging unit, The system includes a memory for storing images processed by the first image processing engine, The aforementioned server, A second communication unit that receives the imaging data transmitted from the first communication unit of the imaging terminal, A second image processing engine that processes the received imaging data and generates an image for recording, comprising a second image processing engine different from the first image processing engine of the imaging terminal, When the imaging terminal receives a still image capture instruction from the user, it transmits the capture instruction information from the first communication unit to the server. When the second image processing engine receives the shooting instruction information via the second communication unit, it processes the image data corresponding to the shooting instruction information from the continuous image data to generate a still image for recording. Imaging system.

2. An imaging system comprising at least one imaging terminal and a server, The aforementioned imaging terminal is The imaging unit includes an image sensor and outputs imaging data, A first communication unit transmits the imaging data output from the imaging unit to the server, A first image processing engine that processes the image data output from the imaging unit, The system includes a memory for storing images processed by the first image processing engine, The aforementioned server, A second communication unit that receives the imaging data transmitted from the first communication unit of the imaging terminal, A second image processing engine that processes the received imaging data and generates an image for recording, comprising a second image processing engine different from the first image processing engine of the imaging terminal, When the imaging terminal receives a video recording instruction or recording termination instruction from the user, it transmits the recording instruction information or recording termination instruction information from the first communication unit to the server. When the second image processing engine receives the recording instruction information or the recording end instruction information via the second communication unit, it processes the continuous image data from the time the recording instruction information is received until the recording end instruction information is received, and generates a video for recording. Imaging system.

3. The second image processing engine generates a live view image based on the continuously received imaging data. The imaging system according to claim 1 or 2.

4. The server records the image for recording, generated by the second image processing engine, in the image recording unit. The imaging system according to claim 1 or 2.

5. The second image processing engine has technical specifications that exceed the technical specifications of the first image processing engine. The imaging system according to claim 1 or 2.

6. The first image processing engine and the second image processing engine have different image processing capabilities. The imaging system according to claim 5.

7. The first image processing engine has less information per pixel that allows it to perform image processing than the second image processing engine. The imaging system according to claim 1 or 2.

8. The first image processing engine has a different number of gradation bits or processing bits compared to the second image processing engine. The imaging system according to claim 1 or 2.

9. The first image processing engine has only some of the functions that enable the second image processing engine to perform image processing. The imaging system according to claim 1 or 2.

10. The first image processing engine performs image processing on the second image processing engine when the number of pixels in the image is small. The imaging system according to claim 1 or 2.

11. The first image processing engine has computing elements with lower thermal design power than the second image processing engine. The imaging system according to claim 1 or 2.

12. The first image processing engine has fewer processing elements than the second image processing engine, The imaging system according to claim 1 or 2.

13. The first image processing engine has fewer processor cores than the second image processing engine. The imaging system according to claim 1 or 2.

14. The first image processing engine has arithmetic elements with a lower operating clock frequency than the second image processing engine. The imaging system according to claim 1 or 2.

15. The first image processing engine has a processing element with a lower rated operating current value than the second image processing engine. The imaging system according to claim 1 or 2.

16. The first image processing engine has a smaller cache memory capacity than the second image processing engine. The imaging system according to claim 1 or 2.

17. The first image processing engine has a configuration that allows it to execute fewer instructions than the second image processing engine. The imaging system according to claim 1 or 2.

18. The first image processing engine has a configuration with fewer processing units for executing processing instructions compared to the second image processing engine. The imaging system according to claim 1 or 2.

19. The first image processing engine has an integrated graphics function, and the second image processing engine has an extended graphics function. The imaging system according to claim 1 or 2.

20. The imaging terminal transmits terminal information indicating the imaging terminal to the server when communication with the server begins. When the server receives the terminal information, it obtains RAW development information corresponding to the received terminal information. The second image processing engine performs image processing to develop the image data into RAW format based on the acquired RAW development information. The imaging system according to claim 1 or 2.

21. The server transmits the live view image generated by the second image processing engine to the imaging terminal via the second communication unit. When the imaging terminal receives the live view image from the server via the first communication unit, it displays the live view image on the display of the imaging terminal. The imaging system according to claim 3.

22. The first image processing engine operates during periods when communication between the imaging terminal and the server is unavailable. The imaging system according to claim 1 or 2.

23. The first image processing engine generates a live view image based on the imaging data continuously output from the image sensor during periods when communication between the imaging terminal and the server is unavailable. The imaging terminal displays the live view image generated by the first image processing engine on a display during periods when communication between the imaging terminal and the server is unavailable. The imaging system according to claim 1 or 2.

24. The imaging terminal communicates with the server, receives an image from the server that has been selected by user operation from among the images recorded in the image recording unit, and displays the received image on the display or saves it to the memory. The imaging system according to claim 4.

25. An imaging system comprising at least one imaging terminal and a server, The aforementioned imaging terminal is The system includes an imaging unit that includes an image sensor and outputs imaging data, and a first communication unit that transmits the imaging data output from the imaging unit to the server. The aforementioned server, A second communication unit that receives the imaging data transmitted from the first communication unit of the imaging terminal, The system includes an image processing engine that processes the received imaging data to generate an image for recording, The data corresponding to one pixel of the image sensor in the aforementioned imaging data has the maximum number of bits of grayscale converted by the analog-to-digital conversion circuit. The image processing engine generates a live view image based on the continuously received imaging data. The server transmits the generated live view image to the imaging terminal via the second communication unit. When the imaging terminal receives the live view image from the server via the first communication unit, it displays the live view image on the display of the imaging terminal. When the imaging terminal receives a still image capture instruction from the user, it transmits the capture instruction information from the first communication unit to the server. When the image processing engine receives the shooting instruction information via the second communication unit, it processes the image data corresponding to the shooting instruction information from the continuous image data to generate a still image for recording. Imaging system.

26. An imaging system comprising at least one imaging terminal and a server, The aforementioned imaging terminal is The system includes an imaging unit that includes an image sensor and outputs imaging data, and a first communication unit that transmits the imaging data output from the imaging unit to the server. The aforementioned server, A second communication unit that receives the imaging data transmitted from the first communication unit of the imaging terminal, The system includes an image processing engine that processes the received imaging data to generate an image for recording, The data corresponding to one pixel of the image sensor in the aforementioned imaging data has the maximum number of bits of grayscale converted by the analog-to-digital conversion circuit. The image processing engine generates a live view image based on the continuously received imaging data. The server transmits the generated live view image to the imaging terminal via the second communication unit. When the imaging terminal receives the live view image from the server via the first communication unit, it displays the live view image on the display of the imaging terminal. When the imaging terminal receives a video recording instruction or recording termination instruction from the user, it transmits the recording instruction information or recording termination instruction information from the first communication unit to the server. When the image processing engine receives the recording instruction information or the recording end instruction information via the second communication unit, it processes the continuous image data from the time the recording instruction information is received until the recording end instruction information is received, and generates a video for recording. Imaging system.

Images