Information processing device, image processing device, control method thereof, and image processing system

Abstract

To provide an information processing device which makes it possible to efficiently use image processing by an external device, and control method of the information processing device.SOLUTION: The information processing device transmits reduced image data of a captured image to a first external device, and receives a result of evaluation processing on the reduced image data from the first external device. Then, the information processing device determines original data to be applied with image processing by a second external device, out of original data of the reduced image data, on the basis of a result of the evaluation processing. The information processing device transmits the determined original data to the second external device, and receives a result of image processing on the original data from the second external device.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device, an image processing device, and a control method therefor, and an image processing system. [Background technology]

[0002] Imaging devices such as digital cameras and digital video cameras have hardware and software that perform various types of image processing to convert image data (RAW data) read from an imaging element into image data in a general-purpose format that can be used by other devices. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2003-259281 A Summary of the Invention [Problem to be solved by the invention]

[0004] The hardware and software of imaging devices remain essentially the same as when they were first released, except for updates to correct malfunctions, and it is common that new functions added to models released later cannot be used, for example because the hardware of the older model does not meet the specifications required to implement the newly added functions.

[0005] Furthermore, if the smartphone or tablet computer captures image data from an imaging device, it is possible to use functions that are developed later by uploading an application. However, functions that require highly advanced calculations or a huge amount of calculations may not be available on the smartphone or tablet computer because the installed hardware cannot handle them. As described above, similar problems can occur not only in digital cameras and digital video cameras, but also in information processing devices in general, including smartphones, tablet computers, and the like.

[0006] Patent Document 1 discloses a configuration in which a processing program for RAW data is output together with image data in RAW format, and image processing is executed by an external device. With this configuration, it is possible to apply image processing that requires a higher performance than that of an information processing device to RAW data.

[0007] However, RAW data is very large in volume. Moreover, since the processing program is also transmitted to the external device, the configuration of Patent Document 1 requires a long time for data transmission. If a fee is charged for data transmission, there is also the problem of increased communication charges. In particular, if all RAW data obtained by imaging is transmitted to the external device, these problems become extremely serious.

[0008] The present invention aims to at least alleviate the problems of the conventional techniques. According to the present invention, there is provided an information processing device that enables efficient use of image processing in an external device, and a control method thereof. According to the present invention, there is also provided an image processing device suitable as an external device for such an information processing device, a control method thereof, and an image processing system having the information processing device and the image processing device. [Means for solving the problem]

[0009] The above-mentioned object can be achieved by an information processing device having a communication means and a control means, wherein the control means transmits reduced image data of a captured image to a first external device through the communication means, receives a result of an evaluation process on the reduced image data from the first external device, determines, based on the result of the evaluation process, original data from the original data of the reduced image data to which image processing is applied in a second external device, transmits the determined original data to the second external device through the communication means, and receives the result of the image processing on the original data from the second external device. Effect of the Invention

[0010] According to the present invention, an information processing apparatus and a control method thereof that enable efficient use of image processing in an external device are provided. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing an image processing system, an information processing device, and a server according to an embodiment of the present invention; [Diagram 2] Flowchart relating to the first embodiment [Diagram 3] FIG. 1 is a diagram for explaining a first embodiment. [Figure 4] Flowchart relating to the first embodiment [Diagram 5] FIG. 1 is a diagram showing an image processing system and an edge device according to second to fifth embodiments. [Figure 6] Flowchart relating to the second embodiment [Figure 7] FIG. 11 is a diagram relating to an edge device in the third embodiment. [Figure 8] Flowchart relating to the third embodiment [Figure 9] Flowcharts for the third and fourth embodiments [Figure 10] FIG. 13 is a diagram for explaining the third and fourth embodiments. [Figure 11] FIG. 10 is a diagram for explaining the third and fourth embodiments. [Figure 12] FIG. 11 is a diagram related to a server in the fourth embodiment. [Figure 13] Flowchart relating to the fourth embodiment [Figure 14] Flowchart relating to the fifth embodiment [Figure 15] FIG. 13 is a block diagram showing an example of the functional configuration of an information processing device according to a sixth embodiment. [Figure 16] FIG. 13 is a block diagram showing an example of the functional configuration of a server apparatus and an edge device according to a sixth embodiment. [Figure 17] Flowchart relating to the sixth embodiment [Figure 18]Flowchart relating to a modified example of the sixth embodiment [Figure 19] Flowchart relating to the seventh embodiment [Figure 20] FIG. 23 is a diagram showing an example of a partial region in the seventh embodiment; [Figure 21] Flowchart relating to the eighth embodiment [Figure 22] Flowchart relating to the ninth embodiment [Figure 23] Flowchart relating to the tenth embodiment [Figure 24] Flowchart relating to the tenth embodiment [Diagram 25] Flowchart relating to the eleventh embodiment [Figure 26] Flowchart relating to the twelfth embodiment [Figure 27] Flowchart relating to the thirteenth embodiment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0013] In the following embodiment, the present invention will be described with respect to a case where the present invention is implemented in an imaging device as an example of an information processing device. However, the present invention is applicable to any electronic device capable of handling image data. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention is also applicable to other electronic devices.

[0014] ●(First embodiment) FIG. 1(a) is a diagram showing a schematic configuration example of an image processing system according to a first embodiment of the present invention. The image processing system has a configuration in which an image capturing device 100 and a server device 200 (first and second external devices) that are external devices are connected to each other so as to be able to communicate with each other through a network 300. Note that a plurality of image capturing devices 100 may be connected to the network 300. The image capturing device 100 and the server device 200 can communicate with each other using a communication protocol according to the type of the network 300. For example, if the network 300 is the Internet, communication can be performed using a protocol such as TCP / IP or UDP. Here, the network 300 is a public network such as the Internet, or a network in which a fee is incurred for use or communication volume.

[0015] 1(b) is a block diagram showing an example of the functional configuration of the imaging device 100. The imaging device 100 is typically a digital (video) camera, but may be any electronic device that has an imaging function and can communicate with a server device 200 (hereinafter also simply referred to as the server 200) through a network 300. The control unit 101 is, for example, a CPU (also called an MPU or microprocessor), and controls the operation of each unit of the imaging device 100 by loading a program stored in a ROM 102 into a RAM 103 and executing it, thereby realizing various functions of the imaging device 100. Note that, if the optical system 104 is replaceable, the control unit 101 controls the operation of the optical system 104 through communication with a controller that the optical system 104 has.

[0016] The ROM 102 stores programs executed by the control unit 101, various setting values ​​of the imaging device 100, GUI data, etc. The ROM 102 may be electrically rewritable. The RAM 103 is a main memory used when the control unit 101 executes the programs. The RAM 103 may also be used as a buffer memory for image data or a video memory for the display unit 109.

[0017] The optical system 104 includes movable lenses such as a zoom lens and a focus lens, and forms an image of a subject on an imaging surface of the imaging unit 105. The control unit 101 controls the driving of the movable lenses of the optical system 104.

[0018] The imaging unit 105 has an imaging element such as a CCD image sensor or a CMOS image sensor. The imaging element has a plurality of photoelectric conversion units arranged two-dimensionally, and the photoelectric conversion units convert the subject image formed on the imaging surface by the optical system 104 into an analog image signal. The A / D conversion unit 106 converts the analog image signal input from the imaging unit 105 into digital image data. The A / D conversion unit 106 may be provided in the imaging unit 105.

[0019] The digital image data is temporarily stored in the RAM 103. The image processing unit 107 applies predetermined image processing to the image data stored in the RAM 103 to generate signals and image data, and acquire and / or generate various information. The image processing unit 107 may be, for example, a dedicated hardware circuit such as an ASIC designed to realize a specific function, or may be configured to realize a specific function by a programmable processor such as a DSP executing software.

[0020] Here, the image processing applied by the image processing unit 107 includes pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing includes signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation is a process that interpolates the values ​​of color components that are not included in the image data read from the pixels, and is also called demosaic processing or synchronization processing. The correction processing includes white balance adjustment, gradation correction (gamma processing), processing for correcting the effects of optical aberration and peripheral light falloff of the optical system 104, and color correction processing. The detection process includes detection of feature regions (for example, face regions and human body regions) and their movements, and person recognition processing. The data processing includes synthesis, scaling, encoding and decoding, header information generation, and the like. The evaluation value calculation process includes the generation of signals and evaluation values ​​used in automatic focus detection (AF), and the calculation of evaluation values ​​used in automatic exposure control (AE). Special effects include blurring, changing color tones, relighting, and the like. It should be noted that these are merely examples of image processing that the image processing unit 107 can apply, and do not limit the image processing that the image processing unit 107 can apply.

[0021] Among the above-mentioned image processes, the digital image data supplied from the A / D conversion unit 106 to the image processing unit 107 is in a state in which the pixel data constituting the digital image data has a value of one color component according to the color arrangement of the color filters of the image sensor. Such image data is called RAW data. Among the image processes applied to the RAW data by the image processing unit 107, a series of image processes for converting the RAW data into image data in a general format that can be used for display or printing as photographic data is called development processing. Generally, development processing includes white balance processing and color complementation processing, and may also include lens aberration correction processing, noise reduction (NR) processing, gamma (tone conversion) processing, and the like.

[0022] The recording unit 108 records captured image data on a recording medium and reads captured image data recorded on the recording medium. The recording medium may be, for example, a removable memory card. The recording unit 108 records the captured image data (moving image or still image data) and / or RAW data for recording that have been subjected to development processing by the image processing unit 107 and stored in the RAM 103, and the proxy image data on the recording medium.

[0023] The proxy image data is image data having a smaller amount of data than the captured image data for recording or RAW data, and may be a thumbnail image for still images. The thumbnail image is an image obtained by reducing the size of the original image after development processing, and the thumbnail image data is reduced image data. The proxy image data may be image data having, for example, a lower resolution and / or a smaller bit depth than the captured image data for recording or RAW data. In the case of still images, the proxy image data may be image data (thumbnail image data) having a lower resolution than RAW data and having been subjected to development processing (developed). In the case of videos, the proxy image data may be developed video data having a lower bit rate than RAW data. Note that the proxy image data does not necessarily have to be developed, but developed image data can reduce the amount of data.

[0024] The display unit 109 is a display device such as an LCD, and performs display corresponding to image data stored in the video memory area of ​​the RAM 103. The display unit 109 displays captured images and information obtained from the images (e.g., a brightness histogram), setting values ​​of the imaging device 100, and GUI elements (such as menu screens and icons and marks superimposed on captured images).

[0025] The communication unit 110 has a communication interface conforming to one or more known wireless or wired communication standards, for example, and realizes data communication between an external device and the imaging device 100. The communication unit 110 can support data communication using a mobile phone network conforming to standards such as LTE and 5G, and data communication conforming to standards such as wireless LAN and Bluetooth (registered trademark). The communication unit 110 can also support communication standards such as USB and Ethernet (registered trademark). In this embodiment, the communication unit 110 realizes communication between the imaging device 100 and the server 200 through the network 300. The control unit 101 transmits and receives image data and the like to and from the server 200 through the communication unit 110.

[0026] The operation unit 111 is a general term for input devices such as buttons and switches for the user to input various instructions to the imaging device 100. When the display unit 109 is a touch display, the touch panel is also included in the input device. The input devices constituting the operation unit 111 have names according to the assigned functions. For example, the operation unit 111 includes a release switch for instructing the start of a shooting preparation operation and the start of shooting, a shooting mode selection switch for selecting a shooting mode, a menu button, direction keys, and a decision key. A plurality of functions may be assigned to the same input device. The input device may also be a software button / key. The operation unit 111 may also include an input device for inputting instructions without contact, such as voice input or gaze input.

[0027] 1C is a block diagram showing an example of the functional configuration of the server 200. The server 200 is typically a general-purpose computer on a network, but may be any electronic device that has a server function and can communicate with the imaging device 100 via the network 300. The control unit 201 is, for example, a CPU (also called an MPU or microprocessor), and realizes various functions of the server by loading programs stored in the ROM 203 or storage unit 205 into the RAM 202 and executing them. In addition to the CPU, the server 200 may use an auxiliary processor suitable for executing image processing, such as a GPU. The ROM 203 stores application programs for realizing the image processing functions provided by the server 200, programs executed at startup, various setting values, and the like.

[0028] Here, an image recognition program 210 and an image processing program 211 are stored in the ROM 203 as examples of application programs for realizing the image processing function. The image processing function provided by the server 200 is typically a function that cannot be applied to the imaging device 100, or a function that is superior to the imaging device 100 in terms of at least one of processing time and processing quality. The server 200 can obtain programs based on the latest image processing technology from other devices that can communicate via the network 300, for example, through the communication unit 204. Therefore, compared to the imaging device 100, which is not easy to update software, the server 200 can easily enjoy the benefits of technological advances. For example, it is easy to use an image recognition program that uses machine learning technology such as deep learning, which has been put to practical use in recent years.

[0029] The RAM 202 is a main memory used when the control unit 201 executes a program. The RAM 202 may also be used as a buffer memory for image data or a video memory for the display unit 207.

[0030] The communication unit 204 has a communication interface conforming to, for example, a known wireless or wired communication standard, and realizes data communication between an external device and the server 200. In this embodiment, the communication unit 204 realizes communication between the server 200 and the imaging device 100 via the network 300. The control unit 201 transmits and receives image data and the like to and from the imaging device 100 via the communication unit 204.

[0031] The storage unit 205 may be a large-capacity storage device such as a hard disk drive (HDD) or a solid-state drive (SSD). The storage unit 205 is used to store, for example, an OS, application programs, user data, etc. The ROM 203 may be a part of the storage unit 205.

[0032] The operation unit 206 is a general term for an input device that allows a user to input instructions to the server 200, and is typically a keyboard, a mouse, a touch pad, or the like.

[0033] The display unit 207 is a display device such as an LCD, and performs display corresponding to image data stored in the video memory area of ​​the RAM 202. The display unit 207 displays user interface elements provided by the OS, information related to applications, and the like.

[0034] The network 300 may have any configuration capable of supporting data communication between the image capture device 100 and the server 200, and the specific configuration is arbitrary, so a description thereof will be omitted.

[0035] Next, the interaction between the imaging device 100 and the server 200 will be described with reference to the flowchart in Fig. 2. In the following, operations described as being mainly performed by the imaging device 100 are actually performed by the control unit 101 (CPU), and operations described as being mainly performed by the server 200 are actually performed by the control unit (CPU) 201. In addition, in the imaging device 100, when the image processing unit 107 generates image data for recording from image data obtained by shooting, it also generates corresponding proxy image data. It is assumed that the image data for recording and the proxy image are stored in association with each other in the recording unit 108.

[0036] In S401, the imaging device 100 transmits proxy image data recorded in the recording unit 108 from the communication unit 110 to the server 200 via the network 300. It is assumed that settings and information necessary for communication with the server 200 are registered in advance and stored in the ROM 102. There is no particular limit to the timing of executing S401, and it may be started, for example, when it is confirmed that unprocessed RAW data exists in the recording unit 108, or it may be started by a user instruction. In addition, the proxy image data to be transmitted may be data corresponding to RAW data, data corresponding to developed image data, or data selected by the user. Alternatively, all proxy image data may be transmitted.

[0037] Also, among the RAW data, proxy image data corresponding to RAW data that has not been developed (unprocessed) may be transmitted. In this case, for example, RAW data that does not have image data recorded in a corresponding general format (e.g., JPEG format) can be determined as unprocessed RAW data. Alternatively, a file name that can be determined as to whether it is unprocessed or not may be given to the data, and the unprocessed RAW data may be determined from the file name.

[0038] In S402, the server 200 receives the proxy image data from the imaging device 100 via the communication unit 204. The server 200 stores the received proxy image data in the RAM 202.

[0039] In S403, the server 200 loads the image recognition program 210 stored in the ROM 203 into the RAM 202 and executes it. The image recognition program 210 provides a function of detecting a face area by applying a known face recognition process to the image data, and a subject recognition function of recognizing a person in the detected face area by comparing the features of the detected face area with the features of the face area of ​​a person registered in advance. By executing the image recognition program 210, the control unit 201 applies, to the proxy image data, an evaluation process that generates information that serves as a judgment criterion for the imaging device 100 to determine image data to which image processing is applied in an external device. The subject recognition process is an example of the evaluation process, and other information may be generated based on the proxy image data.

[0040] It is assumed that the face area or the feature information of the person to be detected is registered in advance in the server 200. This registration can be realized, for example, by transmitting image data including the face of the person to be detected from the imaging device 100 to the server 200, and the server 200 registers feature information extracted from the face area using the image recognition program 210. It is to be noted that the person to be detected may be registered in other ways. Here, it is assumed that persons 501 and 502 are registered as persons to be detected using an image as shown in FIG. 3. In addition, the server 200 assigns unique identification information (ID) to the feature information for each person to be detected and stores the information in, for example, the storage unit 205. Here, it is assumed that ID1 is assigned to person 501 and ID2 is assigned to person 502.

[0041] The object recognition process in S403 will be described in detail with reference to the flowchart shown in Fig. 4. This process is realized by the control unit 201 (CPU) executing the image recognition program 210.

[0042] In S601, the control unit 201 applies face recognition processing to one of the proxy image data stored in the RAM 202. The face recognition processing is processing for detecting a face area from the proxy image data and processing for determining whether the detected face area belongs to a registered person. If the control unit 201 determines that the features of the detected face area are the same as or highly similar to the features of the registered person, it can determine that the face area belongs to the registered person. Note that if multiple face areas are detected from the proxy image data, the control unit 201 determines for each detected face area whether it belongs to a registered person.

[0043] In S602 to S603 and S605 to S606, the control unit 201 assigns a tag to the proxy image data based on the result of the face authentication process in S601. In S602, if the face recognition process of S601 determines that the face area detected from the proxy image data is the face area of ​​the person registered as ID1, the control unit 201 executes S606, and if not, executes S603. In S603, if the face recognition process of S601 determines that the face area detected from the proxy image data is the face area of ​​the person registered as ID2, the control unit 201 executes S605, and if not, executes S604.

[0044] In S605, the control unit 201 assigns a tag of ID 2 to the proxy image data. In addition, in S606, the control unit 201 assigns a tag of ID 1 to the proxy image data. Note that if multiple face areas are detected from the proxy image data, the control unit 201 executes S602 to S603 for each face area.

[0045] In S604, the control unit 201 assigns, to the proxy image data, a tag (mismatch tag) indicating that it has been determined that the image does not include a registered person.

[0046] In S607, if the control unit 201 determines that the object recognition processing has been applied to all received proxy image data, it terminates the object recognition processing; if not, it applies the processing from S601 onwards to the proxy image data to which the object recognition processing has not been applied.

[0047] Although an example in which two people, ID1 and ID2, are registered is shown here, the control unit 201 performs the same tagging process regardless of the number of people registered. In this manner, through the subject recognition process, a tag of the ID of a registered person determined to appear in the proxy image, or a tag indicating that no registered person is appearing in the proxy image, is assigned to the proxy image data received from the imaging device 100.

[0048] Returning to FIG. 2, in S404, the server 200 transmits the tag information (subject recognition result) added in S403 to the imaging device 100 via the network 300. The server 200 transmits, for example, data that associates information for identifying the proxy image data with the tag information added to the proxy image data as tag information. By not transmitting the proxy image data, the amount of transmitted data can be reduced, and processing delays can be reduced. The tag information may be, for example, one or more person IDs or information indicating a mismatch.

[0049] In S405 , the imaging device 100 receives the tag information from the server 200 and stores it in the RAM 103 . In S406, the imaging device 100 determines image data (here, RAW data) to which image processing is to be applied by the server 200 based on the received person tag information. Here, as an example, it is assumed that the original data of the proxy image associated with one or more person IDs is determined as the image data to which image processing is to be applied by the server 200. Note that the original data of the proxy image is image data that is the source of the proxy image, and may be either RAW data or developed image data.

[0050] In this case, in S406, the imaging device 100 determines whether or not one or more person IDs are associated with the tag information received from the server 200 for one of the proxy image data transmitted in S401. If the imaging device 100 determines that one or more person IDs are associated, the imaging device 100 determines the corresponding RAW data as RAW data to which the server 200 applies image processing, and executes S407. On the other hand, if the imaging device 100 does not determine that one or more person IDs are associated, it executes S413 without executing S407.

[0051] In S407, the imaging apparatus 100 transmits the RAW data to the server 200 and waits to receive the development results. In S408 , when the server 200 receives the RAW data from the image capture device 100 , the server 200 stores the RAW data in the RAM 202 or the storage unit 205 .

[0052] In S409, the server 200 loads the image processing program 211 from the ROM 203 to the RAM 202 and executes it. The image processing program 211 applies development processing to the RAW data. The development processing includes, for example, white balance processing, lens aberration correction, noise reduction (NR) processing, color interpolation processing, gamma processing, etc. In addition, after the development processing, an encoding process may be applied to generate an image data file in a predetermined format, for example, JPEG format.

[0053] The development processing applied by the server 200 is based on a newer and / or more complex technique than that of the image processing unit 107 of the imaging device 100, and a better processing result can be obtained than that of the development processing applied by the imaging device 100. This is because the processing capacity of the server 200 is higher than that of the imaging device 100, and the image processing program 211 executed by the server 200 is based on a newer technology than that of the image processing unit 107 of the imaging device 100.

[0054] In S410, the server 200 transmits the image data to which the development processing has been applied in S409 to the imaging device 100 via the network 300. In S411, the imaging device 100 receives the developed image data from the server 200.

[0055] In S412, the imaging device 100 supplies the developed image data to the image processing unit 107, which generates image data for recording and image data for display. Then, the imaging device 100 displays the image data for display on the display unit 109, and records the image data for recording in the recording unit 108.

[0056] In S413, the imaging device 100 determines whether or not all of the proxy image data transmitted in S401 has been processed, and if it is determined that all of the proxy image data has been processed, the development process of the RAW data ends. On the other hand, if it is not determined that all of the proxy image data has been processed, the process from S406 is applied to the unprocessed proxy image data.

[0057] The imaging device of this embodiment determines whether or not to execute image processing on the original data of the proxy image data in an external device based on the result of applying a predetermined image processing to the proxy image data of the recorded image data in the external device. Therefore, it is possible to determine whether or not to apply image processing in the external device based on the result of image processing that cannot be applied by the imaging device or that can only be applied with low accuracy.

[0058] For example, when the imaging device cannot apply a highly accurate object recognition process, the image data to be processed by the external device can be more accurately determined by using the result of applying a highly accurate object recognition process to the proxy image data by the external device. In the present embodiment, an example has been described in which the result of determining whether or not a registered person is captured by the external device is used. However, it is also possible to use the result of other image processing, such as the result of object recognition process other than a person or the result of scene determination process.

[0059] In the present embodiment, a configuration has been described in which the image capturing apparatus 100 does not allow the user to be involved in determining which image data should be processed by the external device. However, a list of image data determined to be processed by the external device may be displayed on the display unit 109 in a selectable manner, and only image data selected by the user from the list may be transmitted to the external device. Furthermore, the image processing finally applied by the external device is not limited to development processing of RAW data, and may be any image processing. Furthermore, the image data to which image processing is applied by the external device is not limited to RAW data, and may be developed image data for recording.

[0060] ●(Second embodiment) Next, a second embodiment of the present invention will be described. Fig. 5(a) is a diagram showing a schematic configuration example of an image processing system according to this embodiment, and components similar to those in the first embodiment are given the same reference numerals as in Fig. 1(a) and will not be described. In this embodiment, image processing performed on proxy image data is not performed by the server 200 on the network 300, but by an edge device 400, which is an external device located before the network 300 as viewed from the imaging device 100. Therefore, in this embodiment, the edge device 400 is a first external device, and the server 200 is a second external device.

[0061] The edge device 400 is an image processing device that can communicate with the imaging device 100 and also with the server 200 through the network 300. Here, a configuration using a computer device having a communication function, such as a smartphone or a tablet computer, will be described as an example of the edge device 400. When a smartphone is used instead of the imaging device 100, it is possible to use a stationary computer, for example, with higher performance than the smartphone as the edge device 400.

[0062] In recent years, many smartphones and tablet computers are equipped with processors that perform high-speed image processing using machine learning technology. Therefore, they can be suitably used as edge devices. Note that a general-purpose computer similar to the server 200 may be used as the edge device 400.

[0063] There is no particular limitation on the communication method between the imaging device 100 and the edge device 400, and it may be wired communication or wireless communication. The imaging device 100 may use the same communication method or different communication methods for communication with the server 200 and the edge device 400. When different communication methods are used, the communication unit 110 has a communication interface corresponding to each communication method (see FIG. 15). A plurality of communication units 110 may be provided according to the communication method. It is assumed here that no cost is incurred for communication between the imaging device 100 and the edge device 400.

[0064] 5(b) is a block diagram showing an example of the functional configuration of the edge device 400. The control unit 401 is, for example, a CPU (also called an MPU or a microprocessor), and realizes various functions of the edge device 400 by loading programs stored in a ROM 402 or a recording unit 407 into a RAM 403 and executing them.

[0065] The ROM 402 stores the programs executed by the control unit 401, various setting values ​​of the edge device 400, GUI data, etc. The ROM 402 may be electrically rewritable. The RAM 403 is a main memory used when the control unit 401 executes the programs. The RAM 403 may also be used as a buffer memory for data or a video memory for the display unit 408.

[0066] The image recognition unit 404 executes an object recognition process to determine whether or not a person registered in advance is captured in the image. The object recognition process performed by the image recognition unit 404 may be the same as the object recognition process performed by the server 200 in the first embodiment.

[0067] The image processing unit 405 applies predetermined image processing to image data stored in the RAM 403, generates signals and image data, and acquires and / or generates various information. Image processing that the image processing unit 405 can apply may be similar to the image processing applied by the image processing unit 107 of the imaging device 100 or the image processing applied by the server 200 using the image processing program 211.

[0068] The operation unit 406 is a general term for an input device for a user to input instructions to the edge device 400, and is typically a keyboard, a mouse, a touch pad, etc. If the display unit 408 is a touch display, the touch panel of the touch display is included in the operation unit 406.

[0069] The recording unit 407 records data on a recording medium and reads data recorded on the recording medium. The recording medium may be, for example, a removable memory card or a fixed storage device.

[0070] The display unit 408 is a display device such as an LCD, and performs display corresponding to image data stored in the video memory area of ​​the RAM 403. The display unit 408 displays data stored in the recording unit 407, GUI elements related to applications and the OS, etc. The display unit 408 may be a touch display.

[0071] The edge device 400 of this embodiment has two communication units 409 and 410. Here, the communication unit 410 is used for communication with an external device (e.g., the server 200) on the network 300, and the communication unit 409 is used for communication with the imaging device 100. Note that one communication unit that supports a plurality of communication methods may be provided. Alternatively, one of the communication units 409 and 410 may be used to communicate with both the external device on the network 300 and the imaging device 100.

[0072] Next, the interoperation between the imaging device 100, the edge device 400, and the server 200 will be described with reference to the flowchart of FIG. 6. Hereinafter, the operations described as being mainly performed by the imaging device 100 are actually performed by the control unit 101 (CPU), the operations described as being mainly performed by the server 200 are actually performed by the control unit (CPU) 201, and the operations described as being mainly performed by the edge device 400 are actually performed by the control unit 401 (CPU). In addition, in FIG. 6, the same reference numerals as in FIG. 2 are used for steps that perform the same operations as those described in the first embodiment. It is assumed that settings and information required for communication with the server 200 and the edge device 400 are registered in advance and stored in the ROM 102. It is assumed that, when it is necessary to register the edge device 400 as a communication partner before communicating with the edge device 400, such registration is also performed in advance.

[0073] S401 and S405 are the same as those in the first embodiment, except that the communication partner is the edge device 400 rather than the server 200. Furthermore, in S402 to S404, the edge device 400 executes the same processes as those executed by the server in the first embodiment. In S402, the edge device 400 receives the proxy image data from the imaging apparatus 100 via the communication unit 409. The edge device 400 stores the received proxy image data in the RAM 403.

[0074] In S403, the edge device 400 executes the above-described object recognition process on the image data stored in the RAM 403 by the image recognition unit 404. The method of the object recognition process does not need to be the same as the method used by the server 200. In S404, the edge device 400 transmits information on the tag added in S403 (subject recognition result) to the imaging device 100 via the communication unit 409.

[0075] In S405 and S406, the imaging apparatus 100 determines, from the RAW data corresponding to the proxy image data transmitted in S401, the RAW data to which image processing is to be applied in an external apparatus (here, the server 200), in the same manner as in the first embodiment. Then, in S407, the image capturing apparatus 100 transmits the determined RAW data to the server 200.

[0076] The operations of the server 200 in S408 to S410 are similar to those in the first embodiment. In addition, in S410, the server 200 may transmit the image data to which the development process has been applied to the edge device 400 instead of or in addition to transmitting the image data to the imaging device 100 via the network 300. In this case, information (such as a destination address) required for transmitting data to the edge device 400 is registered in advance in the server 200, or is notified to the server 200 when the imaging device 100 transmits the RAW data.

[0077] The operations of the imaging device 100 in S411 to S413 are similar to those in the first embodiment. When the edge device 400 receives image data to which development processing has been applied from the server 200, in S1011 the edge device 400 stores the received image data in the RAM 403. Then, in S1012, the edge device 400 displays the image data stored in the RAM 403 on the display unit 408 and records the image data in the recording unit 407.

[0078] In this embodiment, image processing of the proxy image data is performed by the edge device 400, not by the server 200. In this embodiment, the amount of communication data between the imaging device 100 and the server 200 is less than that of the first embodiment, so that in addition to the same effects as the first embodiment, it is possible to reduce communication costs and the effects of congestion on the network 300 and quality degradation.

[0079] In this embodiment, as in the first embodiment, the image processing applied to the proxy image data and the image processing applied to the RAW data by the server 200 are merely examples, and other image processing may be used. Moreover, the image data to which the image processing is finally applied by the server 200 is not limited to RAW data, and may be developed image data for recording.

[0080] Also, the image data that the imaging device 100 has determined as the image data to which the server 200 will finally apply image processing may be sent to the server 200 after the user has confirmed the image data. At this time, the user can change one or more of the image data determined by the imaging device.

[0081] ●(Third embodiment) Next, a third embodiment of the present invention will be described. The image processing system of this embodiment differs from that of the second embodiment in the configuration and operation of the edge device. Therefore, the following description will focus on the differences from the second embodiment.

[0082] 7 is a block diagram showing an example of the functional configuration of an edge device 400′ of this embodiment, which has a configuration in which an evaluation unit 411 is added to the edge device 400 of the second embodiment. The other functional blocks are similar to those of the second embodiment, and therefore description thereof will be omitted.

[0083] In this embodiment, the imaging device 100 transmits RAW data for multiple frames to the edge device 400' instead of the server 200. Then, the edge device 400' selects RAW data for which application of development processing by the server 200 is considered to have a large effect, and transfers the RAW data to the server 200. The evaluation unit 411 is configured to determine the RAW data to be transmitted. For convenience, the evaluation unit 411 is depicted in FIG. 7 as an independent component, but it may be realized by the control unit 401 (CPU) executing an application for the image processing system stored in the ROM 402.

[0084] Next, the interoperation between the imaging device 100, the edge device 400', and the server 200 will be described with reference to the flowchart of FIG. 8. Hereinafter, the operations described as being mainly performed by the imaging device 100 are actually performed by the control unit 101 (CPU), the operations described as being mainly performed by the server 200 are actually performed by the control unit (CPU) 201, and the operations described as being mainly performed by the edge device 400' are actually performed by the control unit 401 (CPU). In addition, in FIG. 8, the same reference numerals as in FIG. 2 and FIG. 6 are used for steps that perform the same operations as those described in the first and second embodiments. It is assumed that settings and information required for communication with the server 200 and the edge device 400' are registered in advance and stored in the ROM 102. It is assumed that, when it is necessary to register the edge device 400' as a communication partner before communicating with the edge device 400', such registration is also performed in advance.

[0085] 8, the operations from S401 to S406 are the same as those in the second embodiment, and therefore the description will be omitted. In S407, the imaging device 100 transmits RAW data to which the server 200 applies image processing to the edge device 400'.

[0086] In S1208, the edge device 400′ stores the received RAW data in the RAM 403 or the recording unit 407. In S1209, the edge device 400' causes the evaluation unit 411 to calculate an evaluation value for each piece of received RAW data.

[0087] Here, an example of the operation of calculating an evaluation value performed by the evaluation unit 411 will be described with reference to the flowchart in Fig. 9. The evaluation unit 411 calculates an evaluation value, which is an index of the magnitude of the effect of applying image processing in the server 200 to the image data received from the imaging device 100.

[0088] Here, as an example, a method of calculating an evaluation value in a case where the effect of applying image processing by server 200 is greater for an image with more edges is described. Note that there is no particular limit to what evaluation value is calculated, and it is possible to change it as appropriate, for example, by updating the application for the above-mentioned image processing system. Since the evaluation value depends on the image processing applied by server 200, the evaluation value calculated by evaluation unit 411 may be changed, for example, in response to an update of image processing program 211 in server 200.

[0089] In S1601, the evaluation unit 411 reads out one frame's worth of RAW data and generates a luminance signal. Here, it is assumed that the imaging element of the imaging device 100 has a primary color Bayer array color filter. In this case, the RAW data is data in which each pixel has the value of one color component, as shown in FIG. 10(a). Note that FIG. 10 shows only an image area of ​​four pixels in the horizontal and vertical directions of the RAW data. Also, R represents red, G represents green, and B represents blue.

[0090] The evaluation unit 411 first separates the RAW data into color components and generates color plane image data (FIG. 10(b)) by inserting zeros into pixel positions where no color component exists. The evaluation unit 411 then applies known interpolation processing to the color plane image data to compensate for the values ​​of the pixels where zeros have been inserted (FIG. 10(c)).

[0091] Next, the evaluation unit 411 uses the interpolated image data for each color plane to calculate the luminance value Y at each pixel position according to the following formula 1, and generates the luminance data (FIG. 10(d)). Y = 0.3 × R + 0.6 × G + 0.1 × B (Equation 1) Here, R, G, and B are values ​​in the respective plane image data.

[0092] In S1602, the evaluation unit 411 applies edge detection processing to one pixel of the luminance data generated in S1601 as a target pixel. The edge detection may be, for example, a spatial filter processing using a Sobel filter shown in FIG. 11. FIG. 11(a) shows the vertical edge strength E y The Sobel filter for calculating the horizontal edge strength E x The Sobel filter is used to calculate the vertical edge strength E y and the horizontal edge strength E x The spatial filter process is a multiplication and accumulation operation between the values ​​of the target pixel, the surrounding pixels centered on the target pixel, and the coefficients of the spatial filter at the corresponding positions. Note that the coefficients and sizes of the spatial filter shown in FIG. 11 are merely examples. Edge strength may be detected using other filters or methods.

[0093] The evaluation unit 411 evaluates the vertical edge strength E y and horizontal edge strength E x Then, the edge strength E (amount of change) for the target pixel is calculated using the following formula 2. E = {(Ex 2 + Ey 2 )} 1 / 2 (Formula 2)

[0094] In S1603, the evaluation unit 411 determines whether the edge strength E calculated in S1602 is equal to or greater than a threshold, and if it is determined that the edge strength E is equal to or greater than the threshold, an edge flag is assigned to the target pixel in S1604, and then S1605 is executed. On the other hand, if it is not determined that the edge strength E is equal to or greater than the threshold in S1603, the evaluation unit 411 skips S1604 and executes S1605.

[0095] In S1605, the evaluation unit 411 determines whether or not edge detection processing has been applied to all pixels of the luminance data. If it is determined that edge detection processing has been applied to all pixels, the evaluation unit 411 calculates the number of pixels to which edge flags have been assigned as an evaluation value for the RAW data in S1606. On the other hand, if it is not determined that edge detection processing has been applied to all pixels, the evaluation unit 411 applies the processing from S1602 to pixels to which edge detection processing has not been applied. The evaluation unit 411 calculates an evaluation value for each of the stored RAW data.

[0096] 8, in S1210, the edge device 400′ determines RAW data to be transmitted to the server 200 based on the evaluation value calculated in S1209. Specifically, the edge device 400′ transmits to the server 200 RAW data whose evaluation value is equal to or greater than a preset threshold value. This threshold value may be configured to be adjustable according to a user instruction obtained via a user interface.

[0097] In S1211, the edge device 400′ transmits the RAW data determined in S1210 to the server 200 via the network 300.

[0098] The operations of S408 to S410 in the server 200, S1011 to S1012 in the edge device 400', and S411 to S413 in the imaging device 100 are similar to those in the second embodiment, and therefore description thereof will be omitted.

[0099] In this embodiment, image data to which image processing is applied by the server 200 is evaluated by the edge device 400 before being transmitted to the server 200, and RAW data that is considered to be highly effective when image processed by the server 200 is transmitted to the server 200. Therefore, the imaging device 100 can receive the processing result of an image for which application of image processing by the server 200 is highly effective.

[0100] Note that the image data to be transmitted may be presented to the user before the edge device 400' transmits the image data to the server 200, and the image data may be transmitted only after confirmation is obtained. In this case, the user may be allowed to select image data that is not to be transmitted.

[0101] ●(Fourth embodiment) Next, a fourth embodiment of the present invention will be described. In the image processing system of this embodiment, the server 200 evaluates the image data executed by the edge device 400' in the third embodiment, and determines the RAW data to which image processing is applied. Therefore, the following description will focus on the differences from the third embodiment.

[0102] 12 is a block diagram showing an example of the functional configuration of the server 200′ in this embodiment, which has a configuration in which an effect determination program 212 is added to the ROM 203 of the server 200 described in the first embodiment. The other functional blocks are similar to those in the first embodiment, and therefore description thereof will be omitted.

[0103] The effect judgment program 212, when executed by the control unit 201, realizes a function similar to that of the evaluation unit 411 of the edge device 400' in the third embodiment. The control unit 201 executes the effect determination program 212 to evaluate the image data (RAW data) received from the imaging device 100. Then, based on the evaluation result, it determines the RAW data to which the processing by the image processing program 211 is to be applied.

[0104] Next, the interoperation between the imaging device 100, the edge device 400, and the server 200' will be described with reference to the flowchart in FIG. 13. In the following, operations described as being mainly performed by the imaging device 100 are actually performed by the control unit 101 (CPU), operations described as being mainly performed by the server 200' are actually performed by the control unit (CPU) 201, and operations described as being mainly performed by the edge device 400 are actually performed by the control unit 401 (CPU). In addition, in FIG. 13, steps that perform the same operations as those described in the first to third embodiments are given the same reference numerals as in FIG. 2, FIG. 6, and FIG. 8. In addition, when it is necessary to register the edge device 400' as a communication partner before communicating with the edge device 400', it is assumed that such registration has also been performed in advance.

[0105] 8, the operations from S401 to S406 are the same as those in the second embodiment, and therefore the description will be omitted. In S407, the imaging device 100 transmits RAW data to which image processing is applied in the server 200 to the server 200 in the same manner as in the first embodiment. In S408, when the server 200' receives the RAW data from the image capture device 100, the server 200' stores the RAW data in the RAM 202 or the storage unit 205.

[0106] In S1209, the control unit 201 executes the effect determination program 212 stored in the ROM 203, and calculates an evaluation value for the RAW data received from the imaging device 100. This process may be the same as that described in the third embodiment.

[0107] In S1210, the control unit 201 determines, based on the evaluation value calculated in S1209, whether or not to apply image processing by the image processing program 211 to the RAW data received from the imaging device 100. Specifically, the control unit 201 determines to apply image processing only to RAW data for which the image processing by the image processing program 211 is considered to be highly effective.

[0108] In S409, the control unit 201 executes the image processing program 211, and applies image processing (here, development processing) to the RAW data determined in S1210. The subsequent operations of S410 in the server 200', S1011 to S1012 in the edge device 400, and S411 to S413 in the imaging device 100 are similar to those in the second embodiment, and therefore description thereof will be omitted.

[0109] In this embodiment, the server 200' evaluates image data to which the server 200' applies image processing, and applies image processing only to RAW data for which the server 200' is likely to be effective. Therefore, the imaging device 100 can receive the processing results of only images for which the server 200 will be effective in applying image processing.

[0110] Note that before applying image processing in the server 200', image data to which the image processing is to be applied may be transmitted to the imaging device 100 and presented to the user via the display unit 109 of the imaging device 100. Then, the image processing may be applied after confirmation is obtained from the imaging device 100. In this case, the imaging device 100 may be configured so that the user can select image data to which the image processing is not to be applied.

[0111] ●(Fifth embodiment) Next, a fifth embodiment of the present invention will be described. The image processing system of this embodiment is configured such that the process that the server 200 receives RAW data and performs in the first embodiment is performed by the edge device 400. The configuration of the image processing system according to this embodiment, the imaging device 100, the server 200, and the functional configuration of the edge device 400 may be similar to those of the second embodiment.

[0112] Next, the interoperation between the imaging device 100, the edge device 400, and the server 200 will be described with reference to the flowchart in Fig. 14. Hereinafter, operations described as being mainly performed by the imaging device 100 are actually performed by the control unit 101 (CPU), operations described as being mainly performed by the server 200 are actually performed by the control unit (CPU) 201, and operations described as being mainly performed by the edge device 400 are actually performed by the control unit 401 (CPU). In Fig. 14, steps that perform the same operations as those described in the first embodiment are given the same reference numerals as in Fig. 2. In addition, when it is necessary to register the edge device 400 as a communication partner before communicating with the edge device 400, it is assumed that such registration has also been performed in advance.

[0113] 14, the operations from S401 to S406 are the same as those in the first embodiment, and therefore the description thereof will be omitted. In S408 , when the edge device 400 receives the RAW data from the imaging apparatus 100 , the edge device 400 stores the RAW data in the RAM 403 or the recording unit 407 .

[0114] In S409, the edge device 400 uses the image processing unit 405 to apply image processing (development processing) to the RAW data similar to that applied by the server 200 executing the image processing program 211 in the first embodiment. In S1510, the edge device 400 displays the developed image data on the display unit 408.

[0115] Thereafter, the edge device 400 transmits the developed image data to the imaging apparatus 100 in S410. The operations of S411 to S413 in the imaging device 100 are similar to those in the first embodiment, and therefore the description thereof will be omitted.

[0116] In this embodiment, image processing of the original data of the proxy image data is performed by the edge device 400, not by the server 200. The volume of the original data of the proxy image data is large, and the volume is particularly large if it is RAW data. Therefore, by applying image processing by the edge device 400, not by the server 200, it is possible to significantly reduce the amount of image data transmitted to and received from the server 200 over the network 300. This makes it possible to significantly reduce communication costs and server costs using the network 300.

[0117] (Modification) In the above-described embodiment, the image data to which the image processing is applied in the external device is determined by the imaging device 100. In this case, as described in the embodiment, it is necessary to implement in the imaging device a function (e.g., a program, a determination criterion, etc.) for the imaging device to determine the image data to which the image processing is applied in the external device based on the processing result of the proxy image data.

[0118] As described above, the hardware resources of an imaging device are significantly limited in both capacity and capability compared to edge devices and servers, and it is not easy to update the implemented functions. Therefore, even if it is desirable to update the function for the imaging device to determine the image data to which the image processing is applied in the external device when the image processing program of the external device is updated, it may not be possible to do so. In this case, the imaging device cannot appropriately determine the image data to be transmitted to the external device.

[0119] Therefore, it is possible to have the server 200 or the edge device 400 perform the process of determining the image data to which the image processing is applied by the external device, which is performed by the imaging device 100 in the above-mentioned embodiment in S405 to S406. In this case, the server 200 or the edge device 400 requests the imaging device 100 to transmit the determined image data, and the imaging device transmits the requested image data to the server 200 or the edge device 400. For example, in the first embodiment, after the server 200 performs the object recognition process in S403, it requests the imaging device 100 in S404 to transmit the original data (which may be developed image data or RAW data) of the proxy image data in which the registered person is detected. Then, the imaging device 100 transmits the original data to the server 200 in S407. Similarly, in other embodiments, the device that performed the image processing on the proxy image data can request the imaging device to transmit the original data of the proxy image data. Alternatively, the device that performed the image processing on the proxy image data (for example, an edge device) may request the imaging device to transmit the original data of the proxy image data to another device (for example, a server).

[0120] The edge device 400 has capabilities closer to the server 200 than the imaging device 100, and software updates are also easy. Therefore, even if the image processing program 211 of the server 200 is updated and the criteria for determining image data that is desired to be processed by the server 200 change, it can be easily dealt with. For example, the server 200 may transmit criteria suitable for the updated image processing program 211 to the edge device 400 to update the criteria used by the edge device 400. Alternatively, the criteria may be updated by updating an application for the image processing system of the edge device 400. As already explained, in each of the above-mentioned embodiments, the imaging device 100 has been used for explanation, but the present invention can be applied to an information processing device capable of handling image data.

[0121] ●(Sixth embodiment) This embodiment relates to an image processing system having an imaging device 100, an edge device 400 (first external device), and a server 200 (second external device) connected to a network 300, as shown in Fig. 5(a). However, since the functional configurations of the edge device 400 and the server 200 are partially different, hereinafter they will be referred to as the edge device 400" and the server 200". In this embodiment, as shown in FIG. 15, the communication unit 110 of the imaging device 100 has a communication interface for the edge device 400" in addition to a communication interface for the network 300 (server 200"). In this embodiment as well, no cost is incurred for communication between the imaging device 100 and the edge device 400".

[0122] 15 is a block diagram showing an example of the functional configuration of the imaging device 100. Except for the configuration of the communication unit 110, the configuration may be the same as that described in the first embodiment, and therefore a description of the functional blocks other than the communication unit 110 will be omitted.

[0123] In this embodiment, the communication unit 110 realizes communication between the imaging device 100 and the server 200" via the network 300, as well as wireless communication between the imaging device 100 and the edge device 400". The control unit 101 transmits and receives image data, etc. to and from the server 200" and the edge device 400" via the communication unit 110.

[0124] FIG. 16(a) is a block diagram showing an example of the functional configuration of a server 200". The server 200" may have a similar configuration to the server 200 of the first embodiment shown in FIG. 1(c), except that the image recognition program does not need to be stored in the ROM 203'. Therefore, a description of matters common to the server 200 will be omitted. In the following, a case will be described in which the image processing function provided by the server 200" is development processing of RAW data, but other image processing functions may also be used. In this embodiment, the communication unit 204 realizes communication between the server 200″ and the imaging device 100 via the network 300, and communication between the server 200″ and the edge device 400″ via the network 300. The control unit 201 transmits and receives image data, etc. to and from the imaging device 100 and the edge device 400″ via the communication unit 204.

[0125] The network 300 may be any known network capable of supporting data communication between the imaging device 100 and the edge device 400'' and the server 200'', and therefore a description of the specific configuration will be omitted.

[0126] FIG. 16(b) is a block diagram showing an example of the functional configuration of an edge device 400″. The edge device 400″ may have the same configuration as the edge device 400 of the second embodiment shown in FIG. 5(b) except that it has an evaluation unit 411 instead of the image recognition unit 404. Therefore, a description of matters common to the edge device 400 will be omitted.

[0127] In some embodiments, the evaluation unit 411 calculates an evaluation value from image data. The evaluation value is used to determine the transmission order of RAW data for which image processing is performed in an external device of the imaging device 100. The evaluation unit 411 may be the one described in the third embodiment using Figs. 9 to 11. Note that in an embodiment in which the evaluation value is not used, the evaluation unit 411 is not necessary.

[0128] Next, the mutual operations between the imaging device 100, the edge device 400", and the server 200" will be described with reference to the flowchart in FIG. 17. Hereinafter, operations described as being mainly performed by the imaging device 100 in this specification are realized by the control unit 101 (CPU) controlling the other components of the imaging device 100. Furthermore, operations described as being mainly performed by the server 200" are realized by the control unit (CPU) 201 controlling the other components of the server 200". Furthermore, operations described as being mainly performed by the edge device 400" are realized by the control unit 401 (CPU) controlling the other components of the edge device 400". Furthermore, in the following description, unless otherwise specified, when specific information or data is communicated between devices, other information or data not explicitly stated may also be communicated.

[0129] It is assumed that settings and information required for the imaging device 100 to communicate with the server 200'' and the edge device 400'' are stored in advance in the ROM 102. Also, if it is necessary for the imaging device 100 to register the edge device 400'' as a communication partner before communicating with the edge device 400'', such registration is also assumed to have been performed in advance.

[0130] In S1701, the imaging device 100 transmits proxy image data recorded in the recording unit 108 to the edge device 400'' from the communication unit 110. This proxy image data is an example of information related to the image data recorded in the recording unit 108. There is no particular limit to the timing of executing S1701, and it may be started, for example, when it is confirmed that unprocessed RAW data exists in the recording unit 108, or it may be started in response to a user instruction. In addition, the proxy image data to be transmitted may correspond to RAW data, may correspond to developed image data, or may be selected by the user. Alternatively, all proxy image data may be transmitted.

[0131] Also, among the RAW data, proxy image data corresponding to RAW data that has not been developed (unprocessed) may be transmitted. In this case, for example, RAW data for which image data in a corresponding general format (e.g., JPEG format) is not recorded can be determined as unprocessed RAW data. Alternatively, a file name that can determine whether the data is unprocessed or not may be given, and unprocessed RAW data may be determined from the file name. In S1701, two or more pieces (frames) of proxy image data are transmitted to the edge device 400”.

[0132] In S1702, the edge device 400'' receives the proxy image data from the imaging apparatus 100 via the communication unit 204. The edge device 400'' stores the received proxy image data in the RAM 403.

[0133] In S1703, the edge device 400" displays a list of images based on the proxy image data stored in the RAM 403 on the display unit 408 in a selectable manner. For example, the edge device 400" displays an image selection screen on the display unit 408 that displays a list of images based on the proxy image data (proxy images) and notifies the user to select an image on which development processing is to be performed. There are no particular limitations on the number of proxy images that the edge device 400" displays on the display unit 408 at one time, the display size of the proxy images, or the display order. If the display size is made smaller, many proxy images are displayed simultaneously, improving the viewability, and if the display size is made larger, it becomes easier to check the details of the images. It is not necessary to display all of the proxy images based on the proxy image data stored in the RAM 403 on one screen. The list of proxy images may be displayed across multiple screens using a known method such as scrolling or page switching.

[0134] A user of the edge device 400″ (which may or may not be the same as the user of the imaging device 100) selects one or more proxy images by operating the operation unit 111 of the edge device 400″. For example, if the display unit 408 is a touch display, the user can select an image by touching it. The user may also select an image by operating the keys or buttons of the operation unit 111. Image selection here corresponds to selection of RAW data to which development processing is applied by the server 200″.

[0135] In S1704, the edge device transmits to the imaging device 100 selected file information (e.g., a file name) that identifies proxy image data corresponding to the proxy image selected by the user. Note that the selected file information transmitted from the edge device 400'' to the imaging device 100 in S1704 may change depending on the information received from the imaging device 100 together with the proxy image data. For example, when information that identifies RAW data corresponding to the proxy image data has been acquired, the edge device 400'' may transmit information that identifies the RAW data corresponding to the selected proxy image (e.g., a file name) to the imaging device 100 as selected file information. In addition, any information that allows the imaging device 100 to identify the RAW data that the user has selected to be developed by the server 200'' may be transmitted to the imaging device 100 as selected file information from the edge device 400''.

[0136] In S1705, the imaging device 100 stores the information (selected file information) received from the edge device 400″ in the RAM 103. In S1706, the image capturing apparatus 100 identifies RAW data based on the selected file information. Then, the image capturing apparatus 100 reads the identified RAW data from the recording unit 108, and transmits it to the server 200'' via the network 300.

[0137] In S1707, the server 200'' receives the RAW data and stores it in the RAM 202 or the storage unit 205. In S1708, the server 200" loads the image processing program 211 from the ROM 203' into the RAM 202 and executes it. The image processing program 211 applies development processing to the RAW data. The development processing includes, for example, white balance processing, lens aberration correction processing, noise reduction (NR) processing, color interpolation processing, gamma processing, and the like. In addition, after the development processing, an encoding processing may be applied to generate an image data file in a predetermined format, for example, JPEG format. The server 200" may receive the RAW data in S1707 and execute the image processing program 211 in S1708 (apply development processing) in parallel.

[0138] The development processing applied by the server 200" is based on a newer and / or more complex technique than that applied by the image processing unit 107 of the imaging device 100, and produces better processing results than the development processing applied by the imaging device 100. This is because the processing capability of the server 200" is higher than that of the imaging device 100, and because the image processing program 211 executed by the server 200" is based on newer technology than that of the image processing unit 107 of the imaging device 100.

[0139] When reception of the RAW data is completed in S1707, the server 200'' transmits a RAW data reception completion notification to the image capturing apparatus 100 via the network 300 in S1709.

[0140] In S1710, the imaging device 100 receives a reception completion notification from the server 200''. In S1711, the imaging device 100 starts transmitting to the server 200″ the RAW data that has not been transmitted in S1707, among the RAW data stored in the recording unit 108. Note that the RAW data transmitted to the server 200″ in S1711 does not necessarily have to be all of the RAW data that has not been transmitted in S1707 (i.e., not selected by the user). For example, the RAW data stored in the recording unit 108 may include RAW data for which proxy image data has not been transmitted to the server 200″ in S1701. Therefore, the RAW data to be transmitted in S1711 may be determined from the RAW data for which proxy image data has been transmitted in S1701, or the RAW data captured on the same day or at a similar time as the RAW data for which proxy image data has been transmitted in S1701.

[0141] In S1712, the server 200'' starts receiving the RAW data, and stores it in the RAM 202 or the storage unit 205. In S1713, the server 200" performs development processing on the RAW data received in S1712 in the same manner as in S1708. Note that the server 200" performs S1713 after completing execution of S1708. The server 200" stores the developed image data obtained in S1708 and S1713 in the memory unit 205.

[0142] In FIG. 17, the developed image data obtained by applying the development process in the server 200'' is stored in the server 200''. This is on the assumption that the communication between the imaging device 100 and the server 200'' is charged according to the amount of data. In such a case, communication costs can be reduced by obtaining the developed image data from the server 200'' after moving to an environment where communication charges are fixed, such as home. However, since it is also possible that the user wants to check the development results immediately, the server 200'' may transmit the image data to the imaging device 100 in order starting from the image data for which the development process has been completed. Since the RAW data to which the development process is applied in S1708 is selected by the user, the user can check the development results of the desired RAW data before the development results of other RAW data.

[0143] In this embodiment, the edge device is used to inquire of the user about the RAW data to which the user would like to preferentially apply development processing among the RAW data present in the imaging device. Then, the imaging device transmits the RAW data selected by the user to an external device that executes development processing before other RAW data. Therefore, even if a large amount of RAW data exists in the imaging device, it is possible to preferentially execute development processing on the RAW data desired by the user. Therefore, it is possible to reduce the waiting time of the user when development processing is executed in an external device of the imaging device. In addition, since development processing is executed in an external device, development processing using processing that cannot be executed in the imaging device and development processing faster than that of the imaging device are possible.

[0144] In the present embodiment, the imaging device 100 is configured to transmit the proxy image directly to the edge device 400'', but the proxy image may be transmitted via the server 200''. For example, as shown in FIG. 18, in S1701, the imaging device 100 transmits the proxy image data to the server 200'', and in S1721, the server 200'' receives the proxy image data. Then, in S1722, the server 200'' transmits the proxy image data to the edge device 400''. From here on, S1702 onwards is the same as in FIG. 17. Note that the edge device 400'' may transmit the selected file information to the imaging device 100 via the server 200'' instead of directly to the imaging device 100. In this way, by configuring the imaging device 100 and the edge device 400'' to communicate with each other via the server 200'', the present invention can be implemented even when the imaging device 100 and the edge device 400'' cannot communicate with each other directly.

[0145] Furthermore, in this embodiment, the development processing is executed by the server 200'', but the development processing may be executed by the edge device 400''. In this case, the processing executed by the server 200'' in FIG. 17 may be executed by the edge device 400''.

[0146] ●(Seventh embodiment) Next, a seventh embodiment of the present invention will be described. In this embodiment, the development processing results for a portion of RAW data are presented to the user, and for the RAW data for which the necessity for development processing has been confirmed, development processing is performed on the entirety of the data. This can reduce the time required to obtain the development results for a partial region and the amount of communication data.

[0147] This embodiment can be implemented by an image processing system similar to that of the sixth embodiment, and therefore a description of the configurations of the imaging device 100, the server 200'', and the edge device 400'' will be omitted.

[0148] The interactions between the imaging device 100, edge device 400", and server 200" in this embodiment will be described with reference to the flowchart in FIG. 19. Steps S1901 to S1903 are the same as steps S1701 to S1703 in the sixth embodiment, and therefore will not be described. Here, it is assumed that when one image is selected in step S1903, the edge device 400" executes step S1904.

[0149] In S1904, the edge device 400" selects a partial region for the proxy image selected by the user in S1903. This selection operation will be further explained with reference to FIG. 20. FIG. 20 shows a proxy image 2000 selected by the user in S1903. The control unit 401 controls the image processing unit 405 to execute detection of a characteristic region for the proxy image data corresponding to the proxy image selected by the user. In this case, the position and size of a region that appears to be a human face (face region) is detected as the characteristic region. FIG. 20 shows an example in which face regions 2001 and 2002 have been detected by the detection process.

[0150] The edge device 400" can present to the user an index (e.g., a rectangular index) indicating the detected characteristic area for the proxy image selected in S1903 in a superimposed manner. For a proxy image in which multiple characteristic areas have been detected, the edge device 400" selects one characteristic area by having the user select it or by automatically selecting it according to predetermined conditions. Note that an example has been shown here in which a partial area to be subjected to development processing is selected from the characteristic areas in advance. However, it is also possible to divide the image into multiple areas using one or more of color, brightness, and distance information, and have the user select one area. It is also possible to have the user select any partial area within the image.

[0151] 20, the user can select one of two face areas 2001 and 2002, for example, by a touch operation on the display unit 408. Here, it is assumed that the face area 2001 is selected.

[0152] Returning to FIG. 19, in S1905, the edge device 400'' transmits, in addition to the selected file information, information identifying the partial area selected for the file (partial area information) to the imaging device 100. There are no particular limitations on the partial area information, but examples include the diagonal vertex coordinates of the partial area in the proxy image. In this case, by using coordinate values ​​normalized by the horizontal and vertical sizes of the proxy image, the position of the partial area in the corresponding RAW data can be easily identified.

[0153] In S1906, the imaging apparatus 100 receives the selected file information and the partial region information, and stores them in the RAM 103. In S1907, the imaging apparatus 100 transmits, for the RAW data identified from the selected file information, RAW data of a partial region based on the partial region information (partial RAW data) to the server 200'' via the network 300.

[0154] In S1908, the server 200'' receives the partial RAW data. In S1909, the server 200'' applies development processing to the partial RAW data received in S1908. The development processing to be applied may be the same as that in the sixth embodiment. In S1910, the server 200'' transmits the development result of the partial RAW data together with information for identifying the RAW data to the edge device 400'' via the network 300.

[0155] In S1911, the edge device 400" receives the development results and stores them in the RAM 403. The edge device 400" displays an image of the development results on the display unit 408. The edge device 400" may display the development results together with the entire corresponding proxy image. The user can check the development results of a partial area of ​​the image that he or she selected in S1903 via the display unit 408.

[0156] In this embodiment, the edge device 400" displays the developed image so that the user can select one of the following: an instruction to execute development processing for the entire image, an instruction to reselect a partial region, or an instruction to cancel the development processing. For example, the edge device 400" can display selectable icons corresponding to each instruction adjacent to the developed image.

[0157] In S1912, the edge device 400" branches the process according to the received user instruction. If the edge device 400" detects an instruction to reselect a partial region, it executes S1904, and if it detects an instruction to execute development processing for the entire image, it executes S1913. Although not shown in the figure, if a cancel instruction is detected, the edge device 400" ends the display of the corresponding image.

[0158] When S1904 is executed in response to an instruction to reselect a partial region, the edge device 400'' can display indicators of feature regions excluding those already selected. When the user selects a new partial region, the processing from S1905 onwards is executed again for the target image.

[0159] On the other hand, in S1913, the edge device 400'' transmits to the imaging apparatus 100 a notification requesting development processing of the entire image (main development request) together with information for identifying the RAW data. In S1914, the imaging apparatus 100 receives the main development request and information for identifying the target RAW data, and stores them in the RAM 103. Thereafter, operations from S1915 onwards for transmitting the entire RAW data are equivalent to the operations from S1706 to S1713 in Fig. 17 described in the sixth embodiment, and therefore detailed description thereof will be omitted.

[0160] For ease of explanation and understanding, the flow of operations for one of the proxy images displayed on the edge device 400" has been described here. When multiple proxy images are displayed on the display unit 408, the same operations are executed for the remaining proxy images selected in S1903. Note that the processes from S1903 to S1911 can also be configured to be executed collectively for multiple proxy images. In other words, when multiple proxy images are selected in S1903, the edge device 400" transmits selected file information and partial area information for the multiple images to the imaging device 100 collectively. Thereafter, transmission of partial RAW data, development processing, and transmission of development results are also executed collectively for the multiple images, and finally the development results for the multiple images are displayed on the display unit 408. Thereafter, a request for development processing for the entire image and reselection of the partial areas can also be executed collectively.

[0161] According to this embodiment, in addition to the effects of the sixth embodiment, the user can check the development results of a partial area of ​​a selected image and then decide whether or not to execute development processing on the entire image. By first executing development processing on the partial area, the amount of communication data is small and the load of the development processing is also low, so the user can quickly check the development results of the partial area. Furthermore, if there is RAW data that is determined not to require development processing on the entire image based on the development results of the partial area, the amount of communication data using the network 300 can be reduced. Note that it is also possible to apply irreversible or lossy data reduction processing to the partial RAW data, in which case the amount of communication data can be further reduced.

[0162] Furthermore, the development process for the partial RAW data or the RAW data does not necessarily have to be executed by the server 200'', and may be configured to execute at least a part of it in the edge device 400''.

[0163] ●(Eighth embodiment) Next, an eighth embodiment of the present invention will be described. This embodiment corresponds to a modified example of the seventh embodiment. Specifically, the amount of communication data is further reduced by preventing the partial RAW data and the development results thereof in the seventh embodiment from being repeatedly transmitted between devices.

[0164] As in the seventh embodiment, this embodiment can be implemented by an image processing system similar to that of the sixth embodiment, and therefore, a description of the configurations of the imaging device 100, the server 200'', and the edge device 400'' will be omitted.

[0165] The interaction between the imaging device 100, edge device 400", and server 200" in this embodiment will be described with reference to the flowchart in FIG. 21. In FIG. 21, the same reference numbers are used for the same processes as in the seventh embodiment. Steps S1901 to S1914 are the same as in the seventh embodiment, and therefore description thereof will be omitted.

[0166] Upon receiving the actual development request in S1914, in S2115, the imaging device 100 transmits the RAW data to be developed to the server 200'', excluding the partial RAW data transmitted to the server 200'' in S1907. For example, if the partial RAW data corresponding to the facial area 2001 has been transmitted to the server 200'' in S1907, the imaging device 100 transmits RAW data corresponding to the area excluding the facial area 2001 (the remaining area) to the server. The imaging device 100 can generate RAW data excluding the partial RAW data that has already been transmitted, by referring to the selected file information and partial area information stored in RAM 103.

[0167] In S2116, the server 200'' receives the RAW data excluding the partial RAW data received in S1908. In S2117, the server combines the partial RAW data received in S1908 and the RAW data received in S2116 to generate RAW data corresponding to the entire image. The subsequent processing from S1917 onwards is similar to that in the seventh embodiment, and therefore a description thereof will be omitted.

[0168] In this embodiment, when performing main development (development processing on the entire image) on RAW data for which development processing has already been performed on a partial area, partial RAW data that has already been transmitted from the imaging device 100 to the server 200'' is not transmitted again. Therefore, in addition to the effect of the seventh embodiment, the amount of communication data from the imaging device 100 to the server 200'' can be reduced.

[0169] In this embodiment as well, the development process for the partial RAW data or the RAW data does not necessarily have to be executed by the server 200'', and may be configured so that at least a part of it is executed by the edge device 400''.

[0170] ●(Ninth embodiment) Next, a ninth embodiment of the present invention will be described. In this embodiment, the order in which RAW data is transmitted to an external device is determined according to a predetermined condition. Since this embodiment can also be implemented by an image processing system similar to that of the sixth embodiment, a description of the configurations of the imaging device 100, the server 200", and the edge device 400" will be omitted.

[0171] The interoperation between the imaging device 100, edge device 400'', and server 200'' in this embodiment will be described with reference to the flowchart of FIG. 22. In FIG. 22, S2201 and S2202 are similar to S1701 and S1702 in the sixth embodiment, and therefore description thereof will be omitted.

[0172] In S2203, the edge device 400" calculates a predetermined evaluation value for the proxy image data stored in the RAM 403. This evaluation value is an index of the magnitude of the effect of developing the RAW image data corresponding to the proxy image data in an external device (the server 200" or the edge device 400"). Therefore, the evaluation value of the proxy image data can also be regarded as the evaluation value of the corresponding RAW data.

[0173] The operation of the evaluation unit 411 that calculates the evaluation value in the edge device 400'' may be the same as that described in the third embodiment with reference to FIGS. The evaluation unit 411 calculates an evaluation value for each of the stored proxy image data. The calculated evaluation values ​​are stored in the RAM 403.

[0174] Returning to FIG. 22, in S2204, the edge device 400" sorts the evaluation values ​​calculated in S2203 in descending order of the evaluation value, and generates rating information associated with the information of the corresponding proxy image data. The edge device 400" then transmits the rating information to the imaging device 100. The rating information is information that indicates the priority order of the proxy image data (and the image data that is the basis of the proxy image data).

[0175] In S2205, the imaging device 100 receives the rating information and stores it in the RAM 103. In S2206, the imaging device 100 determines the transmission order of the RAW data to be transmitted to the server 200" in accordance with the rating information so that the RAW data with the highest rating value is transmitted first. When determining the transmission order, RAW data with a rating value less than a preset threshold value may be excluded from the transmission target. Also, the threshold value may be configured to be adjustable according to a user instruction obtained via a user interface. In S2207, the image capturing apparatus 100 sequentially transmits the RAW data to the server 200'' in accordance with the transmission order determined in S2206.

[0176] In S2208, the server 200'' receives the RAW data and stores it in the RAM 202 or the storage unit 205. In S2209, the server executes the image processing program 211 to apply development processing to the RAW data, similarly to S1708 in the sixth embodiment. Note that the server 200'' may execute the reception of the RAW data in S2208 and the execution of the image processing program 211 (application of development processing) in S2209 in parallel.

[0177] In S2210, the server 200'' transmits the development results (developed image data) to at least one of the imaging device 100 and the edge device 400'' via the network 300, in order from the RAW data for which development processing has been completed.

[0178] When the edge device 400'' receives developed image data from the server 200'', in S2211 the edge device 400'' stores the received image data in the RAM 403. Then, in S2212, the edge device 400'' displays the image data stored in the RAM 403 on the display unit 408 and records the image data in the recording unit 407.

[0179] When the imaging device 100 receives developed image data from the server 200", in S2213 the imaging device 100 receives the developed image data from the server 200" and stores it in RAM 103. Then, in S2214 the imaging device 100 supplies the image data stored in RAM 103 to the image processing unit 107, which generates image data for recording and image data for display. Then the imaging device 100 displays the image data for display on the display unit 109, and records the image data for recording in the recording unit 108.

[0180] In this embodiment, the edge device 400" evaluates the effect of applying image processing to the RAW data held by the imaging device 100 in an external device (e.g., the server 200"), and transmits the RAW data to the server 200" in order of the effect. Therefore, the imaging device 100 can receive the processing results in order of the RAW data that will be most effective when image processing is applied by the external device.

[0181] This embodiment can be combined with the sixth to eighth embodiments. Specifically, the calculation of the evaluation value, the determination of the transmission order, and the implementation of the development process can be performed only for the image data selected by the user. In this case, in addition to the effects of the sixth to eighth embodiments, the processing results can be received for the RAW data selected by the user in the order of the RAW data that will be most effective when image processing is applied in an external device.

[0182] In this embodiment, the evaluation unit 411 calculates the evaluation value based on the edge detection result, but the evaluation value may be calculated based on other criteria according to the characteristics of the development process of the server 200". In addition, the evaluation value may be an index of the quality of the image, not limited to an index of the magnitude of the effect of the development process performed by the server 200". For example, an evaluation value based on the aesthetic value of the composition, an evaluation value based on the degree of focus, an evaluation value based on the appropriateness of the exposure, an evaluation value based on the facial expression of the subject, such as the degree of smile, etc. may be calculated. In addition, a plurality of types of evaluation values ​​may be calculated, and the transmission order may be determined based on the average value or the maximum value thereof. The calculation of the evaluation value is not limited to being performed by the edge device 400", and may be performed by the imaging device 100 or the server 200".

[0183] ●(Tenth embodiment) Next, a tenth embodiment of the present invention will be described. This embodiment differs from the ninth embodiment in the method of calculating an index of the magnitude of the effect of applying image processing in an external device. Since this embodiment can also be implemented by an image processing system similar to that of the sixth embodiment, a description of the configurations of the imaging device 100, the server 200", and the edge device 400" will be omitted.

[0184] In this embodiment, the effect of applying image processing in an external device is evaluated in more detail by evaluating the image in consideration of multiple evaluation values ​​based on the performance of the image processing provided by the external device. It is possible to perform an evaluation that reflects performance improvements due to updates to the image processing program 211 provided by the external device.

[0185] The mutual operations among the imaging device 100, edge device 400", and server 200" in this embodiment will be described with reference to the flowchart in FIG. 23. In FIG. 23, steps that perform the same operations as in the ninth embodiment are given the same reference numerals as in FIG. 22. Descriptions of operations that are the same as in the ninth embodiment will be omitted, and the following description will focus on operations that are characteristic of this embodiment.

[0186] In S2301, the server 200" generates a development prediction flag and transmits it to the edge device 400". The development prediction flag is information that serves as an index of the performance of various image processing functions provided by the image processing program 211 of the server 200". For example, the development process, which is an example of an image processing function provided by the image processing program 211, is composed of multiple processes such as white balance processing, lens aberration correction processing, noise reduction (NR) processing, color interpolation processing, and gamma processing. The development prediction flag may be information that has a "1" for a process that is capable of high performance processing and a "0" for other processes. It is not necessary to generate the development prediction flag each time S2301 is executed, and it can be stored in the ROM 203' in advance. Then, when the image processing program 211 is updated, only the necessary parts are updated.

[0187] There is no particular limit to the method for evaluating the level of performance, but it is considered that applications and modules that realize processing have been updated within the last year, processing that has a high load, etc. are likely to have high performance. Alternatively, the vendor of the image processing program 211 may also provide updates to the development prediction flag when updating the program.

[0188] In S2302, the edge device 400'' receives the development prediction flag from the server 200''. In S2303, the edge device 400'' calculates an evaluation value for each of the proxy image data recorded in the RAM 403.

[0189] The operation of the evaluation unit 411 in S2303 will be described with reference to the flowchart in Fig. 24. In this embodiment, the evaluation unit 411 calculates an evaluation value for each piece of proxy image data by weighting a plurality of types of evaluation values ​​according to the development prediction flag.

[0190] In S2401, the evaluation unit 411 calculates a first evaluation value based on edge detection for the proxy image data in the same manner as in the ninth embodiment. In S2402, the evaluation unit 411 applies a known method to the proxy image data to calculate the person's smile degree as a second evaluation value. The person's smile degree is a numerical value indicating the degree to which a face area detected from the proxy image data is smiling. Applications that calculate smile degrees from images are known, and the second evaluation value can be obtained, for example, by implementing such an application in the image processing unit 405. For a proxy image that has multiple face areas, the second evaluation value is, for example, the average or maximum value of the smile degrees. Both the first and second evaluation values ​​are normalized to the range of 0 to 1.

[0191] In S2403, the evaluation unit 411 determines weights for the first and second evaluation values ​​based on the development prediction flag. A method of calculating weights according to the development prediction flag for each type of evaluation value is registered in advance in the evaluation unit 411. Then, the evaluation unit 411 adjusts the weight ratio so that the sum of the weight for the first evaluation value and the weight for the second evaluation value becomes 1, and then performs a weighted addition of the first and second evaluation values ​​to calculate a final evaluation value.

[0192] By calculating an evaluation value for each piece of proxy image data in this manner, an evaluation value according to the characteristics of the image processing program 211 in the server 200" can be calculated, and the extent of the effect of the development processing in the server 200" can be evaluated in more detail. Furthermore, if the image processing program 211 is updated and the prediction flag changes, an evaluation value that reflects the change can be obtained. Therefore, an appropriate evaluation is always possible.

[0193] For example, suppose that the method of noise reduction (NR) processing in the image processing program 211 has been changed, making it possible to perform noise reduction processing with higher performance than before, and the flag for noise reduction processing in the development prediction flag has also changed from "0" to "1." In this case, since the improvement in performance of the noise reduction processing has a large effect on improving the results of development processing of images with many edges, the weight for the first evaluation value becomes larger than before the image processing program 211 was updated.

[0194] For example, if the first evaluation value is Sedge and the second evaluation value is Sface, the weighted sum evaluation value S is: S= (S_edge×α)+(S_face×(1-α)) (Formula 3) It can be calculated as follows.

[0195] Then, in response to the change in the development prediction flag, the evaluation unit 411 updates the weight α for the first evaluation value to the weight α before the update of the image processing program 211. before Rather, the updated weight α after Increase (α after >α before ) to obtain a weighted and added evaluation value that reflects the update of the image processing program 211.

[0196] Returning to the flowchart of FIG. 23, in S2204, the edge device 400″ generates rating information in the same manner as in the ninth embodiment, except that the evaluation value calculated in S2303 is used, and transmits the rating information to the imaging device 100. The subsequent processing is similar to that in the ninth embodiment, and therefore will not be described.

[0197] According to this embodiment, an evaluation value obtained by weighting a plurality of evaluation values ​​is used, and weights are determined so as to increase the weight of the evaluation value that better reflects the effect of applying image processing in an external device. Therefore, in addition to the same effects as in the ninth embodiment, it is possible to apply image processing in an external device in an order that reflects the latest image processing provided by the external device.

[0198] In this embodiment, a case has been described in which the ratio (weight) of the weighted addition of the evaluation value based on edge detection and the evaluation value based on the smile level is determined based on the development prediction flag. However, the types of evaluation values ​​are not limited to these. As in the ninth embodiment, an evaluation value based on the aesthetic quality of the composition, an evaluation value based on the degree of focus, an evaluation value based on the appropriateness of exposure, an evaluation value based on the facial expression of the subject such as the smile level, etc. may be calculated. Furthermore, three or more types of evaluation values ​​may be weighted and added. Note that the calculation of the evaluation value is not limited to being performed by the edge device 400'', and may be performed by the imaging device 100 or the server 200''.

[0199] ●(Eleventh embodiment) Next, an eleventh embodiment of the present invention will be described. This embodiment differs from the ninth embodiment in that when transmitting RAW data to an external device according to a transmission order determined based on an evaluation value, the upper limit number of RAW data to be transmitted can be limited. Since this embodiment can also be implemented by an image processing system similar to that of the sixth embodiment, a description of the configurations of the imaging device 100, the server 200", and the edge device 400" will be omitted.

[0200] The mutual operations among the imaging device 100, edge device 400", and server 200" in this embodiment will be described with reference to the flowchart in FIG. 25. In FIG. 25, steps that perform the same operations as in the ninth embodiment are given the same reference numerals as in FIG. 22. Descriptions of operations that are the same as in the ninth embodiment will be omitted, and the following description will focus on operations that are characteristic of this embodiment.

[0201] In the ROM 102 of the imaging device 100 of this embodiment, an upper limit number of RAW data to be transmitted to an external device is set. This setting may be changeable by the user. Also, a separate upper limit number may be set according to the communication destination or communication method. For example, an upper limit number is set for communication using a mobile phone communication network (communication in which a charge is incurred according to the amount of communication data), and an upper limit number is not set (or set to no limit) for communication using a wireless LAN (communication in which a charge is not incurred). Also, an upper limit number is set for communication with the server 200'', and an upper limit number is not set for communication with the edge device 400. The upper limit number may be the number of RAW data files (number of data) or the amount of transmitted data. The number of transmitted files and the amount of communication data to be compared with the upper limit number are reset every predetermined period (for example, one month, one week, one day, etc.).

[0202] The operations from receiving the rating information to determining the transmission order (up to S2206) may be similar to those in the ninth embodiment (or the tenth embodiment). In S2501, the imaging device 100 refers to the RAM 103, for example, and determines whether the current number of transmissions is less than the upper limit number. If it is not less than the upper limit number, the imaging device 100 does not transmit any more RAW data. On the other hand, if it is less than the upper limit number, in S2207, the imaging device 100 transmits the next RAW data to the server 200'' in accordance with the transmission order. After transmission, the imaging device 100 reflects information on the most recently transmitted RAW data in the number of transmitted files and the communication data amount stored in the RAM 103. Then, if there is any untransmitted RAW data remaining, the imaging device 100 executes S2501 again. Note that in the case where the upper limit number is the data amount, even if the upper limit number is less than the upper limit number, if the upper limit number is exceeded during transmission of the next RAW data, the next RAW data will not be transmitted.

[0203] The operations from S2207 onwards may be similar to those in the ninth embodiment (or the tenth embodiment), and therefore the description will be omitted. This embodiment also provides the same effect as the ninth embodiment (or the tenth embodiment). In addition, by setting the upper limit number to a value that takes into consideration the monthly upper limit communication fee in a mobile communication contract, it is possible to prevent the upper limit communication fee from being exceeded by transmitting RAW data in an environment using mobile communication.

[0204] ●(Twelfth embodiment) Next, a twelfth embodiment of the present invention will be described. This embodiment differs from the ninth embodiment in that when transmitting RAW data to an external device in accordance with a transmission order determined based on an evaluation value, the remaining amount of data that can be transmitted to a server is taken into consideration. Since this embodiment can also be implemented by an image processing system similar to that of the sixth embodiment, a description of the configurations of the imaging device 100, server 200", and edge device 400" will be omitted.

[0205] The mutual operations among the imaging device 100, edge device 400", and server 200" in this embodiment will be described with reference to the flowchart in FIG. 26. In FIG. 26, steps that perform the same operations as in the ninth embodiment are given the same reference numerals as in FIG. 22. Descriptions of operations that are the same as in the ninth embodiment will be omitted, and the following description will focus on operations that are characteristic of this embodiment.

[0206] In this embodiment, the server 200" is assumed to have a predetermined storage capacity of the storage unit 205 available to each user, and the usage amount is managed for each user. The usage amount is managed, for example, in association with information specific to the user or the imaging device 100. The server 200" can manage the usage amount based on information specific to the imaging device that received the RAW data or the user of that device.

[0207] The operations from receiving the rating information to determining the transmission order (up to S2206) may be similar to those in the ninth embodiment (or the tenth embodiment). In S2601, the server 200" calculates the difference between the upper limit of the available storage capacity set for the imaging device 100 or its user and the current amount of usage as the remaining data capacity, and transmits it to the imaging device 100. Note that S2601 can be executed at any timing from when the server 200" and the imaging device 100 are able to communicate with each other until the imaging device 100 starts transmitting RAW data.

[0208] In S2602, the imaging device 100 receives the remaining data capacity from the server 200″ and stores it in the RAM 103. In S2603, the imaging device 100 determines whether the received remaining data capacity is equal to or larger than the amount of RAW data to be transmitted next. If the remaining data capacity is not equal to or larger than the amount of RAW data to be transmitted next, the imaging device 100 does not transmit subsequent RAW data. On the other hand, if the remaining data capacity is equal to or larger than the amount of RAW data to be transmitted next, the imaging device 100 executes S2207 and transmits the RAW data to the server. After transmission, the imaging device 100 updates the remaining data capacity by reducing the amount of RAW data most recently transmitted from the remaining data capacity stored in the RAM 103. Then, if there is RAW data remaining that has not been transmitted, the imaging device 100 executes S2603 again.

[0209] The operations from S2207 onwards may be similar to those in the ninth embodiment (or the tenth embodiment), and therefore the description will be omitted. This embodiment also provides the same effects as the ninth embodiment (or the tenth embodiment). Furthermore, when there is an upper limit to the amount of data that can be stored in the server 200'' (or the amount of data to which image processing can be applied), image processing can be applied preferentially to RAW data for which image processing by the server 200'' is most effective. Note that instead of the remaining data capacity, a value obtained by converting the remaining data capacity into the number of RAW data files using a representative data size of RAW data may be used.

[0210] ●(Thirteenth embodiment) Next, a thirteenth embodiment of the present invention will be described. This embodiment differs from the ninth embodiment in that rating information generated by the edge device 400'' is transmitted to the server 200'', and the rating information for the RAW data stored in the server 200'' is taken into consideration to generate rating information again. Since this embodiment can also be implemented by an image processing system similar to that of the sixth embodiment, a description of the configurations of the imaging device 100, the server 200'', and the edge device 400'' will be omitted.

[0211] The mutual operations among the imaging device 100, edge device 400", and server 200" in this embodiment will be described with reference to the flowchart in FIG. 27. In FIG. 27, steps that perform the same operations as in the ninth embodiment are given the same reference numerals as in FIG. 22. Descriptions of operations that are the same as in the ninth embodiment will be omitted, and the following description will focus on operations that are characteristic of this embodiment.

[0212] The operation up to generation of rating information in the edge device 400'' (up to S2204) may be similar to that in the ninth embodiment (or tenth embodiment). However, in this embodiment, the edge device 400'' transmits the rating information to the server 200'' instead of the imaging device 100 in S2204.

[0213] In S2701, the server 200'' receives rating information from the edge device 400'' and stores it in the RAM 202. The server 200" holds, in the storage unit 205, rating information about the imaging device 100 or the RAW data associated with its user, which is stored in the storage unit 205 of the server 200". This rating information is continuously updated based on rating information previously received from the edge device 400", and includes a predetermined number of evaluation values ​​starting from the highest one and information about the corresponding RAW data.

[0214] In S2702, the server 200" updates the rating information stored in the memory unit 205 using the rating information received from the edge device 400" in S2701 (re-rating process). Specifically, the server 200" rearranges the multiple evaluation values ​​included in the two pieces of rating information from highest to lowest, and stores a predetermined number of evaluation values ​​in descending order and the corresponding RAW data information in the memory unit 205 as new rating information. Note that the maximum number of evaluation values ​​(RAW data) included in the rating information stored in the memory unit 205 can be set to the maximum number of RAW data that the user can store in the memory unit 205.

[0215] Furthermore, the server 200" extracts only information relating to RAW data that does not exist in the server 200" from the new rating information generated by the re-rating process, and generates rating information to be transmitted to the imaging device 100. Of the RAW data stored in the storage unit 205, the RAW data that is no longer included in the rating information updated by the re-rating process can be deleted in order to secure the capacity of the storage unit 205.

[0216] In S2703, the server 200'' transmits the rating information generated for the image capture device 100 to the image capture device 100. Thereafter, operations from S2205 onwards are the same as those in the ninth embodiment, and therefore the description thereof will be omitted.

[0217] According to this embodiment, when there is a limit to the number or amount of data that can be stored in the server 200'', data for which application of image processing in the server 200'' is highly effective (data with a high evaluation value) can be stored in the server 200''.

[0218] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0219] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0220] 100...imaging device, 200, 200', 200"...server device, 300...network, 400, 400', 400"...edge device, 101...control unit, 102...ROM, 103...RAM, 105...imaging unit, 107...image processing unit, 108...recording unit, 109...display unit, 110...communication unit

Claims

1. An information processing device having a communication means and a control means, The control means Transmitting reduced image data of the captured image to a first external device through the communication means; receiving a result of an evaluation process on the reduced image data from the first external device; determining, based on a result of the evaluation process, original data to which image processing is to be applied in a second external device, from among the original data of the reduced image data; transmitting the determined original data to the second external device via the communication means; receiving a result of the image processing on the original data from the second external device; 23. An information processing apparatus comprising:

2. 2. The information processing apparatus according to claim 1, wherein the first external device and the second external device are devices on a network.

3. 2. The information processing apparatus according to claim 1, wherein the information processing apparatus communicates with the first external device through a network that does not incur any charges for use or communication volume, and communicates with the second external device through a network that incurs charges for use or communication volume.

4. 4. The information processing apparatus according to claim 1, wherein the evaluation process is an object recognition process.

5. 5. The information processing apparatus according to claim 1, wherein the image processing includes image processing that cannot be performed by the information processing apparatus.

6. An image processing device having a communication means and a control means, The control means receiving reduced image data from the information processing device through the communication means; applying an evaluation process to the reduced image data to generate information that serves as a judgment criterion for determining whether or not the information processing device should apply a predetermined image processing to the original data of the reduced image data in an external device; transmitting a result of the evaluation process, including the information, to the information processing device via the communication means; receiving image data from the information processing device through the communication means; applying the predetermined image processing to the image data; transmitting a result of the application of the predetermined image processing to the information processing device via the communication means; 13. An image processing device comprising:

7. 7. The image processing apparatus according to claim 6, wherein communication with said information processing apparatus is performed through a network in which a fee is charged for use or communication volume.

8. The control means further comprises: determining image data to which the predetermined image processing is to be applied from among the image data of the plurality of frames received from the information processing device; applying the predetermined image processing to the determined image data; 7. The image processing device according to claim 6,

9. The control means further comprises: receiving image data for a plurality of frames from the information processing device through the communication means; determining image data to which the predetermined image processing is to be applied from among the image data for the plurality of frames; transmitting the determined image data to a device different from the information processing device through the communication means; 7. The image processing device according to claim 6,

10. An image processing device having a communication means and a control means, The control means receiving reduced image data from the information processing device through the communication means; applying an evaluation process to the reduced image data; requesting the information processing device to transmit original data of the reduced image data based on a result of the evaluation process; receiving the requested original data from the information processing device via the communication means; applying a predetermined image processing to the requested original data; transmitting a result of the application of the predetermined image processing to the information processing device via the communication means; 13. An image processing device comprising:

11. 11. An image processing system in which the information processing device according to claim 1 and the image processing device according to claim 6 are communicably connected to each other.

12. A method for controlling an information processing device, comprising: The control means Transmitting reduced image data of a captured image to a first external device through a communication means of the information processing device; receiving a result of an evaluation process on the reduced image data from the first external device; determining, based on a result of the evaluation process, original data to which image processing is to be applied in a second external device, from among the original data of the reduced image data; transmitting the determined original data to the second external device via the communication means; receiving a result of the image processing on the original data from the second external device; 13. A method for controlling an information processing apparatus comprising the steps of:

13. A control method for an image processing device, comprising: The control means receiving reduced image data from an information processing device through a communication means of the image processing device; applying an evaluation process to the reduced image data to generate information that serves as a judgment criterion for determining whether or not the information processing device applies a predetermined image processing to original data of the reduced image data in an external device; transmitting a result of the evaluation process, including the information, to the information processing device via the communication means; receiving image data from the information processing device through the communication means; applying the predetermined image processing to the image data; transmitting a result of the application of the predetermined image processing to the information processing device via the communication means; 13. A method for controlling an image processing apparatus comprising the steps of:

14. A control method for an image processing device, comprising: The control means receiving reduced image data from an information processing device through a communication means of the image processing device; applying an evaluation process to the reduced image data; requesting the information processing device to transmit original data of the reduced image data based on a result of the evaluation process; receiving the requested original data from the information processing device via the communication means; applying a predetermined image processing to the requested original data; transmitting a result of the application of the predetermined image processing to the information processing device via the communication means; 13. A method for controlling an image processing apparatus comprising the steps of:

15. An information processing device capable of communicating with a first external device and a second external device, A storage means for storing image data; A control unit for controlling an operation of the information processing device, The control means Transmitting information for selecting one or more of the image data stored in the storage means to the first external device; receiving information on selected image data from the first external device; reading the selected image data from the storage means and transmitting the image data to a second external device; receiving, from the second external device, a result of applying image processing to the transmitted image data; 23. An information processing apparatus comprising:

16. An information processing device capable of communicating with a first external device and a second external device, A storage means for storing image data; A control unit for controlling an operation of the information processing device, The control means Transmitting information for selecting one or more of the image data stored in the storage means to the first external device; receiving, from the first external device, information for identifying the selected image data and a partial area thereof as information on the selected image data from the image data stored in the storage means; reading out data of the partial region from the selected image data from the storage means and transmitting the data to the second external device; reads out from said storage means image data requested by said first external device from said selected image data and transmits the image data to said second external device; receiving, from the second external device, a result of applying image processing to the image data requested from the first external device; 23. An information processing apparatus comprising:

17. 16. The information processing apparatus according to claim 15, wherein the control means determines the order in which image data is transmitted to the second external device based on information received from the first external device or the second external device.

18. 18. The information processing apparatus according to claim 15, wherein the control means transmits the image data to the second external device within a range not exceeding a predetermined data amount or number of data.

19. 19. The information processing apparatus according to claim 15, wherein the first external device and the second external device are devices on a network.

20. 20. An information processing device according to claim 15, characterized in that it communicates with the first external device through a network that does not incur any charges for use or communication volume, and communicates with the second external device through a network that incurs charges for use or communication volume.

21. 21. The information processing apparatus according to claim 15, wherein the image processing includes image processing that cannot be performed by the information processing apparatus.

22. An image processing device having a control means, The control means receiving information relating to image data stored in an information processing device from the information processing device; displaying a screen for selecting image data stored in the information processing device based on the information; determining a partial area for the image data selected through the screen; transmitting the selected image data and information about the partial region to the information processing device; 13. An image processing device comprising:

23. The control means receiving a result of applying image processing to the data of the partial region from an external device different from the information processing device; Displaying the results, sending a notification to the information processing device requesting execution of image processing on the entire image data corresponding to the result designated by the user; 23. The image processing device according to claim 22.

24. 24. The image processing apparatus according to claim 22, wherein the information relating to the image data stored in the information processing apparatus is proxy image data based on the image data.

25. An image processing device having a control means, The control means receiving proxy image data based on image data held by the information processing device from the information processing device; calculating an evaluation value for the proxy image data; generating information indicating a priority order of the proxy image data based on the evaluation value; transmitting information indicating the priority order to the information processing device; 13. An image processing device comprising:

26. The control means Calculating a plurality of types of the evaluation values; generating information indicating a priority order of the proxy image data based on an evaluation value obtained by weighting and adding the plurality of types of evaluation values; 26. The image processing device according to claim 25,

27. A program for causing a computer to function as the control means of the information processing device according to any one of claims 1 to 5 and 15 to 21.

28. A program for causing a computer to function as the control means of the image processing device according to any one of claims 6 to 10 and 22 to 26.

29. A method for controlling an information processing device having a storage unit for storing image data and capable of communicating with a first external device and a second external device, comprising the steps of: transmitting information to the first external device for selecting one or more of the image data stored in a storage device; receiving information on selected image data from the first external device; reading the selected image data from the storage means and transmitting the image data to a second external device; receiving a result of applying image processing to the transmitted image data from the second external device; 13. A method for controlling an information processing apparatus comprising the steps of:

30. A method for controlling an information processing device having a storage unit for storing image data and capable of communicating with a first external device and a second external device, comprising the steps of: transmitting information to the first external device for selecting one or more of the image data stored in a storage device; receiving, from the first external device, information that identifies the selected image data and a partial area thereof as information about the selected image data from the image data stored in the storage means; reading out data of the partial region from the selected image data from the storage means and transmitting the data to the second external device; reading out from said storage means image data requested by said first external device from said selected image data and transmitting said image data to said second external device; receiving from the second external device a result of applying image processing to the image data requested from the first external device; 13. A method for controlling an information processing apparatus comprising the steps of:

31. A control method for an image processing device, comprising: receiving information relating to image data stored in an information processing device from the information processing device; displaying a screen for selecting image data stored in the information processing device based on the information; determining a partial area for image data selected through the screen; transmitting the selected image data and information about the partial region to the information processing device; 13. A method for controlling an image processing apparatus comprising the steps of:

32. A control method for an image processing device, comprising: receiving proxy image data based on image data held by an information processing device from the information processing device; calculating an evaluation value for the proxy image data; generating information indicating a priority order of the proxy image data based on the evaluation value; transmitting information indicating the priority order to the information processing device; 13. A method for controlling an image processing apparatus comprising the steps of:

33. An image processing system having an information processing device, an image processing device, and a server device, The information processing device includes: A storage means for storing image data, transmitting information for selecting one or more of the image data stored in the storage means to the image processing device; The image processing device includes: displaying a screen for selecting one or more pieces of image data stored in the information processing device based on the information received from the information processing device; Transmitting information regarding the image data selected through the screen to the information processing device; The information processing device includes: receiving information about the selected image data from the image processing device; reading out the selected image data from the storage means and transmitting the image data to the server device; The server device includes: applying image processing to the image data received from the information processing device; A result of applying the image processing is transmitted to the information processing device; The information processing device includes: receiving a result of the application of the image processing from the server device; 1. An image processing system comprising:

34. An image processing system having an information processing device, an image processing device, and a server device, The information processing device includes: A storage means for storing image data, transmitting information for selecting one or more of the image data stored in the storage means to the image processing device; The image processing device includes: displaying a screen for selecting one or more pieces of image data stored in the information processing device based on the information received from the information processing device; determining a partial area for the image data selected through the screen; transmitting the selected image data and information about the partial region to the information processing device; The information processing device includes: receiving the selected image data and information about the partial region from the image processing device; reading out data of the partial region from the selected image data from the storage means and transmitting the data to the server device; The server device includes: applying image processing to the data of the partial region received from the information processing device; A result of applying the image processing is transmitted to the image processing device; The image processing device includes: receiving a result of the application of the image processing from the server device; Displaying the results, sending a notification to the information processing device requesting execution of image processing on the entire image data corresponding to the result designated by the user; The information processing device includes: transmitting the requested image data to the server device; The server device includes: applying image processing to the image data received from the information processing device; A result of applying the image processing is transmitted to the information processing device; The information processing device includes: receiving, from the server device, a result of applying image processing to the requested image data; 1. An image processing system comprising:

35. An image processing system having an information processing device, an image processing device, and a server device, The information processing device includes: transmitting proxy image data based on the image data stored in the storage means to the image processing device; The image processing device includes: receiving the proxy image data from the information processing device; calculating an evaluation value for the proxy image data; generating information indicating a priority order of the proxy image data based on the evaluation value; transmitting information indicating the priority order to the information processing device; The information processing device includes: receiving information indicating the priority order from the image processing device; determining a transmission order of the image data based on the information indicating the priority order; Transmitting the image data to the server device in accordance with the transmission order; The server device includes: applying image processing to the image data received from the information processing device; A result of applying the image processing is transmitted to the information processing device; The information processing device includes: receiving a result of the application of the image processing from the server device; 1. An image processing system comprising:

36. An image processing system having an information processing device, an image processing device, and a server device, The information processing device includes: transmitting proxy image data based on the image data stored in the storage means to the image processing device; The image processing device includes: receiving the proxy image data from the information processing device; calculating an evaluation value for the proxy image data; generating information indicating a priority order of the proxy image data based on the evaluation value; transmitting information indicating the priority order to the server device; The server device includes: receiving information indicating a priority of image data from the image processing device; updating information indicating the priority order stored in a storage means of the server device using information indicating the priority order received from the image processing device; transmitting information indicating the updated priority order to the information processing device; The information processing device includes: receiving information indicating the updated priority from the server device; determining a transmission order of the image data based on the updated information indicating the priority order; Transmitting the image data to the server device in accordance with the transmission order; The server device includes: applying image processing to the image data received from the information processing device; A result of applying the image processing is transmitted to the information processing device; The information processing device includes: receiving a result of the application of the image processing from the server device; 1. An image processing system comprising:

Images