Expansion of tethered photography

Abstract

The present disclosure relates to a method for enabling a general-purpose computer connected to a handheld digital camera to be used as a computational resource. The handheld digital camera and the general-purpose computer plan the distribution of various processes during photography on the basis of the configurations or states of the handheld digital camera and the general-purpose computer, and execute the plan by transmitting and receiving images or instructions by asynchronous communication. By using streaming to sequentially notify the general-purpose computer of changes in the state of the handheld digital camera, unnecessary processing that would be required if the general-purpose computer were to repeatedly issue requests to the handheld digital camera is eliminated.

Description

Expansion of Tethered Shooting

[0001] This disclosure relates to the cooperation between a handheld digital camera and a general-purpose computer.

[0002] A handheld digital camera (hereinafter also referred to as "camera") is a computer, and its computing power has been improving year by year. This improvement has realized high image quality. On the other hand, in view of technological trends, it is expected that the application of machine learning will progress more diversely in the future. However, at present, the market size of cameras has been shrinking due to the popularity of smartphones, and the research and development costs that can be invested in improving hardware performance are limited. In addition, the problems of increasing power consumption and heat generation associated with high performance cannot be avoided.

[0003] On the other hand, information processing devices such as smartphones, tablet computers, and personal computers are general-purpose computers that are widely popular. The huge market and research and development costs have greatly improved the performance of these devices. Some of these devices are equipped with hardware specialized for processing such as machine learning, parallel computing, and video encoding, and the power efficiency of these hardware is also high.

[0004] At the time of this disclosure, in the shooting of still images or videos (hereinafter, still images and videos are collectively referred to as "images". Videos include audio.) (hereinafter referred to as "tethered shooting") performed by connecting a camera and a general-purpose computer, the uses of the general-purpose computer are limited to uses as a user interface separated from the camera, that is, remote control of the camera and display of images, or uses as a storage destination for image files. Therefore, there is room to reduce the above problems by utilizing the functions or capabilities of general-purpose computers.

[0005] U.S. Patent Application Publication No. 2015 / 0358513, Patent No. 6603513, Brochure

[0006] "AIST Japanese Hand Dimension Data", [online], National Institute of Advanced Industrial Science and Technology, Artificial Intelligence Research Center, [searched on September 4, 2024], Internet <URL: https: / / www.airc.aist.go.jp / dhrt / hand / data / list.html>

[0007] This disclosure relates to a method for enabling a general-purpose computer connected to a handheld digital camera to be used as a computing resource. The handheld digital camera and the general-purpose computer plan the distribution of various processes during shooting based on the configuration or state of these devices, and execute this plan by sending and receiving images or commands via asynchronous communication. By sequentially notifying the general-purpose computer of changes in the state of the handheld digital camera via streaming, unnecessary processing that would occur if the general-purpose computer repeatedly made requests to the handheld digital camera is eliminated.

[0008] Block diagram showing the equipment configuration of the example system. System architecture diagram of the camera in the example system. System architecture diagram of the general-purpose computer in the example system. Sequence diagram illustrating the start of communication between the shooting application and the camera control program (Part 1). Flowchart illustrating the relationship between jobs and tasks. Diagram illustrating the data structure of a task. Sequence diagram illustrating the process of sending state values ​​from the general-purpose computer to the camera control program. Sequence diagram illustrating the execution flow of a remote execution task. Sequence diagram illustrating the start of communication between the shooting application and the camera control program (Part 2). Sequence diagram illustrating the process of sending state values ​​from the camera to the shooting application. Sequence diagram illustrating the execution flow of a task. Sequence diagram illustrating the start of communication between the shooting application and the camera control program (Part 3). Sequence diagram illustrating the data transmission and reception flow in streaming. Flowchart illustrating the processing of data received by streaming. Sequence diagram illustrating asynchronous processing by multiple threads.

[0009] This disclosure reveals a method for distributing computation and input / output processing using the computing resources of a general-purpose computer connected to a camera during tethered imaging. This disclosure also reveals a method for data streaming for controlling a camera from a general-purpose computer during tethered imaging.

[0010] <Definition> In this disclosure, "general-purpose computer" means a computer that can be used for a wide range of applications. This includes smartphones, wearable devices such as smart glasses, tablet computers, laptop computers, and desktop computers.

[0011] In this disclosure, a handheld device means a device that can be grasped by two or more fingers of one hand and is primarily used while held in the hand. In contexts where hand size is relevant, the dimensions of the hand are assumed to be the average values ​​for women as shown in Non-Patent Document 1. In this disclosure, "connected" means "functionally connected," and means that other components that are not connected devices may or may not be used for the connection, directly or indirectly.

[0012] In this specification, “person skilled in the art” means an imaginary person having ordinary knowledge in the art described herein. In this specification, descriptions of methods, numbers or formulas, operations or sets of operations, processes, components, circuits, networks, protocols, structures, materials, and features that are widely known in the art described herein are either omitted entirely or simplified. For example, descriptions of error handling that are obvious to a person skilled in the art are omitted. In this specification, “implementer” means a person who implements the contents of this specification.

[0013] This specification describes the claims by illustrating an imaging system (hereinafter referred to as the "Exemplary System"). The description of the Exemplary System is not a complete or exhaustive description of the requirements or specifications of imaging systems that can be realized by this disclosure. The components appearing in the description of the Exemplary System in this specification are limited to those necessary for describing the claims. However, this limitation does not preclude the existence or addition of other components.

[0014] In this specification, “Embodiment” is a representation that exemplifies either a) any part or all of any one or more claims of this disclosure, or b) prior art, by visual means such as drawings, graphs, photographs, videos, computer graphics, three-dimensional computer models, or physical models. The terms used to describe the various embodiments included herein are intended to describe, and not limit, any particular embodiment. No embodiment is intended to limit what it represents to one preferred form. Rather, the various embodiments herein are intended to encompass substitutes, modifications, and equivalents that may fall within the spirit and scope of the claims.

[0015] The various computers, components, modules, algorithms, protocols, programs, and networks included in the embodiment of this specification can be arranged in various ways by hardware, software, or a combination of both. Listing these various arrangements is not meaningful in this specification, and it will be apparent to those skilled in the art that implementers may arrange them as appropriate in light of various requirements and constraints. Each block included in the embodiment of this specification describes the function of that block collectively, and does not mean that the block is a physically or logically independent component from other components. The system architecture diagrams in this disclosure are conceptual diagrams and do not necessarily represent all or all physical connections or signal flows between components. In this disclosure, when a device “equips” a program, it means that the device physically, logically, or functionally possesses the program, including cases where the device realizes some or all of the functions of the program by obtaining data from other devices via a network.

[0016] In this specification or the claims of this disclosure, the order of blocks, processes, or operations is illustrative and does not preclude the possibility of these orders being rearranged or parallelized. For example, if five processes A, B, C, D, and E are described in this order, and their rearrangement does not affect the overall process, the order of these processes is not limited to A, B, C, D, and E unless explicitly limited, even though the remaining 119 possible orders of these processes are not enumerated.

[0017] In this disclosure, each countable noun refers to one or more objects unless explicitly stated otherwise. In this disclosure, "several" means that there is one or more objects it refers to. In this disclosure, modifiers that generally indicate order, such as "first," are merely labels for distinguishing objects referred to by nouns, unless the order is clear from the context, and do not imply any kind of order (e.g., spatial, temporal, logical).

[0018] In this disclosure, the conjunction "or" is used as a conjunction that encompasses any combination of the elements it brings together. For example, "A, B, or C" could be A alone, A and B, or A, B, and C. Also, when the terms "include" or "constitute" are used in this disclosure, these terms mean the presence of the elements they refer to, but do not exclude the presence or addition of other elements.

[0019] Where there are phrases in this disclosure that include a verb or verbal phrase but do not include an agent, the agent, such as a user or manufacturer of the device or a computer, is not limited.

[0020] In this disclosure, the term used to describe elements of a user interface refers to either a physical user interface or a graphical user interface (GUI). Furthermore, the terms used in this disclosure are chosen to reflect common usage in the art as defined herein. For example, "job" refers to a set of processes performed by a computer, not a professional service.

[0021] Where this disclosure includes descriptions or illustrations of a computer program, its process, or its thread as the operating entity, such descriptions or illustrations summarize, from a functional standpoint, a series of processes that the computer performs by referring to the instructions and data in the program. A computer program itself is a collection of instructions and data and cannot be an operating entity. However, it will be apparent to those skilled in the art that, in general, in descriptions or recognition of processes that a computer performs by referring to a program, a program, its process, or its thread may be represented or understood as an imaginary operating entity. In this disclosure, for example, descriptions of processes that involve a program or its process as the operating entity, such as "Program X performs process Y" or "Program X does Y," mean "A computer equipped with Program X performs process Y by referring to Program X." Similarly, descriptions of processes that have a program or its process as the object, such as "Send Y to Program X," mean a process performed on the computer equipped with this program.

[0022] In general, a running computer program often consists of multiple processes or threads that run in parallel. In this disclosure, unless otherwise specified, the number of processes or threads in a running computer program is one or more, but not limited to one. When a single computer shows multiple processes or threads running in parallel in a single diagram, these processes or threads are depicted as imaginary entities instead of the computer. When some of the multiple processes or threads running in parallel in a single computer are the target of processing (e.g., data transmission or authentication), these processes or threads are described as imaginary entities instead of the computer.

[0023] In this specification, lines beginning with an opening angle bracket and ending with a closing angle bracket are headings. Lines beginning with a double opening angle bracket and ending with a double closing angle bracket are subheadings. These headings or subheadings exist solely for reference and to improve readability and should not be used to interpret the contents.

[0024] <Configuration of the Example System> Figure 1 shows the equipment configuration of the example system. The example system includes a wired camera 0001 and a general-purpose computer 0002. Although not shown in Figure 1, there may be equipment such as a LAN (Local Area Network) or USB (Universal Serial Bus) hub between the camera 0001 and the general-purpose computer 0002.

[0025] Figure 2 illustrates the components that make up camera 0001 and the connections between these components. Camera 0001 includes a lens module 0003, an image sensor 0004, an image processor 0005, a system controller 0006, and a wired communication interface 0007. Camera 0001 may also include non-volatile internal data storage 0008, such as an SSD (Solid-State Drive). Camera 0001 may also include a removable recording media drive 0009. If camera 0001 includes a removable recording media drive 0009, the user can insert a removable recording medium 0010 into it. Hereinafter, the removable recording medium 0010 and the internal data storage 0008 will be collectively referred to as "internal recording media". Some of the parts of camera 0001 are equipped with thermometers.

[0026] The system controller 0006 includes semiconductor components: a CPU (Central Processing Unit) 0011, RAM (Random-Access Memory) 0012, and ROM (Read-Only Memory) 0013. The CPU 0011 loads a program (hereinafter referred to as the "camera control program") recorded in the ROM 0013 or the built-in data storage 0008 into the RAM 0012 and controls various parts of the camera 0001 by executing this program. The camera control program contains various instructions and data necessary for controlling various parts of the camera 0001. The CPU 0011 also uses the RAM 0012 as a temporary storage area for various data, such as metadata associated with images taken by the user.

[0027] The lens module 0003 consists of optical components such as a lens 0014 and aperture blades 0015, and a lens controller 0016 that drives them. The housing of the lens module 0003 may be detachable from the housing of the camera 0001. The lens module 0003 directs incident light from the subject to the image sensor 0004. The image sensor 0004 converts the optical image obtained through the lens module 0003 into an electrical signal. The image sensor 0004 also digitizes this electrical signal by A / D conversion.

[0028] The image processor 0005 performs image processing (hereinafter referred to as "development") such as demosaicing, noise reduction, and resolution conversion on the electrical signal from the image sensor 0004. The image processor 0005 includes semiconductor components such as a CPU 0017, ROM 0018, RAM 0019, and a DSP (Digital Signal Processor) 0020 for performing these processes.

[0029] The wired communication interface 0007 includes a communication controller 0021 and a port 0022, and relays communication between the camera control program and the shooting application program on the general-purpose computer 0002 (hereinafter referred to as the "shooting application"). Port 0022 may conform to a known standard such as USB or Ethernet (registered trademark).

[0030] Figure 3 illustrates the components that make up the general-purpose computer 0002 and the connections between these components. The general-purpose computer 0002 includes a CPU 0023, RAM 0024, ROM 0025, data storage 0026, and a wired communication interface 0027. The general-purpose computer 0002 may also include a touchscreen system 0028. The CPU 0023 loads various programs recorded in ROM 0025 or data storage 0026 into RAM 0024 and executes them. The CPU 0023 records the data obtained from the execution of these programs into data storage 0026. The wired communication interface 0027 includes a communication controller 0029 and a port 0030. Port 0030 is used to connect to port 0022 of camera 0001. Port 0030 may conform to known standards such as USB or Ethernet (registered trademark). The touchscreen system 0028 includes a touchscreen 0031 and its controller 0032. The general-purpose computer 0002 may also include a user input interface 0033. The user input interface 0033 includes an input device such as a keyboard 0034 or mouse 0035 that accepts user input. The general-purpose computer 0002 may also include a display system 0036. The display 0037 is controlled by a display controller 0038. The display 0037 or touchscreen 0031 displays various user interface elements to the user.

[0031] The data storage 0026 of the general-purpose computer 0002 stores the shooting application. In the example system, the user controls the camera 0001 using this shooting application. This shooting application displays a GUI to the user and includes various commands and data for receiving and processing user operations. During shooting, the camera control program continuously transmits images obtained from the image sensor 0004 and various values ​​representing the state of the camera 0001 to this shooting application via the camera's wired communication interface 0007. The GUI of this shooting application displays the images received from the camera control program to the user and accepts user operations. This shooting application interprets user operations as instructions and transmits these instructions to the camera control program. These instructions include instructions for saving images by releasing the shutter or starting recording, determining shooting parameters such as specifying the focus area, or changing settings of the camera 0001.

[0032] The camera control program or shooting application includes a job scheduler. The operation of the job scheduler is described below.

[0033] <Job Scheduler> The following describes a configuration in which the camera control program includes a job scheduler. After that, a configuration in which the shooting application includes a job scheduler is described.

[0034] 《Establishing Communication and Sending Initial Data》 Figure 4 illustrates the initiation of communication between the camera control program 0039 on camera 0001 and the shooting application 0040 on general-purpose computer 0002. To initiate communication with the camera control program 0039, the shooting application 0040 requests the camera control program 0039 to establish asynchronous communication via message 0041. In this communication, asynchronous communication refers to asynchronous communication at layer 5 or higher in the OSI (The Open Systems Interconnection) model. For example, if the physical connection method between camera 0001 and general-purpose computer 0002 is Ethernet®, and TCP / IP (Transmission Control Protocol / Internet Protocol) is used for communication between camera 0001 and general-purpose computer 0002, the shooting application 0040 and the camera control program 0039 establish asynchronous communication at the TCP / IP application layer. In this specification, asynchronous communication means communication in which each program processes messages it receives as expected messages, even if they are not replies to messages sent to the other party.

[0035] The protocol used for asynchronous communication between the shooting application 0040 and the camera control program 0039 may be a publicly known protocol. For example, this protocol may be WebSocket. Alternatively, the implementer may implement their own protocol.

[0036] Once asynchronous communication is established between the shooting application 0040 and the camera control program 0039, the shooting application 0040 sends a message 0042 to the camera control program 0039 that represents the configuration of the general-purpose computer 0002. This message includes items such as the model of the general-purpose computer 0002, the type and number of processors, and the RAM capacity. The configuration of the general-purpose computer 0002 is later used for job scheduling.

[0037] 《Jobs and Tasks》 Figure 5 shows an example of the relationship between jobs and tasks in the exemplary system. A job is a set of tasks and their execution order. Job 0043 includes multiple tasks such as task 0044. A task is a unit that represents various processes performed by the camera control program 0039 from the time it receives an image save instruction from the shooting application 0040 until the saving of the image file is completed, by dividing them into blocks according to type. The image file is written to the internal recording media of the camera 0001 or to the data storage 0026 of the general-purpose computer 0002.

[0038] Figure 6 illustrates the data structure of a task. A task has five elements: an identifier (id), a command, an input, an output, and an execution environment. The task's execution environment indicates whether the task is executed on camera 0001 or general-purpose computer 0002. The execution environment for task 0045 is "camera," meaning it is executed inside the camera. The task's command represents the type of processing to be performed by this task, for example, an averaging filter in image processing. The task's input represents the arguments for this processing, for example, the address and kernel size of the image data to be subjected to the averaging filter processing. The image data address is, for example, a pointer to the address of a variable, a path to a file on the file system, or a string that serves as a key for the shooting application 0040 to request image data from the camera control program 0039. The task's output represents the destination of the data resulting from the processing. This destination is, for example, space on RAM 0019 of the image processor 0005, or a file on the file system of the internal recording media.

[0039] 《Updating the execution plan via the job scheduler》

[0040] After the establishment of the asynchronous communication in FIG. 4, the imaging application 0040 continuously transmits a message including a value representing the state of the general-purpose computer 0002 (hereinafter, the value representing the state of the camera 0001 or the general-purpose computer 0002 is referred to as "state value") to the camera control program 0039, for example, at a fixed time interval. FIG. 7 shows the flow of these messages 0046. The state value represents the state of the general-purpose computer 0002 such as the usage rate of the CPU 0023, the temperature of the CPU 0023, and the free capacity of the RAM 0024. The state value may include an image. When the general-purpose computer 0002 is a handheld device such as a smartphone, the acquisition of these states may be restricted by its operating system. In this case, the imaging application 0040 itself may estimate the state of the general-purpose computer 0002 by aggregating recent processes, generate an alternative to the message 0046 from this estimation, and transmit this to the camera control program 0039.

[0041] FIG. 7 also illustrates the timing of updating the job execution plan. The camera control program0039 passes the message 0046 of the state value of the general-purpose computer 0002 received from the imaging application 0040 to the job scheduler every time it receives it. The job scheduler updates the job execution plan in the process 0047 every time it receives this.

[0042] The job execution plan is a combination of the execution environments of the tasks included in the job. In other words, the job execution plan represents which task is to be executed on which device, either the camera 0001 or the general-purpose computer 0002. In updating the job execution plan, the job scheduler evaluates candidates for the execution plan using a cost function and selects the execution plan estimated to be the best. This cost function is a polynomial consisting of the sum of one or more cost estimation functions (hereinafter referred to as "total cost") as shown in, for example, Equation 1.

[0043]

[0044] Each term of c kThese are cost estimation functions, such as power consumption, temperature rise, and execution time. These estimation functions take as input the status values ​​of each part of the camera 0001, as well as the configuration of the general-purpose computer 0002 transmitted in message 0042 in Figure 4 and the status values ​​of the general-purpose computer 0002 transmitted in message 0046 in Figure 7. The status values ​​of each part of the camera 0001 include the settings of the camera 0001, such as the settings for still image shooting or video shooting. The job scheduler also processes each term c according to the configuration and status values ​​of the camera 0001 or the general-purpose computer 0002. k To weight it, coefficient a k Adjust.

[0045] As an example of a cost function, four examples of execution plan selection are given below, using the sum of cost estimation functions where c1 is the power consumption of camera 0001, c2 is the power consumption of general-purpose computer 0002, c3 is the temperature rise of image processor 0005, c4 is the temperature rise of built-in data storage 0008, c5 is the temperature rise of camera 0001's communication controller 0021, and c6 is the total execution time of the job.

[0046] The first example assumes a situation where camera 0001 is in a state of being driven by an internal battery, still image shooting is selected by setting, and the storage location of the still image file can be either the built-in data storage 0008 of camera 0001 or the data storage 0026 of general-purpose computer 0002. In this case, the power consumption of camera 0001 can be a major cost factor. Therefore, the job scheduler adjusts the coefficient a1 of the power consumption amount of camera 0001 to be larger than the other terms a2 to a6. For example, (a1, a2, a3, a4, a5, a6) = (1, 0.1, **0.1**, 0.1, 0.1, 0.1, 0.1), and compares the execution plan p1 developed using the image processor 0005 in camera 0001 with the execution plan p2 developed using the shooting application 0040 of general-purpose computer 0002. In this example, assume that in p1, (c1, c2, c3, c4, c5, c6) = (1, 0.5, 0.1, 0.1, 0,1, 0.1), and in p**2**, (c1, c2, c3, c**4**, c5, c6) = (0.1, 0.8, 0.1, 0.1, 0.1, 0.1). As a result of the comparison, if the total cost of p1 (1.09 in the example) is larger than the total cost of p2 (0.22 in the example), the job scheduler selects a plan that assigns the development and subsequent tasks to general-purpose computer 0002.

[0047] The second example assumes a situation where continuous shooting of still images is selected. At this time, the jobs related to the processing of each image need to be completed in a short time. The job scheduler increases the coefficient a6 of the term of the total execution time of the job or the coefficient a5 of the term of the temperature rise of the communication controller 0021. As a result, if it is predicted that transmitting the RAW image data before development to the general-purpose computer **0002** will significantly increase the total cost in terms of the total execution time of the job or the temperature rise of the communication controller 0021, the job scheduler can select an execution plan that assigns the development task to the image processor 0005 in the camera 0001 and executes the subsequent tasks of the development on the general-purpose computer 0002.

[0048] It seems there are some formatting or content issues in the original text, such as some overlapping or incorrect numbering in the "c" terms in the first example. I've translated it as accurately as possible based on the provided text. If you have any further clarifications or corrections, please let me know.A third example assumes that both camera 0001 and general-purpose computer 0002 receive external power, video recording is selected, and general-purpose computer 0002 is a laptop computer. In this case, the temperature rise of various parts inside camera 0001 can be a more significant cost factor affecting the recording time than power consumption. Therefore, the job scheduler decreases the coefficients a1 and a2 in the terms of power consumption for camera 0001 and general-purpose computer 0002, and increases the coefficients a3, a4, and a5 in the terms of temperature rise for various parts inside camera 0001, such as the image processor 0005, built-in data storage 0008, and communication controller 0021. Using this adjusted cost function, the job scheduler can compare an execution plan that sends the RAW image to general-purpose computer 0002 with an execution plan that runs inside camera 0001 until the image file is written.

[0049] A fourth example assumes that video recording is selected, and the encoding is not supported by camera 0001, and general-purpose computer 0002 has the hardware for that encoding. In this case, the job scheduler may select an execution plan to send the RAW image to general-purpose computer 0002 and encode it on general-purpose computer 0002.

[0050] Beyond the four examples above, each term and coefficient of the cost function can be implemented in various ways depending on the camera implementation. Regardless of how the cost function is implemented, the job scheduler uses that cost function to select one of one or more execution plan candidates based on the configuration and status of camera 0001 and the configuration and status of general-purpose computer 0002.

[0051] 《Execution of Job Execution Plan》 When the camera control program 0039 receives an image saving instruction, it executes a series of tasks by referring to the execution plan selected at that time.

[0052] For tasks executed on the general-purpose computer 0002 (hereinafter referred to as "remote execution tasks"), the camera control program 0039 sends this task to the shooting application 0040. Figure 8 illustrates the processing flow of a remote execution task. If the task preceding the remote execution task is executed on the camera 0001, the input to this remote execution task includes a key string (for example, the random string "JKy3gNZ6..." in Figure 8) that allows the shooting application 0040 to obtain the output of the preceding task.

[0053] When the shooting application 0040 receives this remote execution task, it uses the key specified as input for this task to obtain the output result of the previous task from the camera control program 0039. The shooting application 0040 then uses this output result as input to execute the remote execution task. In Figure 8, the shooting application 0040 receives the task via message 0048 and obtains the image data for this previous task via message 0049. Once the execution of the remote execution task by process 0050 is complete, the shooting application 0040 notifies the camera control program 0039 of this completion via message 0051. This message 0051 may include a string that serves as a key for the camera control program 0039 to obtain the output of the remote execution task from the shooting application 0040 as input for the subsequent task (for example, the random string "Di9ZempQ..." in Figure 8).

[0054] The job execution is complete when all tasks included in the job have finished executing.

[0055] The above describes job planning and execution in a configuration where the camera control program 0039 has a job scheduler. Conversely, the shooting application 0040 may have a job scheduler. Even in this case, the camera control program 0039 and the shooting application 0040 establish asynchronous communication. This asynchronous communication allows the camera control program 0039 to send messages to the shooting application 0040 at any time. Therefore, although the roles of the camera control program 0039 and the shooting application 0040 are swapped compared to the configuration where the camera control program 0039 has a job scheduler, no additional technical elements are required.

[0056] More specifically, in message 0052 in Figure 9, the camera control program 0039 sends message 0052 to the shooting application 0040 representing the configuration of camera 0001. Next, as shown in message 0053 in Figure 10, the camera control program 0039 continuously sends the status value of camera 0001 to the shooting application 0040, for example, at regular time intervals. Each time the shooting application 0040 receives message 0052, it updates the job execution plan in process 0054 using the same method as described in "Updating the Execution Plan by the Job Scheduler".

[0057] When the shooting application 0040 detects an image saving instruction from the user, it executes the tasks included in the job by referring to the execution plan selected at that time. For tasks executed on camera 0001, the shooting application 0040 sends a task execution request to camera control program 0039. For example, in Figure 11, the shooting application 0040 requests the camera control program 0039 to execute a task by message 0055. The camera control program 0039 executes this task by process 0056 and sends a completion notification to the shooting application 0040 as message 0057. The input for the subsequent task (identifier "AirZhnUC...") requested in message 0058 is the output of the preceding task (identifier "JJuB43To..."). When the execution of all tasks included in the job is complete, the job execution is completed.

[0058] <Streaming> Among the various shooting applications that run on smartphones, some not only control the smartphone's built-in camera but also have functions such as object recognition and focus tracking using machine learning models. These applications extend the capabilities of human judgment and work. On the other hand, the role of known remote control applications for handheld digital cameras that run on general-purpose computers during tethered shooting is limited to presenting images to the user, conveying the results of the user's judgment as instructions to the camera, and saving the images to the general-purpose computer.

[0059] When performing more advanced processing, such as object recognition, on a general-purpose computer during tethered shooting, the image obtained from the image sensor is naturally necessary. Furthermore, the shooting application may use camera status values, such as the coordinates of the focus area or the focus distance, as input.

[0060] To the best of the inventor's knowledge, publicly available handheld digital camera SDKs (Software Development Kits) or APIs (Application Programming Interfaces) only support synchronous communication. When building a shooting application using these APIs or SDKs, it becomes necessary to send requests to the camera frequently, and the camera must also parse each of these requests it receives. Consequently, a lot of unnecessary processing occurs.

[0061] The example system is not based on this request-driven processing, but rather on an event-driven scheme using data streaming. The following describes how the camera control program 0039 sequentially sends status values ​​of the camera 0001 to the shooting application 0040, and how the shooting application 0040 processes these status values.

[0062] The shooting application 0040 contains data including the image encoding obtained from the camera control program 0039, the type of state value of camera 0001, and the arguments necessary to obtain these state values. Hereinafter, this data will be referred to as the "streaming specification." Since the streaming specification is not referenced from outside the shooting application 0040, the shooting application 0040 may store the streaming specification in any format. The shooting application 0040 may also contain data for the streaming specifications of multiple camera models. Table 1 shows an example of a streaming specification.

[0063]

[0064] Figure 12 illustrates the initiation of communication between the shooting application 0040 and the camera control program 0039. Upon startup, the shooting application 0040 establishes asynchronous communication with the camera control program 0039 using the same method as described in "Establishing Communication and Sending Initial Data." Once this communication is established, the shooting application 0040 requests the camera control program 0039 to send a message 0060 representing the configuration of the camera 0001 via message 0059. Message 0060 includes, for example, the model name or firmware version number of the camera 0001.

[0065] Upon receiving message 0060, the shooting application 0040 refers to the streaming specifications illustrated in Table 1 and requests the camera control program 0039 to stream the necessary images and status values. Figure 13 illustrates the data transmission and reception flow in streaming. Streaming here refers to a communication and data processing method in which, after the camera control program 0039 receives a data transmission request from the shooting application 0040 once, it continuously sends sequentially generated data within the camera 0001 to the shooting application 0040, and the shooting application 0040 processes the received data sequentially.

[0066] Figure 14 illustrates the flow of process 0061 or process 0062 in Figure 13. The shooting application 0040 performs preprocessing 0063 on the image or state value received from the camera control program 0039 as needed. Preprocessing here refers to processes such as buffering, shaping, and normalization. Next, the shooting application 0040 performs process 0064, which inputs the image, state value, or preprocessed data received by this streaming into, for example, a machine learning model, and calculates, thereby obtaining a camera control command 0065. The control command is, for example, a focus area specification for autofocus, and includes the coordinates of the focus area obtained as a result of object recognition as an argument. If the general-purpose computer 0002 is equipped with a sensor (e.g., a depth sensor) and the shooting application 0040 can use it, the shooting application 0040 may add data obtained from this sensor to the input of process 0063.

[0067] In process 0066, the shooting application 0040 sends a message containing this control command and arguments to the camera control program 0039 using the asynchronous communication shown in Figure 9. Each message sent includes an identifier, as illustrated by messages 0067 and 0068 in Figure 15, which are random strings aD5ht... and BNaEh.... The process or thread that sent this message does not have to wait for the result of executing the control command or for an error response. For example, after sending a command to specify the focus area by message 0067 in Figure 15, the shooting application 0040 may start any other processing without waiting to receive a focus or error response 0069. As shown by messages 0069 and 0070, the camera control program 0039 includes the identifier of this message in its reply to each message received from the shooting application 0040.

[0068] The camera control program 0039 does not have to execute the control commands contained in the received messages sequentially in the order they were received. For example, if the focus lens and ISO (International Organization for Standardization) sensitivity can be controlled independently, as shown in Figure 15, one thread 0071 may drive the focus lens while another thread 0072 adjusts the ISO sensitivity. Each thread may send the result of its own processing as a reply to the shooting application 0040 without waiting for the completion of processing by other processes or threads.

Claims

1. A method for distributing processing between a handheld digital camera and a general-purpose computer connected thereto via a wired connection during imaging, wherein the processing consists of a series of numerical operations, logical operations, or data input / output operations starting from the detection of an instruction to save an image and ending with the completion of saving an image file, wherein the handheld digital camera or the general-purpose computer plans the distribution based on the configuration or state of the handheld digital camera or the general-purpose computer, and the handheld digital camera or the general-purpose computer executes the plan by sending and receiving images or instructions via the wired connection.

2. A method for communication between a handheld digital camera and a general-purpose computer connected thereto by a wire, wherein the handheld digital camera comprises a first program, the general-purpose computer comprises a second program, the second program is an application program executed on the operating system of the general-purpose computer, the handheld digital camera and the general-purpose computer perform the communication by executing the first program and the second program, and in the communication, the handheld digital camera and the general-purpose computer each process the received message as an expected message even if the message received from the other is not a reply to a message sent to the other.

3. A method for distributing processing between a handheld digital camera and a general-purpose computer wired to the handheld digital camera during imaging, wherein the processing is a series of numerical operations, logical operations, or data input / output operations starting from the detection of an instruction to save an image and ending with the completion of saving an image file, wherein the general-purpose computer transmits its configuration or state to the handheld digital camera via the communication of claim 2, the handheld digital camera plans the distribution using a cost function that includes the configuration or state and its own configuration or state as arguments, and the handheld digital camera causes the general-purpose computer to execute a part of the processing by transmitting an image or command via the communication of claim 2.

4. A method for distributing processing between a handheld digital camera and a general-purpose computer wired to the handheld digital camera during imaging, wherein the processing is a series of numerical operations, logical operations, or data input / output operations starting from the detection of an instruction to save an image and ending with the completion of saving an image file, wherein the handheld digital camera transmits its configuration or state to the general-purpose computer via the communication of claim 2, the general-purpose computer plans the distribution using a cost function that includes the configuration or state and its own configuration or state as arguments, and the general-purpose computer causes the handheld digital camera to execute a part of the processing by transmitting an image or command via the communication of claim 2.

5. A method for communication between a handheld digital camera and a general-purpose computer connected thereto by a wire, characterized in that the general-purpose computer requests the handheld camera to transmit data, the data is generated sequentially by changes in the state of the handheld digital camera after the handheld digital camera receives the request, and the handheld camera transmits the data sequentially to the general-purpose computer using the wired connection after receiving one request.

6. A method for controlling a handheld digital camera wired to a general-purpose computer, characterized in that the general-purpose computer sequentially receives images or data representing the state of the handheld digital camera by the communication method of claim 5; the general-purpose computer sequentially generates commands for controlling the handheld digital camera from the images, the data, or data obtained by processing them; the general-purpose computer sequentially transmits the commands to the handheld digital camera by the communication method of claim 2; and the handheld digital camera executes the commands in series or in parallel.

Images