Image pickup apparatus and control method, storage medium, and program product

Abstract

The invention provides an image pickup apparatus, a control method, a storage medium, and a program product. The image pickup apparatus includes: a first generation unit configured to generate image data at a predetermined period; a second generation unit for generating metadata relating to a process until the generation of the image data; a first output unit for adding the metadata to the image data to output the image data to the outside; and a second output unit for outputting predetermined information, generated until the image data is output, from among the metadata to the outside a plurality of times before the image data is output by the first output unit.

Description

Technical Field

[0001] This disclosure relates to camera equipment, control methods, storage media, and program products, and particularly to control for outputting information related to image data to an external source. Background Technology

[0002] The technique is known as visual effects (VFX). VFX is a technique used to composite computer graphics (CG) with actually captured images.

[0003] In VFX, compositing captured images with CG includes methods that composite pre-generated CG after capturing the image, and methods that composite CG with sequentially captured images in real time. In the latter method, CG is composited with the captured image frame by using metadata associated with the image data, and because of the emphasis on real-time characteristics, a communication protocol called User Datagram Protocol (UDP) is typically used.

[0004] Japanese Patent No. 7190594 describes a method in which a time code of metadata is output from an output unit different from the output unit that outputs image data with added time codes, so that the time code of metadata can be used even when the time code added to the image data cannot be used correctly.

[0005] UDP has the possibility that the metadata of a frame may be lost due to packet loss. In the event of packet loss, since the metadata of the lost frame cannot be used, there is a possibility of generating unnatural images. Summary of the Invention

[0006] This disclosure was made in view of the above-mentioned problems and provides technical advantages that can reduce the generation of unnatural images due to packet loss.

[0007] To address the aforementioned problems, this disclosure relates to a camera device comprising: a first generation unit for generating image data at a predetermined period; a second generation unit for generating metadata related to processing up to the generation of the image data; a first output unit for adding the metadata to the image data to output the image data to an external location; and a second output unit for repeatedly outputting predetermined information generated in the metadata up to the output of the image data to an external location before the first output unit outputs the image data.

[0008] To address the aforementioned problems, this disclosure relates to a control method for a camera device, the control method comprising: generating image data at a predetermined period; generating metadata related to processing up to the generation of the image data; adding the metadata to the image data to output the image data to an external location from a first output unit; and, before outputting the image data from the first output unit, repeatedly outputting predetermined information related to the image data generated up to the generation of the image data from a second output unit.

[0009] To address the aforementioned problems, this disclosure relates to a computer-readable storage medium storing a program for enabling a computer to function as a camera device as described above.

[0010] To address the aforementioned problems, this disclosure relates to a computer program product comprising a program for enabling a computer to function as a camera device as described above.

[0011] The features of this disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings. The following description of embodiments is given by way of example. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the specification, serve to explain the principles of the embodiments.

[0013] Figure 1 This is a block diagram illustrating the configuration of the camera device according to this embodiment.

[0014] Figure 2 This is a block diagram illustrating the configuration of the image processing apparatus according to this embodiment.

[0015] Figure 3 This is a flowchart illustrating the control processing of the camera device in this embodiment.

[0016] Figure 4 This is a diagram showing the metadata output from the camera device in this embodiment.

[0017] Figure 5 This is a diagram illustrating a description example of metadata output from the camera device of this embodiment.

[0018] Figure 6 This is a flowchart illustrating the control process of the image processing apparatus in this embodiment.

[0019] Figures 7A to 7D This is a diagram illustrating the operation of the camera device according to the first embodiment, from image processing to image data output processing.

[0020] Figures 8A to 8DThis is a diagram illustrating the operation of the camera device according to the second embodiment, from image processing to image data output processing. Detailed Implementation

[0021] In the following, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claims. Several features are described in the embodiments, but not all such features are required, and several such features can be appropriately combined. Furthermore, in the drawings, the same reference numerals are given the same or similar configuration, and redundant descriptions thereof are omitted.

[0022] In this embodiment, an example will be described as follows: In a system in which a camera device and an image processing device are communicatively connected, the camera device generates image data at a predetermined period (e.g., frame rate), adds metadata related to the processing up to the point of generating image data for each frame to the image data, and outputs the image data to the outside, and the image processing device combines the image data with computer graphics (CG) data in real time based on the image data and metadata received from the camera device.

[0023] First Embodiment Next, the first embodiment will be described.

[0024] Equipment Configuration Reference Figure 1 The configuration and functions of the camera device in this embodiment are described.

[0025] The camera device 100 in this embodiment is, for example, a digital camera, a digital video camera, a smartphone with camera and communication functions, a tablet computer, or a web camera.

[0026] The camera device 100 in this embodiment has the following functions: Figure 1 The software program to be executed by the hardware and / or control unit 101 shown is implemented.

[0027] The control unit 101 is an arithmetic processing processor, such as a central processing unit (CPU) or microprocessor unit (MPU), that controls the entire camera device 100 as a whole, and implements the control processing described later by executing the program stored in the non-volatile memory 104. Note that instead of the control unit 101 controlling the entire device, multiple hardware units can be used to distribute the processing and control the entire device.

[0028] The imaging unit 102 includes an imaging optical system. The imaging optical system includes a lens group with a zoom lens and a focusing lens, and a shutter with an aperture function. The imaging unit 102 includes an image sensor, including a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) device, that converts the image of the subject into an electrical signal, and an A / D converter that converts the analog image signal output from the image sensor into a digital signal. Under the control of the control unit 101, the imaging unit 102 converts the light from the subject image formed by the lenses included in the imaging unit 102 into an electrical signal through the image sensor, performs noise reduction processing, etc., and outputs a digital image signal. Note that the imaging optical system including the lens group and the shutter may be removable from the imaging device 100.

[0029] The image processing unit 103 includes a graphics processing unit (GPU) and performs various types of image processing on the image signals generated by the camera unit 102. The image processing unit 103 compresses and encodes still image data processed by image processing methods such as JPEG, or encodes moving image data using moving image compression methods such as MPEG2 or H.264 to generate image files, and records the image files to a recording medium such as a memory card. The image processing unit 103 performs predetermined calculations on the image signals generated by the camera unit 102, and the control unit 101 controls the focusing lens, aperture, and shutter of the camera unit 102 based on the obtained calculation results, thereby performing autofocus (AF) processing and automatic exposure (AE) processing.

[0030] The image processing performed by the image processing unit 103 includes preprocessing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, and special effects processing.

[0031] Preprocessing includes signal amplification, reference level adjustment, and defective pixel correction.

[0032] Color interpolation is a process that interpolates the values ​​of color components that cannot be obtained during shooting, and it is also known as de-mosaicing.

[0033] Correction processing includes processes such as white balance adjustment, grayscale correction, correction of image degradation caused by optical aberrations of the camera optical system (image restoration), correction of the effects of peripheral darkening of the camera optical system, and color correction.

[0034] The detection process includes the detection of motion in characteristic regions (e.g., facial regions or body regions) or specific regions, as well as the recognition of people.

[0035] Data processing includes processes such as compositing (combining), scaling, encoding and decoding, and header information generation (data file generation). Data processing also includes the generation of image data for purposes such as display, recording, and external output.

[0036] Evaluation value calculation processing includes processes such as the generation of signals and evaluation values ​​for automatic focus detection (AF) and the generation of evaluation values ​​for automatic exposure control (AE).

[0037] Special effects processing includes techniques such as adding blur effects, changing color tones, and re-lighting.

[0038] Note that the above image processing is an example and does not limit the processing to be performed by the image processing unit 103.

[0039] Non-volatile memory 104 is a type of flash memory, solid-state drive (SSD), or hard disk drive (HDD) that enables data to be erased or written. Non-volatile memory 104 records constants and programs used for the operation of control unit 101. The programs mentioned here are used to execute [further details to be provided later]. Figure 3 The described control processing procedure.

[0040] Volatile memory 105 is a RAM that temporarily stores programs read from non-volatile memory 104, as well as constants and variables used to execute the program. Volatile memory 105 also serves as working memory for control unit 101, image processing unit 103, and metadata control unit 108. Volatile memory 105 also serves as a buffer memory for temporarily holding image signals generated by imaging unit 102, image data processed by image processing unit 103, and metadata generated by metadata control unit 108, and as image display memory for display unit 107.

[0041] The operation unit 106 is an input device that receives various operations from the user, such as various switches, buttons, and dials, and outputs operation information to the control unit 101. The operation unit 106 includes, for example, a power button for turning the power on or off, a shooting button for indicating the start or end of still image or moving image shooting, a playback button for indicating image playback, and a mode switching button for changing the operating mode of the camera device 100. The operation unit 106 also includes a dedicated connection button for initiating communication with external devices such as the image processing device 200 described later. Note that the functions assigned to the same input device can be variable. The input device can be a software button or key using a touch display.

[0042] In still image capture mode, autofocus (AF) and auto exposure (AE) processing are performed based on the image signal generated by the imaging unit 102. The control unit 101 performs the capturing process of recording still image data, which is processed by the image processing unit 103 to process the image signal generated by the imaging unit 102, onto the recording medium.

[0043] In motion image capture mode, control unit 101 performs autofocus (AF) and auto exposure (AE) processing based on the image signals for each frame generated by imaging unit 102. Control unit 101 performs image capture processing, recording motion image data from the image signals generated by imaging unit 102 processed by image processing unit 103 onto a recording medium. Control unit 101 begins motion image data capture processing in response to the initial pressing of the capture button and continues this process until the capture button is pressed again. Control unit 101 stops motion image data capture processing in response to the capture button being pressed again and records motion image data for the time from the start to the stop of the capture process onto the recording medium.

[0044] Display unit 107 displays real-time view images generated by camera unit 102, captured images, images to be reproduced, and a graphical user interface (GUI) for receiving user operations. Display unit 107 is, for example, a display device such as a liquid crystal display (LCD) or an organic EL display. Display unit 107 may be integrated with camera device 100, or it may be an external device connected to camera device 100. Camera device 100 may be able to connect to display unit 107 and control the display of display unit 107.

[0045] The metadata control unit 108 generates, in parallel with the camera processing performed by the camera unit 102, related information concerning the processing up to the generation of image data. This related information is metadata that includes information related to the camera processing and information related to the image data generated by the camera processing. The metadata includes, but is not limited to, information to be referenced later. Figure 4 The description includes the timecode, recording time, shooting conditions, lens information, and the orientation of the camera device 100. Shooting conditions include the focal length and angle of view of the camera optical system, ISO, shutter speed, exposure time, and frame rate. Lens information includes distortion parameters indicating the degree of lens distortion, lens manufacturing error information, and lens name. The metadata control unit 108 extracts metadata from the generated metadata for real-time compositing processing used to synthesize image data generated at a predetermined frame rate with pre-generated CG data.

[0046] The first output unit 109 adds metadata to the image data captured by the camera unit 102 and processed by the image processing unit 103, and outputs the image data to the outside. In this embodiment, the first output unit 109 conforms to a video signal transmission standard called Serial Digital Interface (SDI) to output the image data, but is not limited thereto.

[0047] The second output unit 110 outputs metadata generated by the metadata control unit 108 to the outside. The metadata to be output from the second output unit 110 includes at least information for real-time compositing of image data and CG data performed by the image processing device 200 (e.g., timecode and shot information). In this embodiment, the second output unit 110 outputs metadata using a communication protocol called User Datagram Protocol (UDP) that conforms to the communication standard called Ethernet (registered trademark), but is not limited thereto.

[0048] The signal transmission standards, communication standards, and communication protocols conforming to the first output unit 109 and the second output unit 110 in this embodiment are examples, and other standards and communication protocols can be used. Wired or wireless communication methods can be used. The first output unit 109 and the second output unit 110 have configurations (connectors, antennas, and transmitting / receiving circuits, etc.) corresponding to the conforming standards.

[0049] Reference Figure 2 The configuration and functions of the image processing device according to this embodiment are described.

[0050] The image processing device 200 in this embodiment is a general-purpose computer such as a personal computer (PC) or a tablet computer, or a special-purpose computer such as a server computer.

[0051] The image processing device 200 in this embodiment has the function of: Figure 2 The software program to be executed by the hardware and / or control unit 201 shown is used to implement this.

[0052] The image processing device 200 of this embodiment uses image data and metadata received from the camera device 100 to perform real-time compositing processing that combines CG data with image data.

[0053] The control unit 201 is an arithmetic processing processor, such as a CPU or MPU, that controls the entire image processing device 200, and implements the program stored in the non-volatile memory 204 (described later) to achieve the later... Figure 6 The control processing described herein. Note that instead of control unit 201 controlling the entire device, the entire device can be controlled by distributing the processing among multiple hardware units.

[0054] Non-volatile memory 204 is a flash memory, solid-state drive (SSD), or hard disk drive (HDD) that enables data to be erased or written. Non-volatile memory 204 records the operating system (OS), which is the basic software executed by control unit 201, and applications that cooperate with the OS to implement application functions. In this embodiment, non-volatile memory 204 stores applications used by image processing device 200 to implement the control processing and real-time synthesis processing described later.

[0055] The image processing device 200 in this embodiment performs processing by reading software provided by an application. Note that it is assumed the application includes software for using the basic functions of the OS installed in the image processing device 200. Note that the OS of the image processing device 200 may include software for implementing the processing in this embodiment.

[0056] Volatile memory 205 is RAM that temporarily stores programs read from non-volatile memory 204, as well as constants and variables used to execute the program. Volatile memory 205 also serves as working memory for control unit 201, CG generation unit 209, and CG compositing unit 210. Volatile memory 205 acts as a buffer memory, temporarily holding image data and metadata received from imaging device 100, CG data generated by CG generation unit 209, and composite image data generated by CG compositing unit 210. Volatile memory 205 also serves as image display memory for display unit 207.

[0057] The operation unit 206 receives user input devices such as a mouse, keyboard, or touchpad and outputs operation information to the control unit 201. Note that the functions assigned to the same input device can be variable. The input device can be a software button or key using a touch display.

[0058] The control unit 201 performs processing based on the operations of the operation unit 206. For example, the control unit 201 performs the start and end of the application, as well as processing based on the operations of the GUI provided by the application.

[0059] Display unit 207 displays a GUI provided by the OS or application. Display unit 207 displays, through the GUI, setting information of image processing device 200, image data received by first input unit 202, CG data generated by CG generation unit 209, and composite image data generated by CG compositing unit 210, etc. Display unit 207 is, for example, a display device such as a liquid crystal display or an organic EL display. Display unit 207 may be integrated with image processing device 200, or it may be an external device connected to image processing device 200. Image processing device 200 may be able to connect to display unit 207 and control the display of display unit 207.

[0060] The first input unit 202 is directly or indirectly connected to the first output unit 109 of the camera device 100. The first input unit 202 receives image data with added metadata via the first output unit 109 of the camera device 100. In this embodiment, the first input unit 202 receives data in accordance with the High Definition Multimedia Interface (HDMI) (registered trademark) standard, but is not limited thereto.

[0061] The second input unit 203 is directly or indirectly connected to the second output unit 110 of the camera device 100. The second input unit 203 receives metadata output from the second output unit 110 of the camera device 100. In this embodiment, the second input unit 203 outputs metadata using a communication protocol called User Datagram Protocol (UDP) that conforms to the communication standard known as Ethernet (registered trademark), but is not limited thereto. The metadata to be received by the second input unit 203 includes metadata for real-time combination of image data received by the first input unit 202 and CG data generated by the CG generation unit 209.

[0062] The signal transmission standards, communication standards, and communication protocols conforming to the first input unit 202 and the second input unit 203 in this embodiment are examples, and other standards and communication protocols can be used. Wired or wireless communication methods can be used. The first input unit 202 and the second input unit 203 have configurations (connectors, antennas, and transmitting / receiving circuits, etc.) corresponding to the conforming standards.

[0063] The synchronization control unit 208 controls the synchronization between the metadata received by the second input unit 203 and the image data received by the first input unit 202.

[0064] The CG generation unit 209 generates CG data to be synthesized with the image data received by the first input unit 202 by using the metadata received by the second input unit 203.

[0065] The CG compositing unit 210 combines the CG data generated by the CG generation unit 209 with the image data received by the first input unit 202 to generate composite image data.

[0066] Control processing Next, refer to Figure 3 The control processing of the camera device 100 in this embodiment is described.

[0067] The control unit 101 loads the program stored in the non-volatile memory 104 into the volatile memory 105 and executes the program, and controls the various components of the camera device 100 to achieve this. Figure 3 The processing.

[0068] In this embodiment, it is assumed that the camera device 100 is pre-configured to perform image processing at a predetermined frame rate and output the image data generated by the image processing along with metadata to the outside in real time. In this embodiment, the transmission standard (data format) of the image data and the type of metadata to be output to the outside are also pre-configured.

[0069] In step S300, the control unit 101 causes the camera unit 102 to begin image processing. The control unit 101 controls the camera unit 102 to capture moving images at a predetermined frame rate. Thereafter, the camera unit 102 continues capturing moving images until an indication to end the recording is given. The control unit 101 continuously performs exposure control and focusing control of the camera optical system during recording, for example, based on evaluation values ​​obtained from the image processing unit 103.

[0070] In step S301, the control unit 101 controls the image processing unit 103 to generate image data in a format to be output externally from the image signals of each frame output by the camera unit 102. The image processing unit 103 applies the required image processing to the image signals and generates image data in a format corresponding to the external output settings. The image processing unit 103 sequentially stores the generated image data into the volatile memory 105.

[0071] In step S302, the control unit 101 obtains the camera conditions and lens information set in steps S300 and S301 from the camera unit 102 and the image processing unit 103, and stores them in volatile memory 105 in association with the image data of the frame generated by the camera processing.

[0072] Metadata used for real-time compositing processing includes, for example, information related to the camera optical system (lens information) and the pose information of the camera device 100. For example, parameters related to image processing are camera conditions obtained through image processing of the read-out frames. Aperture values ​​and distortion parameters are lens information for frames during exposure, and shutter speed is camera conditions for frames where exposure will begin in the next camera processing step.

[0073] Even if this information can be obtained simultaneously, the frames associated with it are different. Therefore, the obtained information is stored in association with the individual frames scheduled for synchronization. This allows for the pre-output of a portion of metadata for image data to be output after a given image data at the appropriate time for outputting that image data.

[0074] Note that in this embodiment, the timecode is used as metadata for associating (synchronizing) the image data output from the first output unit 109 with the metadata output from the second output unit 110. The use of the timecode can specify which frame and metadata correspond to each other by the frame number of the timecode output from the second output unit 110 and the frame number of the timecode added to the image data output from the first output unit 109.

[0075] In step S303, control unit 101 causes metadata control unit 108 to begin generating metadata. Metadata control unit 108 reads the timecode of the metadata of the image data to be added to the volatile memory 105, and generates metadata based on the information corresponding to that timecode in the camera conditions and lens information stored in the volatile memory 105 in step S302. Furthermore, metadata control unit 108 reads information from the volatile memory 105 corresponding to the timecodes in the next and subsequent frames of the aforementioned timecodes, and generates metadata in association with these timecodes. Metadata control unit 108 stores the generated metadata in the volatile memory 105.

[0076] Note that in the metadata stored in volatile memory 105, the metadata associated with the same timecode added to the image data is initialized because it will no longer be used in the next and subsequent times. That is, the metadata stored in volatile memory 105 and updated sequentially is maintained and output until image data synchronized with the timecode of that metadata is output from the first output unit 109.

[0077] In step S304, the control unit 101 sequentially obtains the metadata generated by the metadata control unit 108 from the volatile memory 105.

[0078] In step S305, the control unit 101 determines whether the metadata obtained in step S304 is metadata output from the first output unit 109. When the metadata obtained in step S304 is metadata output from the first output unit 109, the control unit 101 supplies it to the first output unit 109, and when the metadata is not metadata output from the first output unit 109, the control unit 101 supplies it to the second output unit 110.

[0079] Note that it is assumed that the relationship between the type of metadata and the corresponding output unit is pre-registered in non-volatile memory 104, for example, in tabular form. Note that the metadata is associated with at least one of the first output unit 109 and the second output unit 110, and can be associated with both output units.

[0080] Only the types of metadata to be output to the first output unit 109 can be registered, and unregistered metadata can be supplied to the second output unit 110. The correspondence between metadata and output units can be changed by the user.

[0081] Steps S306 and S307 are the processing of the first output unit 109, and steps S308 and S309 are the processing of the second output unit 110.

[0082] In step S306, the first output unit 109 adds the metadata supplied from the control unit 101 to the motion image data stored in the volatile memory 105, and generates output data according to the output format. In this embodiment, the first output unit 109 generates SDI format output data with metadata added as auxiliary data.

[0083] In step S307, the first output unit 109 outputs the output data generated in step S306 to the outside.

[0084] In step S308, the second output unit 110 extracts the metadata to be output from the metadata supplied from the control unit 101. It is assumed that the metadata to be output is preset. The metadata to be output includes at least the metadata required for real-time synthesis of image data and CG data output from the first output unit 109, and metadata for associating (synchronizing) the output of the first output unit 109 with the output of the second output unit 110.

[0085] Note that the metadata required for real-time compositing can be notified to the camera device 100 from an external device (image processing device 200) performing the real-time compositing, or it can be pre-registered in the camera device 100. For example, when an application for real-time compositing is executed in the image processing device 200, the control unit 201 notifies the control unit 101 of information related to the required metadata. The control unit 101 can register the type of the metadata notified by the control unit 201 in the second output unit 110, or it can, for example, store the type of the metadata in volatile memory 105 so that the second output unit 110 can refer to it.

[0086] In step S309, the second output unit 110 stores the metadata extracted in step S308 into a UDP packet and outputs the metadata.

[0087] Subsequently, the above operations are performed continuously until a stop condition, such as a user instruction, is met. In this embodiment, metadata generation and output are performed in parallel with camera processing; however, this disclosure is not limited to this, and metadata can be generated and output only for the frames to be recorded in response to a recording instruction for recording image data obtained by camera processing. That is, the operations in step S301 and thereafter can begin for the image data to be recorded if a recording start instruction is received in step S300, and then the operations in steps S301 to S309 can be performed continuously until a recording stop instruction is received.

[0088] Figure 4 This shows the metadata to be output from the camera device 100 for each frame of image data when capturing moving images.

[0089] Metadata includes: management information, which is related to moving images; lens information, which is related to the state of the camera optical system during shooting; exposure information, which is related to the exposure conditions during shooting; and posture information related to the posture of the camera device 100 during shooting.

[0090] In addition to metadata synchronized with the image data output from the first output unit 109, this information also includes the following information ( Figure 4 The information (related to frames that have not yet been synchronized) is metadata scheduled to be synchronized in the future with image data that has not yet been output from the first output unit 109, and can be obtained by the control unit 101.

[0091] Notice, Figure 4 The data configuration is an example and may include other information or may not need to be included. Figure 4 Part of the data. Figure 4 Only one piece of information related to frames that have not yet been synchronized is shown, but it may include information related to multiple frames that have not yet been synchronized.

[0092] exist Figure 4 In the example, information related to the synchronization frame includes a time code as metadata used to associate the output of the first output unit 109 with the output of the second output unit 110. Information related to frames that have not yet been synchronized includes the time code of frames scheduled for synchronization. Note that this information is not limited to a time code and can be a frame number, or information corresponding to the time difference or the number of frames between the time code of the synchronization frame and the time code of the synchronization frame.

[0093] Figure 5 This is shown in JavaScript Object Notation (JSON) format. Figure 4 An example of the metadata shown.

[0094] This is an example of a description method that can be used when outputting metadata to the outside world, and other description methods such as Extensible Markup Language (XML) can be used to describe metadata.

[0095] For ease of description, line numbers will be used in the following text. Figure 5 In this embodiment, it is assumed that the image data in a frame with a "timecode" of "00:00:09.18" is... Figure 5 The data shown is output from the first output unit 109 in a timed synchronization manner from the second output unit 110.

[0096] First, the data in rows 2 to 19 is data synchronized with the image data output from the first output unit 109 (hereinafter referred to as the synchronization metadata section). The synchronization metadata section includes... Figure 4 The data showing all the information.

[0097] Next, rows 20 to 44 contain data related to frames that have not yet been output from the first output unit 109, but which have been obtained by the control unit 101. In this embodiment, the "time code" (hereinafter referred to as the synchronization scheduling metadata unit) is constructed as a data set "scheduled_data" as data to be synchronized with frames after "00:00:09.18".

[0098] The data in lines 22 to 31 of the Synchronization Schedule Metadata section is the data scheduled to be synchronized with the frame following the frame synchronized with the Synchronization Schedule Metadata section ("timecode" is "00:00:09.19") (hereinafter referred to as the Synchronization Schedule Metadata section one frame later). The data in lines 34 to 42 is the data scheduled to be synchronized with the frame two frames after the frame synchronized with the Synchronization Schedule Metadata section ("timecode" is "00:00:09.20") (hereinafter referred to as the Synchronization Schedule Metadata section two frames later).

[0099] exist Figure 5 In the example, the synchronization scheduling metadata section one frame later and the synchronization scheduling metadata section two frames later are only available in control unit 101. Figure 4 The status of the predetermined portion of the metadata shown is available, but it can be obtained. Figure 4 All the data in the file.

[0100] Next, refer to Figure 6 The control processing of the image processing apparatus 200 in this embodiment is described.

[0101] Figure 6The image processing device 200 is shown to receive image data and metadata output from the camera device 100, and the image processing device 200 uses the received image data and metadata to perform real-time compositing processing.

[0102] In the following text, it is assumed that the first output unit 109 of the camera device 100 and the first input unit 202 of the image processing device 200 are communicatively connected, and the second output unit 110 of the camera device 100 and the second input unit 203 of the image processing device 200 are communicatively connected. It is assumed that the SDI format image data output from the first output unit 109 of the camera device 100 is converted to HDMI format by a converter or the like before reaching the first input unit 202. It is assumed that at least the timecode of the metadata added to the SDI format image data can be correctly recognized at the receiving side after being converted to HDMI format.

[0103] The control unit 201 executes a real-time compositing processing application stored in the non-volatile memory 204 to control the components of the image processing device 200. Figure 6 The processing.

[0104] In step S600, when the first input unit 202 or the second input unit 203 receives data from an external device (camera device 100), the control unit 201 determines whether the data was input from the first input unit 202. If the control unit 201 determines that the input is from the first input unit 202, the process proceeds to step S601; otherwise (if the input is determined to be from the second input unit 203), the process proceeds to step S602.

[0105] In step S601, the control unit 201 receives image data with added metadata through the first input unit 202 and stores the received image data and metadata in the volatile memory 205.

[0106] In step S602, the control unit 201 receives metadata through the second input unit 203 and stores the received metadata in the volatile memory 205. At this time, Figure 5The information in the synchronization metadata section and the synchronization scheduling metadata section shown is separated into separate areas guaranteed for each time code and stored in volatile memory 205. When a portion of the metadata associated with a received time code has already been received and stored in volatile memory 205 at the timing of previous metadata reception, the information in the existing metadata section is updated or added using the data received this time. Note that the areas guaranteed for each time code in volatile memory 205 include a ring buffer, and when information associated with a certain time code's area is used for real-time synthesis processing of image data received by the first input unit 202, that area is initialized as an unused area. The ring buffer ensures an area that can adequately handle the difference in data reception timing between the first input unit 202 and the second input unit 203.

[0107] Through the above processing, even if the metadata output from the second output unit 110 of the camera device 100 disappears along the communication path and cannot be received by the second input unit 203 of the image processing device 200, part or all of the synchronization metadata can be recovered by using the previously received metadata.

[0108] In step S603, control unit 201 sends an instruction to synchronization control unit 208 to begin synchronization processing between the image data received by first input unit 202 in step S601 and the metadata received by second input unit 203 in step S602. Synchronization control unit 208, in response to the instruction received from control unit 201, begins synchronization processing by reading the timecode of the image data stored in volatile memory 205 in step S601 and comparing this timecode with the timecode of the metadata stored in volatile memory 205 in step S602. Then, synchronization control unit 208 synchronizes the image data with the metadata by reading the metadata with the matching timecode.

[0109] exist Figure 6 In the process, even if the metadata synchronized with the image data cannot be received, the image data and metadata can be synchronized with each other because part or all of the metadata synchronized with the image data is stored in the volatile memory 205 through metadata received multiple times previously.

[0110] Note that the synchronization method described above is an example, and synchronization can be performed by other methods.

[0111] Next, the control unit 201 supplies a frame of image data to the CG compositing unit 210 from the image data and metadata synchronized by the synchronization control unit 208, and supplies the metadata corresponding to the image data to the CG generation unit 209.

[0112] In step S604, the CG generation unit 209 generates CG data to be composited with image data in the CG compositing unit 210 based on metadata supplied from the control unit 201. The CG generation unit 209 generates natural CG data for compositing with image data by rendering pre-prepared CG model data based on pose information and lens information of the camera device 100 during shooting. The CG generation unit 209 supplies the generated CG data to the CG compositing unit 210 via the control unit 201.

[0113] In step S605, the CG compositing unit 210 combines the image data of one frame supplied from the control unit 201 in step S603 with the CG data generated by the CG generation unit 209 in step S604 to generate a frame of composite image data. The CG compositing unit 210 then supplies the composite image data to the control unit 201.

[0114] In step S606, the control unit 201 stores the composite image data of one frame generated by the CG compositing unit 210 into the video memory area of ​​the volatile memory 205 and displays it on the display unit 207.

[0115] Execute the above continuously Figure 6 The processing continues until the termination conditions of the real-time synthesis process are met.

[0116] Next, refer to Figures 7A to 7D The camera device 100 described in this embodiment Figure 3 The control processing involves operations from camera processing to image data output processing.

[0117] In the following text, for ease of description, the description will be given with reference to the symbols 1 to 5 for the separation of each vertical synchronization signal (VD) when reading the image signal from the image sensor of the camera unit 102.

[0118] In the image processing of a certain frame N (N is a natural number), at the timing of VD1, at T701, the image sensor performs the exposure (hereinafter referred to as sensor exposure).

[0119] At T702, the image signal is read from the image sensor exposed at T701 to obtain a RAW (raw) image signal. For the RAW image signal obtained at T702, developing processing is performed at T703, post-processing at T704, and the developed image data and metadata added to the image data are output externally at T705. Here, the metadata to be output externally includes... Figure 4 Metadata of all information in the synchronization frame.

[0120] The operation of T701 to T705 is for Figure 7A The frames N+1 and N+2 shown are similar, and the same applies to frames N-1 and N-2, where frames N-1 and N-2 are... Figure 7A The frame preceding frame N, which is not shown in the image.

[0121] The camera device 100 uses the RAW image signal obtained by exposure through the sensor to perform various types of image processing (such as detection of exposure and color information, detection of the subject, and detection of motion vector amount for blur correction).

[0122] exist Figure 7B In the metadata acquisition process 700A, during the timing of VD1, at T711, the shooting conditions for performing a sensor exposure of frame N+2 during the timing of VD2, immediately following the sensor exposure of frame N+1, are obtained as pre-exposure metadata for the (N+2)th frame. The pre-exposure metadata includes, for example, shutter speed. During the timing of VD1, the shooting conditions and lens information corresponding to the currently being exposed frame N+1 are obtained as exposure metadata for the (N+1)th frame. The exposure metadata includes, for example, information related to distortion parameters and manufacturing errors, as well as the position information of the imaging device 100. Furthermore, during the timing of VD1, the shooting conditions and lens information corresponding to the Nth frame undergoing readout processing are obtained as readout metadata for the Nth frame. The readout metadata includes, for example, parameters related to the processing performed during readout, such as ISO sensitivity. Furthermore, although not shown, the shooting conditions and lens information corresponding to the (N-1)th frame in which image processing is performed at the timing of VD1 are obtained as metadata for the (N-1)th frame in image processing. This metadata includes, for example, white balance information. Furthermore, although not shown, the shooting conditions and lens information corresponding to the (N-2)th frame in which post-processing is performed at the timing of VD1 are obtained as metadata for the (N-2)th frame in post-processing. This metadata includes, for example, electronic image stabilization information.

[0123] The metadata acquisition process performed at T711 has been described above, and similar metadata acquisition processes described at T711 are performed at T712, T713, T714 and T715 for frames +1, +2, +3 and +4 relative to T711.

[0124] After the metadata acquisition processing of each frame is performed at the time intervals of T711 to T715 as described above, the metadata output processing 700B of T721 to T724 is performed.

[0125] The metadata output processing for frame N at T721 is described as an example. This outputs the metadata already obtained at T721 (metadata 700C to be output, described below T721) as a UDP packet. Note that the timing of the metadata output at T721 is synchronized with the timing of the output of the image data of frame N at T705, and even if the metadata related to frame N is output at or after this timing, real-time synthesis processing of the image data of frame N cannot be performed. Therefore, in frames where the metadata synchronized with the image data of frame N is output as a UDP packet, no metadata related to frame N is output to the outside. That is, the metadata to be output to the outside at T722 is only the metadata related to frames at frame N+1 and later.

[0126] Next, it will be through Figures 7C to 7D The operation is described as follows: the metadata output from the second output unit 110 of the camera device 100 to the outside disappears along the communication path and cannot be received by the second input unit 203 of the image processing device 200.

[0127] For example, consider the case where the metadata related to frame N that was output to the outside at T721 has disappeared.

[0128] When the metadata output to the outside at T721 disappears, the image processing device 200 cannot receive the metadata related to frame N. However, if the metadata related to frame N output to the outside at T723 or T724 before T721 can be received, a portion of the metadata related to frame N (the metadata output at T731, T732, and T733) can be received, and therefore a portion of the received metadata related to frame N can be used for real-time compositing processing. This can reduce the rendering of unnatural images.

[0129] Note that in this embodiment, for ease of understanding, an example of outputting a portion of the metadata required for real-time compositing processing from the second output unit 110 has been described. However, the problem that the metadata output from the camera device 100 cannot be used by external devices is not limited to real-time compositing processing. Therefore, to solve the problem of this embodiment, it is necessary to be able to repeatedly output part or all of the multiple types of metadata generated during recording until image data synchronized with the metadata is output.

[0130] Second Embodiment Next, the second embodiment will be described.

[0131] In the second embodiment, except Figure 7C In addition to the metadata output processing 700B, a processing 800B for immediately outputting metadata is added, and other device configuration and control processing is similar to that in the first embodiment.

[0132] In the second embodiment, an example will be described as follows: not only at the timing when metadata synchronized with the frame is output from the first output unit 109 as in the first embodiment, but also at the timing when the control unit 101 obtains the metadata of the frame scheduled for synchronization from the camera unit 102 or the image processing unit 103, the metadata is sequentially output from the second output unit 110 in association with the frame scheduled for synchronization.

[0133] Next, refer to Figures 8A to 8D The imaging device 100 described in this embodiment performs the following operations: Figure 3 The control processing involves operations from camera processing to image data output processing. Note that in... Figures 8A to 8D In, with Figures 7A to 7D The processing in the middle is similar to the processing by and Figures 7A to 7D The same reference numerals are used in the accompanying drawings, and descriptions will be omitted.

[0134] Will be described in Figure 8B This is an example of the VD period (VD2) of sensor exposure in frame (N+2) during metadata acquisition processing.

[0135] Figure 8B VD2 in the above case is as follows: it can obtain the exposure metadata of frame N+2 at time T801, obtain the readout metadata of frame N+1 at time T802, obtain the display metadata of frame N at time T803, and obtain the post-processing metadata of frame N-1 at time T804.

[0136] When the exposure metadata for frame N+2 can be obtained at timing T801, control unit 101 immediately outputs the metadata and thus obtains all exposure metadata from volatile memory 105 as metadata of the same type associated with the (N-2)th frame and subsequent frames scheduled to be synchronized at VD2. Then, at T821, a set 811 of exposure metadata for frames N+1 and N+2, including the exposure metadata for frame N+1 obtained at timing T801, is output from second output unit 110.

[0137] Similar to the above processing, when the read metadata, display metadata, and post-processing metadata of frame N+1 can be obtained at timings T802, T803, and T804 respectively, all metadata related to the frame scheduled for synchronization N-2 and subsequent frames is obtained from volatile memory 105. T812, T813, and T814 are collections of the metadata obtained at T802, T803, and T804, and are output from the second output unit 110 at T822, T823, and T824.

[0138] According to the second embodiment described above, in addition to the processing of the first embodiment, when a portion of the metadata of multiple types generated from the output processing of camera processing to image data can be obtained, the obtained metadata of the same type as the obtained metadata is immediately output as a set, thereby improving robustness against packet loss.

[0139] Note that in this embodiment, an example of outputting metadata obtained for each process separated by the vertical sync signal (VD), such as during exposure or development, has been described externally; however, this disclosure is not limited thereto. For example, external output in more detailed data units may also be possible each time a T-value or distortion parameter is available.

[0140] In this embodiment, in addition to the metadata output processing synchronized with the frame in the first embodiment, the processing of outputting metadata immediately at the point when the metadata is available has been described, but the metadata output processing synchronized with the frame is not required.

[0141] Third Embodiment Next, the third embodiment will be described.

[0142] In the first and second embodiments, examples of outputting all metadata that is available up to the output of image data for a certain frame have been described. However, when all metadata is output, the processing load at the camera device 100 and the image processing device 200 increases. Therefore, in the third embodiment, an example will be described where the frequency of outputting metadata changes according to the importance level of the metadata in the cases described in the first and second embodiments.

[0143] In the third embodiment, the frequency of output metadata is increased as the importance level of the real-time synthesis processing performed by the image processing device 200 increases.

[0144] For example, even if the pose information of the camera device 100 is lost in a frame, there is a high probability that an unnatural image will be rendered. Therefore, such metadata with a high level of importance can be output for each frame over multiple frames as in the first embodiment, or immediately at the time when the information is available as in the second embodiment, thereby improving robustness to packet loss.

[0145] Distortion parameters and T-values ​​(F-values) are also exemplified as data that are highly likely to render unnatural images even when lost in a single frame. However, compared to the pose information of the camera device 100, changes in image brightness are less likely to be visually identifiable, thus making them less important than the pose information of the camera device 100.

[0146] Furthermore, since the frame rate and resolution are not information that changes for each frame and do not need to be output for each frame, the size of the transmitted data can be prevented from increasing by limiting the number of outputs (e.g., by omitting frames to be output at intervals or outputting only a predetermined number of frames when there are changes in the information).

[0147] Therefore, in the third embodiment, as the importance level of the metadata decreases in the order of the camera device 100's posture information, the information related to the brightness of the image, and the information related to camera processing, the frequency of outputting metadata is reduced accordingly.

[0148] According to the third embodiment described above, the processing load of the camera device 100 and the image processing device 200 can be reduced by not outputting all metadata that can be obtained up to the point of outputting image data of a certain frame to the outside.

[0149] According to the embodiments described above, a portion of the metadata of multiple types generated in the camera device from the image processing to the output processing of image data is output from an output unit different from the output unit that outputs image data. This allows an external device to perform desired processing using a portion of the metadata previously received from the camera device, even if, for example, the format of the image data is converted and the metadata added to the image data is no longer continuously output.

[0150] According to this disclosure, the generation of unnatural images due to packet loss can be reduced. [Other Embodiments]

[0151] Embodiments of the present invention can also be implemented by providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU) or microprocessor unit (MPU) of the system or device) reads and executes the computer program.

[0152] While this disclosure has been described with reference to exemplary embodiments, it should be understood that this disclosure is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. A camera device, comprising: The first generation unit is used to generate image data at a predetermined period; The second generation unit is used to generate metadata related to the processing up to the generation of the image data; The first output unit is used to add the metadata to the image data to output the image data to the outside. as well as The second output unit is used to output predetermined information generated in the metadata up to the point of outputting the image data multiple times before the first output unit outputs the image data.

2. The camera device according to claim 1, further comprising: An extraction unit is configured to extract the predetermined information that the second output unit intends to output to the outside from the metadata generated by the second generation unit before the image data is output by the first output unit.

3. The camera device according to claim 1, in, After the first output unit outputs the image data with the added metadata to the outside, the second output unit does not output any predetermined information related to the image data to be output after the image data.

4. The camera device according to claim 1, in, The metadata added to and output by the first output unit includes the same information as the predetermined information associated with the image data.

5. The camera device according to claim 1, in, The predetermined information includes information for synchronizing the image data to be output by the first output unit with the predetermined information to be output by the second output unit.

6. The camera device according to claim 1, in, The second generation unit generates first metadata synchronized with the first image data to be output by the first output unit, and second metadata scheduled to be synchronized with the second image data to be output after the first image data. The first output unit adds the first metadata to the first image data to output the first image data to the outside, and When the first metadata is added to the first image data to output the first image data to the outside, the second output unit outputs the second metadata.

7. The camera device according to claim 6, in, The second metadata includes predefined information related to the second image data.

8. The camera device according to claim 1, in, The second output unit outputs the predetermined information to the outside when it receives the predetermined information generated by the second generation unit.

9. The camera device according to claim 8, in, The second output unit outputs to the outside a set of first predetermined information related to the first image data generated by the second generation unit before the first output unit outputs the first image data, and second predetermined information related to the second image data of the same type as the first predetermined information generated by the second generation unit before the first output unit outputs the second image data, wherein the second image data is output before the first image data.

10. The camera device according to claim 1, in, The second output unit uses the predetermined information to change the frequency of outputting the predetermined information according to the importance level of the process to be performed externally.

11. The camera device according to claim 10, in, The second output unit increases the frequency of outputting the predetermined information as the importance level of the process to be performed externally increases.

12. The camera device according to claim 11, in, The predetermined information includes the posture information of the camera device, information related to image brightness, and information related to camera processing, as well as... The frequency decreases in the following order: the posture information of the camera device, the brightness information of the image, and the information related to the camera processing.

13. The camera device according to claim 1, further comprising a camera unit, the camera unit being configured to capture images at a predetermined frame rate. in, The image data is motion image data. The metadata refers to the information associated with the motion image data to be generated for each frame of the motion image data, and The metadata includes: metadata prior to exposure of the camera unit, metadata during exposure of the camera unit, metadata during readout of the image signal from the camera unit, metadata during development of the image signal, and metadata related to post-processing of the developed image data.

14. The camera device according to claim 1, in, The predetermined information is information to be used for processing to be performed externally, and The process to be performed externally is the real-time synthesis of the image data and CG data.

15. The camera device according to claim 13, in, The predetermined information includes the timecode or frame number of the image data frame scheduled to be output by the first output unit.

16. The camera device according to claim 13, in, The predetermined information includes information corresponding to the following: the time difference with the time code of the image data to be output from the first output unit or the number of frames with the frame number.

17. The camera device according to any one of claims 1 to 16, in, The first output unit conforms to the Serial Digital Interface (SDI) standard, and The second output unit conforms to the User Datagram Protocol (UDP) standard.

18. A control method for a camera device, the control method comprising the following steps: Image data is generated at a predetermined cycle; Generate metadata related to the processing up to the generation of the image data; Add the metadata to the image data to output the image data from the first output unit to the outside; as well as Before the image data is output from the first output unit, predetermined information related to the image data generated in the metadata up to the point of outputting the image data is output from the second output unit multiple times.

19. A computer-readable storage medium storing a program for causing a computer to perform the control method according to claim 18.

20. A computer program product comprising a program for causing a computer to perform the control method according to claim 18.

Images