IMAGING APPARATUS, DISTRIBUTION SYSTEM, DISTRIBUTION METHOD, AND PROGRAM

Abstract

To simultaneously distribute image data of a plurality of streams by using a photographing device having limitation on the number of hardware encoders.SOLUTION: A photographing device 10 distributes image data of a plurality of streams, and comprises: a photographing processing section 15 (an example acquisition means) that acquires photographed image data photographed by an imaging unit 101 (an example of photographing means); a determination section 14 (an example of determination means) that determines a method for encoding the acquired photographed image data according to an encoding parameter for each of the streams; an encoding processing section 18 (an example of generation means) that generates streaming image data for each of the streams by using the determined encoding method; and a communication section 11 (an example of distribution means) that distributes the plurality of pieces of generated streaming image data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a photographing device, a distribution system, a distribution method, and program Regarding. [Background technology]

[0002] In recent years, technology that enables streaming distribution of images captured by an image capture device using a distribution service on the Internet has become widespread. In addition, a system that enables streaming distribution by directly connecting an image capture device to the Internet without going through a terminal such as a PC or a smartphone is also known.

[0003] Furthermore, in such streaming distribution, a technique is known for simultaneously distributing images of multiple streams with different image quality so that the viewer's terminal can select the stream to receive. In this case, the processing power of the distribution device for encoding the image data to be distributed is important. For example, Patent Document 1 discloses a technique for encoding video data by allocating processing to a software encoder and a hardware encoder, and synthesizing the encoded data into one stream. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional methods had the problem that it was difficult to simultaneously distribute image data of multiple streams because the encoding process for each stream could not be performed in parallel in devices with a limited number of hardware encoders. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, the present invention according to claim 1 provides an image capturing device that distributes a plurality of streams of image data, the image capturing device comprising: an acquisition unit that acquires captured image data captured by the image capturing unit; At least one of a hardware encoder and a software encoder;a determining means for determining an encoding method for the acquired photographed image data in accordance with the encoding parameters for each stream; The above Encoding Method Based on , using the hardware encoder for a stream corresponding to the encoding parameter having a large processing load among the plurality of streams, and using the software encoder for the other streams; a generating means for generating streaming image data for each of the streams; The above and a distribution unit that distributes streaming image data. Effect of the Invention

[0006] According to the present invention, it is possible to simultaneously distribute a plurality of streams of image data using an imaging device that has a limited number of hardware encoders. [Brief description of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating an example of a system configuration of a distribution system according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of an outline of processing of a distribution system according to the first embodiment. [Diagram 3] 1 is a diagram illustrating an example of a hardware configuration of an imaging apparatus according to a first embodiment. [Figure 4] FIG. 1 is a diagram illustrating an example of a hardware configuration of a communication terminal according to a first embodiment; [Diagram 5] 1 is a diagram illustrating an example of a functional configuration of a distribution system according to a first embodiment. [Figure 6] FIG. 2 is a conceptual diagram illustrating an example of an encoding parameter management table according to the first embodiment. [Figure 7] FIG. 4 is a sequence diagram showing an example of an image delivery process in the delivery system according to the first embodiment. [Figure 8] 5 is a flowchart showing an example of an encoding process of image data according to the first embodiment. [Figure 9] 5A and 5B are conceptual diagrams showing another example of the encoding parameter management table according to the first embodiment. [Figure 10]10 is a flowchart showing an example of an encoding process of image data according to a modified example of the first embodiment. [Figure 11] FIG. 13 is a conceptual diagram for explaining an example of an encoding parameter management table when a target encoder is changed in the imaging device according to a modified example of the first embodiment. [Figure 12] FIG. 11 is a diagram illustrating an example of a system configuration of a distribution system according to a second embodiment. [Figure 13] FIG. 11 is a diagram illustrating an example of an outline of processing of a distribution system according to a second embodiment. [Figure 14] FIG. 11 is a diagram illustrating an example of a functional configuration of a distribution system according to a second embodiment. [Figure 15] FIG. 11 is a sequence diagram showing an example of an image delivery process in a delivery system according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated description will be omitted.

[0009] First embodiment System Configuration Fig. 1 is a diagram showing an example of the system configuration of a distribution system according to the first embodiment. The distribution system 1 shown in Fig. 1 is a system capable of streaming image data captured and acquired by an imaging device 10 to a communication terminal 50 used by a user.

[0010] 1, the distribution system 1 is configured by a photographing device 10 and a communication terminal 50. The photographing device 10 and the communication terminal 50 are communicatively connected via a communication network 5. The communication network 5 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), or the like. Note that the communication network 5 may include not only wired communication but also wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity (registered trademark)), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0011] The photographing device 10 is a digital camera capable of photographing a subject and acquiring a photographed image. The photographing device 10 is, for example, a special digital camera for obtaining a 360° panoramic image. The photographing device 10 may be a general digital camera (such as a single-lens reflex camera, a compact digital camera, a security camera, or a video (video) camera). The photographing device 10 performs streaming distribution of image data relating to the photographed image via a communication network 5 such as the Internet. The photographed image may be a moving image or a still image, or may be both a moving image and a still image. The photographed image may also include sound together with the image.

[0012] The communication terminal 50 is a terminal device such as a tablet terminal used by a user. The communication terminal 50 performs data communication with the photographing device 10 by accessing a communication network 5 such as the Internet. The communication terminal 50 can receive streaming image data distributed from the photographing device 10. Note that the communication terminal 50 is not limited to a tablet terminal, and may be a PC (Personal Computer), a smartphone, a mobile phone, a wearable terminal, a game machine, a video conference (television conference) terminal, an IWB (Interactive White Board: an electronic whiteboard function capable of mutual communication), a telepresence robot, or the like.

[0013] Overview Here, an overview of the processing of the distribution system according to the first embodiment will be described. Note that Fig. 2 is a simplified overview of the distribution system according to the first embodiment, and the details of the functions and the like realized by the distribution system 1 will be described with reference to the drawings and the like described later.

[0014] 2 is a diagram showing an example of an outline of processing in the distribution system according to the first embodiment. The distribution system 1 is a system that can distribute image data of a plurality of streams when the image data captured and acquired by the image capture device 10 is distributed in a plurality of streams by assigning an encoder of the image capture device 10 to each stream and processing the data in parallel.

[0015] First, the photographing device 10 acquires photographed image data to be the subject of image distribution. Next, the photographing device 10 prepares a plurality of streams with different image quality so that the communication terminal 50 can receive data with a desired image quality according to, for example, the state of the network bandwidth of the communication network 5. The photographing device 10 determines an encoding method for the image data for each stream based on the number of streams to be streamed and the encoding parameters for each stream. The encoding parameters are information indicating the image quality of the image data to be processed by the encoder. Then, the photographing device 10 performs encoding processing for each frame (e.g., frame X) of the photographed image data using the determined encoding method.

[0016] 2, the image capturing device 10 performs encoding processing on stream 1 using a hardware encoder (HW), and performs encoding processing on streams 2 and 3 using a software encoder (SW). Frame X, which is one frame of captured image data, undergoes a predetermined image processing and is input in parallel to three encoders, and each encoder performs encoding processing in parallel according to the set encoding parameters.

[0017] Then, the photographing device 10 distributes the encoded streaming image data as independent streams to the communication terminal 50 via the communication network 5. In this case, for example, the streaming image data distributed in stream 1 is high-quality data, the streaming image data distributed in stream 2 is medium-quality data, and the streaming image data distributed in stream 3 is low-quality data.

[0018] When multiple streams of image data with different image quality are distributed simultaneously, the distribution device often uses a dedicated hardware encoder. However, for example, in a photographing device such as a normal digital camera, which has physical constraints such as device size or layout within the device, or constraints such as price, the number of hardware encoders installed is limited, making it difficult to distribute multiple streams simultaneously. Therefore, when distributing image data using multiple streams with different image quality, the distribution system 1 assigns either a hardware encoder or a software encoder provided in the photographing device 10 to each stream according to the encoding parameters for each stream, and performs encoding processing. As a result, the distribution system 1 can distribute multiple streams of image data with different image quality using the photographing device 10, which has a limited number of hardware encoders.

[0019] ●Hardware configuration Next, the hardware configurations of the photographing device 10 and the communication terminal 50 in the embodiment will be described with reference to Figures 3 and 4. Note that components may be added or deleted from the hardware configurations shown in Figures 3 and 4 as necessary.

[0020] ○Hardware configuration of the imaging device○ First, the hardware configuration of the imaging device 10 will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the hardware configuration of the imaging device according to the first embodiment. In the following, the imaging device 10 is an omnidirectional (all-directional) imaging device using two imaging elements, but the number of imaging elements may be two or more. Furthermore, the imaging device 10 does not necessarily have to be a device dedicated to omnidirectional imaging, and an omnidirectional imaging unit may be attached to a normal digital camera, smartphone, or the like to have substantially the same functions as the imaging device 10.

[0021] As shown in FIG. 3, the photographing device 10 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a hardware encoder 106, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an external storage I / F 116, an input / output I / F 117, a network I / F 118 and an acceleration / direction sensor 119.

[0022] Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to the wide-angle lenses. The imaging elements 103a and 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image captured by the wide-angle lenses 102a and 102b into image data of an electrical signal and outputs the image data, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters required for the operation of the imaging elements are set.

[0023] The imaging elements 103a and 103b are each connected to the image processing unit 104 via a parallel I / F bus. Also, the imaging elements 103a and 103b are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus line 110. Furthermore, the bus line 110 is connected to the ROM 112, the SRAM 113, the DRAM 114, the operation unit 115, the external storage I / F 116, the input / output I / F 117, the network I / F 118, and the acceleration / direction sensor 119.

[0024] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus. Then, the image processing unit 104 performs a predetermined process on each image data, and then synthesizes the image data to create equirectangular projection image data.

[0025] The imaging control unit 105 generally sets commands and the like in the register groups of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the register groups of the imaging elements 103a and 103b and send it to the CPU 111.

[0026] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button of the operation unit 115 is pressed. The photographing device 10 may also have a preview display function on a display and a function corresponding to the display of still images and moving images. In the case of moving images, the image data is output from the imaging elements 103a and 103b continuously at a predetermined frame rate (frames / minute).

[0027] As described later, the imaging control unit 105 also functions as a synchronization control means for synchronizing the output timing of image data from the imaging elements 103a and 103b in cooperation with the CPU 111. In this embodiment, the imaging device 10 is not provided with a display, but a display unit may be provided.

[0028] The hardware encoder 106 is an encoder that performs encoding (compression) using an encoding method such as H.264 or MPEG-4, etc. The hardware encoder 106 has a limit to the number of encoder processes that can be executed simultaneously.

[0029] The microphone 108 collects sound from the surrounding environment of the image capture device 10 and converts the collected sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs a predetermined process on the sound data. The microphones 108 constitute a sound collection means for collecting sound from the surrounding environment. Although FIG. 3 shows only the case where the microphone 108 is built into the image capture device 10, the microphone 108 may be externally attached to the image capture device 10. Furthermore, the number of microphones 108 is not limited to one, and the image capture device 10 may have a plurality of microphones having a predetermined arrangement. The microphone 108 may be an Ambisonics microphone.

[0030] The CPU 111 controls the overall operation of the image capturing device 10 and executes necessary processing. The ROM 112 is a non-volatile memory that stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are volatile memories that are used as a work area for the CPU 111 and store programs executed by the CPU 111, data in the middle of processing, and the like. In particular, the DRAM 114 stores, for example, image data in the middle of processing by the image processing unit 104 and data of a processed equirectangular projection image.

[0031] The operation unit 115 is a general term for operation buttons such as a shutter button. A user operates the operation unit 115 to input various shooting modes, shooting conditions, and the like. The external storage I / F 116 controls reading and writing (storing) data from and to an external storage such as a recording medium. The external storage may store encoded streaming image data as an image file (moving image file).

[0032] The input / output I / F 117 is an interface for connecting to an external device in order to input / output various data or information between the external device and the input / output I / F 117. The input / output I / F 117 may be wireless or wired, and may be configured, for example, by a Universal Serial Bus (USB) I / F. The network I / F 118 is an interface for performing data communication via the communication network 5. The network I / F 118 may be configured by a wired Network Interface Card (NIC), a wireless NIC, or a USB I / F.

[0033] The acceleration / azimuth sensor 119 calculates the orientation of the image capturing device 10 from the earth's magnetism and outputs the orientation information. The orientation information is an example of related information (metadata) according to the Exif (Exchangeable image file format), and is used for image processing such as image correction of captured images. The related information includes data such as the date and time of capturing the image and the data size of the image data.

[0034] The photographing device 10 is not limited to an all-sphere (omnidirectional) photographing device, and may be a photographing device such as a normal digital camera, a security camera, or a video camera. In this case, the imaging unit 101 as a photographing means is composed of a pair of a lens and an imaging element.

[0035] ○Hardware configuration of communication terminal○ Next, the hardware configuration of the communication terminal 50 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the hardware configuration of the communication terminal according to the embodiment. The communication terminal 50 includes a CPU 501, a ROM 502, a RAM 503, an input device 504, a display 505, an external storage I / F 506, an input / output I / F 507, and a network I / F 508.

[0036] The CPU 501 controls the overall operation of the communication terminal 50. The ROM 502 is a non-volatile memory that stores programs used to drive the CPU 501. The RAM 503 is a volatile memory used as a work area for the CPU 501. The input device 504 is a type of input means that accepts operation input by a user. The input device 504 is, for example, a keyboard, a mouse, a touch panel, a button, a dial, or a microphone for voice input.

[0037] The display 505 is a type of display means such as liquid crystal or organic electroluminescence (EL) that displays various information such as a cursor, a menu, a window, characters, an icon, or an image. The external storage I / F 506 controls reading or writing (storing) data from or to an external storage such as a recording medium. The input / output I / F 507 is an interface for connecting to an external device in order to input / output various data or information between the external device and the display. The network I / F 508 is an interface for performing data communication via the communication network 5. The communication terminal 50 also includes a bus line 510. The bus line 510 is an address bus or a data bus for electrically connecting each component such as the CPU 501.

[0038] Each of the above programs may be recorded in a computer-readable recording medium as an installable or executable file and distributed. Examples of the recording medium include a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), a Blu-ray (registered trademark) disc, an SD card, and a USB memory. The recording medium may be provided domestically or internationally as a program product. For example, the imaging device 10 realizes the distribution method according to the present invention by executing the program according to the present invention.

[0039] ●Function configuration Next, the functional configuration of the distribution system according to the first embodiment will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram showing an example of the functional configuration of the distribution system according to the embodiment. Fig. 5 shows devices and terminals shown in Fig. 1 that are related to processing or operations to be described later.

[0040] ○Functional configuration of the imaging device○ First, the functional configuration of the photographing device 10 will be described with reference to Fig. 5. The photographing device 10 has a communication unit 11, a reception unit 12, an input / output unit 13, a judgment unit 14, a photographing processing unit 15, a parameter setting unit 16, an image processing unit 17, an encoding processing unit 18, and a storage / reading unit 19. Each of these units is a function or a means for performing a function that is realized by operating any of the components shown in Fig. 3 according to an instruction from the CPU 111 according to a program loaded from the ROM 112 or SRAM 113 onto the DRAM 114. The photographing device 10 also has a storage unit 1000 constructed by the ROM 112, SRAM 113, or DRAM 114 shown in Fig. 3.

[0041] 3 and the network I / F 118, and has a function of transmitting and receiving various data (or information) to and from the communication terminal 50 via the communication network 5. The communication unit 11 distributes, for example, image data captured and acquired by the imaging processing unit 15 to the communication terminal 50 via the communication network 5. The communication unit 11 is an example of a distribution means.

[0042] The reception unit 12 is realized by an instruction from the CPU 111 shown in FIG. 3 and an input means such as the operation unit 115, and is a function of receiving various inputs from a user. The judgment unit 14 is realized by an instruction from the CPU 111 shown in FIG. 3 and is a function of making various judgments. The input / output unit 13 is realized by an instruction from the CPU 111 shown in FIG. 3 and an input / output I / F 117, and is a function of inputting and outputting various data or information between an external device. The judgment unit 14 determines an encoding method for the captured image data, for example, according to an encoding parameter for each stream. Specifically, the judgment unit 14 determines whether the captured image data for each stream is to be encoded by a hardware encoder or a software encoder. The judgment unit 14 is an example of a determination means.

[0043] 3, as well as the imaging unit 101, image processing unit 104, imaging control unit 105, microphone 108, and sound processing unit 109, and has a function of acquiring captured image data by capturing images of subjects, scenery, etc. The imaging processing unit 15 is an example of an acquiring means.

[0044] The parameter setting unit 16 is realized by a command from the CPU 111 shown in Fig. 3, and is a function for setting encoding parameters for each stream. The parameter setting unit 16 is an example of a setting means. The image processing unit 17 is realized by a command from the CPU 111 shown in Fig. 3, and is a function for performing various processes on the captured image data acquired by the shooting processing unit 15.

[0045] The encoding processing unit 18 is realized by an instruction from the CPU 111 shown in Fig. 4, and is a function that performs encoding processing of the captured image data using a determined encoding method. The encoding processing unit 18 performs encoding processing using the determined encoding method, for example, to generate streaming image data for each stream. When the encoding method is a method using a hardware encoder, the encoding processing unit 18 performs encoding processing of the captured image data using the hardware encoder 106 shown in Fig. 3. On the other hand, when the encoding method is a method using a software encoder, the encoding processing unit 18 performs encoding processing of the captured image data using a software encoder that is operated by the CPU 111 shown in Fig. 3 executing dedicated software. The encoding processing unit 18 is an example of a generating means.

[0046] The storage / reading unit 19 is realized by instructions from the CPU 111 shown in FIG. 3, and has the function of storing various data in the storage unit 1000 or reading various data from the storage unit 1000.

[0047] Encoding parameter management table Fig. 6 is a conceptual diagram showing an example of an encoding parameter management table according to the first embodiment. An encoding parameter management DB 1001 constituted by an encoding parameter management table as shown in Fig. 6 is constructed in the storage unit 1000. The encoding parameter management table stores and manages an encoding method for each stream using the encoding processing unit 18. The encoding parameter management table shown in Fig. 6 is an example of a case where the imaging device 10 is provided with one hardware encoder 106 as shown in Fig. 3. In this case, the hardware encoder 106 can simultaneously execute only one encoding process.

[0048] In the encoding parameter management table, encoding parameters and a target encoder indicating an encoder used in encoding are stored and managed in association with each other for each stream that distributes image data. Among these, the encoding parameters indicate information indicating the image quality of the image data for each stream. The information indicating the image quality includes, for example, the resolution and frame rate of the image data. The encoding parameters may include the bit rate of the image data. In addition, when the image data is moving image data, the encoding parameters are treated as information indicating the image quality as well as the sound quality of the audio data. Furthermore, the encoding parameters may include an encoding method used in the encoding process. For example, the encoding method may be JPEG (Joint Photographic Experts Group) or its evolved system (JPEG2000) or GIF (Graphics Interchange Format) for still images, or MPEG (Moving Picture Experts Group) or its evolved systems (H.264, H.265) for moving images.

[0049] The target encoder is assigned, for example, to the hardware encoder 106 in the order of the largest encoding parameter processing load among streams 1 to 3. In the encoding parameter management table, for example, the hardware encoder 106 is assigned to stream 1 having the highest resolution, and the software encoder is assigned to streams 2 and 3.

[0050] ○Functional configuration of communication terminal○ Next, the functional configuration of the communication terminal 50 will be described with reference to Fig. 5. The communication terminal 50 has a communication unit 51, a reception unit 52, a display control unit 53, a determination unit 54, and a storage / readout unit 59. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 4 operates according to an instruction from the CPU 501 in accordance with a program loaded on the RAM 503. The communication terminal 50 also has a storage unit 5000 constructed by the ROM 502 or an external storage shown in Fig. 4.

[0051] The communication unit 51 is realized by commands from the CPU 501 shown in FIG. 4 and the network I / F 508, and has a function of transmitting and receiving various data or information to and from the image capturing apparatus 10 via the communication network 5.

[0052] The reception unit 52 is realized by an instruction from the CPU 501 shown in Fig. 4 and the input device 504, and has a function of receiving various input operations from the user. The display control unit 53 is realized by an instruction from the CPU 501 shown in Fig. 3 and the display 505, and has a function of displaying various screens such as an operation screen for receiving an input operation by the user on the display 505. The judgment unit 54 is realized by an instruction from the CPU 501 shown in Fig. 4 and has a function of making various judgments. The storage / reading unit 59 is executed by an instruction from the CPU 501 shown in Fig. 4 and has a function of storing various data in the storage unit 5000 or reading various data from the storage unit 5000.

[0053] Processing or Operation of the First Embodiment Next, the processing or operation of the distribution system according to the first embodiment will be described with reference to Fig. 7 to Fig. 11. First, the overall flow of the processing or operation in the distribution system 1 will be described with reference to Fig. 7. Fig. 7 is a sequence diagram showing an example of an image distribution process in the distribution system according to the first embodiment.

[0054] First, the reception unit 52 of the communication terminal 50 receives a request to start streaming image distribution by receiving a predetermined input operation by the user on the input device 504 (step S11). The communication unit 51 transmits an image distribution request indicating a request for streaming image distribution to the photographing device 10 (step S12). As a result, the communication unit 11 of the photographing device 10 receives the image distribution request transmitted from the communication terminal 50. Note that the image distribution request transmitted from the communication terminal 50 may include information on the number of streams for which image distribution is requested and encoding parameters for each stream. In this case, the information on the number of streams for which image distribution is requested and encoding parameters for each stream included in the image distribution request is received by the reception unit 52 by a user input to the input device 504. Furthermore, the photographing device 10 may automatically start a process for performing data distribution, triggered by a user operation on the operation unit 115 or by power-on of the photographing device 10.

[0055] Next, the photographing processing unit 15 of the photographing device 10 performs photographing processing of the subject and acquires photographed image data (step S13). Then, the photographing device 10 performs encoding processing on the photographed image data acquired by the photographing processing unit 15 (step S14).

[0056] Here, the encoding process of the captured image data by the image capturing device 10 will be described in detail with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the encoding process of the image data according to the first embodiment.

[0057] First, the parameter setting unit 16 sets the number of streams to be streamed and the encoding parameters for each stream (step S101). Specifically, the parameter setting unit 16 sets the number of streams and the encoding parameters for each stream in the encoding parameter management DB 1001 (see FIG. 6) based on information included in the image distribution request received by the communication unit 11 or information input from an external device to the input / output unit 13. Note that the parameter setting unit 16 may set information stored in advance for streaming distribution as the encoding parameters. In the case of the example shown in FIG. 6, the parameter setting unit 16 sets three streams and stores information on the encoding parameters for each stream in the encoding parameter management DB 1001.

[0058] Next, the determination unit 14 determines the number of processes that can be simultaneously executed by the hardware encoder 106 (step S102). The determination unit 14 determines the number of processes that can be simultaneously executed by the hardware encoder 106 based on, for example, the processing capacity of the hardware encoder 106 or the values ​​of the encoding parameters set in step S101. In the case of the encoding parameter management table shown in Fig. 6, the determination unit 14 determines that the number of processes that can be simultaneously executed is one.

[0059] Next, the determination unit 14 determines a target encoder for performing encoding processing for each stream as an encoding method for the captured image data for each stream set in step S101 (step S103). Specifically, the determination unit 14 determines whether to use the hardware encoder 106 or a software encoder for encoding processing for each stream, using the encoding parameters set in step S101. The determination unit 14 determines the target encoder so that the hardware encoder 106 is assigned to a stream with a large processing load, for example. The number of streams that can be assigned to the hardware encoder 106 corresponds to the number of simultaneous processes that can be executed determined in step S102.

[0060] The determination unit 14 assigns the hardware encoder 106 to a high-resolution stream that has a large processing load, and assigns a software encoder to other streams, according to the value of the resolution of the image data, which is one of the encoding parameters. When the encoding parameter management table shown in Fig. 6 is used, the determination unit 14 assigns the hardware encoder 106 to stream 1, which has the highest resolution, and assigns software encoders to streams 2 and 3.

[0061] The image capture processing unit 15 captures one frame of captured image data captured by the imaging unit 101, which is an example of an image capture means (step S104). The image processing unit 17 then executes image processing of the captured captured image data (step S105). The encoding processing unit 18 then executes encoding processing using the target encoder determined in step S103 on the image data processed in step S105, based on the encoding parameters set for each stream (step S106). The encoding processing unit 18 then executes encoding processing using the target encoder for each stream, and generates streaming image data for each stream. The encoding processing unit 18 processes encoding processing using the target encoder for multiple streams and generation of streaming image data in parallel using a hardware encoder and a software encoder.

[0062] Then, the communication unit 11 distributes the streaming image data, which is the captured image data encoded in step S106, to the communication terminal 50 (step S107). Note that the streaming image data may be distributed to the communication terminal 50 directly using the input / output unit 13 without going through the communication network 5.

[0063] Then, when the delivery of all frames of captured image data acquired by the image capture processing unit 15 is completed (YES in step S108), the image capture device 10 ends the process. On the other hand, when there are frames of the captured image data acquired by the image capture processing unit 15 for which delivery has not been completed (NO in step S108), the image capture device 10 repeats the process from step S104 to continue the delivery of streaming image data.

[0064] As a result, when the imaging device 10 distributes multiple streams of image data with different image quality, it can determine an appropriate encoding method (whether to use a hardware or software encoder) depending on the image quality of the image data for each stream.

[0065] Returning to FIG. 7, as shown in step S107, the communication unit 11 of the photographing device 10 transmits (distributes) the multiple streaming image data generated by the encoding process in step S14 to the communication terminal 50 via the communication network 5 (step S15).

[0066] Next, the communication terminal 50 selects data to be received from among the multiple streaming image data distributed from the image capturing device 10 (step S16). Specifically, the reception unit 52 of the communication terminal 50 accepts the selection of the streaming image data to be received by a predetermined input operation of the user on the input device 504. Also, the determination unit 54 of the communication terminal 50 selects the streaming image data to be received according to the information on the network bandwidth from the communication unit 51. In this case, the communication unit 11 monitors the communication state of the communication network 5, for example, based on the presence or absence of communication delays. The determination unit 54 selects streaming image data corresponding to the current communication state of the communication network 5. Then, the display control unit 53 of the communication terminal 50 causes the display 505 to display an image related to the streaming image data selected and received in step S16 (step S17).

[0067] Here, another example of the encoding method of the captured image data determined by the above-mentioned determination unit 14 will be described. First, the encoding parameter management table shown in Fig. 9(A) is an example in which the number of encoding processes that can be simultaneously executed in one hardware encoder 106 as shown in Fig. 3 is two, unlike the example in Fig. 6. In the encoding parameter management table shown in Fig. 9(A), for example, a hardware encoder is assigned to stream 1 having the highest resolution and stream 2 having the second highest resolution, and a software encoder is assigned to stream 3. In this way, the imaging device 10 can assign hardware encoders to the encoding processes of multiple streams depending on the number of hardware encoders 106 or the number of processes that can be simultaneously executed in one hardware encoder 106.

[0068] Also, the encoding parameter management table shown in Fig. 9(B) is an example in which, unlike the example in Fig. 6, the frame rate, among the encoding parameters, differs for each stream. In the encoding parameter management table shown in Fig. 9(B), for example, a hardware encoder is assigned to stream 1 having the highest frame rate, and a software encoder is assigned to streams 2 and 3. In this way, the imaging device 10 can determine the target encoder to be used for encoding processing for each stream by determining the processing load of the encoder using not only the resolution of the captured image data but also the frame rate of the image data as the encoding parameters.

[0069] The type of encoding parameters used to determine the target encoder (encoding method) for each stream can be set arbitrarily. For example, the determination unit 54 may determine the target encoder by prioritizing the resolution value or the frame rate value among multiple encoding parameters.

[0070] Effect of the First Embodiment Therefore, the distribution system of the first embodiment can distribute image data of multiple streams simultaneously by generating streaming image data using an encoding method determined according to encoding parameters for each stream with different image quality.

[0071] Modification of the first embodiment Next, a distribution system according to a modification of the first embodiment will be described with reference to Figures 10 and 11. The same configurations and functions as those of the above-mentioned embodiment are given the same reference numerals, and the description thereof will be omitted. The distribution system 1a according to the modification of the first embodiment is a system that can change the encoding parameters of image data while streaming image data is being distributed from the photographing device 10 by the above-mentioned distribution system 1.

[0072] Fig. 10 is a flow chart showing an example of encoding processing of image data according to a modified example of the first embodiment. Note that the processing of steps S201 to S208 shown in Fig. 10 is similar to the processing of steps S101 to S108 shown in Fig. 8, and therefore a description thereof will be omitted. If the delivery of streaming image data is continuing (NO in step S208), the image capturing device 10 shifts the processing to step S209. The parameter setting unit 16 determines whether or not the encoding parameters have been changed (step S209). The parameter setting unit 16 determines whether or not the encoding parameters have been changed, and changes the values ​​of the encoding parameters.

[0073] If the encoding parameters have been changed (YES in step S209), the parameter setting unit 16 of the imaging device 10 shifts the process to step S210. Specifically, if a parameter change request is received from the communication terminal 50 by the communication unit 11 or the input / output unit 13, or if a parameter change request is received by the reception unit 12 through an input operation by the distributor, the parameter setting unit 16 changes the value of the encoding parameter stored in the encoding parameter management DB 1001. In the case of the example shown in FIG. 11, the parameter setting unit 16 changes the frame rate of stream 2 from "10 fps" to "30 fps" among the encoding parameters in which the frame rate differs for each stream. On the other hand, if the parameter setting unit 16 does not change the encoding parameters (NO in step S209), the parameter setting unit 16 repeats the process from step S204.

[0074] Next, the determination unit 14 changes the encoding method for each stream according to the changed encoding parameters, and changes the target encoder to be used for encoding (step S210). Specifically, in the example shown in Fig. 11, when the frame rate of stream 2 is changed, the determination unit 14 recalculates the encoder allocation, and changes the target encoder of stream 2, which has the highest frame rate and the largest processing load, from a software encoder to a hardware encoder, and changes the target encoder of stream 1, which has a frame rate lower than that of stream 2, from a hardware encoder to a software encoder.

[0075] Then, the image capturing device 10 repeats the process from step S204 using the changed target encoder and continues to distribute the streaming image data. On the other hand, if the encoding parameters have not been changed, the parameter setting unit 16 continues to distribute the streaming image data encoded using the target encoder determined in step S203.

[0076] As a result, even if the encoding parameters are changed during the distribution of image data, the distribution system 1a can recalculate the encoder allocation and automatically change the encoding method, so that distribution can continue with optimal encoder allocation.

[0077] Second embodiment Next, the configuration of a distribution system according to the second embodiment will be described with reference to Fig. 12 to Fig. 15. The same configurations and functions as those of the first embodiment are given the same reference numerals and their description will be omitted. The distribution system according to the second embodiment is a system that can distribute streaming image data generated by an image capture device 10 to a communication terminal 50 via a distribution control device 90.

[0078] System Configuration Fig. 12 is a diagram showing an example of a system configuration of a distribution system according to the second embodiment. In the distribution system 2 shown in Fig. 12, a distribution control device 90 is added to the configuration according to the first embodiment. The distribution control device 90 can communicate with the image capture device 10 and the communication terminal 50 via the communication network 5.

[0079] The distribution control device 90 is a server computer that controls the distribution of streaming image data transmitted from the image capture device 10. The distribution control device 90 may be constructed by a single computer, or may be constructed by multiple computers to which each unit (function or means) is divided and arbitrarily assigned. The hardware configuration of the distribution control device 90 is the same as that of the communication terminal 50 shown in Fig. 4 or a partially modified hardware configuration, so a description thereof will be omitted.

[0080] Overview 13 is a diagram showing an example of an outline of the process of the distribution system according to the second embodiment. Note that the process of generating streaming image data for each stream by the imaging device 10 is similar to the process described in the first embodiment.

[0081] The photographing device 10 transmits the generated streaming image data for each stream to the distribution control device 90 via the communication network 5. The distribution control device 90 stores the received multiple streaming data in the storage unit 9000. Then, in response to a request from the communication terminal 50, the distribution control device 90 distributes (transmits) to the communication terminal 50 the streaming image data transmitted from the photographing device 10 and stored in the storage unit 9000.

[0082] 13, the distribution control device 90 can simultaneously distribute streaming image data to multiple communication terminals 50 (communication terminal 50A and communication terminal 50B). The distribution control device 90 can also distribute streaming image data corresponding to a stream in response to a request from a communication terminal 50, and can distribute streaming image data of a different stream to each communication terminal 50. As a result, by using the distribution control device 90, the distribution system 2 can distribute streaming image data transmitted from the imaging device 10 in a stream appropriate for each communication terminal 50 depending on the network bandwidth state of each communication terminal 50, etc.

[0083] ●Function configuration Next, the functional configuration of the distribution system according to the second embodiment will be described. Fig. 14 is a diagram showing an example of the functional configuration of the distribution system according to the second embodiment. Note that the functional configurations of the imaging device 10 and the communication terminal 50 (communication terminals 50A and 50B) are similar to those shown in Fig. 5, and therefore will not be described.

[0084] The distribution control device 90 has a communication unit 91 and a storage / readout unit 99. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 4 operates in response to an instruction from the CPU 501 in accordance with a program loaded on the RAM 503. The distribution control device 90 also has a storage unit 9000 constructed by the ROM 502 shown in Fig. 4 or an external storage.

[0085] The communication unit 91 is realized by commands from the CPU 501 shown in Fig. 4 and the network I / F 508, and has a function of transmitting and receiving various data or information between the photographing device 10 and the communication terminal 50 via the communication network 5. The storage / reading unit 99 is executed by commands from the CPU 501 shown in Fig. 4, and has a function of storing various data in the storage unit 9000 or reading various data from the storage unit 9000. The storage unit 9000 stores streaming image data transmitted from the photographing device 10.

[0086] Processing or Operation of the Second Embodiment Next, the processing or operation of the delivery system according to the second embodiment will be described with reference to Fig. 15. Fig. 15 is a sequence diagram showing an example of image delivery processing in the delivery system according to the second embodiment.

[0087] First, the communication unit 91 of the delivery control device 90 transmits an image acquisition request indicating a request to acquire streaming images to the image capture device 10 (step S51). As a result, the communication unit 11 of the image capture device 10 receives the image acquisition request transmitted from the delivery control device 90. Note that the image acquisition request transmitted from the delivery control device 90 may include information on the number of streams for which image acquisition is requested and encoding parameters for each stream. In this case, the delivery control device 90 may, for example, read and transmit information previously stored in the storage unit 9000, or may transmit information accepted by a user's input to the delivery control device 90. In addition, the image capture device 10 may automatically start processing for image acquisition, triggered by a user operation on the operation unit 115 or by power-on of the image capture device 10.

[0088] Next, the photographing processing unit 15 of the photographing device 10 performs a photographing process (step S52). Then, the photographing device 10 executes an encoding process on the photographed image data acquired by the photographing processing unit 15 (step S53). Note that the encoding process in step S53 is similar to the content shown in FIG. 8, and therefore a description thereof will be omitted.

[0089] Next, the communication unit 11 of the image capture device 10 transmits (distributes) the multiple pieces of streaming data generated and encoded in step S14 to the distribution control device 90 via the communication network 5 (step S54). As a result, the communication unit 91 of the distribution control device 90 receives the multiple pieces of streaming image data transmitted from the image capture device 10. Then, the storage / readout unit 99 of the distribution control device 90 stores the streaming image data received by the communication unit 91 in the storage unit 9000 (step S55). In this way, the distribution control device 90 temporarily stores the streaming image data transmitted from the image capture device 10.

[0090] Next, in response to the user request accepted by the accepting unit 52, the communication unit 51A of the communication terminal 50A transmits an image distribution request indicating a request for distribution of streaming images to the distribution control device 90 (step S56). As a result, the communication unit 91 of the distribution control device 90 receives the image distribution request transmitted from the communication terminal 50A. Next, the communication unit 91 of the distribution control device 90 transmits (distributes) the multiple streaming image data stored in the storage unit 9000 to the communication terminal 50A (step S57).

[0091] The communication terminal 50A selects data to receive from among the multiple streaming image data distributed from the distribution control device 90 (step S58). Specifically, the reception unit 52A of the communication terminal 50A receives the selection of the streaming image data to receive by a predetermined input operation of the user on the input device 504. The determination unit 54A of the communication terminal 50A selects the streaming image data to receive according to the network bandwidth information from the communication unit 51A. Then, the display control unit 53A of the communication terminal 50A causes the display 505 to display an image related to the streaming image data selected and received in step S58 (step S59).

[0092] Meanwhile, in response to the user request accepted by the accepting unit 52B, the communication unit 51B of the communication terminal 50B transmits an image distribution request indicating a request for distribution of streaming images to the distribution control device 90 (step S60). As a result, the communication unit 91 of the distribution control device 90 receives the image distribution request transmitted from the communication terminal 50B. Next, the communication unit 91 of the distribution control device 90 transmits (distributes) the multiple streaming image data stored in the storage unit 9000 to the communication terminal 50B (step S61).

[0093] Similar to the process of step S58, the communication terminal 50B selects data to be received from the multiple streaming image data distributed from the distribution control device 90 (step S62). Then, the display control unit 53B of the communication terminal 50B causes the display 505 to display an image related to the streaming image data selected and received in step S62 (step S63). Note that the order of the processes of steps S56 to S59 and steps S60 to S63 may be reversed, or may be performed in parallel.

[0094] Effect of the second embodiment Therefore, by using the distribution control device 90, the distribution system of the second embodiment can distribute streaming image data transmitted from the photographing device 10 in a stream appropriate for each communication terminal 50, depending on the network bandwidth status of each communication terminal 50, etc.

[0095] ●Summary● As described above, the photographing device according to one embodiment of the present invention is a photographing device 10 that distributes image data of multiple streams, and includes a photographing processing unit 15 (an example of an acquisition means) that acquires photographed image data photographed by an imaging unit 101 (an example of a photographing means), a judgment unit 14 (an example of a determination means) that determines an encoding method for the acquired photographed image data according to encoding parameters for each stream, an encoding processing unit 18 (an example of a generation means) that generates streaming image data for each stream using the determined encoding method, and a communication unit 11 (an example of a distribution means) that distributes the generated multiple streaming image data. As a result, the photographing device 10 can simultaneously distribute image data of multiple streams by performing encoding processing using an appropriate encoding method according to image quality.

[0096] Moreover, the image capture device according to one embodiment of the present invention includes at least one hardware encoder 106 and a software encoder. In the image capture device 10, the encoding processing unit 18 (an example of a generating means) generates streaming image data using the hardware encoder 106 for a stream corresponding to encoding parameters of high resolution or high frame rate, which imposes a large processing load. This allows the image capture device 10 to simultaneously distribute image data of multiple streams even when the number of hardware encoders is limited.

[0097] Furthermore, in the image capture device according to one embodiment of the present invention, the encoding unit 18 (an example of a generating means) generates multiple streaming image data in parallel using the hardware encoder 106 and a software encoder, thereby enabling the image capture device 10, which has a limited number of hardware encoders, to simultaneously distribute multiple streams of image data.

[0098] Moreover, the imaging device according to one embodiment of the present invention includes a parameter setting unit 16 (an example of a setting means) that sets encoding parameters for each stream. In the imaging device 10, a judgment unit 14 (an example of a deciding means) changes the decided encoding method when the set encoding parameters are changed. As a result, even if the encoding parameters are changed during the distribution of streaming image data, the imaging device 10 can recalculate the encoder allocation and automatically change to the encoding method, so that streaming distribution with optimal encoder allocation can be continued.

[0099] ●Additional Information● Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0100] In addition, the various tables in the above-described embodiment may be generated by the learning effect of machine learning, and the data of each associated item may be classified by machine learning, making it unnecessary to use tables. Here, machine learning is a technology for making a computer acquire human-like learning capabilities, and refers to a technology in which a computer autonomously generates an algorithm required for judgment such as data identification from learning data previously acquired, and applies this to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods, and any learning method for machine learning may be used.

[0101] So far, an imaging device, a distribution system, a distribution method, and a recording medium according to one embodiment of the present invention have been described. However, the present invention is not limited to the above-described embodiment, and other embodiments can be added, modified, or deleted, etc., within the scope that can be conceived by a person skilled in the art. In any aspect, as long as the functions and effects of the present invention are achieved, it is included in the scope of the present invention. [Explanation of symbols]

[0102] 1. Distribution System 5. Communication Network 10. Imaging Device 11 Communication unit (an example of a distribution means) 14 Judgment unit (an example of a decision means) 15. Shooting processing unit (an example of an acquisition means) 16 Parameter setting unit (an example of a setting means) 18 Encoding processing unit (an example of a generating means) 50 Communication terminal 90 Distribution control device 101 Imaging unit (an example of a photographing means) [Prior art documents] [Patent documents]

[0103] [Patent Document 1] JP 2011-77564 A

Claims

1. 1. An image capturing device that delivers a plurality of streams of image data, An acquisition means for acquiring photographed image data photographed by the photographing means; At least one of a hardware encoder and a software encoder; a determination means for determining an encoding method for the acquired captured image data in accordance with an encoding parameter for each stream; a generating means for generating streaming image data for each of the plurality of streams based on the determined encoding method, by using the hardware encoder for a stream corresponding to the encoding parameter having a large processing load among the plurality of streams and by using the software encoder for the other streams; a distribution means for distributing the generated plurality of streaming image data; An imaging device comprising:

2. the encoding parameter is a resolution of the streaming image data; 2. The image capturing apparatus according to claim 1, wherein the generating means generates the streaming image data for the high resolution stream by using the hardware encoder.

3. the encoding parameter is a frame rate of the streaming image data; 3. The image capturing apparatus according to claim 1, wherein the generating means generates the streaming image data using the hardware encoder for the stream with the high frame rate.

4. 4. The image capturing apparatus according to claim 1, wherein the generating means generates a plurality of streaming image data in parallel using the hardware encoder and the software encoder.

5. A setting means for setting an encoding parameter for each of the streams, 5. The photographing apparatus according to claim 1, wherein the determination means determines an encoding method for the acquired photographic image data in accordance with a set encoding parameter.

6. 6. The imaging apparatus according to claim 5, wherein the determining means changes the determined encoding method when a set encoding parameter is changed.

7. A distribution system for distributing a plurality of streams of image data, comprising: An acquisition means for acquiring photographed image data photographed by the photographing means; At least one of a hardware encoder and a software encoder; a determination means for determining an encoding method for the acquired captured image data in accordance with an encoding parameter for each stream; a generating means for generating streaming image data for each of the plurality of streams based on the determined encoding method, by using the hardware encoder for a stream corresponding to the encoding parameter having a large processing load among the plurality of streams and by using the software encoder for the other streams; An imaging device comprising: a distribution means for distributing the plurality of streaming image data generated by the imaging device; A distribution system comprising:

8. A distribution method executed by an image capture device that distributes a plurality of streams of image data, comprising: An acquisition step of acquiring photographed image data photographed by a photographing means; a determining step of determining an encoding method for the acquired captured image data in accordance with an encoding parameter for each stream; a generating step of generating streaming image data for each of the plurality of streams based on the determined encoding method, by using at least one hardware encoder provided in the imaging device for a stream corresponding to the encoding parameter having a large processing load among the plurality of streams, and by using a software encoder provided in the imaging device for the other streams; a distribution step of distributing the generated plurality of streaming image data; The delivery method to be performed.

9. A computer of an imaging device that delivers multiple streams of image data, An acquisition step of acquiring photographed image data photographed by a photographing means; a determining step of determining an encoding method for the acquired captured image data in accordance with an encoding parameter for each stream; a generating step of generating streaming image data for each of the plurality of streams based on the determined encoding method, by using at least one hardware encoder provided in the imaging device for a stream corresponding to the encoding parameter having a large processing load among the plurality of streams, and by using a software encoder provided in the imaging device for the other streams; a distribution step of distributing the generated plurality of streaming image data; A program for executing the above.

Images