Imaging method, image processing device, and imaging apparatus and system

Abstract

To easily obtain a desired image effect for an imaged subject.SOLUTION: An imaging method includes: a step for imaging a subject image and generating a captured image (20) by an imaging part (115); a step for performing image processing to the captured image generated by the imaging part by an image processing part (260); and a step for controlling the image processing by the image processing part on the basis of the captured image by a control part (210). In the method, the control part allows the image processing part to perform the imaging processing, so as to acquire at least one user instruction for specifying a subject in a moving image generated from the captured image and generate the moving image including processed images (41 and 42) to which an image effect which is different between the inside and the outside of subject areas (R31 and R32) corresponding to the subject specified by the user instruction in the captured image is imparted, according to at least one user instruction.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to an imaging method, an image processing device, an imaging device, and a system for performing image processing on a captured image. [Background technology]

[0002] Patent Document 1 discloses an imaging device that aims to enable a user to easily perform color conversion during shooting, retaining a color freely designated by the user and erasing all other color components. In this imaging device, the user's designated color is determined based on color information contained in a predetermined area of ​​an image displayed on an electronic viewfinder, which displays an image based on image data captured by an imaging unit and output from an image processing unit. The imaging device then sets parameters for the image processing unit to perform color conversion that erases all colors other than the designated color. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-211525 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides an imaging method, an image processing device, an imaging device, and a system that can easily achieve a desired image effect for an imaged subject. [Means for solving the problem]

[0005] In the present disclosure, an imaging method includes a step of an imaging unit capturing an image of a subject and generating a captured image, a step of an image processing unit performing image processing on the captured image generated by the imaging unit, and a step of a control unit controlling the image processing by the image processing unit based on the captured image. In this method, the control unit acquires at least one user instruction specifying a subject in a moving image generated from the captured image, and in response to the at least one user instruction, causes the image processing unit to perform image processing so as to generate a moving image including processed images to which different image effects are applied between the inside and outside of a subject area in the captured image corresponding to the subject specified by the user instruction.

[0006] In the present disclosure, an image processing device includes a receiving unit that receives image data including a captured image of a subject captured by an imaging device, an image processing unit that performs image processing on the captured image included in the image data received by the receiving unit, and a control unit that controls the image processing by the image processing unit based on the captured image. The control unit acquires at least one user instruction that specifies a subject in a moving image generated from the captured image, and in response to the at least one user instruction, causes the image processing unit to perform image processing so as to generate a moving image including processed images to which different image effects are applied between the inside and outside of a subject area in the captured image that corresponds to the subject indicated by the user instruction.

[0007] In the present disclosure, an imaging device includes an imaging unit that captures an image of a subject and generates a captured image, an image processing unit that performs image processing on the captured image generated by the imaging unit, and a control unit that controls the image processing by the image processing unit based on the captured image. The control unit acquires at least one user instruction that specifies a subject in a moving image generated from the captured image, and in response to the at least one user instruction, causes the image processing unit to perform image processing so as to generate a moving image including processed images to which different image effects are applied between the inside and outside of a subject area in the captured image that corresponds to the subject specified by the user instruction.

[0008] In the present disclosure, an imaging system includes an imaging device and an image processing device. The imaging device includes an imaging unit that captures an image of a subject and generates a captured image, and a transmission unit that transmits image data representing the captured image to the image processing device. The image processing device includes a receiving unit that receives image data from the imaging device and an image processing unit that performs image processing on the captured image represented by the image data based on the image data received by the receiving unit. The imaging device or the image processing device includes a control unit that acquires at least one user instruction that specifies a subject in a moving image to be generated from the captured image, and causes the image processing unit to perform image processing in response to the at least one user instruction so as to generate a moving image including processed images to which different image effects are applied between the inside and outside of a subject area in the captured image corresponding to the subject specified by the user instruction. [Effects of the Invention]

[0009] The imaging method, image processing device, imaging device, and system disclosed herein can facilitate the realization of a desired image effect for an imaged subject. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an imaging system according to a first embodiment of the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating the configuration of a digital camera in an imaging system; [Figure 3] FIG. 1 is a diagram illustrating a configuration of an image processing server in an imaging system. [Figure 4] FIG. 1 is a diagram for explaining the operation of the imaging system according to the first embodiment. [Figure 5] FIG. 1 is a sequence diagram illustrating an example of the operation of the imaging system according to the first embodiment. [Figure 6] FIG. 1 is a diagram illustrating an example of a data structure of service management information in an imaging system. [Figure 7] FIG. 1 is a diagram showing an example of the arrangement of multiple cameras in an imaging system. [Figure 8] FIG. 1 is a diagram for explaining a detection operation of a subject area in an imaging system. [Figure 9]1 is a flowchart illustrating a parallel color preservation process in an imaging system; [Figure 10] FIG. 1 is a diagram for explaining image processing in an imaging system. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0012] (Embodiment 1) 1. Configuration An imaging system according to a first embodiment of the present disclosure will be described with reference to FIG.

[0013] 1, an imaging system 10 according to this embodiment includes one or more cameras 100, an image processing server 200, and one or more user terminals 300. In this system 10, the image processing server 200 is, for example, a cloud server, and is connected to the cameras 100 and the user terminals 300 via a communication network 15 such as the Internet so as to enable data communication. The camera 100 and the user terminal 300 may also be connected via a wired or wireless connection so as to enable data communication.

[0014] The present system 10 can be applied to various uses, for example, to deliver videos captured by a camera 100 to multiple user terminals 300. The present system 10 provides information support useful for realizing video delivery that highlights subjects that attract the user's attention for each user terminal 300.

[0015] The user terminal 300 may be any of various information terminals, such as a personal computer (PC), a smartphone, or a tablet terminal. In the present system 10, the user terminal 300 may be omitted, or the user terminal 300 may be an external component of the present system 10. The output destination of the video in the present system 10 is not limited to the user terminal 300, and for example, the present system 10 may store the output video data in various storage devices.

[0016] 1-1.Camera configuration The camera 100 of this embodiment may be any of various imaging devices such as a digital camera, a video camera, a camcorder, etc. The configuration of the camera 100 of this embodiment will be described with reference to FIG.

[0017] FIG. 2 is a diagram illustrating an example of the configuration of a camera 100 in the system 10. The camera 100 is an example of an imaging device in this embodiment. The camera 100 in this embodiment includes an image sensor 115, an image processing engine 120, a display monitor 130, and a control unit 135. The camera 100 also includes a buffer memory 125, a card slot 140, a flash memory 145, an operation unit 150, a communication module 155, and a microphone 160. The camera 100 also includes, for example, an optical system 110 and a lens driving unit 112. In the camera 100 of this embodiment, the optical system 110 and the lens driving unit 112 may be external components, and the camera 100 may be an interchangeable lens type.

[0018] The optical system 110 includes a focus lens, a zoom lens, an optical image stabilization lens (OIS), an aperture, a shutter, etc. The focus lens is a lens for changing the focus state of the subject image formed on the image sensor 115. The zoom lens is a lens for changing the magnification of the subject image formed by the optical system. Each of the focus lens, etc. is composed of one or more lenses.

[0019] Lens driving unit 112 drives the focus lens and the like in optical system 110. Lens driving unit 112 includes a motor, and moves the focus lens along the optical axis of optical system 110 under the control of control unit 135. The configuration for driving the focus lens in lens driving unit 112 can be realized by a DC motor, a stepping motor, a servo motor, an ultrasonic motor, or the like.

[0020] The image sensor 115 captures an image of a subject formed via the optical system 110 and generates imaging data. The imaging data constitutes image data representing an image captured by the image sensor 115. The image sensor 115 generates new frame image data at a predetermined frame rate (e.g., 30 frames / second). The timing of generating imaging data and the electronic shutter operation in the image sensor 115 are controlled by the control unit 135. The image sensor 115 can be any of a variety of image sensors, such as a CMOS image sensor, a CCD image sensor, or an NMOS image sensor.

[0021] The image sensor 115 performs operations such as capturing still images and through images. Through images are mainly moving images, and are displayed on the display monitor 130 so that the user can determine the composition for capturing the still image. The through images and still images are each an example of a captured image in this embodiment. The image sensor 115 is an example of an imaging unit in this embodiment.

[0022] The image processing engine 120 performs various processes on the imaging data output from the image sensor 115 to generate image data, and performs various processes on the image data to generate an image to be displayed on the display monitor 130. The various processes include, but are not limited to, white balance correction, gamma correction, YC conversion processing, electronic zoom processing, compression processing, and expansion processing. The image processing engine 120 may be configured with a hardwired electronic circuit, or may be configured with a microcomputer, processor, or the like using a program.

[0023] In this embodiment, the image processing engine 120 includes an image recognition unit 122 that realizes a function for detecting an area where a specific subject, such as a person, is located by image recognition of a captured image. The image recognition unit 122 includes a trained model constructed to perform image recognition such as segmentation using machine learning such as a neural network. The image processing engine 120 may be configured integrally with the control unit 135, or the image processing engine 120 may be omitted from the camera 100.

[0024] The display monitor 130 is an example of a display unit that displays various information. For example, the display monitor 130 displays an image (through image) represented by image data captured by the image sensor 115 and processed by the image processing engine 120. The display monitor 130 also displays a menu screen or the like that allows the user to make various settings for the camera 100. The display monitor 130 can be configured, for example, by a liquid crystal display device or an organic EL device.

[0025] The operation unit 150 is a general term for user interfaces such as hard keys, software keys, etc., such as operation buttons and operation levers provided on the exterior of the camera 100, and accepts operations by the user. The operation unit 150 includes, for example, a release button, a mode dial, and a touch panel. When the operation unit 150 accepts an operation by the user, it transmits an operation signal corresponding to the user operation to the control unit 135. The operation unit 150 is an example of a setting unit of the camera 100 in this embodiment.

[0026] The control unit 135 controls the overall operation of the camera 100. The control unit 135 includes a CPU and the like, and the CPU executes a program (software) to realize predetermined functions. Instead of a CPU, the control unit 135 may include a processor configured with a dedicated electronic circuit designed to realize the predetermined functions. That is, the control unit 135 can be realized by various processors such as a CPU, MPU, GPU, DSP, FPGA, ASIC, etc. The control unit 135 may be configured with one or more processors. Furthermore, the control unit 135 may be configured on a single semiconductor chip together with the image processing engine 120, etc.

[0027] The buffer memory 125 is a recording medium that functions as a work memory for the image processing engine 120 and the control unit 135. The buffer memory 125 is realized by a DRAM (Dynamic Random Access Memory) or the like. The flash memory 145 is a non-volatile recording medium. Although not shown, the control unit 135 may have various types of internal memory, for example, a built-in ROM. The ROM stores various programs executed by the control unit 135. The control unit 135 may also have a built-in RAM that functions as a work area for the CPU.

[0028] The card slot 140 is a means for inserting a removable memory card 142. The card slot 140 can electrically and mechanically connect the memory card 142. The memory card 142 is an external memory equipped with a recording element such as a flash memory inside. The memory card 142 can store data such as image data generated by the image processing engine 120.

[0029] The communication module 155 is a module (circuit) that connects to an external device in accordance with a predetermined communication standard for wired or wireless communication. The predetermined communication standard includes, for example, USB, HDMI (registered trademark), IEEE802.11, Wi-Fi, Bluetooth, etc. The camera 100 can communicate with other devices via the communication module 155.

[0030] The microphone 160 includes, for example, a plurality of microphone elements built into the camera 100. The microphone 160 outputs an audio signal indicating audio picked up on a plurality of channels to the control unit 135 so that, for example, the direction from which the picked up audio is coming can be estimated. The camera 100 may use an external microphone 160. Instead of or in addition to the built-in microphone 160, the camera 100 may include a connection unit such as a terminal for connecting to the external microphone 160.

[0031] 1-2. Image processing server configuration The configuration of the image processing server 200 in this embodiment will be described with reference to FIG.

[0032] 3 is a diagram illustrating an example of the configuration of the image processing server 200. The image processing server 200 illustrated in FIG.

[0033] The control unit 210 includes, for example, a CPU or MPU that cooperates with software to realize predetermined functions. The control unit 210 controls, for example, the overall operation of the image processing server 200. The control unit 210 reads data and programs stored in the storage unit 220 and performs various arithmetic processing to realize various functions.

[0034] The control unit 210 executes, for example, a program including a set of instructions for implementing each of the above functions. The program may be provided via the communication network 15 or may be stored on a portable recording medium. The control unit 210 may also be a hardware circuit such as a dedicated electronic circuit or a reconfigurable electronic circuit designed to implement each of the above functions. The control unit 210 may also be configured with various semiconductor integrated circuits such as a CPU, MPU, GPU, GPGPU, TPU, microcomputer, DSP, FPGA, and ASIC.

[0035] The storage unit 220 is a storage medium that stores programs and data necessary to realize the functions of the image processing server 200. The storage unit 220 includes a storage unit 221 and a temporary storage unit 222, as shown in FIG.

[0036] The storage unit 221 stores parameters, data, control programs, etc. for realizing predetermined functions. The storage unit 221 is configured, for example, with an HDD or SSD. For example, the storage unit 221 stores the above-mentioned programs and various image data, etc.

[0037] The temporary storage unit 222 is configured with a RAM such as a DRAM or an SRAM, and temporarily stores (i.e., holds) data. For example, the temporary storage unit 222 holds image data that is in the middle of being edited. The temporary storage unit 222 may also function as a work area for the control unit 210, and may be configured as a storage area in the internal memory of the control unit 210.

[0038] The communication unit 250 is a module (circuit) that connects to an external device in accordance with a predetermined communication standard for wired or wireless communication. The predetermined communication standard includes, for example, USB, HDMI, IEEE802.11, Wi-Fi, Bluetooth, etc. The communication unit 250 may connect the image processing server 200 to a communication network such as the Internet. The communication unit 250 is an example of an acquisition unit that receives various information from an external device or a communication network.

[0039] The image processing unit 260 generates processed image data by performing various image processing operations on image data received from, for example, the communication unit 250. The image processing unit 260 may be configured with a hardwired electronic circuit, or may be configured with a microcomputer, processor, or the like using a program.

[0040] The above-described configuration of the image processing server 200 is an example, and the configuration of the image processing server 200 is not limited to this. For example, the control unit 210 and the image processing unit 260 may be configured integrally. Furthermore, the acquisition unit in the image processing server 200 may be realized by cooperation with various software in the control unit 210, etc. The input unit in the image processing server 200 may acquire various pieces of information by reading out various pieces of information stored in various storage media (e.g., the storage unit 221) into a working area (e.g., the temporary storage unit 222) of the control unit 210. Furthermore, the image processing server 200 of the present system 10 may be configured by a plurality of server devices.

[0041] 2.Operation The operation of the system 10 configured as above will be described below.

[0042] The system 10 performs image processing such as color preservation processing that preserves color only in a specific subject, by utilizing the subject recognition results of the camera 100. An overview of the operation of the system 10 will be described with reference to FIG.

[0043] 4 is a diagram illustrating the operation of the imaging system 10 of the embodiment 1. For example, by changing the subject to be colored for each viewer using each user terminal 300, the system 10 colors the subject that each viewer pays attention to when shooting a horse racing broadcast, a sports broadcast, or live video, thereby distributing video with different coloring targets for each viewer.

[0044] Fig. 4(A) shows an example of a captured image 20 in the camera 100. Fig. 4(B) shows an example of a processed image 41 resulting from image processing performed on the captured image 20 in Fig. 4(A) in the system 10. Fig. 4(C) shows an example of a processed image 42 resulting from image processing different from that in Fig. 4(B) performed on the captured image 20 in Fig. 4(A).

[0045] 4(A) illustrates a captured image 20 of a frame in a shooting scene in which multiple pairs of racehorses and their jockeys are captured as subjects in a video (hereinafter also referred to as the "original video") shot by the camera 100. For ease of explanation, FIG. 4(A) illustrates a captured image 20 in which two subjects 31 and 32 are captured, but the original video may include three or more subjects 31-32.

[0046] For example, while shooting an original video, the camera 100 of this embodiment sequentially detects subject regions R31-R32 that follow the outlines of the subjects 31-32. These subject regions R31-R32 are obtained using various techniques for controlling shooting in the camera 100, such as AF (autofocus) or AE (auto exposure).

[0047] The system 10 uses the detection results of these subject regions R31-R32 to perform image processing on the captured image 20 to highlight the specific subject 31, as shown in FIGS. 4(A) and 4(B), to generate a processed image 41. The specific subject 31 is, for example, a subject that is attracting attention on a certain user terminal 300. The image processing by the system 10 is, for example, a color-preserving process that preserves the color inside the subject region R31 corresponding to the specific subject 31 while achromatizing the area outside the subject region R31. The system 10 distributes video data including the processed image 41 that has been subjected to the color-preserving process for the specific subject 31 to the user terminal 300 that is attracting attention to the subject 31.

[0048] 4(C) illustrates a processed image 42 delivered by the system 10 to a user terminal 300 different from the delivery destination of FIG. 4(B). The system 10 assumes a situation in which, for example, multiple user terminals 300 focus on different subjects 31 and 32. To accommodate such a situation, the system 10 generates multiple videos (hereinafter also referred to as "processed videos") from the original video of FIG. 4(A), for example, as shown in FIGS. 4(B) and (C), each of which includes processed images 41 and 42 in which different subjects 31 and 32 are colored.

[0049] In this way, the present system 10 can provide a service of individually distributing processed videos in which the subjects 31, 32 that each viewer focuses on are highlighted on each user terminal 300. Below, an example will be described in which the present system 10 is applied to a live streaming service of video footage of horse racing, etc.

[0050] 2-1. Overall operation The overall operation of the camera 100 in the present system 10 will be described with reference to FIGS.

[0051] 5 is a sequence diagram illustrating the operation of the present system 10. The present system 10 provides a service for distributing video that highlights, for example, a racehorse (and its jockey) for which a user has purchased a betting ticket. In such a video distribution service, the present system 10 may also provide an option that enables a user to issue a switching instruction during a race to highlight a racehorse other than the racehorse for which the user has purchased a betting ticket.

[0052] The following describes an example of operation when each user focuses on one subject, that is, when placing a winning betting ticket, in the present system 10. Note that the present service may be available for application even if the user does not purchase a betting ticket.

[0053] For example, in the present system 10, before video distribution, the image processing server 200 accepts applications for video distribution services from multiple user terminals 300A to 300Z via the communication network 15 (S1 to S2). Hereinafter, the "user terminals 300A to 300Z" will be collectively referred to as "user terminal 300." User terminal 300A is the user terminal 300 of user A, and user terminal 300Z is the user terminal 300 of another user Z (see FIG. 6).

[0054] The plurality of user terminals 300A-300Z each transmit application information for the video distribution service to the image processing server 200 via, for example, the communication network 15 (S1). The application information from each user terminal 300 includes, for example, identification information of the racehorse for which the corresponding user has purchased a betting ticket (or a racehorse of interest), and the presence or absence of an application for the video distribution service and a switching option.

[0055] The control unit 210 of the image processing server 200 receives application information for a video distribution service from the user terminal 300 via the communication network 15 (S1), and then generates service management information D1 for managing the application information from each user terminal 300 (S2). Figure 6 shows an example of the service management information D1 in the system 10.

[0056] The service management information D1 manages "users," "subjects of interest," "switching options," and "live streaming destinations" in association with one another, as shown in FIG. 6, for example. The service management information D1 is an example of management information in this embodiment. The service management information D1 is stored in the storage unit 220 of the image processing server 200, for example.

[0057] In the service management information D1, "user" stores the identification information of the user of the user terminal 300 who has applied for the video distribution service. "subject of interest" stores the identification information of the object of interest in the application information from each user terminal 300, such as the bib number of a racehorse. "switching option" stores whether switching is possible depending on the application for the switching option from each user terminal 300. "live distribution destination" stores address information for distributing videos to the user, and indicates, for example, the user terminal 300 from which the application information was sent.

[0058] 6, for example, for user A, the target subject is number "1," the switching option is "unswitchable," and the live distribution destination is user terminal 300A. Also, for user Z, the target subject is number "16," the switching option is "switchable," and the live distribution destination is user terminal 300Z.

[0059] In this system 10, the camera 100 captures an original video, for example, in live streaming of a video (S3 to S7). In this embodiment, an example of the operation of the imaging system 10 performing live streaming using multiple cameras 100A to 100E will be described. In the following, the "cameras 100A to 100E" will be collectively referred to as "camera 100."

[0060] 7 shows an example of the arrangement of multiple cameras 100A-100E in the present system 10. The course Co on which the horses of subjects 31-32 run at the racetrack includes, for example, a first corner C1, a second corner C2, a third corner C3, and a fourth corner C4, as shown in FIG.

[0061] The course Co also includes a start point Ps following the fourth corner, a home stretch Sh between the fourth and first corners C4 and C1, a finish line Pg between the home stretch Sh and the first corner C1, and a back stretch Sb between the second and third corners C2 and C3. Opposite the home stretch Sh, stands St for spectators are provided.

[0062] FIG. 7 shows an example of an arrangement in which five cameras 100A to 100E are used in the system 10. The first camera 100A is positioned, for example, in the highest position in the stand St, so as to capture the entire course Co. The second camera 100B is positioned, for example, at a lower position than the first camera 100A, so as to capture the finish line Pg. The third camera 100C is positioned, for example, at the same height as the first camera 100A, so as to capture the backstretch Sb.

[0063] The fourth camera 100D is positioned outside the third corner C3 to capture, for example, the herd of horses (subjects 31-32) passing toward the third corner C3. The fifth camera 100E is positioned outside the fourth corner C4 to capture, for example, the herd of horses at the fourth corner C4. In the present system 10, the first to fifth cameras 100A-100E each perform video capture processing (S3-S7).

[0064] For example, in each camera 100, first, the control unit 135 causes the image sensor 115 to perform an imaging operation at a predetermined frame rate (e.g., 30 to 60 fps) to generate a captured image 20 for each frame (see FIG. 4(A)) (S3). In this embodiment, such a video capturing operation of the camera 100 is performed in pan focus using a deep depth of field in order to focus on each of the subjects 31 to 32, for example.

[0065] Next, the control unit 135 operates, for example, the image recognition unit 122 to recognize various pieces of information related to the subjects 31-32 in the generated captured image 20 (S4). In step S4, the image recognition unit 122 detects, for example, subject regions R31-R32. The subject region detection operation in the present system 10 will be described with reference to FIG. 8.

[0066] 8 illustrates an example of an area division map M20 of the detection result for the captured image 20 of Fig. 4(A). For example, the image recognition unit 122 of the camera 100 inputs the captured image 20 for each frame from the image sensor 115 into a trained model of image recognition such as a segmentation technique, and sequentially generates an area division map M20 from the output of the trained model (S4).

[0067] The trained model is machine-learned to calculate a probability value for each pixel in an input image as to whether a specific type of subject, such as a racehorse (including its jockey), is included in the image. For example, in step S4, the image recognition unit 122 performs threshold determination on the probability value for each pixel in the output of the trained model to generate a region division map M20.

[0068] In this case, even if there is overlap between the multiple subject regions R31-R32, the image recognition unit 122 can separate the subject regions R31-R32 from one another based on the difference in distance values ​​among the subject regions R31-R32 using various distance measurement techniques in the camera 100. Such distance measurement techniques may be, for example, image plane phase difference detection or DFD, or a distance measurement sensor such as a TOF sensor.

[0069] According to this region division map M20, it is possible to detect, for example, subject regions R31 to R32 each having a shape that follows the outline of each subject 31 to 32, as shown in Fig. 8, corresponding to each subject 31 to 32 captured in the captured image 20 of Fig. 4(A). Furthermore, by using this region division map M20, for example, the camera 100 can use any of the subject regions R31 to R32 for various types of shooting control (for example, AE control) (S3).

[0070] Also, in step S4, for example, the control unit 135 applies image recognition technology such as optical character recognition (OCR) to the captured image 20 to recognize the identification information of the target subject, such as the bib number "1" in the subject area R31 and the bib number "16" in the subject area R32 in Figure 4(A).

[0071] 5, for example, the control unit 135 records the result of the image recognition (S4) as meta information of the video data (S5), and transmits the video data with the meta information from the communication module 155 to the image processing server 200 via the communication network 15 (S6). The video data transmitted in step S6 includes a predetermined number of frames, for example, in units of GOP (Group Of Picture).

[0072] The control unit 135 repeats the processes of steps S3 to S6 at a predetermined cycle, for example, until the moving image shooting ends (S7). The predetermined cycle is, for example, a predetermined multiple of the frame period (for example, several seconds).

[0073] In the image processing server 200, for example, when the communication unit 250 receives video data from the camera 100 (S6), the control unit 210 performs image processing in parallel on the received video data to generate multiple processed videos (S8). In the parallel color-preserving processing (S8) of this embodiment, image processing such as color-preserving processing is performed in parallel among multiple subjects 31-32 so as to cover all "subjects of interest" in the service management information D1 (S2) (see FIG. 4). The processing of step S8 will be described in detail later.

[0074] Next, the control unit 210 performs a switching process (S9) to switch or combine the distribution targets in the results of shooting by the multiple cameras 100A to 100E. In the present system 10, the switching process (S9) is performed automatically or manually, for example, by grasping the progress of the race from the results of shooting by the multiple cameras 100A to 100E.

[0075] For example, in the switching process (S9), the fifth, first or second camera 100E, 100A or 100B is used as appropriate depending on the position of the horses as they pass through the fourth, first and second corners C4, C1 and C2 from the start point Ps (Fig. 7) and reach the backstretch Sb. Also, when the horses pass the backstretch Sb, the image processing server 200 generates a composite image by juxtaposing the image from the first camera 100A and the image from the third camera 100C, for example.

[0076] Additionally, when the herd of horses approaches the third corner from the backstretch Sb and passes through, the fourth camera 100D is used, and then it switches to the first camera 100A. When the herd of horses enters the fourth corner C4, the fifth camera 100E is used, and when the herd of horses exceeds the field of view of the fifth camera 100E, it switches to the first camera 100A, and when the herd of horses approaches the finish line Pg, it switches to the second camera 100B.

[0077] The above switching process (S9) may be performed, for example, by the control unit 210 performing image recognition on the image capture results of each camera 100. Alternatively, the switching process may be performed by a person in charge of switching instructing the image processing server 200 on the timing of switching using a monitor that displays each video from the multiple cameras 100A to 100E.

[0078] Next, based on the service management information D1, the control unit 210 transmits (S10) the video data of the edited results (S8, S9) for each subject of interest in the service management information D1 from the communication unit 250 to each user terminal 300 via the communication network 15. For example, in step S10, the video data of FIG. 4(B) is transmitted to the user terminal 300A in which the number "1" is registered for the subject of interest in the service management information D1. Also, the video data of FIG. 4(C) is transmitted to the user terminal 300Z in which the number "16" is registered for the subject of interest.

[0079] In addition, in the present system 10, for example, during video distribution, a user terminal 300Z that has applied for a switching option in advance can transmit an instruction to change the subject of interest to the image processing server 200 (S11). In the image processing server 200, the control unit 210 accepts an instruction to change the subject of interest from the user terminal 300 via the communication unit 250 (S12).

[0080] When an instruction to change the subject of interest is received from the user terminal 300 (YES in S12), the control unit 210 updates, for example, the service management information D1 in accordance with the instruction (S13). The subject of interest before and after the change may be managed in the service management information D1.

[0081] The control unit 210 repeats the processing of steps S8 to S13 in a predetermined cycle, for example, until the moving image shooting ends (S14). The predetermined cycle includes, for example, a plurality of frame periods.

[0082] For example, if there is an instruction to change the subject of interest (YES in S12), video data emphasizing the subject of interest updated in step S13 is transmitted to the user terminal 300Z that sent the instruction (S11) from the image processing server 200 in the subsequent step S10. Also, if there is no instruction to change the subject of interest (NO in S12), the control unit 210 does not particularly perform the process of step S13 and proceeds to step S14.

[0083] According to the above operation of the present system 10, for example, for each of the subjects 31, 32 in the subject areas R31, R32 recognized by image recognition (S4) during video shooting by the camera 100, a plurality of processed videos are created (S8) in which different image effects are applied to one subject and another subject. This allows for the creation of a plurality of processed videos in which the subjects to which the image effects are applied are changed, so as to meet the needs of a plurality of users.

[0084] Furthermore, according to the present system 10, a plurality of processed videos are delivered to each user terminal 300 so that the user is provided with a video in which only the subject of the user's interest has been subjected to a visual effect (S10). This makes it easier for the user to see the subject of the user's interest, and enhances the sense of realism of the video viewing experience.

[0085] 2-2. Parallel color preservation processing The parallel color retaining process in step S8 of FIG. 5 will be described in detail with reference to FIGS.

[0086] Fig. 9 is a flowchart illustrating parallel color preservation processing in the present system 10. The processing shown in the flow of Fig. 9 starts, for example, when the image processing server 200 has stored in the storage unit 220 the video data received from each of the cameras 100A to 100E in step S6. Fig. 9 illustrates processing for one frame of captured image 20 in the original video from one camera 100.

[0087] For example, the control unit 210 first causes the image processing unit 120 to perform a monochrome process to change the color of the captured image 20 of the frame to be processed in the original moving image to an achromatic color (S21). Fig. 10(A) shows an example of the processing result of step S21 on the captured image 20 of Fig. 4(A). In step S21, for example, the image processing unit 120 generates a monochrome image G20 by achromatizing the captured image 20, as shown in Fig. 10(A).

[0088] Furthermore, the control unit 210 refers to, for example, the region division map M20 (FIG. 8) and causes the image processing unit 120 to perform mask processing for each of the subject regions R31 and R32 on the captured image 20 of the same frame as in step S21 (S22). FIG. 10(B) illustrates an example of the processing result of step S22 on the captured image 20 of FIG. 4(A).

[0089] In step S22, for example, as shown in Fig. 10(B), the image processing unit 120 generates a mask image G21 by extracting the interior of one subject region R31 of the plurality of subject regions R31 to R32 in the captured image 20. In the example of Fig. 4(A), the image processing unit 120 also generates a mask image of the other subject region R32 in the same manner as above.

[0090] The control unit 210 can manage mask images corresponding to each of the subjects 31 and 32 based on the subject identification information in the meta information of the video data. The masking process in step S22 is executed in parallel for, for example, a plurality of subject regions R31 and R32. If there is a subject that does not appear in the captured image 20 in each frame, the masking process for that subject can be omitted as appropriate.

[0091] Furthermore, the control unit 210 causes the image processing unit 120 to composite a mask image G21 (FIG. 10(B)) of a certain subject 31 with a monochrome image G20 (FIG. 10(A)) for each subject region R31 to R32 (S23). In step S23, for example, the image processing unit 120 generates a plurality of processed images 41 and 42 that retain the colors of the respective subjects 31 and 32, as shown in FIGS. 4(B) and 4(C), for example.

[0092] The compositing process in step S23 is performed in parallel for the multiple mask images generated in step S22. For example, the compositing process for each mask image G21 is performed so that the mask image G21 is superimposed on the monochrome image G20. In this superimposed compositing, the mask image G21 is in a layer above the monochrome image G20, and the transparent attribute is used for the area of ​​the mask image G21 other than the corresponding subject area R31.

[0093] For example, in the parallel color preservation process (S8) of Fig. 5, the control unit 210 executes the processes of steps S21 to S23 shown in the flow of Fig. 9 for multiple frames from multiple cameras 100A to 100E. The processing order at this time is, for example, the chronological order of video shooting. Furthermore, frames from different cameras 100A to 100E that are simultaneously captured in time may be processed in parallel.

[0094] According to the above parallel color preserving process (S8 in FIG. 5), a plurality of processed images 41, 42 in which a plurality of subjects 31, 32 are respectively highlighted can be obtained in parallel (S23).

[0095] The parallel color-preserving process (S8 in FIG. 5) is not limited to the above example. For example, in step S21, instead of the monochrome image G20, multiple mask images may be generated by extracting the areas outside of the individual subject regions R31 and R32, and the monochrome conversion process may be performed on each mask image. In this case, the process of step S21 may also be performed in parallel on the mask images. In addition, in step S23 in this case, by applying a transparent attribute to the areas outside the regions extracted in each mask image, the processed images 41 and 42 can be generated without considering the hierarchical relationship of the layers. Alternatively, the process of step S22 may be omitted, and in step S23, the captured image 20 and the mask images of step S21 may be superimposed and combined.

[0096] In step S22, image processing may be performed to impart various image effects to the mask image G21 (FIG. 10B) that retains the inside of the subject region R31. For example, different color corrections may be performed inside and outside the subject region R31, such as increasing the saturation of the mask image G21 that retains the inside of the subject region R31 and decreasing the saturation of the other mask image.

[0097] In the parallel color preservation process (S8 in FIG. 5), for example, one frame of captured image 20 may be input to multiple image correction circuits, and image processing for the inside and outside of the subject area may be performed in parallel in each image correction circuit. The images thus obtained may be combined to generate processed images 41 and 42.

[0098] In the above description, an example of operation has been described in which the area division map M20 obtained when the camera 100 captures a video has been used. The system 10 does not need to use the area division map M20 obtained by the camera 100, and for example, the control unit 210 may perform the operation of detecting the subject area. In this case, distance measurement information obtained when the camera 100 captured a video may be included in the meta information of the video data to be transmitted (S5 in FIG. 5).

[0099] The processing order of steps S8 and S9 may be reversed. In this case, the processing of step S8 may be performed for the photographing results adopted in the switching processing (S9), and the processing of step S8 may be omitted for the photographing results not adopted in the switching processing.

[0100] Furthermore, in the above explanation, an example of operation in the present system 10 has been described in which each user focuses on one subject, but the present system 10 is not limited to this, and a user may have multiple subjects of interest. For example, a user's betting ticket is not limited to a win bet, but may also be a quinella, exacta, trifecta, trifecta, or box bet, and the image processing server 200 of the present system 10 may appropriately manage identification information of multiple subjects corresponding to such betting tickets in the service management information D1 via information communication. Furthermore, a user can send application information from the user terminal 300 with multiple subjects as subjects of interest (S1) without specifically purchasing a betting ticket.

[0101] In the above case, the image processing server 200 of the present system 10 can generate a processed image by simultaneously highlighting multiple subject areas in the parallel color preservation process (S8) for a combination of subjects registered as a "subject of interest" for each user in the service management information D1. Such a processed image can be transmitted from the image processing server 200 of the present system 10 to a user terminal 300 in which the combination of subjects corresponding to the subject of interest is registered in the service management information D1.

[0102] 3. Summary As described above, the imaging method in this embodiment includes a step (S3) in which the image sensor 115 of the camera 100, as an example of an imaging unit, captures an image of a subject and generates the captured image 20, a step (S8) in which the image processing unit 260 performs image processing on the captured image 20 generated by the image sensor 115, and a step (S8) in which the control unit 210 controls the image processing by the image processing unit 260 based on the captured image 20. In this method, the control unit 210 acquires application information of an example of at least one user instruction that specifies a subject in a moving image to be generated from the captured image 20 (S2), and causes the image processing unit 260 to perform image processing in response to the at least one user instruction so as to generate a moving image including processed images (41, 42) to which different image effects are applied between the inside and outside of a subject area (R31, R32) in the captured image 20 that corresponds to the subject specified by the user instruction (S8).

[0103] According to the above imaging method, image processing is performed for each of the subject regions R31 and R32 in response to a user instruction, making it easier to achieve a desired image effect for the captured subject.

[0104] In the imaging method of this embodiment, at least one user instruction includes, for example, a plurality of user instructions from a plurality of user terminals 300A-300Z, each specifying a different subject among a plurality of subjects 31-32 in a moving image. In response to the plurality of user instructions, control unit 210 controls image processing unit 260 to generate a plurality of moving images in which image effects are respectively applied to a plurality of subject regions R31-R32 corresponding to different subjects (S21-S23). This makes it possible to generate a plurality of moving images in response to the plurality of user instructions, making it easier to achieve the image effects desired by the user.

[0105] In the imaging method of this embodiment, the control unit 210 acquires a user instruction via the communication unit 250 that performs data communication with at least one user terminal 300 corresponding to the at least one user instruction. This method further includes a step (S10) of transmitting, by the communication unit 250, video data representing a video generated in response to the user instruction to the user terminal corresponding to the user instruction.

[0106] In the imaging method of this embodiment, the subject regions R31, R32 are regions that follow the outlines of the subjects 31, 32 in the captured image 20. The image processing unit 260 generates processed images 40, 41 so that the image effects vary along the outlines of the subjects 31, 32 specified by a user instruction in the captured image 20. This makes it possible to vary the image effects between regions that follow the outlines of the subjects in the captured image 20, making it easier to achieve a desired image effect for the captured subject.

[0107] In this embodiment, the imaging method further includes a step (S4) of recognizing subject regions R31, R32 corresponding to subjects 31, 32 in the captured image 20 based on the captured image 20 generated by the image sensor 115. Based on the recognition result of the subject regions R31, R32, the control unit 210 causes the image processing unit 260 to perform image processing so as to impart an image effect to a subject region among the recognized subject regions R31, R32 that corresponds to a subject specified by a user instruction (S8). This makes it easier to achieve a desired image effect for the captured subject, for example, by using the image recognition unit 122 of the camera 100.

[0108] In the imaging method of the present embodiment, the image effect is an effect in which the color tone is adjusted separately between the inside and outside of the subject area. The present system 10 makes it easy to achieve a desired color tone effect for the imaged subject.

[0109] A program including a group of instructions for causing a processor such as the control unit 210 to execute the imaging method of the present embodiment as described above may be provided. Such a program may include a step in which the processor acquires a captured image 20 in which an image of a subject is captured.

[0110] In this embodiment, an image processing server 200, which is an example of an image processing device, includes a communication unit 250, which is an example of a receiving unit, that receives image data including a captured image 20 in which a subject image is captured by the camera 100, an image processing unit 260 that performs image processing on the captured image 20 included in the image data received by the receiving unit, and a control unit 210 that controls the image processing by the image processing unit 260 based on the captured image 20. The control unit 210 acquires at least one user instruction that specifies a subject in a moving image to be generated from the captured image 20 (S2), and in response to the at least one user instruction, causes the image processing unit to perform image processing so as to generate a moving image including processed images to which different image effects are applied between the inside and outside of a subject area in the captured image 20 corresponding to the subject indicated by the user instruction (S8). This makes it easier to achieve a desired image effect for the captured subject.

[0111] In the image processing server 200 of this embodiment, the communication unit 250 may receive a plurality of image data from a plurality of cameras 100 (S6). The control unit 210 may cause the image processing unit 260 to perform image processing so as to generate a moving image based on the plurality of image data received from the plurality of cameras 100 via the communication unit 250 (S8, S9). This makes it easier to achieve a desired image effect for a captured subject in a moving image using a plurality of cameras 100.

[0112] (Other embodiments) As described above, the first embodiment has been described as an example of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. Furthermore, it is also possible to combine the components described in the first embodiment above to create new embodiments.

[0113] In the first embodiment described above, an example in which the imaging system 10 is applied to live streaming of horse racing has been described, but the present disclosure is not particularly limited to this. The imaging system 10 of this embodiment is not limited to the above example, and may also be applied to live streaming of various races, sports, concerts, etc. Furthermore, the subject of interest in this system 10 is not particularly limited to the example in the first embodiment, and may also be a person or an animal, or a moving object such as a vehicle.

[0114] For example, when the present system 10 is applied to live streaming of a soccer match, multiple cameras 100 may be arranged in different areas of the field. For example, a camera 100 that overlooks the entire field, a camera 100 that tracks the ball, a camera for photographing one goal, a camera for photographing the other goal, and cameras that divide the field into four and photograph each of these may be used. Alternatively, multiple cameras 100 may be used that track each player on the pitch. In this case, multiple cameras 100 may be used, such as on all four sides of each player.

[0115] In such soccer video distribution, the present system 10 may not only basically distribute video by switching on the video distribution side as in the first embodiment, but may also individually distribute video requested by the user for video that is not being distributed. For example, if a user is receiving the basic video distribution from the present system 10 showing a player with uniform number 10, and is also interested in the movements of a player with uniform number 2 who is not shown in the distributed video, the present system 10 may be configured to switch to distribution of video showing the latter from the results of shooting by the multiple cameras 100. In this case, the present system 10 can perform image processing such as color preservation processing on the latter subject and distribute it to the user.

[0116] As described above, in a soccer broadcast, for example, the system 10 can deliver, in response to a user's request, footage that follows a player who is not shown in the basic broadcast footage, so that when a user wants to see the movements of a player of interest, the user can view footage that highlights the player of interest that is not included in the basic video delivery. The method of the system 10 can also be applied to, for example, a concert video of a singer group, in which a user can deliver footage focusing on a specific singer from the singer group.

[0117] In addition, in each of the above embodiments, an example has been described in which the imaging system 10 is applied to live streaming. However, the present system 10 is not limited to live streaming, and may also be adapted to post-edit a video that has already been shot and distribute the processed video.

[0118] In addition, in each of the above embodiments, an operation example has been described in which a plurality of cameras 100 are used in the imaging system 10. In this embodiment, the imaging system 10 may perform the same image processing as in each of the above embodiments on a video captured by one camera 100, and deliver the video.

[0119] Furthermore, in each of the above embodiments, an example has been described in which the image processing server 200 in the imaging system 10 performs image processing for each of the subjects 31 and 32, but the present disclosure is not limited to this. In the imaging system 10 of this embodiment, the control unit 135 may cause the image processing engine 120 in the camera 100 to perform some or all of the various processes that the control unit 210 causes the image processing unit 260 in the image processing server 200 of each of the above embodiments to perform. Furthermore, the camera 100 of this embodiment may distribute various processed videos, in which image processing has been performed on each of a plurality of subjects, from the communication module 155 to various user terminals 300 via the communication network 15.

[0120] In this embodiment, the camera 100 includes an image sensor 115 that captures an image of a subject and generates a captured image 20, an image processing engine 120 that is an example of an image processing unit that performs image processing on the captured image 20 generated by the image sensor 115, and a control unit 135 that controls image processing by the image processing engine 120 based on the captured image 20. The control unit 135 may acquire at least one user instruction that specifies a subject in a video to be generated from the captured image 20, and may cause the image processing engine 120 to perform image processing in response to the at least one user instruction so as to generate a video including processed images to which different image effects are applied between the inside and outside of a subject area in the captured image 20 that corresponds to the subject specified by the user instruction. This makes it easier to achieve a desired image effect for the captured subject.

[0121] In this embodiment, camera 100 may further include image recognition unit 122 that recognizes a subject area corresponding to a subject in captured image 20 based on captured image 20 generated by image sensor 115. Control unit 135 controls the image capturing operation by image sensor 115 based on subject areas R31, R32 recognized by image recognition unit 122. This makes it easier to achieve a desired image effect for the captured subject using image recognition unit 122 of camera 100.

[0122] In this embodiment, the imaging system 10 includes a camera 100, which is an example of an imaging device, and an image processing server 200 of an image processing device. The camera 100 includes an image sensor 115 that captures an image of a subject and generates a captured image 20, and a communication module 155, which is an example of a transmitter, that transmits image data representing the captured image 20 to the image processing device. The image processing device includes a communication unit 250 that receives image data from the camera 100 and an image processing unit 260 that performs image processing on the captured image 20 represented by the image data based on the image data received by the communication unit 250. The camera 100 or the image processing device includes a control unit 135, 210 that acquires at least one user instruction specifying a subject in a video to be generated from the captured image 20, and causes the image processing unit 260 to perform image processing in response to the at least one user instruction to generate a video including processed images to which different image effects are applied between the inside and outside of a subject area corresponding to the subject specified by the user instruction in the captured image 20. This makes it easier to achieve a desired image effect for the captured subject.

[0123] Furthermore, in the above-described embodiments, color preservation processing has been described as an example of image processing in the imaging system 10. However, the present system 10 is not limited to this, and various other image processing methods may be employed. For example, the present system 10 may perform image processing that creates different color tonal effects inside and outside the subject area. Such color tonal effects may employ various color correction methods, such as color grading or tone mapping. Furthermore, the present system 10 may perform image processing with a bokeh effect instead of or in addition to such a color tonal effect. Such image processing is performed by applying a bokeh effect to selectively blur the outside of the subject area. The bokeh effect may be applied as an image effect similar to motion blur caused by movement of a subject other than the one being emphasized.

[0124] The image effects applied by the present system 10 are not limited to those described above, and may be visual effects such as various video effects created by editing or processing images or videos. For example, time effects such as slow motion and time lapse, or transition effects such as whiteout and fade-in / fade-out may be employed in the present system 10. Special effects for creating scenes or images that are impossible to realize in reality, such as explosions, collapses, flight, or the appearance of ghosts or monsters, may also be included. Special effects that create realistic images or fantasy worlds using CG may also be applied to the present system 10.

[0125] (Example) Various aspects of the present disclosure are listed below.

[0126] A first aspect of the present disclosure is an imaging method including the steps of: an imaging unit capturing an image of a subject and generating a captured image; an image processing unit performing image processing on the captured image generated by the imaging unit; and a control unit controlling the image processing by the image processing unit based on the captured image. In this method, the control unit acquires at least one user instruction specifying a subject in a moving image generated from the captured image, and in response to the at least one user instruction, causes the image processing unit to perform image processing so as to generate a moving image including processed images to which different image effects are applied between an inside and an outside of a subject area in the captured image corresponding to the subject specified by the user instruction.

[0127] In a second aspect, in the imaging method according to the first aspect, the at least one user instruction includes a plurality of user instructions specifying different subjects among a plurality of subjects in the moving image, and the control unit controls the image processing unit in response to the plurality of user instructions to generate a plurality of moving images in which image effects are applied to a plurality of subject areas corresponding to the different subjects, respectively.

[0128] In a third aspect, in the imaging method according to the first or second aspect, the control unit acquires user instructions via a communication unit that performs data communication with at least one user terminal corresponding to the at least one user instruction, and the method further includes a step of transmitting, by the communication unit, video data representing a video generated in response to the user instructions to the user terminal corresponding to the user instruction.

[0129] In a fourth aspect, in the imaging method according to any one of the first to third aspects, the subject region is a region that follows the outline of the subject in the captured image, and the image processing unit generates a processed image so as to vary the image effect along the outline of the subject specified by a user instruction in the captured image.

[0130] In a fifth aspect, the imaging method according to any one of the first to fourth aspects further includes a step of recognizing a subject area in the captured image, based on the captured image generated by the imaging unit, and the control unit causes the image processing unit to perform image processing based on the subject area recognition result so as to impart an image effect to a subject area, among the recognized subject areas, that corresponds to a subject designated by a user instruction.

[0131] A sixth aspect is the imaging method according to any one of the first to fifth aspects, wherein the image effect is an effect in which color tones are adjusted separately between the inside and outside of the subject area.

[0132] A seventh aspect is an image processing device including a receiving unit that receives image data including a captured image of a subject captured by an imaging device, an image processing unit that performs image processing on the captured image included in the image data received by the receiving unit, and a control unit that controls the image processing by the image processing unit based on the captured image. The control unit acquires at least one user instruction that specifies a subject in a moving image generated from the captured image, and in response to the at least one user instruction, causes the image processing unit to perform image processing so as to generate a moving image including processed images to which different image effects are applied between the inside and outside of a subject area in the captured image that corresponds to the subject specified by the user instruction.

[0133] In an eighth aspect, in the image processing device described in the seventh aspect, the receiving unit receives multiple image data from multiple imaging devices, and the control unit causes the image processing unit to perform image processing so as to generate a moving image based on the multiple image data received from the multiple imaging devices via the receiving unit.

[0134] A ninth aspect is an imaging device including an imaging unit that captures an image of a subject and generates a captured image, an image processing unit that performs image processing on the captured image generated by the imaging unit, and a control unit that controls the image processing by the image processing unit based on the captured image. The control unit acquires at least one user instruction that specifies a subject in a moving image to be generated from the captured image, and in response to the at least one user instruction, causes the image processing unit to perform image processing so as to generate a moving image including processed images to which different image effects are applied between the inside and outside of a subject area in the captured image that corresponds to the subject specified by the user instruction.

[0135] In a tenth aspect, the imaging device according to the ninth aspect further includes an image recognition unit that recognizes a subject area corresponding to a subject in the captured image based on the captured image generated by the imaging unit, and a control unit that controls an image capturing operation by the imaging unit based on the subject area recognized by the image recognition unit.

[0136] An eleventh aspect is an imaging system including an imaging device and an image processing device. The imaging device includes an imaging unit that captures an image of a subject and generates a captured image, and a transmission unit that transmits image data representing the captured image to the image processing device. The image processing device includes a receiving unit that receives image data from the imaging device and an image processing unit that performs image processing on the captured image represented by the image data based on the image data received by the receiving unit. The imaging device or the image processing device includes a control unit that acquires at least one user instruction that specifies a subject in a moving image to be generated from the captured image, and causes the image processing unit to perform image processing in response to the at least one user instruction so as to generate a moving image including processed images to which different image effects are applied between the inside and outside of a subject area corresponding to the subject specified by the user instruction in the captured image.

[0137] As described above, the embodiments have been described as examples of the technology in the present disclosure, and for that purpose, the accompanying drawings and detailed description have been provided.

[0138] Therefore, the components shown in the accompanying drawings and detailed description may include not only essential components for solving the problem, but also components that are not essential for solving the problem in order to illustrate the above technology. Therefore, the fact that these non-essential components are shown in the accompanying drawings or detailed description should not be interpreted as immediately indicating that these non-essential components are essential.

[0139] Furthermore, since the above-described embodiments are intended to illustrate the technology of the present disclosure, various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents. [Industrial Applicability]

[0140] The present disclosure is applicable to various uses in which image processing is performed on an image of a captured subject. [Explanation of symbols]

[0141] 10. Imaging System 100 cameras 115 Image Sensor 120 Image Processing Engine 130 Display Monitor 135 Control Unit 150 Operation section 155 Communication Module 200 Image Processing Server 210 Control Unit 250 Communications Department 260 Image Processing Unit 300 User Terminals

Claims

1. an imaging unit capturing an image of a subject and generating a captured image; an image processing unit performing image processing on the captured image generated by the imaging unit; a step in which a control unit controls image processing by the image processing unit based on the captured image, The control unit acquiring at least one user instruction that specifies a subject in a moving image generated from the captured images; In response to the at least one user instruction, the image processing unit is caused to perform the image processing so as to generate the moving image including a processed image to which different image effects are applied between an inside and an outside of a subject area in the captured image corresponding to the subject designated by the user instruction. Imaging method.

2. the at least one user instruction includes a plurality of user instructions specifying different subjects among a plurality of subjects in the video; The control unit controls the image processing unit to generate a plurality of moving images in which image effects are applied to a plurality of subject areas corresponding to the different subjects, in response to the plurality of user instructions. The imaging method according to claim 1 .

3. the control unit acquires the user instruction via a communication unit that performs data communication with at least one user terminal corresponding to the at least one user instruction; The method further includes transmitting, by the communication unit, video data representing the video generated in response to the user instruction to a user terminal corresponding to the user instruction. The imaging method according to claim 1 .

4. the subject region is a region in the captured image that follows the outline of the subject, The image processing unit generates the processed image so that the image effect varies along the outline of the subject specified by the user instruction in the captured image. The imaging method according to claim 1 .

5. The method further includes a step of recognizing a subject area corresponding to the subject in the captured image based on the captured image generated by the imaging unit, The control unit causes the image processing unit to perform the image processing so as to impart the image effect to a subject area corresponding to a subject designated by the user instruction among the recognized subject areas based on the recognition result of the subject area. The imaging method according to claim 1 .

6. The image effect is an effect in which the color tone is adjusted separately between the inside and outside of the subject area. The imaging method according to claim 1 .

7. a receiving unit that receives image data including a captured image of a subject captured by an imaging device; an image processing unit that performs image processing on a captured image included in the image data received by the receiving unit; a control unit that controls image processing by the image processing unit based on the captured image, The control unit acquiring at least one user instruction that specifies a subject in a moving image generated from the captured images; In response to the at least one user instruction, the image processing unit is caused to perform the image processing so as to generate the moving image including a processed image to which different image effects are applied between an inside and an outside of a subject area in the captured image corresponding to the subject indicated by the user instruction. Image processing device.

8. an imaging unit that captures an image of a subject and generates a captured image; an image processing unit that performs image processing on the captured image generated by the imaging unit; a control unit that controls image processing by the image processing unit based on the captured image, The control unit acquiring at least one user instruction that specifies a subject in a moving image generated from the captured images; In response to the at least one user instruction, the image processing unit is caused to perform the image processing so as to generate the moving image including a processed image to which different image effects are applied between an inside and an outside of a subject area in the captured image corresponding to the subject designated by the user instruction. Imaging device.

9. An imaging system including an imaging device and an image processing device, The imaging device is an imaging unit that captures an image of a subject and generates a captured image; a transmission unit that transmits image data representing the captured image to the image processing device, The image processing device includes: a receiving unit that receives the image data from the imaging device; an image processing unit that performs image processing on a captured image indicated by the image data based on the image data received by the receiving unit, The imaging device or the image processing device is acquiring at least one user instruction that specifies a subject in a moving image generated from the captured images; and a control unit that causes the image processing unit to perform the image processing in response to the at least one user instruction so as to generate the moving image including a processed image to which different image effects are applied between an inside and an outside of a subject area in the captured image that corresponds to the subject specified by the user instruction. Imaging system.

Images