Display control apparatus, control method, and storage medium

Abstract

To provide a display control device capable of desirably controlling a display state of a subject included in a photographed image.SOLUTION: A display control device according to the present disclosure includes detection means for detecting, in a captured image including a predetermined subject, a region of an object that is a subject different from the predetermined subject, and display control means for causing display means to display an image in which a display mode of the object is controlled with respect to the region of the object in the image, in which the display control means controls, in response to detection of a trigger for switching the display mode of the object while the captured image is sequentially acquired, the display mode of the object to a display mode corresponding to the trigger.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a display control device, a control method, and a program. [Background technology]

[0002] In recent years, technologies for sharing captured images in real time have become widespread, such as through online meetings using web cameras and personal video distribution. While such technologies improve the quality of communication by visually sharing images of the objects being explained, they can also result in the sharing of information that should not be shared, such as private or confidential information.

[0003] Patent Document 1 discloses a technology for controlling the focus position of a camera to capture an image of an object presented in front of the camera and the photographer in focus as intended by the photographer. Patent Document 2 discloses a technology for protecting the privacy of a subject by applying mosaic processing to subjects determined to have been unintentionally captured in an image capture device that records information for protecting the privacy of the subject along with the image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-77931 [Patent Document 2] Japanese Patent Application Laid-Open No. 2011-101161 Summary of the Invention [Problem to be solved by the invention]

[0005] The technology disclosed in Patent Document 1 controls the depth of field of the camera to adjust the focus of the target subject, so that other subjects within the depth of field are also displayed in focus, just like the target subject. Furthermore, the technology disclosed in Patent Document 2 sets a privacy protection level based on information determined after shooting, such as the degree of presence within the angle of view and the number of times the subject has been out of frame, and applies mosaic processing during playback of the video based on that setting. In other words, these patent documents did not consider controlling the display of the area of ​​the intended subject while sharing the captured image.

[0006] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a technique that makes it possible to control the display state of a subject included in a captured image as desired. [Means for solving the problem]

[0007] In order to solve this problem, for example, a display control device of the present invention has the following configuration: That is, it has a detection means that detects an area of ​​an object that is a subject different from a predetermined subject in a captured image including the predetermined subject, and a display control means that causes a display means to display an image in which a display mode of the object is controlled for the area of ​​the object in the image, wherein the display control means controls the display mode of the object to a display mode corresponding to the trigger in response to detection of a trigger for switching the display mode of the object while captured images are being sequentially acquired. [Effects of the Invention]

[0008] According to the present invention, it is possible to control the display state of a subject included in a captured image as desired. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of each device in a system according to an embodiment of the present invention. [Figure 2]FIG. 1 is a block diagram showing an example of a functional configuration realized by an electrical circuit of a display control device according to a first embodiment. [Figure 3] 1 is a flowchart showing a series of operations in a display control process according to the first embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of control of the display mode of an object (from display to non-display) in the first embodiment. [Figure 5] FIG. 10 is a diagram illustrating another example of control of the display mode of an object (from hidden to displayed by operation) in the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating another example of control of the display mode of an object (from hidden to displayed by an action) in the first embodiment. [Figure 7] FIG. 10 is a diagram for explaining an example of replacing output audio data in the first embodiment. [Figure 8] FIG. 10 is a block diagram showing an example of a functional configuration realized by an electric circuit of a display control device according to a second embodiment. [Figure 9] 10 is a flowchart showing a series of operations in a display control process according to the second embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of control of the display mode of an object using shooting environment information in the second embodiment. [Figure 11] FIG. 10 is a diagram illustrating another example of control of the display mode of an object using shooting environment information in the second embodiment. [Figure 12] FIG. 10 is a diagram illustrating an example of control of the display mode of an object in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] (Embodiment 1) Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] (Outline of the embodiment) As mentioned above, online conferences using webcams and personal video distribution have become increasingly common in recent years, and in online conferences and live video distribution, images captured using webcams are shared with participants in real time. Taking advantage of the advantage of being able to share information visually through images, in online conferences, the broadcaster may present objects, documents, or other presentation items in front of the camera and introduce or explain them to the participants using the images.

[0012] On the other hand, when sharing information visually using images, unnecessary information may also be shared. For this reason, in online conferences, users may use features built into online conference software to erase the background or replace it with a dummy image in order to protect privacy and confidential information. In such cases, objects that users want to show to other conference participants may be judged as background and not displayed, contrary to the user's intention, or conversely, people or objects that users want to hide may be captured in the image. In other words, it is desirable to appropriately control the display mode of subjects that the broadcaster wants to display or hide in accordance with their intentions.

[0013] Therefore, in this embodiment, a trigger for switching the display mode of an object is detected while sequentially acquiring captured images, and in response to the detection of the trigger, the display mode of the object is controlled to a display mode corresponding to the trigger. In this way, in this embodiment, it is possible to control the display state of a subject included in a captured image as desired.

[0014] (Configuration example according to this embodiment) In the following, an example of a display control device will be described in which an electronic device such as a personal computer capable of controlling the display mode of a subject is used. However, this embodiment can also be applied to other devices capable of controlling the display mode of a subject. These devices may include, for example, digital cameras, smartphones, game consoles, tablet devices, wearable devices, and devices for broadcasting or video distribution systems.

[0015] The configuration of each device in a system including a display control device will be described below with reference to Fig. 1. The display control device of this embodiment includes a camera 100, which is an example of an image acquisition device, and a display control device main body 200. The camera 100 and the display control device main body 200 are connected via an information communication path. The camera 100 is, for example, a web camera or an external camera. Note that, although a detachable configuration is described as an example in this embodiment, the camera 100 may also be configured integrally with the display control device main body 200.

[0016] The display control device main body 200 can be connected to a video distribution device 300 on a network via wireless or wired communication. The video distribution device 300 is, for example, a video distribution server having a video distribution function.

[0017] (Camera configuration) The camera 100 functions as the image acquisition means of the display control device. The camera 100 includes an aperture 11, an image stabilization lens group 12, a focus / zoom lens group 13, and other components as an imaging optical system, and directs an optical image of a subject to an image sensor 15. A drive control circuit 16 controls an actuator (not shown) based on the results of arithmetic processing by an arithmetic processing circuit 20a of the display control device main body, which are transmitted and received via a communication unit 17 on the camera side (described later) and a communication unit 25 on the display control device main body side. This control controls the aperture, lens group, and mechanical shutter 14 of the camera 100 and the display control device main body 200. The camera 100 also includes an image sensor 15 that photoelectrically converts the formed optical image to generate an image signal, and the mechanical shutter 14 that adjusts the exposure time for exposing the image sensor 15. A video signal is formed by sequentially acquiring multiple image signals from the image sensor 15.

[0018] Camera 100 further includes a communication unit 17. Camera 100 controls the aperture and lens group based on control signals transmitted and received via communication unit 17 on the camera side and communication unit 25 on the display control device main body side. Camera 100 also controls the drive timing of image sensor 15 and the shutter speed of mechanical shutter 14 based on the control signal to capture an image with appropriate exposure. Note that if image sensor 15 has an electronic shutter function that can adjust exposure time by controlling signal accumulation time and signal readout time, mechanical shutter 14 is not necessary. Also, if a camera has both mechanical shutter 14 and electronic shutter function and adjusts exposure time using the electronic shutter, mechanical shutter 14 is set to a fully open state.

[0019] The camera 100 also includes an audio input unit 18. The audio input unit 18 includes a microphone or the like, converts input audio into an electrical signal, and outputs the electrical signal as audio data to the display control device main body 200 via the camera-side communication unit 17. When the display control device main body 200 receives the audio data, it adds the audio data to a video signal, stores the video signal in the storage unit 29, and records the audio data as electronic data. In the example of this embodiment, the audio input unit 18 is built into the camera side, but it may also be built into the display control device main body 200, or may be connected to an external terminal (not shown) as an external device.

[0020] (Configuration of display control device) The display control device main body 200 includes a first display unit 21 and a second display unit 22 that can display images captured by the camera 100 and various settings for the camera when capturing images. The first display unit 21 and the second display unit 22 are configured as display devices such as a liquid crystal panel or an organic electroluminescence (EL) panel. When the camera 100 is attached to or built into the display control device main body 200, the first display unit 21 and the second display unit 22 may be provided on the rear surface of the display control device main body 200, opposite the camera 100. In this embodiment, the first display unit 21 and the second display unit 22 are two display units integrated with the display control device main body 200. However, the first display unit 21 and the second display unit 22 may be configured to divide the display area within the screen of a single display device. Alternatively, the first display unit 21 and the second display unit 22 may be configured as external devices that are detachable from the display control device main body 200.

[0021] The display control device main body 200 includes an electric circuit 20. The electric circuit 20 includes an arithmetic processing circuit 20a, a memory circuit 20b, a video processing circuit 20c, a video compression circuit 20d, etc. The arithmetic processing circuit 20a includes one or more processors such as a CPU or an MPU that performs various types of arithmetic processing for controlling the operations of the camera 100 and the display control device main body 200. The arithmetic processing circuit 20a controls each part of the camera 100 and the display control device main body 200 by executing a control program stored in a storage unit 29. The control program here includes a program that performs the display control processing of this embodiment.

[0022] The memory circuit 20b is used as a working memory for expanding programs read from the storage unit 29, a buffer memory for temporarily storing images received from the camera 100, and a memory for displaying images on the first display unit 21 and the second display unit 22.

[0023] The video processing circuit 20c converts the video signal based on the image captured by the imaging element 15 into digital data and performs various types of video processing. The video data output from the video processing circuit 20c is output to the first display unit 21 and the second display unit 22, or is compressed into a predetermined data format by the video compression circuit 20d and output to the storage unit 29 for recording.

[0024] The video compression circuit 20d compresses and encodes the video data output from the video processing circuit 20c into a predetermined data format to generate a video file.

[0025] The display control device main body 200 includes an operation input unit 28 such as a switch, button, or touch panel that accepts user operations. In this embodiment, the operation input unit 28 includes a shutter switch that instructs preparation for shooting or the start of shooting. Lightly pressing the shutter switch to the first position, a so-called "half-press," initiates operations such as autofocus processing, autoexposure processing, and autowhite balance processing. Furthermore, pressing the shutter switch from the half-press to the second position, a so-called "full press," activates the mechanical shutter 14 or the electronic shutter function of the image sensor 15. Thereafter, a series of shooting processing operations, from reading out signals from the image sensor 15 to writing video data to the memory unit 29, is initiated.

[0026] Operation input unit 28 may be provided with a switch that allows the user to set exposure conditions for shooting with camera 100. Operation input unit 28 may also be provided with switches that turn on / off input from audio input unit 18, turn on / off output from audio output unit 23, and switch the display on first display unit 21 and second display unit 22. Operation input unit 28 may also be provided with a switch that allows the user to turn on or off a video distribution function of video distribution device 300, which will be described later. Operation input unit 28 may be configured to receive user operations as a touch panel integrated with first display unit 21 and second display unit 22.

[0027] The display control device main body 200 also includes a communication unit 25. The communication unit 25 includes an interface circuit for communicatively connecting the display control device main body 200 to external devices via a network such as the Internet. The display control device main body 200 can transmit and receive data to and from external devices connected to a wired or wireless network via the communication unit 25. For example, the display control device main body 200 can control the communication unit 25 to output video data processed by the video processing circuit to a video distribution device 300 on the network. The communication unit 25 also communicates with the communication unit 17 on the camera side to transfer images and audio captured by the camera 100.

[0028] The display control device main body 200 includes a storage unit 29 such as a memory card or a hard disk. The storage unit 29 stores programs executed by the arithmetic processing circuit 20a. The storage unit 29 also stores video files compressed into a predetermined format by the video compression circuit 20d, and previously recorded video files are read out as needed. The storage unit 29 may be detachable from the display control device main body 200, or may be built into the display control device main body 200.

[0029] (Configuration of video distribution device) Next, the configuration and functions of the video delivery device 300 of this embodiment will be described with reference to Fig. 1. The video delivery device 300 includes a control unit 30, a communication unit 31, and a streaming processing unit 32. The control unit 30 includes one or more processors, such as a CPU or MPU, that perform various types of arithmetic processing to control the operation of the video delivery device 300. The control unit 30 controls each unit of the video delivery device 300 by executing a predetermined program. The predetermined program may, for example, realize processing related to video delivery.

[0030] The communication unit 31 is connected to the communication unit 25 of the display control device main body 200 via a network, and is capable of transmitting and receiving data to and from the display control device main body 200 and external devices. The communication unit 31 outputs video data transmitted from the communication unit 25 of the display control device main body 200 to the streaming processing unit 32.

[0031] The streaming processing unit 32 creates a video for distribution based on the video data transmitted from the communication unit 25 and transmits it to the communication unit 31. The streaming-processed video data is transmitted via the communication unit 31 to a viewer's device (not shown in FIG. 1).

[0032] (Example of functional configuration realized by electrical circuits) 2 conceptually illustrates an example of a functional configuration realized by the electric circuit 20. The electric circuit 20 includes an image acquisition unit 101, a main subject detection unit 102, an object detection unit 103, a background separation unit 104, a display switching instruction unit 105, a background replacement unit 106, a display switching unit 107, a voice recognition unit 109, and an action detection unit 108. Each of these processing units is stored as a program in the memory circuit 20b, for example, and is realized by the arithmetic processing circuit 20a executing the program.

[0033] Image acquisition unit 101 sequentially acquires images from image sensor 15 of camera 100 in real time via communication unit 17 of camera 100 and communication unit 25 of display control device main body 200 (that is, acquires captured video).

[0034] Main subject detection unit 102 detects the area of ​​the main subject from images sequentially acquired by image acquisition unit 101. Here, the main subject is the central subject within the image, for example, a human subject such as the photographer when the photographer is photographing himself. Main subject detection unit 102 stores feature amounts such as shape information and color information of a person's face or body in advance, and detects the area of ​​the main subject included in the image in real time based on this stored information. Note that the main subject may also be a subject that is the target of processes such as AF and frame display.

[0035] The object detection unit 103 detects the area of ​​an object, which is a subject different from the main subject, from sequentially acquired images. The object detection unit 103 stores feature quantities such as the shape, color, and information of the subject in advance, and detects the area of ​​the object included in the image based on this information. The object in this embodiment may be any object having a shape and color that can be detected from the captured video, and the specific form is not limited to this embodiment. Furthermore, if there are multiple objects in the captured image, the areas of multiple objects may be detected. Note that the detection of the main subject and object areas may be performed by one or more machine learning models that have previously learned feature quantities such as the shape, color, and information of subjects such as people and objects.

[0036] Background separation unit 104 recognizes and separates, as a background region, the image region other than the main subject detected by main subject detection unit 102 and the object detected by object detection unit 103. The background region may also include small subjects on the screen that were not detected as the main subject or object.

[0037] Display switching instruction unit 105 acquires display switching trigger information, which is display switching instruction information from the user (instruction information for controlling the display mode of the main subject or object), and issues a display switching control instruction to display switching unit 107. The display switching trigger information may be operation information of a button or touch panel from operation input unit 28, or may be speech information from the user, which will be described later.

[0038] The background replacement unit 106 replaces the background region of the image separated by the background separation unit 104 with an image obtained by blurring the original background image, or with another image.

[0039] The display switching unit 107 controls the display mode of the object detected by the object detection unit 103 (e.g., performs display switching) based on an instruction from the display switching instruction unit 105. The display switching unit 107 can perform a hide process in which the area of ​​the object is replaced with an image interpolated from the surrounding background area, thereby displaying the object as if it were not present. The display switching unit 107 can also perform a process in which the object is displayed in a blurred image to the extent that detailed information about the object cannot be discerned, for example, to change the object from a identifiable state to an indistinguishable state. When the display switching unit 107 receives an instruction to re-display (i.e., to display) an object that has already been hidden, it can perform a process in which the original image of the object is displayed. In this way, the control of the display mode according to this embodiment may include controlling the display mode of the object from either a state in which the object is displayed or a state in which the object is hidden to the other. The control of the display mode according to this embodiment may also include controlling the display mode from either a state in which the object is identifiable to a state in which the object is indistinguishable (including blurring the object) to the other.

[0040] Action detection unit 108 detects actions, which are specific movements of the main subject or object, from the real-time captured video acquired by image acquisition unit 101. When the main subject or object is a person, the detected actions include hand or finger movements indicating the direction of the object whose display is to be switched. Alternatively, the detected actions include specific arm or body movements (also called gestures) made by the person who is the subject to switch their own display.

[0041] The voice recognition unit 109 detects and recognizes voice indicating the name of an object, voice indicating the content of display switching, and specific voice related to shooting environment information suggesting the shooting location and surrounding information from the voice data acquired from the voice input unit 18. Note that it may also detect and recognize utterances indicating the direction of the object whose display is to be switched.

[0042] The voice processing unit 110 processes the voice data acquired from the voice input unit 18 based on the voice recognized by the voice recognition unit 109. For example, the voice processing unit 110 can change the voice emitted by an object whose display mode is controlled to be hidden to a mode different from the voice emitted when the object is displayed. As an example, the voice processing unit 110 can process the voice data corresponding to the voice recognized by the voice recognition unit 109 to silence or replace it with a different voice.

[0043] (A series of operations related to display control processing) Next, the display control process in the first embodiment will be described with reference to Fig. 3. The flowchart shown in Fig. 3 shows a series of operations in the display control process in this embodiment. Note that each step in the display control process is realized by the arithmetic processing circuit 20a, which is part of the electric circuit 20, executing a program stored in the memory circuit 20b. Furthermore, this display control process is started by, for example, an input operation on the operation input unit 28 to start the shooting and image display operations.

[0044] In step S301, the image acquisition unit 101 of the electric circuit 20 acquires real-time captured video from the image sensor 15 (sequentially acquires captured images). In step S302, the main subject detection unit 102 and the object detection unit 103 start detecting the area of ​​the main subject and the area of ​​an object, which is a subject other than the main subject, in the acquired image. A known technique such as deep learning can be used to detect the area of ​​the object. For example, a machine learning model is trained in advance using multiple images to detect the object to be detected, and the area of ​​the main subject or object in the image is detected using the trained machine learning model. In this embodiment, the areas of multiple objects may be detected. Next, the background separation unit 104 separates and recognizes (detects) the image area other than the detected main subject and object as a background area. Note that step S302 may be executed each time an image constituting the real-time captured video is acquired (e.g., for each frame), or may be executed periodically.

[0045] In step S303, the background replacement unit 106 determines whether the user has issued an instruction to hide the background (an instruction to hide the background). If the background replacement unit 106 determines that the user has issued an instruction to hide the background, the process proceeds to step S304; otherwise, the process proceeds to step S305. The instruction from the user may be issued via an operation member such as a button or lever provided on the operation input unit 28. Alternatively, if the first display unit 21 or the second display unit 22 has a touch panel function, the instruction from the user may be issued by touching with a finger a position or area where the display is to be switched. Furthermore, the display / non-display of the background area may be set in advance by settings or the like.

[0046] In step S304, the background replacement unit 106 executes a process of hiding the image area recognized as the background in step S302 (background hiding process). The background hiding process is a process of generating an image in which the background area is replaced with a different image unrelated to the original background image included in the captured image, or an image in which the original background image is blurred.

[0047] In step S305, the display switching instruction unit 105 determines whether there is a display switching instruction for the object detected in step S302. If it is determined that there is a display switching instruction for the object from the user, the display switching instruction unit 105 proceeds to step S306; otherwise, the display control process ends. The display switching instruction from the user may be an instruction via an operation member such as a button or lever of the operation input unit 28. Alternatively, if the first display unit 21 or the second display unit 22 has a touch panel function, the instruction from the user may be an instruction to touch with a finger the area of ​​the subject for which the display is to be switched, or the position of information indicating the target of the display switching (for example, the object name).

[0048] In step S306, display switching instruction unit 105 determines whether the display switching target (e.g., an object) is a person subject. To determine whether the display switching target is a person subject, for example, a known technique using deep learning or the like can be used. If display switching instruction unit 105 determines that the display switching target is a person subject, it proceeds to step S307, and if not, it proceeds to step S308.

[0049] In step S307, display switching instruction unit 105 determines whether to switch the display by detecting an action of the person subject itself as a display switching method for the person subject. If display switching instruction unit 105 determines that an action detection instruction has been issued by the user, it proceeds to step S309; ​​otherwise, it proceeds to step S308.

[0050] In step S308, the display switching unit 107 executes display switching processing by displaying / hiding the image area of ​​the object determined in step S306 as not being a human subject. The hiding processing may include replacing the target area with an image generated by interpolation using known technology from the image surrounding the hidden target area, thereby displaying the target object as if it does not exist. Alternatively, the hiding processing may be replacing the target area with an image in which the object is blurred, making it impossible to identify detailed information about the object. Furthermore, the object may be replaced with an image of another alternative object. After executing the display switching processing, the display switching unit 107 then ends the series of operations of the display control processing.

[0051] In step S309, if it is determined in step S307 that an action detection instruction has been issued by the user, display switching unit 107 executes display switching processing by detecting an action of the display switching target itself, which is a person subject. After executing the display switching processing, display switching unit 107 then ends the series of operations of the display control processing. However, the processing of steps S301 to S309 may be executed repeatedly. In this way, by controlling the display mode of the object separately from the display mode of the main subject, display switching unit 107 can perform appropriate display control on the object, such as hiding the display mode of the object, while maintaining the display mode of the main subject.

[0052] <Example of display switching process based on direct display switching instructions from the user> <<Example of switching from visible to hidden>> Both the first display unit 21 and the second display unit 22 of the display control device main body 200 are live view screens equipped with touch panel functionality, and constitute part of the operation input unit 28, which inputs instructions from the user by touching the screen. The first display unit 21 displays images to be live streamed. The second display unit 22 displays images that can only be seen by the photographer or broadcaster. The photographer or broadcaster can also recognize information about hidden objects by visually viewing the second display unit 22. Note that, in the example of the present embodiment, an example is shown in which the live streamed images and the images that can only be seen by the photographer or broadcaster are displayed separately on separate display units, but the two types of images may also be displayed side by side on the screen of the first display unit 21.

[0053] FIG. 4(a) shows an example of an original image (captured image) acquired by the image acquisition unit 101, and is displayed as a live view screen on the first display unit 21. In this example, all subjects are displayed. FIG. 4(c) shows information about the display state of FIG. 4(a), which is displayed on the live view screen of the second display unit 22 simultaneously with FIG. 4(a). The information about the display state may indicate, for example, that three objects, Obj1 (house), Obj2 (tree), and Obj3 (apple), have been detected by the object detection unit 103, and that all three detected objects are displayed. The information about the display state may also indicate that the main subject detection unit 102 has detected a human main subject, Sub, and that the detected main subject is displayed. Furthermore, in the example shown in FIG. 4(c), the approximate position of each object on the screen is indicated by text. Note that FIG. 4(c) shows that Etc (cloud) has not been detected as an object, but has been recognized as part of the background region by the background separation unit 104.

[0054] An example of a process in which a user switches Obj1 (house) from display to non-display when all objects are displayed will be described below. First, when a user touches the area of ​​Obj1 (house) on the touch panel screen with a finger, the display switching instruction unit 105 acquires display switching trigger information and issues a display switching control instruction to the display switching unit 107. Upon receiving the instruction to hide Obj1 (house), the display switching unit 107 generates and displays an image in which the area of ​​Obj1 (house) on the first display unit 21 is replaced with an image interpolated from the surrounding background image. At the same time, the display switching unit 107 also switches the second display unit 22 to a corresponding display. FIG. 4(b) shows a live view screen in which Obj1 (house) has been hidden. For illustrative purposes, Obj1 (house) is depicted with a dashed line in the figure, but in reality, the dashed line is not displayed and the image is the same as the background. FIG. 4(d) is an example of a live view screen displayed on the second display unit 22 simultaneously with FIG. 4(b). By looking at the display information in FIG. 4(d), the photographer or distributor can easily understand that Obj1 (house) is hidden and the approximate position on the screen when it was displayed on the first display unit 21. The display control process described above can also be used to switch the display of objects other than Obj1 (house). In the above example, the main subject Sub is not shown as a display switching target, but the main subject Sub may also be treated as a display switching target like other objects. In this way, the display of multiple objects included in a captured image can be switched.

[0055] <<Example of switching an object from hidden to visible>> Figure 5(a) shows the same situation as Figure 4(a) where the objects are detected, but all objects are hidden.

[0056] 4, Figures 5(a) and 5(b) are display examples of the live view screen on the first display unit 21, and Figures 5(c) and 5(d) are display examples of the live view screen on the second display unit 22 that is visible only to the photographer and the distributor. An example of the process in which the user switches Obj3 (apple), which is currently hidden, from hidden to displayed will be described.

[0057] For example, the user touches a portion of the live view screen of the second display unit 22 corresponding to Obj3 (apple) shown in Fig. 5(c). The display switching instruction unit 105 acquires display switching trigger information from the user and instructs the display switching unit 107 to control the display switching. The display switching unit 107 receives the instruction to display Obj3 (apple), and re-displays the area of ​​Obj3 (apple) on the first display unit 21 in the original image as shown in Fig. 5(d).

[0058] In the example described here, an example of switching the display using a touch panel is shown, but if the operation input unit 28 is provided with operation buttons, a configuration in which an object to switch the display is selected by operating the button and the display is switched on and off may also be used. Also, a configuration in which the display is switched in response to a voice instruction from the user may also be used. An example of a process for switching the display by voice will now be described.

[0059] <Example of switching the display by voice> An example of a process for switching the display in response to a voice instruction from the user will be described below in the situation shown in FIG. 4. The voice recognition unit 109 recognizes (detects) a specific voice related to the display switching from the voice data acquired from the voice input unit 18. For example, the user first utters a voice such as "voice switching mode on," which serves as a trigger for starting the voice-based display switching process, and the voice recognition unit 109 recognizes (detects) this trigger voice. In this case, the display switching instruction unit 105 starts control for enabling the voice-based display switching. Next, the user utters a voice including a designation of the object for which the display is to be switched (e.g., the object name) and a designation of the display mode of the object (e.g., a designation of whether to switch to display or hide). The voice recognition unit 109 recognizes (detects) information about the object for which the display is to be switched and information about the display mode (display switching content). In this way, by detecting an instruction to switch the display mode of the object after a voice instruction for enabling control of the display mode of the object, it is possible to reduce the risk of erroneously controlling the display based on an utterance that the user does not intend to switch. The object to be switched may be specified by the object name shown in FIG. 4(c), an object number, or other information that can identify the object. In specifying the display mode, for example, recognizing the voice "on" means controlling (switching) the object to a displayed state, and recognizing the voice "off" means controlling (switching) the object to a hidden state. For example, when a user utters the voice "turn on the house," the voice recognition unit 109 instructs the display switching unit 107 to control the display switching. Upon receiving an instruction to hide Obj1 (the house), the display switching unit 107 replaces the area of ​​Obj1 (the house) on the first display unit 21 with an image interpolated from the surrounding background image, as shown in FIGS. 4(b) and 4(d). Simultaneously with controlling the display on the first display unit 21, the display switching unit 107 also updates the display on the second display unit 22 to the corresponding information. Even when voice-based display switching is enabled, a switching instruction by a user's operation via the operation input unit 28, such as a touch panel, may be prioritized to reduce the impact of incorrect voice recognition.When the voice recognition unit 109 recognizes a trigger voice "voice switching mode off" that disables voice-based display switching, the display switching instruction unit 105 disables voice-based display switching processing.

[0060] <Example of processing to switch displays by detecting actions> An example of the process of detecting an action and switching the display will be described with reference to FIG. 6. FIG. 6(e) shows an example of an image capturing two human subjects: a main subject 60 detected by the main subject detection unit 102, and a human subject 61 detected as an object by the object detection unit 103. The main subject 60 is the photographer and is capturing a photo of himself. In the example of FIG. 6, the photographer confirms with the human subject 61 whether or not the image of the human subject 61 himself / herself can be displayed, and then the display is switched. First, the photographer instructs the human subject 61 (for example, verbally or by gesture) to make a "circle" gesture with his / her arm if the human subject 61 is OK to be displayed, or to make a "cross" gesture if the human subject 61 is not OK to be displayed. 61a shown in FIG. 6(a) shows an example of a gesture when the instructed human subject 61 is OK to be displayed, and 61b shows an example of a gesture indicating that the human subject 61 is not OK to be displayed.

[0061] The action detection unit 108 detects the joints of the relevant human subject 61 and detects a gesture related to whether the human subject 61 itself can be displayed. If the subject's gesture is determined to be a gesture that allows display, as shown in FIG. 6(b), the display switching instruction unit 105 continues the display of FIG. 6(a). If the subject's gesture is determined to be a gesture that prevents display, as shown in FIG. 6(d), the display switching instruction unit 105 switches to a display in which the human subject 61 is hidden, as shown in FIG. 6(f). Note that, in the example shown in FIG. 6, an explicit gesture by the subject is detected. However, the detection target is not limited to an explicit gesture by the subject, and may also include a predetermined movement (action) of the subject. The display switching instruction unit 105 may control switching of the display mode of the subject (e.g., hiding the subject) in response to detecting an action (indicating a state of non-participation), such as the subject talking while looking to the side or looking down.

[0062] <Example of processing to convert the corresponding part of the output audio to dummy audio when switching to hidden mode> When streaming video with audio, a process of replacing audio corresponding to the name of the hidden object with dummy audio and outputting it (hereinafter referred to as "silencing process") may be performed simultaneously with the switching of the object between hidden and unhidden. The user can switch whether or not to perform the silencing process by operating the operation input unit 28 via a menu or the like. When performing the silencing process, the audio recognition unit 109 detects the name of the hidden object from the input audio data in real time. Next, the audio processing unit 110 creates audio data in which the corresponding portion of the input audio data to be silenced is replaced with a dummy audio such as a single-frequency audio, and transmits the audio data to the video distribution device 300. The replaced audio may be output from the audio output unit 23. Note that in the above example, an example of replacing audio corresponding to the name of the hidden object with a dummy audio has been described. However, audio emitted by the hidden object may also be replaced with a dummy audio in real time. Also, the process of silencing objects corresponding to hidden objects has been described here. However, as shown in FIG. 7, the touch panel of the second display unit 22 may be used to switch whether or not to execute the muting process for each object independently of the display state of the object.

[0063] In this way, real-time display according to the intention of the user, such as the photographer or distributor, becomes possible. Even when multiple objects are detected, the user can freely and easily switch the display of the objects at the timing intended by the user by issuing an instruction to switch the display mode for each object.

[0064] As described above, in this embodiment, in a captured image including a predetermined subject, an area of ​​an object that is a subject different from the subject is detected, and an image in which the display mode of the object area is controlled is displayed on the first display unit 21. At this time, in response to a trigger (for switching the display mode of the object) being detected while the captured images are being sequentially acquired, the display mode of the object is controlled to a display mode corresponding to the trigger. In this way, it is possible to control the display state of the subject included in the captured image as desired.

[0065] As described above, the trigger includes an instruction to switch the display mode of the object (for example, a user operation, voice, or action), and when the instruction is detected, the display mode of the object is controlled to a display mode corresponding to the instruction. In this way, even when there are multiple subjects, the user can display the display state of a desired subject as desired.

[0066] (Embodiment 2) A second embodiment will be described below. In the first embodiment, a case has been described in which the user directly switches the display by using a touch operation, voice, or other instructions as trigger information for switching the display. On the other hand, as will be described in the second embodiment, the display may be automatically switched based on acquired voice or image information. In the second embodiment, the display is automatically switched based on shooting environment information. Note that the configuration of the display control device of this embodiment is the same as that of the first embodiment, and therefore the same components are assigned the same reference numerals and their description will be omitted, with the differences being mainly described.

[0067] Fig. 8 conceptually illustrates an example of the configuration of the electric circuit 20 in embodiment 2. The components 101 to 110 in Fig. 8 are the same as those in embodiment 1. In this embodiment, in addition to the components in embodiment 1, the electric circuit 20 includes a shooting environment information detection unit 111.

[0068] The shooting environment information detection unit 111 detects shooting environment information, which will be described later, based on information such as the captured image, audio during shooting, detected objects, and background areas. The shooting environment information can be detected using a known technology such as Deep Learning. For example, a machine learning model can be used that inputs the image acquired by the image acquisition unit 101, the audio acquired by the audio input unit 18, information on the object detected by the object detection unit 103, and the background image extracted by the background separation unit 104, and outputs the type of shooting scene. The machine learning model can be trained in advance using multiple sets of data each consisting of labeled output and input, and the model parameters of the machine learning model can be determined in advance.

[0069] In a situation where many subjects are detected, manually switching the display of each subject individually can be time-consuming, or when initializing the display at the start of shooting or video distribution, it may be desirable to automatically control the display mode in a manner that generally matches the user's intentions. In such a situation, in this embodiment, a display control process is performed that automatically controls the display mode (for example, turns the display on and off) based on acquired images and audio. By performing the automatic display switching described in this embodiment, the user only needs to modify the display as needed based on the results of the automatic display switching, thereby reducing the effort required for user operations.

[0070] (A series of operations related to display control processing) The display control process in this embodiment will be described with reference to Fig. 9. Note that each step in the display control process is realized by the arithmetic processing circuit 20a, which is part of the electric circuit 20, executing a program stored in the memory circuit 20b.

[0071] First, as in the first embodiment, the image acquisition unit 101 and the like execute the processes of steps S301 to S302. Next, in step S901, the image capture environment information detection unit 111 detects image capture environment information based on the various detection information detected in step S302. The image capture environment information is audio information and image information that suggest the image capture location and image capture conditions, or that can infer the image capture location and image capture conditions based on this information. For example, from audio representing the sound of waves, it is possible to infer that the image capture location is the sea or a beach, and from an image with a green or mountain background, it is possible to infer that the image capture location is outdoors, particularly in a mountainous area or suburbs. Furthermore, in a video in which many moving human subjects appear behind the main subject, each smaller on the screen than the main subject, it is possible to infer that the photographer intended to capture only the main subject, and that the individual subjects reflected in the background should not be identified.

[0072] In step S902, the display switching instruction unit 105 switches the display in accordance with the detected shooting environment information, triggered by the shooting environment information detected in step S901. A specific example will be described below with reference to FIGS.

[0073] <Automatic display switching according to shooting environment information: Example of shooting in the mountains> 10 shows an example of a distributed image captured outdoors in a mountainous region. A main subject 1000 is photographed with mountains and trees 1002 in the background. A bird 1001 also appears in the image. In step S302, main subject detection unit 102 detects main subject 1000, and object detection unit 103 detects multiple trees 1002 and bird 1001 as objects to be switched over to the display.

[0074] In step S901, the shooting environment information detection unit 111 detects (outputs) a shooting scene of a "mountain" using the above-described Deep Learning technology. For example, in this embodiment, when a shooting scene of a "mountain" is detected, if an area containing a predetermined number or more of the same type of object is detected, the display switching instruction unit 105 performs a process of excluding these objects from the display switching targets and changing the corresponding area to a background area. On the other hand, if the display switching instruction unit 105 detects fewer than the predetermined number NTh of objects of the same type, it controls the display of the object by distinguishing it from the background area. Note that if the detected object corresponds to a hidden object specified by the user as a pre-setting, the display switching instruction unit 105 prioritizes the setting and hides the detected object. At this time, the display switching instruction unit 105 may display the background area. For example, if the number of trees is more than the predetermined number NTh, in step S902, the display switching instruction unit 105 excludes the trees 1002 from the objects and displays them as part of the background area. On the other hand, since the number of birds 1001 is less than the predetermined number NTh, they are displayed as individual objects. As a result of these processes, the original image is displayed as is in the example shown in Fig. 10. When the user switches the background region or the birds 1001 to non-display, the switching can be controlled by the processes from S303 onward shown in the first embodiment.

[0075] <Automatic display switching according to shooting environment information: Example of shooting in the city> FIG. 11A shows an example of a distributed image captured in a city where multiple people are coming and going. In this example, in step S302, the main subject detection unit 102 detects a main subject 1100, and the object detection unit 103 detects multiple human subjects 1101 as objects. In step S901, the shooting environment information detection unit 111 detects the shooting environment information as a "city" shooting scene. In this embodiment, if the shooting environment information is detected as a "city," the display switching instruction unit 105 performs processing to hide objects other than the main subject and the background area. This processing is performed to protect privacy, assuming that an unspecified number of people may accidentally appear in the image. In step S902, the multiple human subjects 1101 are hidden. FIG. 11B shows an example of an image in which the multiple human subjects 1101 are replaced with images blurred to the extent that individuals cannot be identified. When a user switches to a state in which the background area and the areas of the multiple human subjects 1101 are displayed, the switching can be controlled by the processing from S303 onward described in the first embodiment.

[0076] As described above, in this embodiment, the trigger includes shooting environment information. Furthermore, in response to the detection of the shooting environment information, the display mode of the object is controlled to a display mode corresponding to the shooting environment information. Furthermore, after the shooting environment information is detected, an instruction to switch the display mode of the object is detected, and the display mode of the object is controlled to a display mode corresponding to the instruction. This allows the user to modify the display only as needed based on the results of the automatic display switching, thereby reducing the effort required of the user. In other words, even when multiple subjects are present, the display state of a desired subject can be displayed as desired.

[0077] (Third embodiment) Next, a third embodiment will be described. As described above, a configuration having a second display form different from the display form seen by viewers of the broadcast is desirable so that the user (photographer or broadcaster) can easily understand which objects are displayed / hidden. In this embodiment, a specific example of the second display form will be described. Each display form may be selected appropriately depending on the shooting environment and shooting conditions. Note that the configuration of the display control device of this embodiment may be the same as that of the above-described embodiments. For this reason, the same reference numbers will be used for the same components, and their description will be omitted, with the differences being mainly described. The display example described below is realized by the display switching unit 107 in S308 described above, for example.

[0078] FIG. 12(a) shows a display state similar to that shown in FIG. 4(b) in the first embodiment. That is, FIG. 12(a) shows a live view screen in which Obj1 (house) has been hidden. For illustrative purposes, Obj1 (house) is depicted with a dashed line in the figure; however, in reality, the dashed line is not displayed and the image is the same as the background. Therefore, Obj1 (house) cannot be visually observed on the first display unit 21. FIG. 12(b) shows an example of a live view screen displayed on the second display unit 22 simultaneously with FIG. 12(a), similar to FIG. 4(d). In the example shown in FIG. 12(b), the current display / hide status of the object is displayed as a "text list display." When the photographer or distributor views the display information, they can easily understand that Obj1 (house) is hidden and the approximate location on the screen when it was displayed on the first display unit 21. This display format is effective, for example, when the shooting location is bright and the brightness and saturation of the live view screen are difficult to discern.

[0079] In the example shown in FIG. 12(c), the hidden object Obj1 (house) is displayed in "highlighted outline" on the live view screen of the second display unit 22. This display format is effective when the size of the object on the screen is relatively large or when the shape of the object is simple. In this display format, the position, size, and shape of the hidden object are clear at a glance. The user can intuitively recognize the type of object whose display is being switched, allowing for smooth display switching.

[0080] In the example shown in FIG. 12(d), the hidden object Obj1 (house) is displayed in a "circumscribing frame" on the live view screen of the second display unit 22. This display format is effective when the size of the object on the screen is relatively small or when the shape of the object is complex. However, the type of the hidden object cannot be grasped by displaying only the circumscribing frame. For this reason, the display switching unit 107 can display text indicating the object name and type inside or around the circumscribing frame, making it easier to grasp the object.

[0081] In the example shown in FIG. 12(e), the hidden object Obj1 (house) is displayed in a "color / brightness / transmittance change display" on the live view screen of the second display unit 22. This display format is effective for shooting indoors or in a relatively dark environment where the color and brightness of the live view screen can be clearly seen. Like the "outline highlight display," this display format also allows the user to intuitively grasp the type of object and enables smooth display switching.

[0082] The above-described display modes are representative display modes, and the display switching unit 107 may perform display in other similar display modes. Furthermore, the display switching unit 107 may combine one or more of the above-described display modes.

[0083] As described above, in the above-described embodiment, information indicating the display mode of an object is displayed on the first display unit 21 or the second display unit 22. In this case, the information indicating the display mode of the object includes information indicating that the display mode of the object is not displayed by at least one of text, object outline emphasis, a circumscribing frame, a color change, a brightness change, and a transparency change. This allows the user to intuitively grasp the type, shape, etc. of the object, and enables smooth display switching.

[0084] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0085] (Disclosure of the present specification) The disclosure of this specification includes the following display control device, control method, and program. (Item 1) a detection means for detecting an area of ​​an object that is a subject other than a predetermined subject in a captured image including the predetermined subject; a display control means for controlling a display means to display an image in which a display mode of the object is controlled for a region of the object in the image, The display control device is characterized in that, in response to detection of a trigger for switching the display mode of the object while sequentially acquiring captured images, the display control means controls the display mode of the object to a display mode corresponding to the trigger. (Item 2) 2. The display control device according to item 1, wherein the display control means is further capable of controlling a display mode of the predetermined subject. (Item 3) 3. The display control device according to item 1 or 2, wherein the display control means controls the display mode of the object separately from the display mode of the predetermined subject. (Item 4) 4. The display control device according to any one of items 1 to 3, wherein the trigger includes an instruction to switch the display mode of the object. (Item 5) The display control device according to any one of items 1 to 3, characterized in that the trigger includes shooting environment information, and the shooting environment information includes at least one of the state of an object in the image at the time of shooting and a predetermined sound included in the audio at the time of shooting. (Item 6) the display control means, in response to detecting a first trigger including shooting environment information, controls a display mode of the object related to the first trigger to a display mode corresponding to the first trigger; after detecting the first trigger, in response to detecting a second trigger including an instruction to switch a display mode of the object, controlling the display mode of the object related to the second trigger to a display mode corresponding to the second trigger; The display control device according to any one of items 1 to 3, characterized in that the shooting environment information includes at least one of the state of an object in the image at the time of shooting and a specific sound contained in the audio at the time of shooting. (Item 7) 5. The display control device according to item 4, wherein the instruction includes a designation of the object and a designation of a display mode of the object. (Item 8) 5. The display control device according to item 4, wherein the instruction to switch the display mode of the object includes an instruction by a user operation on an operation input means. (Item 9) 5. The display control device according to item 4, wherein the instruction to switch the display mode of the object includes an instruction by voice. (Item 10) Item 10. The display control device of item 9, characterized in that the instruction to switch the display mode of the object includes a second spoken instruction detected after a first spoken instruction to enable control of the display mode of the object. (Item 11) 5. The display control device according to item 4, wherein the instruction to switch the display mode of the object includes a predetermined action by a person who is the object. (Item 12) A display control device according to any one of items 1 to 11, further comprising a change means for changing the sound emitted by an object whose display mode has been controlled to be hidden to a mode different from the sound emitted when the object is displayed. (Item 13) 13. The display control device according to any one of items 1 to 12, characterized in that the display control means displays information indicating the display mode of the object on the display means or a second display means different from the display means. (Item 14) The display control device described in item 13, characterized in that the information indicating the display mode of the object includes information indicating that the display mode of the object is not displayed by at least one of text, contour emphasis of the object, a circumscribing frame, a color change, a brightness change, and a transparency change. (Item 15) 15. The display control device according to any one of items 1 to 14, wherein the display control means controls the display mode of the object from one of a mode in which the object is identifiable and a mode in which the object is unidentifiable, including blurring the object, to the other. (Item 16) 15. The display control device according to any one of items 1 to 14, characterized in that the display control means controls the display mode of the object from one of a state in which the object is displayed and a state in which the object is not displayed to the other. (Item 17) An imaging means; An imaging device having the display control device according to any one of items 1 to 16. (Item 18) a detection step of detecting an area of ​​an object that is a subject other than a predetermined subject in a captured image including the predetermined subject; a display control step of displaying an image in which a display mode of the object in a region of the object in the image is controlled on a display means, A control method for a display control device, characterized in that in the display control step, when a trigger for switching the display mode of the object is detected while sequentially acquiring captured images, the display mode of the object is controlled to a display mode corresponding to the trigger. (Item 19) A program for causing a computer to function as each means of the display control device according to any one of items 1 to 16.

[0086] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0087] 100 camera, 200 display control device, 300 video distribution device, 20a arithmetic processing circuit, 20b memory circuit, 20c video processing circuit, 21 first display unit, 22 second display unit, 23 audio output unit

Claims

1. a detection means for detecting an area of ​​an object that is a subject other than a predetermined subject in a captured image including the predetermined subject; a display control means for controlling a display means to display an image in which a display mode of the object is controlled for a region of the object in the image, The display control device is characterized in that, in response to detection of a trigger for switching the display mode of the object while sequentially acquiring captured images, the display control means controls the display mode of the object to a display mode corresponding to the trigger.

2. 2. The display control device according to claim 1, wherein the display control means is further capable of controlling a display mode of the predetermined subject.

3. 2. The display control device according to claim 1, wherein the display control means controls the display mode of the object separately from the display mode of the predetermined subject.

4. The display control device according to claim 1 , wherein the trigger includes an instruction to switch the display mode of the object.

5. 2. The display control device according to claim 1, wherein the trigger includes shooting environment information, and the shooting environment information includes at least one of the state of an object in the image at the time of shooting and a predetermined sound included in the audio at the time of shooting.

6. the display control means, in response to detecting a first trigger including shooting environment information, controls a display mode of the object related to the first trigger to a display mode corresponding to the first trigger; after detecting the first trigger, in response to detecting a second trigger including an instruction to switch a display mode of the object, controlling a display mode of the object related to the second trigger to a display mode corresponding to the second trigger; The display control device according to claim 1 , wherein the image capturing environment information includes at least one of a state of an object in the image at the time of capturing and a specific sound included in audio at the time of capturing.

7. The display control device according to claim 4 , wherein the instruction includes a designation of the object and a designation of a display mode of the object.

8. 5. The display control device according to claim 4, wherein the instruction to switch the display mode of the object includes an instruction by a user's operation on an operation input means.

9. The display control device according to claim 4 , wherein the instruction to switch the display mode of the object includes an instruction by voice.

10. The display control device according to claim 9, characterized in that the instruction to switch the display mode of the object includes a second spoken instruction that is detected after a first spoken instruction for enabling control of the display mode of the object.

11. The display control device according to claim 4 , wherein the instruction to switch the display mode of the object includes a predetermined action by a person who is the object.

12. 2. The display control device according to claim 1, further comprising a change means for changing the sound emitted by an object whose display mode has been controlled to be hidden to a mode different from the sound emitted when the object is displayed.

13. 2. The display control device according to claim 1, wherein the display control means displays information indicating a display mode of the object on the display means or a second display means different from the display means.

14. The display control device according to claim 13, characterized in that the information indicating the display mode of the object includes information indicating that the display mode of the object is not displayed by at least one of text, contour emphasis of the object, a circumscribing frame, a color change, a brightness change, and a transparency change.

15. 2. The display control device according to claim 1, wherein the display control means controls the display mode of the object from one of a mode in which the object is identifiable and a mode in which the object is unidentifiable, including blurring the object, to the other.

16. 2. The display control device according to claim 1, wherein the display control means controls the display mode of the object from one of a state in which the object is displayed and a state in which the object is not displayed to the other.

17. An imaging means; An imaging device comprising: the display control device according to claim 1 .

18. a detection step of detecting an area of ​​an object that is a subject other than a predetermined subject in a captured image including the predetermined subject; a display control step of displaying an image in which a display mode of the object in a region of the object in the image is controlled on a display means, A control method for a display control device, characterized in that in the display control step, when a trigger for switching the display mode of the object is detected while sequentially acquiring captured images, the display mode of the object is controlled to a display mode corresponding to the trigger.

19. A program for causing a computer to function as each of the means of the display control device according to any one of claims 1 to 16.

Images