Display control system, display control method, and program

Abstract

To improve viewability of writing on an online basis.SOLUTION: A display control system 10 according to the present invention has a display control unit 101 and a writing reception unit 102. The display control unit 101, in a first terminal device and a second terminal device connected to each other through a communication network, displays an image captured by the first terminal device on the first terminal device and the second terminal device. The writing reception unit 102 receives writing by an operation on at least either of the first terminal device and the second terminal device. The display control unit 101 displays the writing so that it is superimposed on the image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a display control system and the like.

Background Art

[0002] Instructions and guidance may be provided online. For example, Patent Document 1 describes that a reference still image is shared between a user terminal and an instructor terminal, and guidance content for an area to be made up, specified by the instructor, is displayed on the reference still image, and the user terminal further displays a real-time image of the user being photographed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When writing instructions, guidance, comments, etc. online, the writing may be difficult to see. For example, when actually performing work, if guidance content is written on a still image as in Patent Document 1, it is difficult to understand because the actual work is different from the still image.

[0005] An example of the object of the present disclosure is to provide a display control system and the like that improve the visibility of online writing.

Means for Solving the Problems

[0006] A display control system in one aspect of the present disclosure comprises a display control means for causing an image captured by the first terminal device to be displayed on the first terminal device and the second terminal device, which are connected via a communication network, and a write receiving means for receiving writes made by operations on at least one of the first terminal device and the second terminal device, wherein the display control means displays the writes superimposed on the image.

[0007] A display control method in one aspect of the present disclosure involves a first terminal device and a second terminal device connected via a communication network, which cause the first terminal device to display an image captured by the first terminal device on the first terminal device and the second terminal device, receive a write operation to at least one of the first terminal device and the second terminal device, and display the write superimposed on the image.

[0008] A program in one aspect of the present disclosure causes a computer to perform a process in which, in a first terminal device and a second terminal device connected via a communication network, an image captured by the first terminal device is displayed on the first terminal device and the second terminal device, a write operation is performed on at least one of the first terminal device and the second terminal device, and the write is superimposed on the image and displayed.

[0009] The program may be stored on a non-temporary storage medium that is readable by the computer. [Effects of the Invention]

[0010] According to this disclosure, it is possible to improve the readability of online writings. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing an example configuration of a display control system according to Embodiment 1. [Figure 2] This flowchart shows an example of operation of the display control system according to Embodiment 1. [Figure 3] It is an explanatory diagram showing a connection example between the display control system according to Embodiment 2 and each terminal device. [Figure 4] It is a block diagram showing a configuration example of the display control system according to Embodiment 2. [Figure 5] It is a block diagram showing a configuration example of the terminal device according to Embodiment 2. [Figure 6] It is an explanatory diagram showing an example in which captured images are shared in real time. [Figure 7] It is an explanatory diagram showing an example in which writing is shared. [Figure 8] It is an explanatory diagram showing an example of accepting the follow mode. [Figure 9] It is an explanatory diagram showing an example of causing follow-up to writing. [Figure 10] It is an explanatory diagram showing an example in which writing does not follow the target according to the symbol. [Figure 11] It is an explanatory diagram showing an example in which writing follows the target according to the symbol. [Figure 12] It is an explanatory diagram showing an example in which the ratio of the writing overlapping the target area is high. [Figure 13] It is an explanatory diagram showing an example in which a teacher guides a student. [Figure 14] It is a flowchart showing an operation example of displaying an image of the display control system according to Embodiment 2. [Figure 15] It is a flowchart showing an operation example of superimposing and displaying the writing of the display control system according to Embodiment 2 on an image. [Figure 16] It is an explanatory diagram showing a hardware configuration example of a computer device.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of a display control system, a display control method, a program, and a non-temporary recording medium for recording the program according to the present disclosure will be described in detail with reference to the drawings. This embodiment does not limit the disclosed technology.

[0013] (Embodiment 1) First, in Embodiment 1, the basic functions of the display control system will be described. FIG. 1 is a block diagram showing a configuration example of the display control system according to Embodiment 1. The display control system 10 includes a display control unit 101 and a write reception unit 102.

[0014] The display control unit 101 causes the first terminal device and the second terminal device connected via a communication network to display an image captured by the first terminal device while the first terminal device is imaging a target. The target is not particularly limited, such as at least a part of a person or at least a part of an object. A part of a person is, for example, the head, face, hand, arm, etc. For example, the first terminal device is continuously imaging an image. That is, the first terminal device may be imaging a moving image. Specifically, for example, the display control unit 101 causes the image to be displayed on the display device of the first terminal device and the display device of the second terminal device. Thereby, the captured image is shared in real time between the first terminal device and the second terminal device.

[0015] The write reception unit 102 receives a write operation performed on at least one of the first terminal device and the second terminal device. The write includes at least any one of characters, numbers, symbols, lines, figures, etc. Note that the write may be a series of writes, such as a combination of characters, numbers, symbols, lines, and figures. The write is not particularly limited, such as instructions, guidance, comments, etc. The reception method by the write reception unit 102 is not particularly limited. The write reception unit 102 receives a write operation performed on at least one of the input device of the first terminal device and the input device of the second terminal device. Each input device is not particularly limited. For example, the first terminal device and the second terminal device may include a touch panel display in which the display device and the input device are integrated.

[0016] Furthermore, the write reception unit 102 may, for example, accept the position on the image where the write should be displayed when accepting a write. For example, when a touch panel display is used, the write reception unit 102 may accept a write to a screen displaying an image on at least one of the first terminal device and the second terminal device.

[0017] The display control unit 101 displays the received write by superimposing it on the displayed image. Specifically, for example, if the location of the write on the image is specified, the display control unit 101 displays the write superimposed on the image at the specified location. As mentioned above, when a touch panel display is used, the display control means displays the write at the location on the screen where the write was received when the image is displayed across the entire screen.

[0018] Furthermore, for example, even if a new image is acquired in real time, the display control unit 101 overlays the previously received writes onto the newly acquired image. This makes the writes to the transparent board appear superimposed on the image.

[0019] Figure 2 is a flowchart illustrating an example of the operation of the display control system 10 according to Embodiment 1. The display control unit 101 causes the image captured by the first terminal device to be displayed on the first terminal device and the second terminal device (step S101). The write reception unit 102 determines whether a write has been received (step S102). If the write reception unit 102 has received a write (step S102: Yes), the display control unit 101 displays the write superimposed on the image currently displayed on the first terminal device and the second terminal device (step S103). After step S103, the display control system 10 returns to step S101. If the write reception unit 102 has not received a write (step S102: No), the display control system 10 returns to step S101. If a new image is acquired after returning to step S101, the display control unit 101 displays the previously received write superimposed on the newly acquired image.

[0020] In the first embodiment described above, the display control system 10 shares the image of the target captured in real time by the first terminal device with the first terminal device and the second terminal device, and displays the written data received by either the first terminal device or the second terminal device superimposed on the image.

[0021] Embodiment 1 is not limited to the example described above and can be modified in various ways. Furthermore, each functional unit may be implemented by a single device (e.g., a display control device). For example, each functional unit may be implemented by a single server or a single user-operable terminal device. Alternatively, each functional unit may be implemented by multiple devices. For example, each functional unit may be implemented by multiple servers.

[0022] (Embodiment 2) Next, Embodiment 2 will be described in detail with reference to the drawings. Embodiment 2 will describe an example of overlaying annotations onto an image, using specific examples such as makeup instruction. Embodiment 2 will also describe an example of displaying annotations that follow an object within the image. To the extent that the description of Embodiment 2 does not become unclear, explanations that overlap with the above description will be omitted.

[0023] Figure 3 is an explanatory diagram showing an example of the connection between the display control system and each terminal device according to Embodiment 2. The display control system 20 is connected to the first terminal device 21-1 and the second terminal device 21-2 via the communication network NT.

[0024] The display control system 20 may be, for example, a server. Furthermore, the first terminal device 21-1 and the second terminal device 21-2 are not particularly limited to, for example, smartphones, tablet devices, or PCs (Personal Computers). In the following description, unless specifically limited, the terminal device will be referred to as terminal device 21.

[0025] For example, the first terminal device 21-1 and the second terminal device 21-2 may have specific application programs installed that use the display control system 20. For example, the first terminal device 21-1 may start the imaging device and perform imaging via a specific application program. Also, the first terminal device 21-1 and the second terminal device 21-2 may display various information via a specific application program.

[0026] Figure 4 is a block diagram showing an example configuration of the display control system 20 according to Embodiment 2. The display control system 20 comprises an acquisition unit 203, a display control unit 201, a write reception unit 202, a determination unit 204, a follow mode reception unit 205, a target reception unit 206, and a target identification unit 207. Compared to Embodiment 1, the display control system 20 newly includes an acquisition unit 203, a determination unit 204, and a follow mode reception unit 205.

[0027] The display control unit 201 and the write reception unit 202 have the basic functions of the display control unit 101 and the write reception unit 102 described in Embodiment 1, respectively.

[0028] Figure 5 is a block diagram showing an example configuration of the terminal device according to Embodiment 2. The terminal device 21-n comprises an imaging unit 211-n, a reception unit 212-n, and a display unit 213-n. n is an integer of 1 or more. In the example in Figure 3, n is 1 and 2. When n is 1, it represents the first terminal device 21-1, and when n is 2, it represents the second terminal device 21-2.

[0029] The imaging unit 211-n causes the imaging device of the terminal device 21-n to capture an image of the target.

[0030] The reception unit 212-n receives various types of information. The method of receiving information by the reception unit 212-n is not particularly limited. For example, the reception unit 212-n receives various types of information through operations on the input device of the terminal device 21. As mentioned above, the input device is not particularly limited, but it may be an input / output device such as a touch panel display integrated with a display device.

[0031] The display unit 213-n displays various information on the display device of the terminal device 21. The display device is not particularly limited, but may be an input / output device such as a touch panel display integrated with an input device.

[0032] In the following explanation, when the terminal device 21 is not specifically identified, it will simply be referred to as the imaging unit 211, the reception unit 212, and the display unit 213.

[0033] The first terminal device 21-1 may be configured to include an imaging unit 211-1, a reception unit 212-1, and a display unit 213-1, while the second terminal device 21-2 may be configured to include a reception unit 212-2 and a display unit 213-2.

[0034] Returning to the explanation of Figure 4, the acquisition unit 203 acquires the image that the first terminal device 21-1 is capturing of the target. The target is not particularly limited and can be a person, a part of a person's body, an object, a part of an object, etc. For example, the imaging unit 211-1 of the first terminal device 21-1 obtains an image by capturing the target with an imaging device. The imaging unit 211-1 of the first terminal device 21-1 then transmits the captured image to the display control system 20 in real time. The acquisition unit 203 acquires the image that the first terminal device 21-1 is capturing of the target by receiving the image from the first terminal device 21-1. Note that the image may be a moving image.

[0035] The display control unit 201 then causes the image acquired by the acquisition unit 203 to be displayed on the first terminal device 21-1 and the second terminal device 21-2. Specifically, the display control unit 201 may transmit the acquired image to the first terminal device 21-1 and the second terminal device 21-2. The display unit 213 of each terminal device 21 then displays the received image.

[0036] Figure 6 is an explanatory diagram illustrating an example of real-time sharing of captured images. In Figure 6, an example of a salesperson giving makeup instruction to a customer is used for explanation. For example, the customer uses the first terminal device 21-1. The salesperson uses the second terminal device 21-2.

[0037] The imaging unit 211-1 of the first terminal device 21-1 captures an image of the customer's face. The imaging unit 211-1 of the first terminal device 21-1 transmits the captured image to the display control system 20. The display control unit 201 then causes the first terminal device 21-1 and the second terminal device 21-2 to display the captured image. The display unit 213-1 of the first terminal device 21-1 displays the captured image. The display unit 213-1 of the second terminal device 21-2 also displays the captured image. As a result, the captured image is shared and displayed.

[0038] In Figure 6, the screens of the first terminal device 21-1 and the second terminal device 21-2 display the captured image of the customer's face being captured by the first terminal device 21-1.

[0039] In Embodiment 2, the acquisition unit 203 of the display control system 20 acquires an image from the first terminal device 21-1, the display control unit 201 transmits the image to each terminal device 21, and the display unit 213 of each terminal device 21 displays the image, but the invention is not limited to this. For example, since the first terminal device 21-1 is capturing an image, the display unit 213-1 of the first terminal device 21-1 may, in response to an image display instruction from the display control unit 201 of the display control system 20, display the captured image as is without receiving an image from the display control system 20. Alternatively, for example, in response to an image display instruction from the display control unit 201 of the display control system 20 to the image capture unit 211-1 of the first terminal device 21-1, the image capture unit 211-1 of the first terminal device 21-1 may transmit the image to the second terminal device 21-2, and the display unit 213-2 of the second terminal device 21-2 may display the received image. Thus, it is sufficient that the image captured by the first terminal device 21-1 is shared and displayed in real time to both the first terminal device 21-1 and the second terminal device 21-2, and the method of display control is not particularly limited.

[0040] Next, returning to the explanation of Figure 4, the write reception unit 202 accepts writes made by operations on at least one of the first terminal device 21-1 and the second terminal device 21-2. The method by which the write reception unit 202 accepts writes is not particularly limited, as explained in Embodiment 1.

[0041] The reception unit 212 accepts writes based on user input. The reception unit 212 also transmits location information indicating the location where the write was received on the image, along with the write itself, to the display control system 20. For example, the write reception unit 202 accepts a write by receiving both the write and the location information.

[0042] The display control unit 201 displays the received write by superimposing it onto the image. For example, the display control unit 201 instructs the first terminal device 21-1 and the second terminal device 21-2 to display the received write by superimposing it onto the position indicated by the position information on the image. More specifically, the display control unit 201 transmits an instruction to the first terminal device 21-1 and the second terminal device 21-2 to display the received write by superimposing it onto the position indicated by the position information on the image. Then, the display unit 213 of each terminal device 21 displays the write on the display device (input / output device) by superimposing it onto the position indicated by the position information on the image.

[0043] Figure 7 is an explanatory diagram illustrating an example of shared writing. The reception unit 212-2 of the second terminal device 21-2 receives the writing. The reception unit 212-2 of the second terminal device 21-2 then transmits, for example, the writing and location information indicating the location where the writing was received to the display control system 20. The writing reception unit 202 receives the writing and location information, for example, by receiving the writing and location information. The display control unit 201 transmits instructions to the first terminal device 21-1 and the second terminal device 21-2 to display the writing superimposed on the location information on the image. The display unit 213-1 of the first terminal device 21-1 displays the writing superimposed on the location information on the image on the display device. The display unit 213-2 of the second terminal device 21-2 displays the writing superimposed on the location information on the image on the display device. In this way, the writing is shared.

[0044] In the above explanation, annotations are superimposed on the image, improving the readability of the annotations. For example, in the case of makeup instruction, demonstrations are often conducted in stores where a salesperson applies makeup to the customer's face and explains the process face-to-face. Online, it is not possible to apply makeup to a customer's face in real time, making it difficult to explain to the customer. For example, even if a finger is pointed at on the screen, it is difficult to tell what is being pointed at. Therefore, by superimposing annotations on the image, the readability of the annotations can be improved, making instruction easier.

[0045] (Example where the object moves) Next, if the subject is moving, the writing process may or may not be made to follow the subject. If the subject is a person, the subject will move during imaging. Also, if the subject is an object, the object may be moved during imaging.

[0046] For example, the write reception unit 202 accepts writes that follow an object included in the image. The display control unit 201 displays the writes superimposed on the image so that the relative positional relationship between the writes and the objects is maintained.

[0047] Furthermore, for example, the write reception unit 202 accepts writes that do not follow the objects included in the image. The display control unit 201 displays the writes superimposed on the positions in the image where the writes were received.

[0048] The write reception unit 202 may also accept a request for whether to follow the input each time a write is made.

[0049] Furthermore, for example, there may be a mode that allows setting whether to track the write operation, and the write operation may be tracked according to the mode. Alternatively, for example, the write operation may be tracked according to the symbols included in the write operation. Alternatively, for example, it may be automatically determined whether to track the write operation according to the area in the image where the write operation is displayed.

[0050] <Examples of tracking depending on the mode> First, depending on the mode, it may be possible to control whether the writing process follows the target's movement.

[0051] The follow mode reception unit 205 accepts a setting for whether to enable follow mode or disable follow mode. The timing at which the follow mode reception unit 205 accepts the mode is not particularly limited; it may be after the write operation is accepted, before the write operation is accepted, or simultaneously with the write operation is accepted. The follow mode reception unit 205 may also accept the mode setting for each write operation.

[0052] More specifically, for example, the display control unit 201 causes the terminal device 21 that accepts writes to display a screen that can accept mode settings.

[0053] Figure 8 is an explanatory diagram showing an example of accepting the follow mode. In Figure 8, in order for the second terminal device 21-2 to accept a write operation, the display control unit 201 displays a screen on the second terminal device 21-2 that allows the mode setting to be accepted. The display unit 213-2 of the second terminal device 21-2 accepts the screen that allows the mode setting to be accepted. The screen displays a "Follow" checkbox for selecting the follow mode and a "Do not follow" checkbox for selecting the mode that does not follow. The checkboxes are just an example, and buttons or other means may be used, and are not particularly limited. In Figure 8, "Follow" is selected, so the mode that allows writing to follow is selected.

[0054] When the reception unit 212-2 receives the "follow" selection, it transmits the selection result to the display control system 20. The follow mode reception unit 205 accepts the mode setting upon receiving the selection result.

[0055] When the display control unit 201 is set to track the target, it displays the write in such a way that the relative positional relationship between the write and the target is maintained. Here, we will describe the specific process for displaying the write in such a way that the relative positional relationship between the write and the target is maintained when the imaging device of the first terminal device 21-1 that captures images is fixed. Specifically, for example, the display control unit 201 identifies the position of the target in the image. As a process for identifying the position of the target, the display control unit 201 may identify the position of the target by extracting the area of ​​the target included in the image. Then, the display control unit 201 identifies the relative positional relationship between the write and the target based on the position where the write was received and the position of the target. Next, when the target moves, the display control unit 201 identifies the position of the target in the new image. Then, based on the position of the target, it identifies the position of the write in the new image where the relative positional relationship between the target and the write is the same. The display control unit 201 displays the write superimposed on the identified position in the new image. In this way, the relative positional relationship can be maintained. Furthermore, if you want to overlay the writing onto the image so that the relative positions are the same, you can perform the same processing, and a detailed explanation will be omitted in the following description. Also, the technique for tracking the object can be any existing technique and is not limited to the example described above.

[0056] Furthermore, if the display control unit 201 is set not to track the target, it will display the write at the received position. For example, even if the target moves, the display control unit 201 will display the write at the received position. More specifically, for example, if an image is displayed across the entire screen, the display control unit 201 will display the write at the received position on the screen, even if the target's position moves on the screen. In other words, the positional relationship between the write and the target is not maintained.

[0057] Figure 9 is an explanatory diagram illustrating an example of making writing follow the target. As shown in Figure 9, even if the customer moves, the writing is displayed in a way that follows the customer's movements so that the relative positional relationship between the writing and the target is maintained.

[0058] <Examples of tracking based on symbols> Depending on the symbols included in the write, it may be determined whether to track the write to the target.

[0059] The determination unit 204 determines whether to make the write follow the target based on the symbols included in the write. The determination unit 204 determines to make the write follow the target if the symbols included in the write are predetermined symbols. The predetermined symbols are symbols that have been defined in advance and are not particularly limited. For example, the predetermined symbols may be symbols that indicate position, such as arrows. The determination unit 204 determines not to make the write follow the target if the symbols included in the write are not predetermined symbols. The determination unit 204 also determines not to make the write follow the target if the write does not contain any symbols.

[0060] The display control unit 201 displays the write in such a way that the relative positional relationship between the write and the target is maintained, if the symbol included in the write is a predetermined symbol. For example, the display control unit 201 displays the write in such a way that the relative positional relationship between the write and the target is maintained even if the target moves. On the other hand, if the symbol included in the write is not a predetermined symbol, the display control unit 201 displays the write at the position it was received in the image. For example, the display control unit 201 displays the write at the position it was received in the image even if the target moves.

[0061] Figure 10 is an explanatory diagram illustrating an example where the writing does not follow the target depending on the symbol. In Figure 10, the writing includes the symbol "Point," but since the symbol is not a predetermined symbol, the display control unit 201 displays the writing at the received position. Furthermore, even if the target customer moves, the writing does not follow the target's movement and is displayed at the received position.

[0062] Figure 11 is an explanatory diagram illustrating an example where the writing follows the target according to the symbol. In Figure 11, the writing includes an arrow symbol, and since this symbol is a predetermined symbol, the display control unit 201 displays the writing superimposed on the image so that the relative positional relationship between the writing and the target is maintained. When the target, the customer, moves, the writing is displayed in accordance with the movement of the target.

[0063] <Example of tracking based on overlap with the target area> Whether to track the write operation based on whether the write operation was performed within the target area may be used as a determination.

[0064] The determination unit 204 determines whether to make the write track the target based on whether it has received a write request within the target area included in the image. For example, if the determination unit 204 receives a write request within the target area included in the image, it determines that the write should track the target. On the other hand, if the determination unit 204 does not receive a write request within the target area included in the image, it determines that the write should not track the target.

[0065] If the display control unit 201 determines that a write has been received within the target area, it displays the write superimposed on the image so as to maintain the relative positional relationship between the write and the target. If it determines that a write has been received outside the target area, it displays the write at the location where it was received on the image.

[0066] For example, if the subject is a person's face, if the drawing overlaps the face area, the drawing will follow the face's movements; however, if the drawing is outside the face area, the drawing will not follow the face's movements.

[0067] In the above example, if a portion of the write overlaps with the target area, it is determined that the write should follow the target. However, the determination is not limited to this, and the determination of whether to follow the write may be made based on the proportion of the write overlapping with the target area. The determination unit 204 determines whether to follow the write based on the proportion of the write overlapping with the target area. For example, if the proportion of the write overlapping with the target area is high, the determination unit 204 may determine that the write should follow the target, and if the proportion is low, it may determine that the write should not follow the target. More specifically, for example, the determination unit 204 may determine that the write should follow the target if the proportion of the write overlapping with the target area is greater than or equal to a threshold. The threshold is not particularly limited. For example, the threshold may differ depending on the type of target and the application of the display control system 20. On the other hand, for example, the determination unit 204 may determine that the write should not follow the target if the proportion of the write overlapping with the target area is less than or equal to a threshold.

[0068] If the display control unit 201 determines that the proportion of the write overlaps the target area is high (for example, if the proportion is above a threshold), it displays the write superimposed on the image so as to maintain the relative positional relationship between the write and the target. If the display control unit 201 determines that the proportion of the write overlaps the target area is not high (for example, if the proportion is not above a threshold), it displays the write at the received position on the image.

[0069] Figure 12 is an explanatory diagram illustrating an example where the amount of writing overlaps with the target area is high. In Figure 12, the display control unit 201 displays the writing superimposed on the image so that the relative positional relationship between the writing and the target is maintained, because the amount of writing overlaps with the target area is high. In Figure 12, when the target customer moves, the writing is displayed in accordance with the movement of the target.

[0070] In this way, for example, the system can automatically determine whether a comment is related to a face based on its proportions, and the comment can be made to track facial movements.

[0071] This concludes the explanation of the example that determines whether to follow the write operation.

[0072] (Example of identifying the target) In the above examples, the explanation was based on the premise that the target is predetermined, but the target may be specified by the user or may be identified according to the application of the display control system 20.

[0073] For example, the target reception unit 206 receives the designation of a target via at least one of the first terminal device 21-1 and the second terminal device 21-2. Specifically, for example, the target reception unit 206 may receive the designation of a target via at least one of the reception units 212 of the first terminal device 21-1 and the second terminal device 21-2.

[0074] Furthermore, for example, the target reception unit 206 may accept a target designation from among the candidate targets included in the image. The target reception unit 206 may also identify candidate targets by extracting objects or people from the image. The method for extracting objects or people from the image is not particularly limited and can be any existing technology.

[0075] Specifically, for example, if multiple objects are extracted from an image, the target receiving unit 206 can accept the user's input to specify a target object from among the multiple candidates, treating the extracted objects as multiple candidates. For example, if multiple people are extracted from an image, the target receiving unit 206 can accept the user's input to specify a target person from among the multiple candidates, treating the extracted people as multiple candidates. The number of targets that can be specified is not particularly limited. For example, multiple objects may be specified as targets, multiple people may be specified as targets, or people and objects may be specified as targets.

[0076] Furthermore, the target reception unit 206 may also accept the intended use. For example, the target reception unit 206 may extract a target from the image according to the intended use. The intended use of the display control system 20 is not particularly limited. For example, if the intended use is to instruct on makeup, the target is a person's face. For example, if the intended use is to instruct on manufacturing, the target may be a person's hands, an object being made, etc.

[0077] Furthermore, the timing at which the target reception unit 206 accepts the designation of the target and its intended use may be before or after the timing at which it accepts the writing, and is not particularly limited.

[0078] Furthermore, the target reception unit 206 may be configured to accept the designation of the target and its purpose only once initially. Specifically, for example, the target reception unit 206 may accept the purpose when a specific application program is launched in the first terminal device 21-1 and the second terminal device 21-2.

[0079] Furthermore, while the above is an example where the target and purpose are accepted through user interaction, the target may also be identified automatically.

[0080] The target identification unit 207 identifies a target according to the location where the write was received. Specifically, for example, the target reception unit 206 extracts candidate targets from the image and identifies the candidate closest to the location where the write was received as the target. Alternatively, specifically, for example, the target reception unit 206 extracts candidate targets from the image and identifies the candidate that overlaps with the write as the target.

[0081] Furthermore, while the use of the display control system 20 is explained using the example of makeup instruction in each drawing, it is not limited to this. Figure 13 is an explanatory diagram showing an example of a teacher instructing a student. In Figure 13, the first terminal device 21-1 captures images of the student making something, and the display control system 20 shares the captured images in real time with the first terminal device 21-1 and the second terminal device 21-2. The teacher then writes instructions via the second terminal device 21-2. The display control system 20 then overlays the received instructions onto the image and displays them. As shown in Figure 13, the subject may be the student's hands or the object the student is making. Thus, the use of the display control system 20 is not particularly limited.

[0082] Figure 14 is a flowchart illustrating an example of the operation of displaying an image using the display control system 20 according to Embodiment 2. The imaging unit 211-1 of the first terminal device 21-1 causes the imaging device to capture an image (step S201). The imaging unit 211-1 may also transmit the captured image to the display control system 20. As a result, the acquisition unit 203 acquires the captured image (step S202).

[0083] Next, the display control unit 201 causes the first terminal device 21-1 and the second terminal device 21-2 to share and display the image (step S203). In step S203, the display control unit 201 may also transmit the image to the first terminal device 21-1 and the second terminal device 21-2 for display. Alternatively, as described above, the display control unit 201 may transmit an instruction to the first terminal device 21-1 to display the image, and then transmit the image to the second terminal device 21-2 for display.

[0084] Then, the display unit 213-1 of the first terminal device 21-1 displays an image (step S204). Also, the display unit 213-2 of the second terminal device 21-2 displays an image (step S205).

[0085] As mentioned above, the imaging unit 211-1 of the first terminal device 21-1 may transmit the image to the second terminal device 21-2 in response to instructions from the display control unit 201, without transmitting the image to the display control system 20. The display unit 213-2 of the second terminal device 21-2 may display the image received from the first terminal device 21-1 on the display device of the second terminal device 21-2 in response to instructions from the display control unit 201. Alternatively, the display unit 213-1 of the first terminal device 21-1 may display the image captured by its own device on the display device of the first terminal device 21-1 in response to instructions from the display control unit 201.

[0086] Figure 15 is a flowchart illustrating an example of the operation of the display control system 20 according to Embodiment 2, in which the write is superimposed on an image. Here, an example is described in which the write is accepted by an operation on the second terminal device 21-2, but the write may also be accepted by an operation on the first terminal device 21-1. Figure 15 also shows an example of the operation in which the determination unit 204 determines whether to make the write follow the movement of the target, such as an example of tracking according to the overlap of the target area or an example of tracking according to a symbol.

[0087] The reception unit 212-2 of the second terminal device 21-2 accepts the write (step S211). In step S211, when the reception unit 212-2 of the second terminal device 21-2 accepts the write, it acquires location information indicating the location where the write was accepted. The reception unit 212-2 then transmits the write and location information to the display control system 20. As a result, the write reception unit 202 of the display control system 20 accepts the write (step S222).

[0088] Next, the determination unit 204 determines whether to make the write follow the movement of the target (step S223). In step S223, the determination unit 204 may determine whether to make the write follow the target based on whether the symbols included in the write are predetermined symbols. Alternatively, in step S223, the determination unit 204 may determine whether to make the write follow the target based on the proportion to which the write overlaps with the area of ​​the target.

[0089] If it is determined that the write should be tracked (step S223: Yes), the display control unit 201 superimposes the write onto the image so as to maintain the relative positional relationship between the write and the object (step S224). In step S224, for example, when the write is tracked, when a new image is captured, the display control unit 201 superimposes the write onto the image so as to maintain the relative positional relationship between the write and the object included in the new image.

[0090] On the other hand, if it is determined that the write should not be tracked (step S223: No), the display control unit 201 overlays the write onto the image (step S225). In steps S224 and S225, the display control unit 201 may also notify each terminal device 21 of the position in the image where the write should be displayed.

[0091] The display unit 213-1 of the first terminal device 21-1 displays the written data on the image at the specified position (step S226). The display unit 213-2 of the second terminal device 21-2 displays the written data on the image at the specified position (step S227).

[0092] Steps S224 and S225 may be repeated until the screen changes. This ensures that even if a new image is acquired, the written data is displayed overlaid on it. Furthermore, if the display control system 20 receives a command to delete the written data, the written data may be deleted.

[0093] Furthermore, while Figure 15 illustrates an example of operation in which the determination unit 204 determines whether to make the write follow the movement of the target, the operation is not limited to this, and as mentioned above, whether to make the write follow the movement of the target may be controlled according to the mode. When whether to make the write follow the movement of the target is controlled according to the mode, for example, in step S211, the reception unit 212-2 of the second terminal device 21-2 may also accept the mode setting when it accepts the write. The reception unit 212-2 may then transmit the write, position information indicating the location where the write was accepted, and the mode setting to the display control system 20. Then, in step S222, the write reception unit 202 of the display control system 20 accepts the write and position information, and the follow mode reception unit 205 accepts the mode setting. Then, in step S223, the follow mode reception unit 205 may determine whether to make the write follow the movement of the target.

[0094] In the second embodiment described above, the display control system 20 overlays the written information onto the image at the location where the writing was received. This allows the user to display the written information at their desired location while overlaying it onto the image. Therefore, the readability of online writing can be improved. Consequently, online instructions and guidance can be facilitated.

[0095] Furthermore, the display control system 20 may accept writes in such a way that it maintains the positional relationship between the object included in the image and the write. This allows the write to follow the object included in the image.

[0096] Furthermore, the display control system 20 may accept writes that do not follow the objects included in the image. This allows the writes to be displayed superimposed on the image while also being displayed at the position desired by the user. Therefore, the readability of online writes can be improved.

[0097] Furthermore, the display control system 20 may accept a setting to maintain the positional relationship between the objects included in the image and the writes. This allows the writes to follow the objects included in the image in response to the writes, or to display the writes at the position desired by the user.

[0098] Furthermore, the display control system 20 may determine whether to make the write follow an object included in the image based on the symbols included in the write. This allows for automatic determination of whether to make the write follow an object based on the content of the write.

[0099] Furthermore, the display control system 20 may determine whether to track the write based on whether it has received a write operation within the target area included in the image. More specifically, the display control system 20 may determine whether to track the write based on the proportion to which the write overlaps with the target area. This allows for automatic determination of whether to track the write based on the content of the write operation.

[0100] <Example 1> If the writing location is not specified, the display control unit 201 may superimpose the writing at a location in the image that does not overlap with a predetermined area. The predetermined area may be, for example, the target area or an area specified by the user.

[0101] <Modification 2> In Embodiment 2, an example was described in which images and writings are shared and displayed between the first terminal device 21-1 and the second terminal device 21-2. However, images and writings may be shared and displayed across three or more terminal devices 21. For example, in a school, an image captured on one student's terminal device 21 may be shared and displayed on each student's terminal device 21 and the teacher's terminal device 21, and comments received on each terminal device 21 may be superimposed on the image and shared and displayed.

[0102] This concludes the description of each embodiment and each modified example. Note that each embodiment and the modified examples described therein may be used in combination. Also, for example, in each embodiment, the display control systems 10 and 20 may be configured to include each functional unit and some of the information. For example, in Embodiment 2, the display control system 20 may be configured to include an acquisition unit 203, a display control unit 201, a write reception unit 202, and a determination unit 204. Also, for example, the display control system 20 may be configured to include an acquisition unit 203, a display control unit 201, a write reception unit 202, and a follow mode reception unit 205.

[0103] Furthermore, the embodiments and modifications described above are not limited to the examples given and can be modified in various ways. Also, the configuration of the display control systems 10 and 20 in each embodiment is not particularly limited. The display control systems 10 and 20 may be implemented by a single device, or by different devices (servers) for each type of information or function. The display control systems 10 and 20 may also be composed of several of these servers. For example, a single terminal device 21 may be equipped with the functional part of the display control system 20 (server) described in Embodiment 2.

[0104] Furthermore, in the transmission and reception between devices such as the terminal device 21 and the display control systems 10, 20 (servers) described in each embodiment and modified example, the object of transmission and reception may be the information itself, or it may be an identifier representing the information.

[0105] In each embodiment and modification, each piece of information may include a portion of the aforementioned information. Furthermore, each piece of information may include information other than that described above. Each piece of information may be further divided into multiple pieces of information. Thus, the method of implementing each piece of information is not particularly limited.

[0106] Furthermore, the position, color, and size of each item, such as buttons, input fields, and display fields, on each screen are not particularly limited. Buttons, information display fields, input fields, etc., not shown in the illustrations may be added to each screen. Also, the background color of the screen may be changed.

[0107] For example, in each embodiment, the process of generating screen information to be displayed on the display device of the terminal device 21 may be performed by the display control units 101 and 201 of the display control systems 10 and 20, or by the display unit 213 of the terminal device 21. Specifically, for example, the display control units 101 and 201 may generate screen information with the writing superimposed on an image and transmit the generated information to the terminal device 21. The display unit 213 of the terminal device 21 then displays the received information on the display device. Alternatively, the display control units 101 and 201 may transmit an instruction to the terminal device 21 to superimpose the writing on an image, and the display unit 213 of the terminal device 21 may generate screen information with the writing superimposed on an image in response to the instruction and display the generated information on the display device.

[0108] (Computer device) Next, we will describe hardware configuration examples when each device, such as the display control systems 10 and 20 (servers) and terminal devices 21 described in each embodiment, is implemented using a computer device. Figure 16 is an explanatory diagram showing an example of the hardware configuration of a computer device. For example, some or all of each device can be implemented using any combination of a computer device 30 and a program, as shown in Figure 16.

[0109] The computer device 30 includes, for example, a processor 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, a storage device 304, a communication interface 305, and an input / output interface 306. Each component is connected via a bus 307.

[0110] The processor 301 controls the entire computer device 30. Examples of processors 301 include a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). There may be multiple processors 301. The computer device 30 includes memory units such as ROM 302, RAM 303, and a storage device 304. Examples of storage devices 304 include semiconductor memory such as flash memory, an HDD (Hard Disk Drive), and an SSD (Solid State Drive). For example, the storage device 304 stores OS (Operating System) programs, application programs, and programs related to each embodiment. Alternatively, ROM 302 stores application programs and programs related to each embodiment. The RAM 303 is used as the work area for the processor 301.

[0111] The processor 301 also loads programs stored in the memory device 304, ROM 302, etc. Then, the processor 301 executes each process or instruction coded in the program. The processor 301 may also download various programs via the communication network NT. Furthermore, the processor 301 functions as part or all of the computer device 30. The processor 301 may also execute processes or instructions in the illustrated flowchart based on the program.

[0112] The communication interface 305 is connected to a communication network NT, such as a LAN (Local Area Network) or WAN (Wide Area Network), via a wireless or wired communication line. The communication network NT may consist of multiple communication networks. This allows the computer device 30 to connect to external devices and external computer devices 30 via the communication network NT. The communication interface 305 manages the interface between the communication network NT and the internal workings of the computer device 30. Furthermore, the communication interface 305 controls the input and output of data from external devices and external computer devices.

[0113] Furthermore, the input / output interface 306 is connected to at least one of the input device, output device, and input / output device. The connection method may be wireless or wired. Examples of input devices include keyboards, mice, and microphones. Examples of output devices include display devices, lighting devices, and speakers which are audio output devices that output sound. Examples of input / output devices include touch panel displays. The input device, output device, and input / output device may be built into the computer device 30 or may be external.

[0114] The hardware configuration of the computer device 30 is an example. The computer device 30 may have some of the components shown in Figure 16. The computer device 30 may have components other than those shown in Figure 16. For example, the computer device 30 may have a drive device. The processor 301 may read programs and data stored on a recording medium mounted on the drive device or the like into the RAM 303. Examples of non-temporary tangible recording media include optical discs, flexible discs, magneto-optical discs, and USB (Universal Serial Bus) memory. Also, as mentioned above, for example, the computer device 30 may have input devices such as a keyboard and a mouse. The computer device 30 may have output devices such as a display. Furthermore, the computer device 30 may have input devices, output devices, and input / output devices, respectively.

[0115] The computer device 30 may have various sensors not shown in the diagram. The type of sensor is not particularly limited. Examples of sensors include acceleration sensors, gyroscopes, magnetic sensors, GPS (Global Positioning System), and other sensors. Other examples of sensors include fingerprint sensors, LiDAR (Light, Detection and Ranging), proximity sensors, and ambient light sensors.

[0116] This concludes the description of the hardware configuration of each device. Furthermore, there are various variations in how each device can be implemented. For example, the display control systems 10 and 20 may be implemented by any combination of different computer devices and programs for each component. Alternatively, the multiple components of each device may be implemented by any combination of a single computer device and program.

[0117] Furthermore, some or all of the components of each device may be implemented using application-specific circuits. Also, some or all of the display control systems 10, 20 may be implemented using general-purpose circuits, including a processor such as an FPGA (Field Programmable Gate Array). Furthermore, some or all of the display control systems may be implemented using a combination of application-specific circuits and general-purpose circuits. These circuits may also be a single integrated circuit, or they may be divided into multiple integrated circuits. These multiple integrated circuits may be connected via a bus or the like.

[0118] Furthermore, if some or all of the components of each device are realized by multiple computer devices or circuits, these multiple computer devices or circuits may be centrally located or distributed.

[0119] The display control methods described in each embodiment are implemented by the execution of the display control systems 10 and 20 (servers). Alternatively, the display control methods can be implemented by the execution of pre-prepared programs by computer devices such as terminal devices and servers. The programs described in each embodiment are recorded on a recording medium readable by a computer device, such as an HDD, SSD, flexible disk, optical disk, magneto-optical disk, or USB memory. The programs are then executed by being read from the recording medium by the computer device. Furthermore, the programs may be distributed via a communication network NT.

[0120] Each component of the display control systems 10 and 20 in the embodiments described above may be implemented in hardware, such as a computer device. Alternatively, each component may be implemented in a computer device or firmware based on program control.

[0121] The present disclosure has been described above with reference to the embodiments described herein, but the present disclosure is not limited to the embodiments described above. The structure and details of each present disclosure may include embodiments that apply various modifications that can be grasped by those skilled in the art within the scope of the present disclosure. The present disclosure may include embodiments that combine or substitute the matters described herein as appropriate. For example, matters described using a particular embodiment may be applied to other embodiments to the extent that they do not cause a contradiction. For example, although multiple operations are described sequentially in the form of a flowchart, the order in which they are described does not limit the order in which the multiple operations are performed. Therefore, when implementing each embodiment, the order of the multiple operations may be changed to the extent that it does not impair the content.

[0122] Some or all of the above embodiments may also be described as follows. However, some or all of the above embodiments are not limited to the following.

[0123] (Note 1) In a first terminal device and a second terminal device connected via a communication network, a display control means is provided to cause the first terminal device to display the image it is capturing on the first terminal device and the second terminal device. A write receiving means that accepts writes made by operations on at least one of the first terminal device and the second terminal device, Equipped with, The display control means causes the writing to be superimposed on the image and displayed. Display control system. (Note 2) The display control means displays the written data superimposed on the position in the image where the written data was received. The display control system described in Appendix 1. (Note 3) The writing receiving means receives the writing that follows the object included in the image, The display control means displays the writing superimposed on the image so as to maintain the relative positional relationship between the writing and the target. The display control system described in Appendix 1. (Note 4) The writing receiving means receives the writing that does not follow the object included in the image, The display control means displays the written information superimposed on the position in the image where the written information was received. The display control system described in Appendix 1. (Note 5) A tracking mode receiving means that accepts a setting to determine whether the aforementioned writing should follow the object included in the aforementioned image. Equipped with, When the display control means is set to track the target, it displays the write in such a way that the relative positional relationship between the write and the target is maintained, and when it is set not to track the target, it displays the write at the position in the image where the write was received. The display control system described in Appendix 1 or 2. (Note 6) A determination means that determines whether the writing follows an object included in the image based on the symbols included in the writing. The display control means, when the symbol is a predetermined symbol, displays the write in such a way that the relative positional relationship between the write and the target is maintained, and when the symbol is not a predetermined symbol, displays the write at the position in the image where the write was received. The display control system described in Appendix 1 or 2. (Note 7) A determination means for determining whether to make the write follow the target based on whether the write was received within the target area included in the image. Equipped with, If the display control means determines that the write has been received within the target area, it displays the write in such a way that the relative positional relationship between the write and the target is maintained; if it determines that the write has been received outside the target area, it displays the write at the position in the image where the write was received. The display control system described in Appendix 1 or 2. (Note 8) The determination means determines whether to make the write follow the target based on the proportion to which the write overlaps with the target area. The display control system described in Appendix 7. (Note 9) The aforementioned subject is a person's face, The area of ​​the subject is the area of ​​the face. The display control system described in Appendix 7 or 8. (Note 10) The subject is at least a part of a person, or at least a part of an object. A display control system as described in any of the appendices 3 to 8. (Note 11) A target acceptance means for accepting the designation of the aforementioned target, A display control system according to any one of the appendices 3 to 10, comprising: (Note 12) The target receiving means receives a designation of a target from among the candidate targets included in the image. The display control system described in Appendix 11. (Note 13) A target identification means for identifying the target corresponding to the location where the aforementioned writing was received, A display control system according to any one of the appendices 3 to 10, comprising: (Note 14) In a first terminal device and a second terminal device connected via a communication network, the image captured by the first terminal device is displayed on the first terminal device and the second terminal device. The system accepts a write operation to at least one of the first terminal device and the second terminal device. The aforementioned writing is superimposed and displayed on the aforementioned image. Display control method. (Note 15) On the computer, In a first terminal device and a second terminal device connected via a communication network, the image captured by the first terminal device is displayed on the first terminal device and the second terminal device. The system accepts a write operation to at least one of the first terminal device and the second terminal device. The aforementioned writing is superimposed and displayed on the aforementioned image. A program that executes a process. (Note 16) On the computer, In a first terminal device and a second terminal device connected via a communication network, the image captured by the first terminal device is displayed on the first terminal device and the second terminal device. The system accepts a write operation to at least one of the first terminal device and the second terminal device. The aforementioned writing is superimposed and displayed on the aforementioned image. A non-temporary recording medium readable by the computer, which records a program that executes a process. [Explanation of symbols]

[0124] 10,20 Display Control System 21,21-n Terminal device 21-1 First Terminal Device 21-2 Second Terminal Device 30 Computer equipment 101,201 Display Control Unit 102,202 Write reception section 203 Acquisition Department 204 Judgment section 205 Follow-up mode reception unit 206 Target Reception Department 207 Targeted Department 211,211-1,211-n imaging unit 212, 212-1, 212-2, 212-n Reception Department 213,213-1,213-2,213-n Display section 301 Processor 302 ROM 303 RAM 304 Storage device 305 Communication Interface 306 Input / Output Interfaces 307 Bus NT communication network

Claims

1. In a first terminal device and a second terminal device connected via a communication network, a display control means is provided to cause the first terminal device to display an image being captured by the first terminal device on the first terminal device and the second terminal device. A write receiving means for receiving writes made by operations on at least one of the first terminal device and the second terminal device, A determination means for determining whether the symbols included in the writing should be used to track the object included in the image, Equipped with, The display control means, when the symbol is a predetermined symbol, displays the writing superimposed on the image so as to maintain the relative positional relationship between the writing and the object; and when the symbol is not a predetermined symbol, displays the writing superimposed on the image at the position in the image where the writing was received. Display control system.

2. A first terminal device and a second terminal device connected via a communication network, comprising display control means for causing the first terminal device to display an image being captured by the first terminal device on the first terminal device and the second terminal device, A write receiving means for receiving writes made by operations on at least one of the first terminal device and the second terminal device, A determination means for determining whether to make the write follow the target based on whether the write was received within the target area included in the image, Equipped with, If the display control means determines that the write has been received within the target area, it displays the write superimposed on the image so as to maintain the relative positional relationship between the write and the target; if it determines that the write has been received outside the target area, it displays the write superimposed on the image at the position where the write was received in the image. Display control system.

3. The determination means determines whether to make the write follow the target based on the proportion to which the write overlaps with the target area. The display control system according to claim 2.

4. The aforementioned subject is a person's face, The area of ​​the subject is the area of ​​the face. The display control system according to claim 2 or 3.

5. The subject is at least a part of a person or at least a part of an object. A display control system according to any one of claims 1 to 3.

6. A target acceptance means for accepting the designation of the aforementioned target, A display control system according to any one of claims 1 to 5, comprising:

7. The target receiving means receives a designation of a target from among the candidate targets included in the image. The display control system according to claim 6.

8. A target identification means for identifying the target corresponding to the location where the aforementioned writing was received, A display control system according to any one of claims 1 to 5, comprising:

9. In a first terminal device and a second terminal device connected via a communication network, the image captured by the first terminal device is displayed on the first terminal device and the second terminal device. The system accepts a write operation to at least one of the first terminal device and the second terminal device. Based on the symbols included in the writing, it is determined whether the writing should follow the object included in the image. If the symbol is a predetermined symbol, the writing is displayed superimposed on the image so as to maintain the relative positional relationship between the writing and the object; if the symbol is not the predetermined symbol, the writing is displayed superimposed on the image at the position in the image where the writing was received. Display control method.

10. In a first terminal device and a second terminal device connected via a communication network, the image captured by the first terminal device is displayed on the first terminal device and the second terminal device. The system accepts a write operation to at least one of the first terminal device and the second terminal device. Whether the write operation is made to follow the target is determined based on whether the write operation was received within the target area included in the aforementioned image. If it is determined that the write has been received within the target area, the write is displayed superimposed on the image so as to maintain the relative positional relationship between the write and the target; if it is determined that the write has been received outside the target area, the write is displayed superimposed on the image at the position where the write was received in the image. Display control method.

11. On the computer, In a first terminal device and a second terminal device connected via a communication network, the image captured by the first terminal device is displayed on the first terminal device and the second terminal device. The system accepts a write operation to at least one of the first terminal device and the second terminal device. Based on the symbols included in the writing, it is determined whether the writing should follow the object included in the image. If the symbol is a predetermined symbol, the writing is displayed superimposed on the image so as to maintain the relative positional relationship between the writing and the object; if the symbol is not the predetermined symbol, the writing is displayed superimposed on the image at the position in the image where the writing was received. A program that executes a process.

12. A computer, In a first terminal device and a second terminal device connected via a communication network, the image captured by the first terminal device is displayed on the first terminal device and the second terminal device. The system accepts a write operation to at least one of the first terminal device and the second terminal device. Whether the write operation is made to follow the target is determined based on whether the write operation was received within the target area included in the aforementioned image. If it is determined that the write has been received within the target area, the write is displayed superimposed on the image so as to maintain the relative positional relationship between the write and the target; if it is determined that the write has been received outside the target area, the write is displayed superimposed on the image at the position where the write was received in the image. A program that executes a process.

Images