Information presentation control device and program

Abstract

In one aspect of the present invention, diminishing / removal process data for causing a video diminishing / removal effect in a target area of a part of acquired video data is generated, an interval period for switching from the video data to the diminishing / removal process data is set, and switching process data for causing an oversight phenomenon of a video change due to the switching during the interval period is generated. Then, the video data, the switching process data, and the diminishing / removal process data are composited, and composite video data obtained by the compositing is displayed.

Description

Information presentation control device and program

[0001] One aspect of the present invention relates to an information presentation control device and a program that apply disappearing reality technology to a process of presenting visual information to a user who is performing desk work, for example.

[0002] The human attention function works to limit unnecessary information from the multiple pieces of information perceived by the five senses and recognize only the necessary information. Therefore, for example, when doing desk work, a person can concentrate on the work by selectively recognizing only the information necessary for the work.

[0003] However, if the function to limit perceived information does not work well, people may end up perceiving unnecessary information as necessary, and become distracted by this unnecessary information, making it difficult to concentrate on their work, resulting in what is known as distraction. For example, this tendency is particularly evident in children and adults with attention deficit hyperactivity disorder (ADHD).

[0004] As a countermeasure for visual distraction, for example, Non-Patent Document 1 describes a method of substituting a person's attention function by limiting the presentation of information using diminished reality (DR) technology. Diminished reality technology, for example, applies image processing to a part of a region on a display screen to make that region physically invisible, i.e., to prevent it from being visually perceived.

[0005] Hagiyama et al., "Study on a system for suppressing inattention during work using distorted reality," Journal of IEICE Technology, vol. 123, no. 228, MVE2023-22, pp. 1-5, 2023.

[0006] However, in the method described in Non-Patent Document 1, when a partial area on a display screen is subjected to a masking process, the masking process is immediately reflected on the screen in response to a user's selection operation. As a result, the appearance of the image displayed in the masking target area changes suddenly during the masking process. As a result, the user may experience a visual discomfort, and the masking process itself may become a new distraction.

[0007] The present invention has been made in light of the above-mentioned circumstances, and aims to provide a technique for reducing the sense of discomfort that the erasing process gives to the user and suppressing the occurrence of new distractions.

[0008] In order to solve the above problem, one aspect of an information display control device according to the present invention generates concealment processing data for producing a concealment effect on a target region of acquired video data, sets an interval period for switching from the video data to the concealment processing data, and generates switching processing data for causing an image change due to the switching to be overlooked during the interval period, and then combines the video data, the switching processing data, and the concealment processing data to display the combined video data.

[0009] According to one aspect of the present invention, when an area in video data that does not substantially need to be confirmed is concealed by applying a concealment effect technique, an interval period of a predetermined length is set in the switching period from the video data to the concealment processing data, and switching processing data for causing an oversight phenomenon of the image change during this interval period is generated and inserted between the video data and the concealment processing data.

[0010] Therefore, it is possible to make the change in image when switching from video data to masked data less noticeable, and as a result, new distractions caused by applying masking processing to the original video data are less likely to occur, making it possible to maintain the user's concentration and improve work efficiency.

[0011] That is, according to one aspect of the present invention, it is possible to provide a technology that reduces the sense of discomfort that the erasure processing gives to the user and suppresses the occurrence of new distractions.

[0012] FIG. 1 is a block diagram showing an example of a hardware configuration of an information presentation control device according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of a software configuration of the information presentation control device according to an embodiment of the present invention. FIG. 3 is a flowchart showing an example of a processing procedure and processing content of an information presentation control process executed by a control unit of the information presentation control device shown in FIG. 2. FIG. 4 is a flowchart showing an example of a processing procedure and processing content of a concealment processing process included in the information presentation control process shown in FIG. 3. FIG. 5 is a flowchart showing an example of a processing procedure and processing content of a switching processing process included in the information presentation control process shown in FIG. 3. FIG. 6 is a diagram showing an example of a switching method management table stored in a switching method storage unit of the information presentation control device shown in FIG. 2. FIG. 7 is a diagram for explaining an example of a switching processing process when a flicker method is applied. FIG. 8 is a diagram for explaining an example of a switching processing process when a splash method is applied. FIG. 9 is a diagram for explaining an example of a switching processing process when a gradual method is applied.

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] [One Embodiment] (Configuration Example) FIGS. 1 and 2 are block diagrams showing an example of the hardware configuration and software configuration, respectively, of an information presentation control device according to one embodiment of the present invention.

[0015] The information presentation control device CS according to one embodiment is configured, for example, by a head-mounted display (HMD) worn on the user's head. The information presentation control device CS may also be configured by smart glasses, a smartphone, a tablet terminal, or a personal computer, and the type of device can be selected depending on the application.

[0016] The information presentation control device CS has a control unit 1 that uses a hardware processor such as a central processing unit (CPU), and this control unit 1 is connected via a bus 5 to a storage unit having a program storage unit 2 and a data storage unit 3, and an input / output interface (hereinafter, interface will be abbreviated as I / F) unit 4.

[0017] A camera CM and a display device DP are connected to the input / output I / F unit 4. The camera CM captures the scenery in the user's line of sight and outputs the video data. The camera CM may be an externally attached type or may be built into the HMD. The display device DP is a transmissive display and is used, for example, to display information required when the user is performing a task.

[0018] The video data to be displayed may be, for example, a video captured by a camera CM with additional data that realizes augmented reality added, and this video may be subjected to the concealment and switching processes described below. Alternatively, the video data to be displayed may be video data transmitted from a server computer via a network, such as conference video data in the metaverse, or may be video data read from a storage medium.

[0019] The program storage unit 2 is, for example, a combination of a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD) as a storage medium that can be written to and read from at any time, and a non-volatile memory such as a read only memory (ROM), and stores application programs necessary for executing various processes related to one embodiment of the present invention, in addition to middleware such as an operating system (OS).

[0020] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an HDD or SSD as a storage medium that can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), and its storage area includes a video data storage unit 31, a concealment effect method storage unit 32, a concealment processing data storage unit 33, and a switching processing method storage unit 34.

[0021] The video data storage unit 31 temporarily stores video data captured by the camera CM for information presentation control processing.

[0022] The disappearance effect method storage unit 32 stores identification information of a plurality of disappearance effect methods that can be applied as a diminished reality (DR) technology, and control data required to execute each disappearance effect method.

[0023] The erasure processing data storage unit 33 stores the erasure processing data generated using the above-mentioned erasure effect method.

[0024] The switching process method storage unit 34 stores a switching process application table. The switching process application table stores information indicating application conditions for image processing defined for each switching process method, in association with identification information for each of a plurality of switching process methods.

[0025] The control unit 1 includes a video data acquisition processing unit 11, a concealment processing unit 12, a switching processing unit 13, a video synthesis processing unit 14, and a synthesized video output processing unit 15 as processing units according to one embodiment of the present invention.

[0026] The processing units 11 to 15 are all realized by causing a hardware processor in the control unit 1 to execute an application program stored in the program storage unit 2. Note that some or all of the processing units 11 to 15 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0027] The video data acquisition processing unit 11 acquires, for example, video data output from a camera CM while a user is working via the input / output I / F unit 4 , and temporarily stores the acquired video data in the video data storage unit 31 .

[0028] The erasure processing unit 12 determines whether or not there is an area to be erased in the video data. If it determines that there is an area to be erased, the erasure processing unit 12 selects one erasure effect method from the multiple types of erasure effect methods stored in the erasure effect method storage unit 32, generates erasure processing data used to erase the area to be erased in accordance with the selected erasure effect method, and stores the generated data in the erasure processing data storage unit 33. An example of the erasure processing data generation process will be described in the operation example.

[0029] The switching processing unit 13 selects a switching processing method that satisfies the application conditions from among a plurality of switching processing methods stored in the switching processing application table of the switching processing method storage unit 34, and sets a predetermined interval period for the switching period from the video data to the concealment processing data according to the selected switching processing method.Then, switching processing data is generated to cause the image change to be overlooked during the interval period.An example of the switching processing data generation process will be described in the operation example.

[0030] The video synthesis processing unit 14 sequentially synthesizes the video data stored in the video data storage unit 31, the switching processing data generated by the switching processing unit 13, and the concealment processing data stored in the concealment processing data storage unit 33 along the time axis to generate synthesized video data.

[0031] The composite video output processing section 15 outputs the composite video data generated by the video composition processing section 14 from the input / output I / F section 4 to the display device DP for display.

[0032] (Example of Operation) Next, an example of operation of the information presentation control device CS configured as above will be described.

[0033] FIG. 3 is a flowchart showing an example of the processing procedure and processing content of information presentation control executed by the control unit 1 of the information presentation control device CS.

[0034] (1) Acquisition of video data When the user starts work and the start of this work is detected in step S10, the control unit 1 of the information presentation control device CS, under the control of the video data acquisition processing unit 11, acquires video data corresponding to the user's line of sight output from the camera CM via the input / output I / F unit 4 in step S20, and stores the acquired video data in the video data storage unit 31.

[0035] As mentioned in the configuration example, the video data may be acquired not only from the camera CM but also from a server computer on the Web or cloud, other information terminals, or external storage media.

[0036] (2) Erasing Process When acquisition of the video data is started, the control unit 1 of the information presentation control device CS, under the control of the erasing process unit 12, executes the erasing process on the video data in step S30 as follows.

[0037] FIG. 4 is a flowchart showing an example of the processing procedure and processing contents executed by the concealment processing unit 12.

[0038] That is, first, in step S31, the erasure processing section 12 reads the video data from the video data storage section 31 for each frame, and determines whether or not there is a region to be erased from the read video data.

[0039] The area to be obscured is an area where an object or the like that may distract the user's attention is present, and can be recognized by performing image analysis on a video frame using existing technology such as object shape recognition technology, color identification or brightness identification technology, and determining whether the information calculated as a result, such as the object shape, color or brightness, matches a preset condition. The above conditions consist of data representing the shape, color, pattern, etc. of an object that is expected to distract the user's attention, and are stored in advance in the data storage unit 3.

[0040] When the object to be erased is recognized, the erasure processing unit 12 sets, for example, an area including the object to be erased as the area to be erased in step S32. Alternatively, the contour of the object may be detected, and an area enclosed by the detected contour may be set as the area to be erased.

[0041] Next, in step S33, the erasure processing unit 12 sets the disappearance effect, for example, as follows. That is, the erasure processing unit 12 selects and sets one applicable disappearance effect method from the disappearance effect method storage unit 32 based on a predetermined selection method. As a selection method, for example, a method is used in which multiple types of disappearance effect method candidates read from the disappearance effect method storage unit 32 are displayed on the display device DP via the input / output I / F unit 4, and a method selected and specified by the user from the multiple types of disappearance effect method candidates displayed. Note that a specific type of disappearance effect method prepared in advance may always be set as the disappearance effect method.

[0042] Specific examples of the disappearance effect method include various existing methods such as "transparency" and "outline" which change the appearance of an object by superimposing it on a background, "blurring" which changes the resolution, "low saturation" and "low salience" which change the saturation, and "low contrast" which changes the brightness. Specific examples of the disappearance effect methods are described in detail in Non-Patent Document 1.

[0043] Finally, in step S34, the erasure processing unit 12 generates erasure processing data for applying the erasure effect to the area to be erased in the video data in accordance with the set erasure effect method, and stores the generated erasure processing data in the erasure processing data storage unit 33.

[0044] The erasure processing unit 12 may first perform the process of selecting and setting the erasure effect method in step S33, and then perform the process of setting the area to be erased in step S32.

[0045] Furthermore, if it is determined in step S31 that there is no area to be erased, the erasure processing unit 12 does not perform the series of erasure processing steps S32 to S34 described above, and notifies the image synthesis processing unit 14, which will be described later, of this fact.

[0046] (3) Switching Processing After the above-mentioned masking processing is completed, the control unit 1 of the information presentation control device CS then, under the control of the switching processing unit 13, executes a switching process from the video data to the above-mentioned masking processing data in step S40 as follows.

[0047] Fig. 5 is a flowchart showing an example of the processing procedure and processing contents executed by the switching processing unit 13. Fig. 6 shows an example of multiple types of switching processing methods that are expected to be applied and application conditions for selecting these methods, and this information is stored in a switching processing application table in the switching processing method storage unit 34.

[0048] That is, when the switching processing unit 13 receives the concealment processing data from the concealment processing unit 12 in step S41, it selects a switching processing method that satisfies the application conditions from the multiple types of switching processing methods stored in the switching processing application table in the switching processing method storage unit 34 in step S42.

[0049] Examples of switching processing methods include the "flicker method," "splash method," and "gradual method," as shown in Figure 6, and application conditions include characteristics of the area or object to be obscured, characteristics of the obscuring effect to be applied, characteristics of the user himself, etc.

[0050] For example, if the object to be obscured is determined to be a stationary object in the process of determining the obscured area described above, the condition "1" shown in Fig. 6 is met. Therefore, the switching processing unit 13 selects the "gradual method," which is a switching processing method set in advance in accordance with the condition "1."

[0051] On the other hand, if the area to be obscured is equal to or larger than a certain size relative to the screen size, for example, equal to or larger than 1 / 4 of the screen size, the switching processing unit 13 determines that the above characteristic meets condition "2," and in this case selects the "flicker method" as the switching processing method.

[0052] Moreover, if the characteristics of the region to be concealed or erased do not match any of the conditions, the switching processing unit 13 selects the condition "n" and selects the "splash method" as the switching processing method.

[0053] Next, in step S43, the switching processing unit 13 generates switching processing data in accordance with the selected switching processing method. The switching processing data includes control parameters necessary for the switching processing, which are stored in the switching processing method storage unit 34. The control parameters include an interval period that defines a period during which the video data is switched to the concealment processing data.

[0054] Finally, in step S44, the switching processing unit 13 sends the generated switching processing data to the video composition processing unit 14.

[0055] (4) Image Composition Processing Under the control of the image composition processing unit 14, the control unit 1 of the information presentation control device CS reads image data from the image data storage unit 31 and reads the concealment processing data from the concealment processing data storage unit 33 in step S50. Then, the image composition processing unit 14 sets an interval period based on the switching processing data sent from the switching processing processing unit 13 during a switching period from the read image data to the concealment processing data, and generates and inserts a switching processing image specified by a control parameter of the switching processing data during this interval period.

[0056] For example, when data based on the "flicker method" is sent as switching processing data, the video composition processing unit 14 sets an interval period of 80 msec between the image frame of the original video data and the frame of the concealment processing data, and generates and inserts a blank image during this interval period, as shown in Fig. 7. The image to be inserted is not limited to a blank image, but may be an image of a simple design that does not attract the user's attention, or an image of a familiar scene.

[0057] On the other hand, when data based on the "splash method" is sent as the switching processing data, the video composition processing unit 14 sets an interval period of 80 msec between the image frame of the original video data and the frame of the concealment processing data, and during this interval period, generates a plurality of small figures smaller than the size of the image frame and superimposes them on the original image frame, as shown in Figure 8. The small figures may be, for example, rectangular figures with a monochrome dot pattern, but are not limited to this.

[0058] Furthermore, for example, when data based on the "gradual method" is sent as the switching processing data, the video synthesis processing unit 14 sets an interval period of 12 seconds between the image frame of the original video data and the frame of the concealment processing data, and uses this interval period to gradually change the original video data into video data that has been subjected to concealment processing, as shown in Figure 9. More specifically, the color, transparency, etc. of the area to be concealed in the original video data is changed gradually or in stages over the above 12 seconds.

[0059] In any of the above switching processing methods, the interval period is not limited to the above times, and may be set to any time period necessary to cause the phenomenon of overlooking a change in the image.

[0060] (5) Output of composite video data The control unit 1 of the information presentation control device CS outputs the composite video data generated by the video synthesis processing unit 14 from the input / output I / F unit 4 to the display device DP under the control of the composite video output processing unit 15 in step S60, and displays it.

[0061] In addition, if the area to be obscured is not recognized by the obscuration processing unit 12 and the obscuration processing unit 12 notifies the user of this, the composite video output processing unit 15 outputs the video data read from the video data storage unit 31 directly from the input / output I / F unit 4 to the display device DP and displays it.

[0062] (Effect) As described above, in one embodiment, when an area to be erased exists in acquired video data, erasure processing data for erasing the area to be erased is generated, and switching processing data for causing an image change to be overlooked when switching from the video data to the erasure processing data is generated. Then, in accordance with the generated switching processing data, an interval period is set for a switching period from the video data to the erasure processing data, and an image specified by the switching processing data is generated and inserted during this interval period.

[0063] Therefore, it is possible to make the change in image when switching from image data to image data with a concealment effect less noticeable, and as a result, new distractions caused by the concealment processing are less likely to occur, making it possible to maintain the user's concentration and improve work efficiency.

[0064] In one embodiment, a plurality of types of switching processing methods are prepared, and a switching processing method that satisfies pre-set application conditions according to the characteristics of the concealment / erasure processing is selected to perform the switching processing. This allows for more appropriate switching processing according to the concealment / erasure processing method, thereby making it possible to achieve a high degree of oversight prevention for the image changes when video data is converted into concealment / erasure processed data.

[0065] [Other Embodiments] (1) In one embodiment, a case where a switching processing method is selected based on common application conditions for all users has been described. However, it is assumed that users have individual differences in how easily they notice changes in images. Therefore, in this case, a switching processing method that is thought to make it difficult for each user to notice the concealment processing, i.e., to cause little distraction, is set as the switching processing method for condition "n." In this way, a more appropriate switching processing method can be selected according to the user's sensibility, thereby making it possible to reduce variation in the distraction reduction effect between users.

[0066] (2) Any conditions may be set as application conditions in the switching process application table other than the conditions exemplified in Fig. 6. Furthermore, the data in the switching process application table may be statically set in advance, or the table may be dynamically updated as needed.

[0067] In the case of dynamic updating, the table is updated based on the user's reaction to the switching process. For example, a method for measuring the user's concentration state (such as eye tracking) is used to observe the change in the user's concentration state after the switching process is applied, and if the degree of change in the concentration state is greater than a predetermined value, i.e., if it is estimated that the effect of the switching process is small, the applicable conditions and switching process method are excluded from the switching process application table.

[0068] In addition, when a method for measuring the user's concentration state is used, the method may be used not only to dynamically update the application conditions and the switching processing method, but also to modify the control parameters of each switching processing method. For example, when the flicker method or the splash method is applied as the switching processing method, the application time may be increased or decreased from, for example, 80 milliseconds depending on the amount of change in the concentration state.

[0069] In addition, in one embodiment, an example was described in which a switching processing application table was used, but any configuration may be used as long as it allows selection from among multiple switching processing techniques.

[0070] (3) In the embodiment, the process of setting the obscuration area in the video data has been described as being performed by the control unit 1. However, the obscuration area in the video data may be set by the user or a system administrator. In the case where the obscuration area is set by the user, for example, the user may actually view the video displayed on the display device DP, identify an object or area in the video that distracts the user, and specify the identified object or area using an input device such as a keyboard or mouse, thereby setting the object or area in the information presentation control device CS.

[0071] (4) In one embodiment, the processing of information presented to a user while the user is performing deskwork has been described as an example. However, the application is not limited to this. For example, the present invention can also be applied to presenting images necessary for work to a user working at a manufacturing site or construction site, or to a user remotely controlling a robot working in a remote location.

[0072] (5) In addition, the functional configuration of the information presentation control device, the processing procedures and processing contents of the concealment processing, switching processing and video synthesis processing, the types of concealment processing methods and switching processing methods, etc. can be modified and implemented in various ways within the scope of the gist of this invention.

[0073] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0074] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0075] CS...information presentation control device CM...camera DP...display device 1...control unit 2...program storage unit 3...data storage unit 4...input / output I / F unit 5...bus 11...video data acquisition processing unit 12...concealment processing processing unit 13...switching processing processing unit 14...video synthesis processing unit 15...synthesized video output processing unit 31...video data storage unit 32...concealment effect method storage unit 33...concealment processing data storage unit 34...switching processing method storage unit

Claims

1. An information presentation control device comprising: a first processing unit that acquires video data; a second processing unit that generates concealment processing data for causing a concealment effect of the video on a target area of a part of the video data; a third processing unit that sets an interval period for switching from the video data to the concealment processing data, and generates switching processing data for causing an oversight phenomenon of the video change due to the switching during the interval period; and a fourth processing unit that synthesizes the video data, the switching processing data, and the concealment processing data.

2. The information presentation control device according to claim 1, wherein the third processing unit generates blank image data as the switching processing data during the interval period.

3. The information presentation control device according to claim 1, wherein the third processing unit generates splash image data in which a plurality of figures smaller than the frame size are dispersedly arranged within the frame of the video data as the switching processing data during the interval period.

4. The information presentation control device according to claim 1, wherein the third processing unit generates change video data for gradually changing the target area of the video data to the concealment processing data as the switching processing data during the interval period.

5. A program for causing a processor included in the information presentation control device to execute processing by the first processing unit, the second processing unit, the third processing unit, and the fourth processing unit included in the information presentation control device according to any one of claims 1 to 4.

Images