Information processing device, image processing device, and program

Abstract

To provide an information processing device, an image processing system, and a program that evaluate a risk of information leakage caused by a third-party peeking at information displayed on an image output device.SOLUTION: In an information leakage prevention system, a control device 170 of an image output device includes: a peripheral state acquisition unit that acquires peripheral state information indicating a peripheral state of a display area where an output image output by an image output device is displayed; a safety management unit which is a browsability deriving unit that derives, on the basis of the peripheral state of the display area indicated by the acquired peripheral state information, degree of browsability of an output image by a third party different from a user of the image output device; a moving image acquisition unit that acquires moving image data including a plurality of frames; and a variation index derivation unit that divides each frame of a moving image into a plurality of predetermined areas, and derives, for each of the plurality of areas, on the basis of pixel values of pixels included in the areas, a variation index value, which is an index indicating a variation in color and / or brightness of each area between a plurality of frames.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an image processing device, and a program. [Background technology]

[0002] Patent Document 1 discloses an information processing device that includes a display unit having a filter layer that can change the viewing angle of the display, and an anti-peeping function that uses the filter layer to reduce visibility from any side other than the front of the display unit. [Prior art document] [Patent documents] [Patent Document 1] JP 2021-182069 A Summary of the Invention [Means for solving the problem]

[0003] In a first aspect of the present invention, there is provided an information processing device. The information processing device includes, for example, a peripheral state acquisition unit that acquires peripheral state information indicating a peripheral state of a display area where an output image output by an image output device is displayed. The information processing device also includes, for example, a viewability derivation unit that derives a degree of viewability of the output image by a third party other than the user of the image output device based on the peripheral state of the display area indicated by the peripheral state information acquired by the peripheral state acquisition unit.

[0004] In any of the above information processing devices, the browsability derivation unit may include an index value derivation unit that derives values ​​of a plurality of indexes related to the degree of browsability, and an evaluation unit that evaluates the degree of browsability based on the values ​​of the plurality of indexes derived by the index value derivation unit.

[0005] In any of the above information processing devices, the index value derivation unit may determine the types of the multiple indices based on (i) the state of the surroundings of the display area indicated by the surrounding state information acquired by the surrounding state acquisition unit, and / or (ii) the current time or the time zone to which the current time belongs. In any of the above information processing devices, the evaluation unit may determine weight values ​​to be set for each of the multiple indices based on (i) the state of the surroundings of the display area indicated by the surrounding state information acquired by the surrounding state acquisition unit, and / or (ii) the current time or the time zone to which the current time belongs. The evaluation unit may evaluate the degree of viewability based on the values ​​of each of the multiple indices derived by the index value derivation unit and the weight values ​​to be set for each of the multiple indices.

[0006] In any of the above information processing devices, the surrounding state acquisition unit may include a video acquisition unit that acquires video data including a plurality of frames from an imaging device that captures an image of the periphery of the display area. The index value derivation unit may include a first index derivation unit that derives a value of a first index included in the multiple indexes. The first index derivation unit may include a variation index derivation unit that (a) divides each frame of the video into multiple predetermined regions, and (b) derives, for each of the multiple regions, a value of a variation index that indicates a variation in color and / or brightness of the region between multiple frames, as a value of the first index, based on pixel values ​​of pixels included in the region. Each of the multiple regions may include one or more pixels. The evaluation unit may evaluate a degree of viewability when the display region is located at a position where the imaging device captured the video, based on the value of the variation index for each of the multiple regions derived by the variation index derivation unit.

[0007] In any of the above information processing devices, the evaluation unit may calculate, for each of the multiple regions, a viewing index value that indicates a degree of likelihood that the output image will be viewed from a position in real space corresponding to each region, based on the value of a variation index for each of the multiple regions. The evaluation unit may evaluate the degree of viewability based on the value of the viewing index for each of the multiple regions. The evaluation unit may calculate the viewing index value so that (i) the viewing index value for a region with a large variation index value is larger than the viewing index value for a region with a small variation index value, and / or (ii) the viewing index value for a region close to the position of an optical axis of the imaging device in the video is larger than the viewing index value for a region far from the position of the optical axis of the imaging device in the video.

[0008] Any of the above information processing devices may include a variation image generation unit that generates a variation image representing the magnitude of the variation index value of each region or each pixel based on the variation index value of each of the multiple regions derived by the variation index derivation unit and a predetermined number of gradations. In any of the above information processing devices, the video acquisition unit may acquire data of multiple videos captured in different imaging directions, the multiple videos being videos of the periphery of a specific position in real space. The variation image generation unit may generate multiple variation images for each of the multiple videos. Any of the above information processing devices may include a combined image generation unit that generates a combined image by panoramic-combining the multiple variation images generated by the variation image generation unit, and a unit image extraction unit that extracts, from the combined image, unit images that correspond to the viewing angle of the display region when the display region is positioned at a specific position and in a specific orientation. In any of the above information processing devices, the evaluation unit may evaluate the degree of viewability when the display region is positioned at a specific position and in a specific orientation based on the variation index values ​​of each of the multiple regions in the unit image. In any of the above information processing devices, the unit image extraction unit may extract multiple unit images with different specific orientations. The evaluation unit may evaluate the degree of viewability for each of the plurality of unit images.

[0009] Any of the above information processing devices may include a marker detection unit that detects a marker reflected in a video. The marker may include information indicating a degree of viewability. A value of a variation index corresponding to a region among the multiple regions in which the marker is reflected may be determined based on information indicating the degree of viewability indicated by the marker. In any of the above information processing devices, the marker may further include information indicating a size of a region to which the degree of viewability applies. A value of a variation index corresponding to a region among the multiple regions in which the marker is reflected may be determined based on the degree of viewability indicated by the marker and the size of the region. In any of the above information processing devices, the image output device may include a housing, a display region arranged on the housing, and an output device arranged on the housing that outputs peripheral state information to the peripheral state acquisition unit. The viewability derivation unit may derive the degree of viewability at multiple points in time during a period in which the user continuously uses the image output device.

[0010] In any of the above information processing devices, the peripheral state acquisition unit may include a peripheral data acquisition unit that acquires image data of an image having one or more objects disposed around the display area as subjects, or peripheral data that is three-dimensional point cloud data or distance data of the one or more objects. Any of the above information processing devices may include an object position determination unit that analyzes the peripheral data acquired by the peripheral data acquisition unit and determines a positional relationship between each of the one or more objects and the display area. In any of the above information processing devices, the viewability derivation unit may derive a degree of viewability based on the positional relationship determined by the object position determination unit. In any of the above information processing devices, the object position determination unit may include a point cloud data acquisition unit that acquires three-dimensional point cloud data for each of the one or more objects based on the peripheral data acquired by the peripheral data acquisition unit, and a distance calculation unit that calculates a distance between a representative point of the display area and each of a plurality of points virtually arranged on the surface of each of the one or more objects represented by the three-dimensional point cloud data acquired by the point cloud data acquisition unit. In any of the above information processing devices, the index value derivation unit may include a second index derivation unit that derives a value of a second index included in the multiple indices based on the distance for each of the multiple points calculated by the distance calculation unit. In any of the above information processing devices, the second index derivation unit may derive a value of a suppression index for each of the multiple points, the suppression index being an index indicating the degree to which each point suppresses viewing of the output image by a third party, based on the distance for each of the multiple points. The second index derivation unit may output the value of the suppression index for each of the multiple points as the value of the second index. In any of the above information processing devices, the evaluation unit may evaluate the degree of viewability by summing the values ​​of the suppression index for each of the multiple points.

[0011] In any of the above information processing devices, the peripheral state acquisition unit may include an environmental information acquisition unit that acquires environmental information indicating the status of the environment surrounding the display area. The index value derivation unit may include a third index derivation unit that derives a value of a third index included in the multiple indexes based on the status of the surrounding environment indicated by the environmental information acquired by the environmental information acquisition unit. The evaluation unit, the viewability derivation unit, may evaluate the degree of viewability based on the value of the third index. In any of the above information processing devices, the peripheral state acquisition unit may include a peripheral data acquisition unit that acquires image data of an image having one or more objects arranged around the display area as subjects, or peripheral data that is three-dimensional point cloud data or distance data of the one or more objects. Any of the above information processing devices may include an object position determination unit that analyzes the peripheral data acquired by the peripheral data acquisition unit and determines a positional relationship between each of the one or more objects and the display area. The viewability derivation unit may derive a degree of viewability based on the positional relationship determined by the object position determination unit.

[0012] Any of the above information processing devices may include a startup detection unit that detects startup of the image output device or startup of a BIOS, OS, or application program running on the image output device, and a derivation procedure determination unit that determines a derivation procedure, which is a procedure by which the viewability derivation unit derives the degree of viewability. The derivation procedure determination unit may determine a derivation procedure such that the derivation procedure is different when the elapsed time since the startup detection unit detected the startup satisfies a predetermined condition and when the elapsed time does not satisfy the predetermined condition. Any of the above information processing devices may include an output control unit that controls output of images by the image output device. In any of the above information processing devices, the output control unit may, when the degree of viewability derived by the viewability derivation unit exceeds a predetermined level, stop output of an output image that was displayed before the degree of viewability exceeded the predetermined level, adjust the brightness or contrast of the output image, or display an image different from the output image in the display area. In any of the above information processing devices, the output control unit may permit output of the output image when the degree of viewability derived by the viewability derivation unit is smaller than a predetermined degree.

[0013] In a second aspect of the present invention, there is provided an image processing device. The image processing device includes, for example, a moving image acquisition unit that acquires data of a moving image including a plurality of frames. The image processing device includes, for example, (a) a moving image acquisition unit that divides each frame of the moving image into a plurality of predetermined regions, and (b) a fluctuation index derivation unit that derives, for each of the plurality of regions, a value of a fluctuation index that is an index indicating fluctuations in color and / or brightness of each region between the plurality of frames based on pixel values ​​of pixels included in each region. In the image processing device, each of the plurality of regions includes, for example, one or more pixels.

[0014] In any of the above image processing devices, the fluctuation index may be at least one of variance, standard deviation, and coefficient of variation, and values ​​obtained by dividing these by a reference value. In any of the above image processing devices, the fluctuation index derivation unit may include a fluctuation index calculation unit that calculates a value of a fluctuation index for each of the multiple regions, a movement state determination unit that analyzes the video and determines a movement state of an imaging device that captured the video based on movement states of feature points in the images between multiple frames, and a correction unit that corrects the value of the fluctuation index for each of the multiple regions based on the movement state of the imaging device determined by the movement state determination unit. Any of the above image processing devices may include a fluctuation image generation unit that generates a fluctuation image representing the magnitude of the value of the fluctuation index for each region or each pixel based on the value of the fluctuation index for each of the multiple regions derived by the fluctuation index derivation unit and a predetermined number of gradations.

[0015] In a third aspect of the present invention, a program is provided. The program may be a program for causing a computer to function as the information processing device according to the first embodiment. The program may be a program for causing a computer to function as the image processing device according to the second embodiment. A computer-readable medium for storing the program may be provided. The computer-readable medium may be a non-transitory computer-readable medium. The computer-readable medium may be a computer-readable recording medium.

[0016] The above summary of the invention does not list all of the necessary features of the present invention, and subcombinations of these features may also constitute inventions. [Brief explanation of the drawings]

[0017] [Figure 1] 1 shows an example of a system configuration of an information leakage prevention system 100. [Figure 2] The viewing angles of the display area 142 and the peripheral data generating device 152 are shown schematically. [Figure 3] 1 shows a schematic diagram of an example of the placement of user terminals 120. [Figure 4] 10 shows an example of information processing in the control device 170. [Figure 5] 10 shows an example of information processing in the control device 170. [Figure 6] 2 shows an example of the internal configuration of the control device 170. [Figure 7] 10 shows an example of the internal configuration of a structural information acquisition unit 612. [Figure 8] 10 shows an example of the internal configuration of the virtual space construction unit 630. [Figure 9] 10 shows an example of information processing in the first calculation unit 830. [Figure 10] 10 shows an example of information processing in the second calculation unit 840. [Figure 11] 10 shows an example of the internal configuration of the safety management unit 640. [Figure 12] 10 shows an example of information processing in the safety management unit 640. [Figure 13] 10 shows an example of information processing in the control device 170. [Figure 14] 13A and 13B show an example of a process for generating the dynamic image data 1340. [Figure 15] 13 shows an example of the internal configuration of the fluctuation index derivation unit 1374. [Figure 16] 13 shows an example of the internal configuration of a safety management unit 1378. [Figure 17] 13A and 13B show an example of information processing in the safety management unit 1378. [Figure 18] An example of a two-dimensional image 1800 is shown schematically. [Figure 19] An example of a panoramic image 1900 is shown schematically. [Figure 20] 3 shows an example of a system configuration of a computer 3000. DETAILED DESCRIPTION OF THE INVENTION

[0018] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. In the drawings, the same reference numerals are used to designate the same or similar parts, and redundant explanations may be omitted.

[0019] An information processing terminal displays various types of information on an image output device such as a display, projector, or screen. If the information displayed by the information processing terminal on the image output device is viewed by a third party other than the user of the information processing terminal, the information may be leaked to the outside. The manner of viewing by a third party is not limited to visual inspection. For example, the third party may view the information displayed by the information processing terminal on the image output device using an imaging device located near the image output device. Examples of such imaging devices include a surveillance camera and a hidden camera. For example, the third party may access the imaging device via a network to view the information displayed by the information processing terminal on the image output device.

[0020] In recent years, various technologies for preventing information leakage from image output devices have been developed, but the inventors thought that if the image output device could be installed in a location where it is relatively difficult for a third party to view the information, the risk of information leakage could be significantly reduced. According to this embodiment, for example, an information leakage prevention system is provided that can evaluate the risk of information leakage caused by a third party peeking at information displayed on the image output device, depending on the surrounding conditions of the image output device.

[0021] According to one embodiment, the information leakage prevention system evaluates the security of an information processing terminal. More specifically, the information leakage prevention system evaluates the level of information security (sometimes simply referred to as security) at the planned installation location or installation location (sometimes referred to as the installation location of the information processing terminal) of an image output device that outputs images generated by the information processing terminal. This allows the user of the information processing terminal to begin using the information processing terminal after installing the image output device of the information processing terminal in a safe location. As a result, the risk of information leakage is significantly reduced.

[0022] According to another embodiment, the information leakage prevention system monitors the security of an information processing terminal, for example, while a user is using the information processing terminal. The information leakage prevention system may constantly check the security of the information processing terminal, or may check the security of the information processing terminal periodically or at any timing. This can reduce the risk of information leakage, for example, even if the security level of the information processing terminal fluctuates due to changes in the surrounding environment over time.

[0023] The safety of an information processing terminal is evaluated based on, for example, the location of the image output device. The safety of an information processing terminal may also be evaluated based on the location of the image output device and the orientation of the image output device at that location (sometimes referred to as the installation direction of the information processing terminal). Note that, in this specification, when an expression "installing an information processing terminal" is used, it may also mean installing the image output device of the information processing terminal, to the extent that no technical contradiction occurs.

[0024] Examples of the image output device include image display devices such as displays and screens, and image projection devices such as projectors. The screen outputs an image, for example, by reflecting the image output by the image projection device. As a result, the image output by the image projection device is displayed on the screen.

[0025] The image output by the image output device (sometimes referred to as the output image) may be displayed on a flat surface, a curved surface, a spherical surface or a portion thereof, and may be a two-dimensional image or a three-dimensional image.

[0026] The installation direction of the information processing terminal is represented by, for example, at least one of rotation around the front-to-back axis of the image output device (sometimes referred to as roll), rotation around the left-to-right axis (sometimes referred to as pitch), and rotation around the up-to-down axis (sometimes referred to as yaw). The front-to-back axis of the image output device may be an axis that passes through the image output device from front to back. The up-to-down axis of the image output device may be an axis that passes through the image output device from top to bottom. The left-to-right axis of the image output device may be an axis that passes through the image output device from left to right. The above three axes may be perpendicular to each other. If the image output device has a curved surface, a space curve extended along the curved surface may be used as an axis.

[0027] (Safety assessment based on object movement or light fluctuation) In one embodiment, it is conceivable to evaluate the safety of an information processing terminal based on the movement of objects or the fluctuation of light in the vicinity of the installation location of the information processing terminal. In this embodiment, the safety of the information processing terminal is evaluated based on the analysis results of a video obtained by capturing an image of the vicinity of the installation location of the information processing terminal.

[0028] When an object located near the installation location of the information processing terminal moves, the pixel value of a pixel corresponding to the object fluctuates among the frames constituting the video, even if the imaging device that captured the video is stationary. Also, even if the object located near the installation location of the information processing terminal and the imaging device that captured the video are stationary, the pixel value of a specific pixel fluctuates among the frames constituting the video due to fluctuations in light reflected from the surface of the object, light that passes through the object, or light emitted from the object.

[0029] For example, even if the average light intensity is constant macroscopically, if there is a small deviation between the light intensity and the average light intensity microscopically, light fluctuations are detected. Light fluctuations indicate, for example, that the light intensity and / or color fluctuate partially randomly while following a specific average. Light fluctuations observed around the installation location of an information processing terminal represent, for example, (i) the degree of movement of various other objects that exist or have existed between a light source and a subject that reflects or transmits light from the light source, and / or (ii) the degree of movement of other objects located between the subject and an imaging device. The other objects may be solid, gaseous, or liquid. Light fluctuations may (i) include changes with a high degree of periodicity to some extent, or (ii) include movements that are close to random.

[0030] An example of a method for evaluating the safety of an information processing terminal is to evaluate the degree to which object movement or light fluctuation is detected around the installation location of the information processing terminal. Examples of the degree include the magnitude of the detected movement or fluctuation, the frequency at which the movement or fluctuation is detected, etc.

[0031] When object movement is detected around the installation location of the information processing terminal, there is a high possibility that a third party is present around the installation location of the information processing terminal. Similarly, when relatively large light fluctuations are detected around the installation location of the information processing terminal, there is a high possibility that a third party is present around the installation location of the information processing terminal. Furthermore, when the pattern of light fluctuations around the installation location of the information processing terminal changes, there is a high possibility that a third party is present around the installation location of the information processing terminal. Therefore, the lower the degree to which object movement or light fluctuations are detected, the higher the degree of security of the information processing terminal can be evaluated (i.e., the lower the possibility that the output image will be viewed by a third party).

[0032] If the light fluctuation pattern observed around the installation location of the information processing terminal resembles the fluctuation pattern of natural light, there is a high possibility that a window or opening leading to the outdoors is located near the installation location, or that the installation location is outdoors. On the other hand, if the light fluctuation pattern observed around the installation location of the information processing terminal resembles the fluctuation pattern of artificial light, there is a low possibility that a window or opening leading to the outdoors is located near the installation location, or that the installation location is outdoors. Therefore, the lower the degree to which light fluctuation similar to natural light is detected, the higher the security of the information processing terminal can be evaluated (i.e., the lower the possibility that the output image will be viewed by a third party).

[0033] Furthermore, if the light fluctuation pattern observed around the installation location of the information processing terminal is similar to the light fluctuation pattern when the information processing terminal is mounted on a predetermined type of mobile object, there is a high possibility that the information processing terminal is mounted on that mobile object. Examples of predetermined types of mobile objects include vehicles, aircraft, and ships. The predetermined type of mobile object may be public transportation such as trains, buses, passenger planes, and cruise ships. Therefore, if the detected light fluctuation has a predetermined first pattern, the information processing terminal may be evaluated as having a low level of security (i.e., there is a high possibility that the output image will be viewed by a third party). Similarly, if the detected light fluctuation has a predetermined second pattern, the information processing terminal may be evaluated as having a high level of security (i.e., there is a low possibility that the output image will be viewed by a third party). The light fluctuation pattern may be an example of the level at which light fluctuation is detected.

[0034] For example, consider a case where natural light entering a room is reflected by the surface of an object placed in the room, and the reflected light is received by a camera placed at the location where an information processing terminal is installed. In this case, the fluctuations in the light received by the camera are affected by factors between the sun (the light source) and the object, and between the object and the camera, such as (i) the swaying of tree branches and leaves, (ii) the swaying of the water surface, (iii) the flow of dust, water droplets, water vapor, etc. suspended in the air, and (iv) fluctuations in the refractive index of the air. Therefore, when the light fluctuations are analyzed using a frequency analysis, the fluctuations include a relatively wide range of frequencies. For example, the frequency distribution of the light fluctuations has a relatively broad, mountain-like or normal distribution shape.

[0035] Similarly, consider a case where artificial light emitted from a light source placed in a room is reflected by the surface of an object placed in the room, and the reflected light is received by a camera placed at the installation location of an information processing terminal. In this case, the distance between the light source and the object is shorter than when the light source is the sun. As a result, when the light fluctuation is analyzed by frequency analysis, the fluctuation contains a relatively narrow range of frequencies, or contains many specific frequency components. For example, the frequency distribution of the light fluctuation has a sharp peak shape or a normal distribution shape, or a shape in which specific frequency components are prominent, compared to when the light source is the sun.

[0036] The inventors have found that (i) natural light contains more light fluctuations in the frequency band of about several Hz to several tens of Hz than light from artificial light sources, (ii) the light fluctuations can be detected by analyzing moving images captured by an imaging device at about 30 frames per second, and (iii) it is possible to estimate an event occurring at a location where the light is observed based on the pattern of the light fluctuations (for example, a pattern of the frequency distribution of the fluctuations).The inventors have also conceived of evaluating the safety of a location where the light is observed based on the observed light fluctuations.

[0037] (Safety assessment based on the presence or absence of shielding or the amount of shielding) In another embodiment, it is possible to evaluate the safety of an information processing terminal based on the arrangement of structures (sometimes referred to as shields) arranged around the installation location of the information processing terminal. The shields may be solid, liquid, gas, or a combination thereof. Another example of a method for evaluating the safety is to evaluate the arrangement of the shields.

[0038] For example, the greater the degree to which a third party's view of a display, screen, etc. is blocked by an obstruction, the greater the degree of safety of the information processing terminal is evaluated. The shorter the distance between the installation position of the information processing terminal and the obstruction, the less likely it is that a third party will get in between the installation position and the obstruction. Therefore, the shorter the distance between the installation position of the information processing terminal and the obstruction, the greater the degree of safety of the information processing terminal may be evaluated. Similarly, the greater the distance, the less the degree of safety of the information processing terminal may be evaluated.

[0039] (Safety assessment based on the surrounding environment) In yet another embodiment, it is conceivable to evaluate the safety of an information processing terminal based on the status of the environment around the installation location of the information processing terminal (sometimes referred to as the surrounding environment). The environment around the installation location may include the environment of the installation location. Examples of the status of the surrounding environment include the temperature distribution status, brightness status, weather status, air status, and the status of people flow or population in the surrounding environment. The status of the surrounding environment may be advanced. An example of a method for evaluating the safety of an information processing terminal is to evaluate the status of the surrounding environment. The surrounding environment may be an example of the environment around the display area.

[0040] (Temperature distribution) The temperature of the surrounding environment can be measured using, for example, an infrared camera, a thermo camera, a thermal imaging camera, or the like. The condition of the surrounding environment can be evaluated by comparing the temperature at the installation location of the information processing terminal with the temperature around the installation location of the information processing terminal. The infrared camera may be a mid-infrared camera or a far-infrared camera.

[0041] For example, people, tools used by people (such as electronic devices), and places exposed to natural light tend to be hotter than the surrounding environment. This tendency is particularly pronounced indoors. Therefore, in a specific space, areas that are hotter than other areas in the space can be evaluated as having a low level of security for the information processing terminal (i.e., there is a high possibility that the output image will be viewed by a third party).

[0042] For example, if a small camera is secretly installed in a room, the temperature at the location where the small camera is installed may be higher than the temperature in the vicinity of the installation location. In this case, the safety of information processing terminals in the vicinity of the installation location of the small camera may be reduced. As a result, the risk of information leakage due to the small camera can be reduced.

[0043] (Brightness conditions) Examples of indicators for evaluating brightness include (i) the current time, the time zone to which the current time belongs, and / or the weather, and (ii) illuminance. The current time may or may not include information indicating the date. The outdoor brightness at a specific time may vary depending on the season. When the current time includes information indicating the date (sometimes referred to as the current date and time), brightness can be evaluated more accurately.

[0044] For example, (i) the greater the brightness of the surrounding environment, or (ii) the greater the ratio of the area of ​​the area set around the installation location where the illuminance is higher than a predetermined standard, the more likely the user of the information processing terminal is to notice the presence of a third party nearby, while also making it easier for the third party to view information output by the information processing terminal. Therefore, the greater the brightness or the greater the ratio, the greater the degree of safety of the information processing terminal may be evaluated. Furthermore, the greater the brightness or the greater the ratio, the less safe the information processing terminal may be evaluated.

[0045] (Weather conditions) Examples of meteorological conditions include weather, temperature, humidity, wind speed, and air pressure. Examples of weather include the type of weather, amount of solar radiation, amount of rainfall, amount of snowfall, and thickness of fog. For example, the greater the amount of rainfall, amount of snowfall, thickness of fog, etc., the smaller the visibility. The smaller the visibility, the higher the degree of safety of the information processing terminal can be evaluated (i.e., the lower the possibility that an output image will be viewed by a third party).

[0046] Furthermore, for example, when the temperature is lower than a predetermined first threshold (e.g., when it is very cold) or when the temperature is lower than a predetermined second threshold (e.g., when it is very hot), the possibility of a third party being present around the installation location is low. Therefore, when the temperature is lower than the predetermined first threshold or the predetermined second threshold, the information processing terminal can be evaluated as having a high level of security (i.e., there is a low possibility that the output image will be viewed by a third party).

[0047] For example, when the wind speed is high, the possibility of spying using a drone decreases. Also, when the air pressure is low, the altitude may be high. The higher the altitude, the less likely it is that spying from the surroundings will occur. Therefore, the degree of safety of an information processing device can be evaluated based on wind speed, air pressure, altitude, etc.

[0048] (Air quality) Examples of air conditions include (i) the degree of air clarity, visibility, or the concentration of suspended or airborne substances, and (ii) the degree of odor or the concentration of odorous substances. For example, the lower the degree of air clarity or the higher the concentration of suspended or airborne substances, the lower the visibility. The lower the visibility, the higher the degree of safety of the information processing terminal (i.e., the lower the possibility that an output image will be viewed by a third party). Suspended or airborne substances may be solid, liquid, or gaseous. Examples of suspended or airborne substances include water droplets (e.g., fog or rain), chemical substances, allergens, dust, PM2.5, etc. An example of an allergen is pollen.

[0049] Furthermore, for example, the stronger the odor intensity in the surrounding environment, the less likely it is that a third party is present near the installation location. Therefore, the greater the intensity of the odor or the concentration of the odorous substance, the greater the degree of safety of the information processing terminal (i.e., the less likely it is that the output image will be viewed by a third party). The odorous substance may be solid, liquid, or gas.

[0050] (Personnel flow situation) Examples of the situation of people flow or population include the volume or flow of pedestrians and / or vehicles. The situation of people flow or population may be the volume or flow of pedestrians. The fewer pedestrians and / or vehicles there are, the higher the degree of safety of the information processing terminal can be evaluated (i.e., the lower the possibility that the output image will be viewed by a third party).

[0051] (Safety assessment based on object detection results) In yet another embodiment, it is conceivable to evaluate the safety of an information processing terminal based on the detection results of objects or events at the installation location of the information processing terminal and its surroundings. The safety of the information processing terminal may be evaluated based on the type and / or attributes of the detected objects or events. The method for detecting objects or events is not particularly limited, but examples include image analysis and audio analysis. This method differs from the methods described in connection with the other embodiments described above in that the safety of the information processing terminal is evaluated based on the characteristics of objects located at the installation location or its surroundings, rather than the characteristics of the installation location or its surroundings.

[0052] (An example of image analysis) For example, by analyzing an image obtained by capturing an image of the installation location or its surroundings (sometimes referred to as the installation location, etc.) of an information processing terminal, it is possible to detect an object located at the installation location, etc. or an event occurring at the installation location, etc., and to determine the type and / or attributes of the detected object or event. If a color value and / or a feature unique to a specific object is detected in the image, the specific object can be detected and determined. Examples of specific objects include a human face, a TV, a picture frame, an AR marker, a window, a door, an imaging device, a drone, etc. An object may be solid, liquid, gas, or a combination thereof. In this specification, even if it does not have a fixed shape, such as the sky, darkness, the sea, waves, or flames, it can be treated as an object as long as it has a color or refractive index.

[0053] If the image is captured by a far-infrared camera, it may be possible to detect, for example, a human being present behind a cloth, a wall, etc. Therefore, if a predetermined first type of object such as a human being, an imaging device, or a drone is detected, it may be evaluated that the security of the information processing terminal is low (i.e., there is a high possibility that the output image may be viewed by a third party).

[0054] On the other hand, even if a human face is detected, if the detected human face is the user's face or a pre-registered human face, the risk of information leakage is relatively small. A human face may be an example of a human feature. Therefore, a process may be executed to compare the features of the detected object with the features of objects whose safety has been confirmed in advance. For example, the features of objects whose safety has been confirmed may be recorded in a whitelist created in advance. If the features of the detected object do not match the features of objects whose safety has been confirmed in advance, the information processing terminal may be evaluated as having a low level of safety (i.e., there is a high possibility that the output image may be viewed by a third party).

[0055] Furthermore, even if a human, an imaging device, a drone, or the like is detected, the risk of information leakage is relatively small if the detected position is inside the TV screen or inside the frame. Therefore, when a predetermined second type of object is detected in an image, processing may be performed to reduce the influence of other objects located in the area in which the object in the image is detected on the safety evaluation of the information processing terminal. For example, processing may be performed to adjust weights and / or thresholds related to the safety evaluation inside and outside the above-mentioned area.

[0056] Furthermore, when an encoded code such as an AR marker, a QR code (registered trademark), or a barcode is detected in the image, a process for evaluating the safety of the information processing terminal may be executed based on information indicated by the encoded code. Examples of the information indicated by the encoded code include that the information processing terminal may be evaluated as safe when the encoded code is detected, that the information processing terminal may be evaluated as unsafe when the encoded code is detected, that the risk of information leakage at the location where the encoded code is located is low, or that the risk of information leakage at the location where the encoded code is located is high.

[0057] (An example of voice analysis) For example, by analyzing the sound recorded at the installation location, etc. of the information processing terminal, it is possible to detect an object placed at the installation location, etc. or an event that occurred at the installation location, etc., and to determine the type and / or attribute of the detected object or event. For example, if a dog barking, human footsteps, etc. are detected and determined from the above sound, there is a high possibility that a human is present near the installation location, etc.

[0058] Therefore, when a specific type of sound is detected, the information processing terminal may be evaluated as having a low level of security (i.e., there is a high possibility that the output image may be viewed by a third party). On the other hand, even when human footsteps are detected, the risk of information leakage differs greatly between when it is determined that the human is approaching the installation location or the like and when it is determined that the human is moving away from the installation location or the like. Therefore, a process for evaluating the security of the information processing terminal may be executed based on the position of the sound source and / or a change in the position. For example, a process for adjusting a weight and / or a threshold value related to the security evaluation may be executed based on the position of the sound source and / or a change in the position.

[0059] (Other evaluation methods) The method for evaluating the security of an information processing terminal is not limited to the above-described method. Furthermore, the security of an information processing terminal may be evaluated by a combination of two or more of the above-described methods. The information leakage prevention system 100 may determine a final evaluation of the security of the information processing terminal by comprehensively assessing the evaluations derived by each combination of two or more of the above-described methods. For example, the information leakage prevention system 100 determines a final evaluation of the security of the information processing terminal based on weights assigned to each of the multiple methods and the evaluations derived by each method. The information leakage prevention system 100 may also determine a final evaluation of the security of the information processing terminal by combining one or more conditional branches with multiple methods. This allows for an accurate evaluation of the security of the information processing terminal. For example, even if a third party attempts to impersonate the user by presenting a photo of the user in front of camera 154, information leakage prevention system 100 can prevent information leakage due to the above-mentioned impersonation act by making a comprehensive judgment on the safety of the information processing terminal, taking into consideration the results of analysis such as infrared image analysis, analysis of object movement or light fluctuation.

[0060] (Information Leakage Prevention System 100) According to this embodiment, details of the information leakage prevention system 100 will be described using Figures 1 to 12. Furthermore, details of a method for evaluating the degree to which object movement or light fluctuation is detected will be described using Figures 13 to 20. Those skilled in the art who have read the description of this specification will understand that technical matters described in one embodiment can also be applied to other embodiments to the extent that they are not technically inconsistent.

[0061] (Outline of Information Leakage Prevention System 100) FIG. 1 schematically illustrates an example of the system configuration of an information leakage prevention system 100. In this embodiment, the information leakage prevention system 100 evaluates the safety of a user terminal 120. More specifically, the information leakage prevention system 100 evaluates the safety of a third party viewing an output image of the user terminal 120. The safety of the user terminal 120 may represent the safety of the user terminal 120 installed in a specific position and direction, may represent the safety of the installation position of the user terminal 120, or may represent the safety of the installation position and installation direction of the user terminal 120. The installation position of the user terminal 120 may be an example of the installation position of the display area 142. The installation direction of the user terminal 120 may be an example of the installation direction of the display area 142.

[0062] For example, the information leakage prevention system 100 evaluates the security of the user terminal 120 based on at least one of the following: (i) an index indicating the degree to which object movement or light fluctuation is detected around the installation location of the user terminal 120; (ii) an index indicating the placement status of obstructions present around the installation location of the user terminal 120; and (iii) an index indicating the status of the surrounding environment around the installation location of the user terminal 120. Instead of or in addition to the at least one index described above, the information leakage prevention system 100 may evaluate the security of the user terminal 120 based on the characteristics or type of objects detected around the installation location of the user terminal 120. An example of the characteristics of the objects is a color value unique to the object. For example, a black object and a night sky have different color values. This allows a painting of a night sky to be distinguished from the night sky seen through a window.

[0063] In this embodiment, the information leakage prevention system 100 supports the installation of the user terminal 120. In this embodiment, the information leakage prevention system 100 monitors the safety of the user terminal 120 while the user terminal 120 is in use.

[0064] In this embodiment, the details of the information leakage prevention system 100 will be described using as an example a case where the user terminal 120 executes the various information processes described above. However, the information leakage prevention system 100 is not limited to this embodiment. In other embodiments, the user terminal 120 may be configured to be able to communicate with another computer and may execute the various information processes described above in cooperation with the other computer.

[0065] (Outline of each part of the information leakage prevention system 100) In this embodiment, the information leakage prevention system 100 includes a user terminal 120. In this embodiment, the user terminal 120 has, for example, a housing 130, a display device 140, an environmental sensor 150, an input device 160, and a control device 170. In this embodiment, the display device 140 includes, for example, a display area 142. In this embodiment, the environmental sensor 150 includes a peripheral data generating device 152, a camera 154, and an illuminance sensor 156. In this embodiment, the installation position of the display area 142 may be referred to as the installation position of the user terminal 120. Furthermore, the installation direction of the display area 142 may be referred to as the installation direction of the user terminal 120.

[0066] In this embodiment, the user terminal 120 outputs information to the user. The user terminal 120 may output information as an image or as sound. The user terminal 120 may also accept information input from the user. The user terminal 120 executes various types of information processing.

[0067] In this embodiment, the user terminal 120 outputs an image. The image output by the user terminal 120 is displayed in the display area 142. A person (sometimes referred to as a viewer) present within the viewing angle range of the display area 142 can view the image displayed in the display area 142. If an imaging device is disposed within the viewing angle range of the display area 142, the imaging device can capture the image displayed in the display area 142. A person (sometimes referred to as a viewer) who acquires image data captured by the imaging device can view the image displayed in the display area 142.

[0068] In one embodiment, the above image is output in response to a user's operation of the user terminal 120. For example, the user inputs instructions to the user terminal 120 via the input device 160 for the BIOS, operating system (sometimes referred to as an OS), or application program (sometimes referred to as a program as a general term for all of these) running on the user terminal 120. The user terminal 120 executes information processing in accordance with the instructions, and displays an image showing the results of the information processing in the display area 142.

[0069] In another embodiment, the image is output when the security level of the user terminal 120 does not meet a predetermined standard. In such a case, for example, a warning message, a warning screen, a lock screen, or a screen with some information hidden is displayed in the display area 142.

[0070] Examples of the user terminal 120 include a personal computer and a mobile terminal. Examples of the mobile terminal include a mobile phone, a smartphone, a PDA (registered trademark), a tablet, a notebook computer or laptop computer, and a wearable computer.

[0071] In this embodiment, the housing 130 supports the display device 140, the peripheral data generating device 152, the input device 160, and the control device 170. Each of the display device 140, the peripheral data generating device 152, the input device 160, and the control device 170 may be at least partially housed inside the housing 130 or may be attached to the outside of the housing 130.

[0072] In this embodiment, the display device 140 outputs an image. The display device 140 may output an image or stop outputting an image according to an instruction from the control device 170. The display device 140 may adjust the brightness or contrast of the image according to an instruction from the control device 170. Examples of the display device 140 include a display and a projector.

[0073] In this embodiment, an image (sometimes referred to as an output image) output by the display device 140 is displayed in the display area 142. In this embodiment, the display area 142 is disposed in the housing 130. This causes the direction of the optical axis of the peripheral data generating device 152 and the direction of the normal vector of the display area 142 to substantially coincide.

[0074] In this embodiment, the environmental sensor 150 acquires various types of information related to the environment surrounding the display area 142. The environmental sensor 150 may include one or more sensors for acquiring the various types of information. Examples of the sensors include a sensor for acquiring various images, a sensor for acquiring sound, a sensor for detecting brightness, a sensor for detecting temperature, and a sensor for detecting distance or depth. The environmental sensor 150 may output the information acquired by the one or more sensors to the control device 170.

[0075] In this embodiment, the peripheral data generating device 152 generates information (sometimes referred to as peripheral data) about one or more objects (sometimes referred to as obstructions) arranged around the display area 142. The peripheral data generating device 152 outputs the generated peripheral data to the control device 170. Examples of the periphery of the display area 142 include positions within the range of the viewing angle of the display area 142, and positions whose distance from the display area 142 is within a predetermined numerical range.

[0076] For example, the surrounding data generating device 152 generates information indicating the relative position of one or more obstructing objects arranged around the display area 142 and the display area 142. Examples of the information indicating the relative position of each obstructing object and the display area 142 include (i) the distance between a representative point of the display area 142 and a representative point of each obstructing object, (ii) the direction from the representative point of the display area 142 to the representative point of each obstructing object, and (iii) the magnitude and direction of each of one or more vectors directed from the representative point of the display area 142 to the representative point of each obstructing object. The surrounding data generating device 152 may generate information indicating at least one of the position, shape, and size of the obstructing object.

[0077] The shielding object may be an object that satisfies certain dimensional conditions. An example of the dimensional conditions is that at least one of the dimensions of width, height, and depth must be greater than a predetermined value. The shielding object may be an object large enough to block the viewer's line of sight when placed between the viewer and the display area 142. The shielding object is preferably an object that suppresses light transmission. However, the degree to which the shielding object suppresses light transmission is not particularly limited. The shielding object may include a transparent object such as glass and / or an object with through holes such as a lattice. The shielding object may be a stationary object or an object that is movable within a predetermined range. Examples of the shielding object include furniture, decorative objects, electrical appliances, floors, ceilings, walls, pillars, beams, doors, and windows.

[0078] In one embodiment, the peripheral data generating device 152 generates, as the peripheral data, image data (sometimes referred to as image data) of one or more objects located around the display area 142 as subjects. Examples of images of obstructing objects as subjects include two-dimensional images and stereo images. In another embodiment, the peripheral data generating device 152 generates, as the peripheral data, three-dimensional point cloud data or distance data (sometimes referred to as three-dimensional data) of one or more obstructing objects located around the display area 142.

[0079] The measurement accuracy of the three-dimensional data is not particularly limited. Furthermore, the three-dimensional data may lack information about a part of the obstruction, or the information about the part of the obstruction may be inferred from other information. The distance data may include information indicating a relative distance, information indicating a distance ranking index, etc.

[0080] When the surrounding data generation device 152 generates three-dimensional data, a three-dimensional virtual space can be constructed around the display area 142. This simplifies the calculation procedure for the degree to which a third party's line of sight is blocked by an obstruction. As a result, the amount of calculation required to derive the index indicating the positioning status of the obstruction can be reduced.

[0081] Note that the peripheral data generating device 152 does not need to generate three-dimensional data. Even in this case, the degree to which the line of sight of a third party is blocked by an obstruction can be derived based on image data generated by the peripheral data generating device 152. In particular, when the position coordinates of the peripheral data generating device 152 and the display area 142 are substantially the same and the direction of the optical axis of the peripheral data generating device 152 and the direction of the normal vector of the display area 142 are substantially parallel, the degree to which the line of sight of a third party is blocked by an obstruction can be derived with high accuracy.

[0082] Examples of the peripheral data generating device 152 include (i) a sensor that captures an image, and (ii) a sensor that measures the distance to an object using sound waves, ultrasonic waves, or electromagnetic waves. More specifically, examples of the peripheral data generating device 152 include an image sensor, a camera, a stereo camera, a LiDAR, and a monocular camera depth distance measurement device (sometimes referred to as a monocular depth estimation device). An optical component such as a fisheye lens or an ultra-wide-angle lens may be attached to the camera. This increases the angle of view of the camera.

[0083] The viewing angle (sometimes referred to as the angle of view) of the peripheral data generating device 152 may be larger than the viewing angle of the display area 142. In this case, the information leakage prevention system 100 can evaluate the safety of the display area 142 for a plurality of different installation directions (sometimes referred to as the installation direction) of the display area 142, for example, based on a single image data or a single three-dimensional data generated by the peripheral data generating device 152. This allows the safety of the display area 142 to be evaluated with high accuracy. Examples of the safety of the display area 142 include the safety of the display area 142 installed at a specific position and direction, the safety of the installation position of the display area 142, and the safety of the installation position and installation direction of the display area 142.

[0084] The installation orientation of display area 142 may be an example of the orientation of the information processing terminal described above. The installation orientation of display area 142 may be represented by the direction of a normal vector of display area 142, or may be represented by rotation of display area 142 around at least one of the front-to-back axis, the up-down axis, and the left-to-right axis.

[0085] When the display area 142 is housed in the housing 130, the installation position of the display area 142 substantially coincides with the installation position of the user terminal 120. In this case, the installation direction of the display area 142 substantially coincides with the installation direction of the user terminal 120.

[0086] In this embodiment, camera 154 generates video data (sometimes referred to as video data) including a plurality of still images (sometimes referred to as frames). For example, camera 154 captures an image of the periphery of the planned installation location or installation location of display area 142 (as described above, sometimes simply referred to as the installation location) and generates the video data. Camera 154 outputs the generated video data to control device 170. Camera 154 may be an image sensor for capturing two-dimensional images, or may be an infrared camera for capturing infrared images (sometimes referred to as thermal images).

[0087] In this embodiment, the camera 154 is disposed in the housing 130. In this embodiment, the display area 142 is also disposed in the housing 130, so the position where the camera 154 captures the video (sometimes referred to as the imaging position) and the installation position of the display area 142 or the user terminal 120 are approximately the same. Furthermore, the direction of the optical axis of the camera 154 and the direction of the normal vector of the display area 142 are approximately the same. Therefore, the direction where the camera 154 captures the video (sometimes referred to as the imaging direction) and the installation direction of the display area 142 or the user terminal 120 are approximately the same.

[0088] If the peripheral data generating device 152 includes an imaging element, the user terminal 120 does not need to include the camera 154. In this case, the imaging element may have the same function as the camera 154 according to this embodiment.

[0089] In this embodiment, the illuminance sensor 156 measures the illuminance around the installation position of the display area 142. The illuminance sensor 156 outputs information indicating the measurement result to the control device 170.

[0090] In this embodiment, the input device 160 accepts input from the user of the user terminal 120. The input device 160 outputs information input by the user to the control device 170. Examples of the input device 160 include a keyboard, a pointing device, a touch panel, and a microphone.

[0091] In this embodiment, the control device 170 controls the user terminal 120. The control device 170 executes various information processing in the user terminal 120. For example, the control device 170 evaluates the safety of the installation position and / or installation direction of the user terminal 120 based on data acquired by the peripheral data generation device 152. The control device 170 may input the data acquired by the peripheral data generation device 152 into multiple application programs. For example, the control device 170 includes a virtual camera driver for inputting the data acquired by the peripheral data generation device 152 into multiple application programs. Details of the control device 170 will be described later.

[0092] The information processing method executed by the control device 170 includes, for example, peripheral state acquisition for acquiring peripheral state information indicating the state of the periphery of a display area where an output image output by the image output device is displayed. The information processing method includes, for example, a viewability derivation step for deriving the degree of viewability of the output image by a third party other than the user of the image output device based on the state of the periphery of the display area indicated by the peripheral state information acquired in the peripheral state acquisition step.

[0093] The information processing method executed by the control device 170 may be an image processing method. The image processing method may, for example, include a moving image acquisition step of acquiring data of a moving image including a plurality of frames. The image processing method may, for example, include a fluctuation index derivation step of (a) dividing each frame of the moving image into a plurality of predetermined regions, and (b) deriving, for each of the plurality of regions, a value of a fluctuation index that is an index indicating fluctuations in color and / or brightness of each region between the plurality of frames, based on pixel values ​​of pixels included in each region. In the image processing method, each of the plurality of regions may, for example, include one or more pixels.

[0094] The information leakage prevention system 100 may be an example of an information processing device. The user terminal 120 may be an example of an information processing device or an image output device. The display device 140 may be an example of an image output device. The camera 154 may be an example of an imaging device. The imaging device may be an example of a surrounding state acquisition unit. The imaging device may be an example of a device that generates surrounding state information and outputs the surrounding state information to the surrounding state acquisition unit. The illuminance sensor 156 may be an example of an environment information acquisition unit. The illuminance sensor 156 may be an example of a device that generates environment information and outputs the environment information to the environment information acquisition unit. The control device 170 may be an example of an information processing device, an image output device, an imaging device, an environment information acquisition unit, or an output control unit.

[0095] The image output by the display device 140 may be an example of an output image. The degree of safety of the user terminal 120 may be an example of the degree of viewability of the output image. The degree of safety of the display area 142 may be an example of the degree of viewability of the output image. Information about obstructions generated by the peripheral data generating device 152 may be an example of peripheral data. Image data may be an example of peripheral data. Three-dimensional data may be an example of peripheral data.

[0096] (An example of another embodiment) In the present embodiment, an example of the information leakage prevention system 100 has been described using the case where the display area 142 is part of the display device 140 as an example. However, the information leakage prevention system 100 is not limited to this embodiment. In other embodiments, the display area 142 is disposed at a position physically separated from the display device 140. For example, when the display device 140 is a projector, a screen is used as the display area 142.

[0097] FIG. 2 schematically shows the viewing angles of the display area 142 and the peripheral data generating device 152. In the embodiment described with reference to FIG. 2, the relationship between the viewing angles of the display area 142 and the peripheral data generating device 152 will be described using as an example a case where a representative point 240 of the display area 142 and a representative point 250 of the peripheral data generating device 152 approximately coincide with each other. As described with reference to FIG. 1, in this embodiment, the display area 142 is part of the display device 140, and the display device 140 and the peripheral data generating device 152 are housed in the same housing 130. Therefore, as shown in FIG. 2, the direction of the normal vector 242 of the display area 142 and the direction of the optical axis 252 of the peripheral data generating device 152 approximately coincide with each other.

[0098] In this embodiment, the viewing angle (viewing angle θ D ) is the angle θ between the normal vector 242 and the vector 244. DR and the angle θ between the normal vector 242 and the vector 246 DL Similarly, in this embodiment, the viewing angle (viewing angle θ C ) is the angle θ between the optical axis 252 and the vector 254. CR and the angle θ between the optical axis 252 and the vector 256 CL It is derived as the sum of the absolute values ​​of

[0099] (An example of another embodiment) In this embodiment, an example of the information leakage prevention system 100 has been described using the case where the viewing angle of the display area 142 is smaller than the viewing angle of the peripheral data generating device 152 as an example. For example, if a film is attached to the surface of the display area 142 to prevent third parties from peeking, the viewing angle of the display area 142 may be smaller than the viewing angle of the peripheral data generating device 152. However, the information leakage prevention system 100 is not limited to this embodiment. In other embodiments, the viewing angle of the display area 142 may be larger than the viewing angle of the peripheral data generating device 152.

[0100] 3 is a schematic diagram showing an example of the arrangement of the user terminal 120. In this embodiment, an example of the arrangement of the user terminal 120 will be described using as an example a case where the user terminal 120 is arranged at position A, position B, or position C inside the work room 300.

[0101] In this embodiment, work room 300 includes wall 312, wall 314, wall 316, wall 318, and floor 320. In this embodiment, a shield 330 is arranged inside work room 300. Markers 342, 344, and 346 are arranged on the surface of wall 316. A glass window 322 is arranged in wall 318.

[0102] (Safety at Position A) 3, when user 22 of user terminal 120 places user terminal 120 at position A so that normal vector 242 of display area 142 faces downward in the figure, the space in which the image displayed in display area 142 can be viewed is the space surrounded by vector 244, vector 246, wall 314, and floor 320. In this case, third parties 32 and 34 different from user 22 cannot view the image displayed in display area 142.

[0103] Therefore, even if the third parties 32 and 34 do not have the proper authority to view the information included in the output image of the user terminal 120, the possibility of the information being leaked is small, and the safety of the user terminal 120 at position A can be evaluated as high. In this way, the control device 170 can evaluate the safety of the user terminal 120, for example, by evaluating the placement of obstructions around the installation position of the user terminal 120.

[0104] (Safety at Position B) When user 22 of user terminal 120 places user terminal 120 at position B so that normal vector 242 of display area 142 faces right in the figure, the space in which the image output in display area 142 can be viewed is the space surrounded by vector 244, vector 246, wall 316, obstruction 330, and floor 320. In this case, third party 32 can view the image displayed in display area 142. On the other hand, third party 34 cannot view the image displayed in display area 142.

[0105] In this embodiment, the distance Lb between position B and wall 318 is greater than the distance La between position A and wall 314. Therefore, when user terminal 120 is installed at position B, the size of the space in which the above images can be viewed becomes larger than when user terminal 120 is installed at position A, and the possibility that a third party 32 will enter the space also increases (i.e., safety decreases).

[0106] Therefore, it is preferable that the control device 170 evaluates the safety of the user terminal 120 at position B, taking into consideration the presence or absence of a third party within the space, the intrusion of a third party into the space, etc. As described above, the control device 170 can evaluate, for example, the degree to which object movement or light fluctuation is detected around the installation position of the user terminal 120. This allows the control device 170 to accurately evaluate the safety of the user terminal 120.

[0107] Additionally, according to this embodiment, a glass window 322 is provided on the wall 318 located behind the user 22. Therefore, it is preferable that the control device 170 evaluates the safety of the user terminal 120 at position B, taking into consideration the possibility that a third party outside the glass window 322 may view the output image.

[0108] In particular, when the outside of the glass window 322 is darker than the inside of the workroom 300, it is more difficult to detect a third party outside the glass window 322 based on the information acquired by the peripheral data generating device 152 or the camera 154, compared to when the outside of the glass window 322 is brighter than the inside of the workroom 300. Therefore, when the glass window 322 is present in the above-mentioned space and / or when the outside of the glass window 322 is darker than the inside of the workroom 300, the safety of the user terminal 120 at position B is reduced.

[0109] As described above, the control device 170 can, for example, evaluate the brightness around the installation location of the user terminal 120. The control device 170 may evaluate the brightness outside the glass window 322 by evaluating the brightness of the glass window 322. This allows the control device 170 to accurately evaluate the safety of the user terminal 120.

[0110] (Safety at Position C) When a user 22 of the user terminal 120 places the user terminal 120 at position C so that the normal vector 242 of the display area 142 faces right in the figure, the space in which the image output to the display area 142 can be viewed is the space surrounded by vectors 244 and 246, wall 318, and floor 320. Third parties 32 and 34 cannot view the image displayed in the display area 142.

[0111] According to this embodiment, markers 342, 344, and 346 are arranged on a wall 316 located behind the user 22. In this embodiment, the markers 342 and 344 are arranged inside the above-mentioned space. Meanwhile, the marker 346 is arranged outside the above-mentioned space. While the user 22 is using the user terminal 120, the control device 170 analyzes data acquired by the environmental sensor 150 to detect the markers 342 and 344. If the control device 170 is no longer able to detect at least one of the markers 342 and 344, the control device 170 determines that the safety of the user terminal 120 has decreased due to some cause, for example. Examples of such causes include a third party entering between the user terminal 120 and the wall 316, or a change in at least one of the installation position and installation orientation of the user terminal 120.

[0112] In this way, the control device 170 can use various markers placed around the user terminal 120 to accurately evaluate the safety of the user terminal 120. This can highly effectively prevent information leakage.

[0113] Various types of information may be assigned to the markers 342, 344, and 346. The control device 170 can evaluate the safety of the user terminal 120 using the information assigned to each marker. This further improves the accuracy of safety determination. Therefore, the safety of the user terminal 120 at position C is further improved. Details of each marker will be described later.

[0114] (Outline of information processing in the control device 170) As shown in FIG. 3, the degree of possibility (sometimes referred to as "viewability") that an output image will be viewed by a third party 32 or 34 other than the user 22 of the user terminal 120 varies greatly depending on the installation location and installation orientation of the user terminal 120. However, it is difficult for the user 22 to objectively grasp the degree of viewability. Therefore, even if the user 22 installs the user terminal 120 in a location that the user 22 perceives as safe, the degree of viewability may be objectively high depending on the placement of obstructions around the installation location. Note that the higher the degree of viewability, the greater the risk of information leakage due to peeking by a third party, and the lower the safety.

[0115] Therefore, in this embodiment, the control device 170 derives the degree of viewability based on various types of peripheral data generated by the peripheral data generating device 152. This objectively ensures the safety of the installation position and installation direction of the user terminal 120. The procedure for deriving the degree of viewability will be described in detail later.

[0116] (Marker Overview) Each of the markers 342, 344, and 346 may include various types of information. Each of the markers 342, 344, and 346 may include identification information that allows the control device 170 to identify each marker. Each of the markers 342, 344, and 346 may be an encoded code such as an AR (Augmented Reality) marker, a QR code (registered trademark), or a barcode.

[0117] When markers 342, 344, and 346 are used as AR markers, each marker includes, for example, information about a three-dimensional model of the object corresponding to the marker, which allows control device 170 to place a three-dimensional model corresponding to the object indicated by each marker at the position of each marker in the three-dimensional virtual space corresponding to work room 300.

[0118] Marker 342 includes, for example, information for generating a three-dimensional model of obstructing object 330 or a point cloud simulating obstructing object 330 to the right of the detection position of marker 342. This allows control device 170 to construct a three-dimensional model corresponding to obstructing object 330 in the three-dimensional virtual space. Marker 342 may also include information for generating a three-dimensional model or point cloud having a predetermined width and height to the left of the detection position of marker 342. This allows control device 170 to construct a three-dimensional model corresponding to a portion of wall 316 in the three-dimensional virtual space.

[0119] Marker 344 includes, for example, information for generating a three-dimensional model or point cloud having a predetermined width and height to the right of the detection position of marker 344, and information for generating a three-dimensional model or point cloud having a predetermined width and height to the left of the detection position of marker 344. This allows control device 170 to construct a three-dimensional model of a portion of wall 316 in the three-dimensional virtual space.

[0120] The marker 346 includes, for example, information for generating a three-dimensional model or point cloud having a predetermined width and height to the right of the detected position of the marker 346. This allows the control device 170 to construct a three-dimensional model corresponding to a portion of the wall 316 in the three-dimensional virtual space.

[0121] Examples of AR markers include ArUco and Chameleon Code. However, AR markers are not limited to these. According to other embodiments, for example, QR codes (registered trademarks) and barcodes are used as AR markers. In still other embodiments, objects in an image may be used as AR markers. For example, by using technology (e.g., various deep learning technologies) that detects known objects from among the objects in an image through image analysis, the known objects can be used as AR markers. According to the above technology, first, a classifier is trained in advance to learn the characteristics of known objects such as road signs, vending machines, microwave ovens, books, and chairs. Next, the trained classifier is used to infer the orientation of the known objects among the objects in the image, the distance between the object and the imaging device, and the like. This allows the three-dimensional coordinates of the object to be derived. As described below, if the shape and size of the object are known, a three-dimensional model of the object can be constructed in a three-dimensional virtual space.

[0122] For example, when marker 342, marker 344, or marker 346 is a QR code (registered trademark) or a barcode, or when a known object is used as an AR marker, control device 170 may refer to a database that stores the identification information of the marker in association with the content of information processing to be performed when the marker is detected, and perform information processing when the marker is detected. As described above, the information processing may be information processing about a three-dimensional virtual space in which workroom 300 or a part thereof is electronically reproduced.

[0123] The information processing related to the three-dimensional virtual space includes, for example, a step of identifying a position in the three-dimensional virtual space that corresponds to a detected position of a marker in the real world, and a step of generating a point cloud or a three-dimensional model that is pre-associated with the marker at the identified position in the three-dimensional virtual space. In this case, the database stores, for example, identification information of the marker and information indicating the shape and size of the point cloud or the three-dimensional model in association with each other.

[0124] By attaching a marker to the subject of the image, the information processing load on the control device 170 can be significantly reduced. For example, if (i) the dimensions of the marker and (ii) the relative positional relationship between the display area 142 and the peripheral data generating device 152 are known, the control device 170 can determine the relative positional relationship between the display area 142 and the object to which the marker is attached, based on a two-dimensional image in which the marker is captured. The control device 170 may also determine the direction of the normal vector of the marker (sometimes referred to as the attitude of the marker) based on the two-dimensional image in which the marker is captured.

[0125] For example, if the dimensions of a marker are known, the relative positional relationship between the peripheral data generating device 152 and the marker can be determined based on a two-dimensional image that the marker is captured in. Furthermore, if the relative positional relationship between the display area 142 and the peripheral data generating device 152 is known, the relative positional relationship between the display area 142 and the marker can be determined based on the relative positional relationship between the peripheral data generating device 152 and the marker.

[0126] The procedure for determining the relative positional relationship between the peripheral data generating device 152 and the marker based on a two-dimensional image in which the marker appears includes, for example, a procedure for analyzing the two-dimensional image and determining the camera coordinates of the marker. The process for determining the camera coordinates of the marker can be realized, for example, using a publicly known library. An example of the publicly known library is the OpenCV library. The OpenCV library is configured to allow one or more camera-related items to be set. Examples of the above items include resolution, FPS, focal length, vertical distortion, and horizontal distortion.

[0127] The process for determining the camera coordinates of the markers includes, for example, a camera calibration process and a 3D reconstruction process. The 3D reconstruction process is realized by calculating a coordinate transformation matrix using, for example, the camera coordinate system, the image coordinate system, and the marker coordinate system. The coordinate transformation matrix may include a rotation matrix and a translation matrix.

[0128] If the error in the coordinate system calculation process is large, the control device 170 may calculate the coordinates of the marker by averaging the coordinate information of the marker in multiple frames captured by the peripheral data generation device 152. The number of frames may be four or more, or eight or more. This may reduce the error. The average may be calculated before the instance is identified. If the variation in the marker coordinates in each of the multiple frames exceeds a predetermined level, the control device 170 may determine that the peripheral data generation device 152 has moved and interrupt the averaging process.

[0129] As described above, according to this embodiment, the control device 170 can generate a point cloud or a three-dimensional model specified by the marker at the position where the marker is detected. This, for example, eliminates the need for processing to identify the outline of the object, or allows the object to be replaced with a point cloud or a model of a simple shape. As a result, the amount of calculation required for processing to determine the relative positional relationship between the display area 142 and the object to which the marker is attached is significantly reduced. In particular, when a set of markers (for example, a pair of markers, although the number of markers per set is not limited) is attached to the edge of an obstructing object, the amount of calculation required for processing to identify the location of the obstructing object is significantly reduced.

[0130] As described above, each of markers 342, 344, and 346 includes, for example, identification information for identifying each of one or more markers and / or a pattern representing information about a point cloud or a three-dimensional model generated around the position of each marker. The information about the point cloud may include information indicating dimensions in a world coordinate system of the generated point cloud or three-dimensional model. The information about dimensions may include information indicating a shape and information indicating a size.

[0131] Each of markers 342, 344, and 346 may include information indicating the dimensions of the marker, information about the object to which the marker is attached, and information about the degree of visibility of the location to which the marker is attached. The information about the object to which the marker is attached may include information indicating that the object is light-transmitting. Examples of light-transmitting objects include glass, lattices, and perforated metal.

[0132] According to this embodiment, an object without an AR marker in the real space is not reflected in the three-dimensional space. For example, if six AR markers representing a 1-meter-wide, 3-meter-high wall are arranged horizontally every 1 meter on a 6-meter-wide, 3-meter-high wall in the real space, a 6-meter-wide, 3-meter-high wall will appear in the three-dimensional space. On the other hand, if three AR markers representing a 1-meter-wide, 3-meter-high wall are arranged every 2 meters on the wall in the real space, three 1-meter-wide, 3-meter-high walls will appear in the three-dimensional virtual space, with a 1-meter gap between each wall. In this way, when calculating the degree of viewability using a three-dimensional virtual space, the arrangement of the AR markers in the real space may affect the calculation result of the degree of viewability.

[0133] Wall 312 may be an example of one or more objects or obstructions. Wall 314 may be an example of one or more objects or obstructions. Wall 316 may be an example of one or more objects or obstructions. Wall 318 may be an example of one or more objects or obstructions. Floor 320 may be an example of one or more objects or obstructions. Glass window 322 may be an example of one or more objects or obstructions. Obstruction 330 may be an example of one or more objects or obstructions.

[0134] Marker 342 may be an example of one or more markers. Marker 344 may be an example of one or more markers. Marker 346 may be an example of one or more markers. Wall 316 may be an example of one or more objects or subjects with one or more markers attached.

[0135] An example of information processing in the control device 170 will be described using Figures 4 and 5. Figure 4 shows an example of processing by the control device 170 to evaluate the safety of the user terminal 120. Figure 5 shows an example of processing in step 416.

[0136] As shown in FIG. 4, according to this embodiment, first, in step 412 (step may be abbreviated as S), the control device 170 receives an instruction from the user 22 to the user terminal 120 to start the user terminal 120 or a program running on the user terminal 120 (this instruction may also be referred to as a start-up instruction). The program may be a BIOS, an OS, or an application program. The application program may be a pre-designated application program, or may be an application program for evaluating the security of the user terminal 120.

[0137] When the control device 170 receives a startup instruction from the user 22, the control device 170 starts executing the startup process corresponding to the instruction from the user 22 and also starts executing an application program (sometimes referred to as a safety evaluation program) for evaluating the safety of the user terminal 120.

[0138] When the safety assessment program is executed, the control device 170 acquires various pieces of information about the surroundings of the user terminal 120 from the environmental sensor 150. Immediately after the above-described startup process is executed, a sufficient amount of data may not be collected to accurately assess the safety of the user terminal 120. In such cases, it may be possible to suspend execution of the program indicated by the startup process until the safety of the user terminal 120 is confirmed. However, the above-described approach degrades the user experience of the user 22. Furthermore, immediately after the user terminal 120 or the program is started, it is highly likely that the user 22 is checking the safety of their surroundings.

[0139] Therefore, according to this embodiment, in S414, the control device 170 determines settings related to the evaluation of the safety of the user terminal 120. For example, the control device 170 has (i) a first setting for evaluating the safety of the user terminal 120 based on a small amount of data, and (ii) a second setting for accurately evaluating the safety of the user terminal 120 based on a sufficient amount of data. Each of the first setting and second setting may include at least one of a setting for deriving values ​​of various indices for evaluating safety (sometimes referred to as an index setting) and a setting for deriving an evaluation value related to safety based on the values ​​of the various indices (sometimes referred to as an evaluation setting).

[0140] As will be described later, the amount of data collected by the environmental sensor 150 increases in the process of deriving the values ​​of various indices. Therefore, in S414, the control device 170 may determine the index setting but not the evaluation setting, thereby reducing the amount of calculation.

[0141] Examples of index settings include a setting indicating the type of index used in processing to evaluate the safety of the user terminal 120, out of one or more indexes related to the safety of the user terminal 120, a setting related to a function or algorithm for deriving the value of the index, a setting related to parameters included in the function or algorithm, etc. Examples of evaluation settings include a setting related to a function or algorithm for deriving an evaluation value based on the value of the index, a setting related to parameters included in the function or algorithm, etc.

[0142] For example, if the time elapsed since the start-up instruction was accepted satisfies a predetermined condition, the control device 170 determines to evaluate the safety of the user terminal 120 based on the first setting. On the other hand, if the time elapsed since the start-up instruction was accepted does not satisfy the predetermined condition, the control device 170 determines to evaluate the safety of the user terminal 120 based on the second setting. An example of the predetermined condition is that the length of the elapsed time is shorter than a predetermined first threshold.

[0143] As a result, even if a sufficient amount of data is not collected to accurately evaluate the safety of the user terminal 120, such as immediately after the above-described startup process is executed, the control device 170 can continue the startup process of the user terminal 120 or the program while ensuring a certain level of safety, thereby improving the user experience of the user 22.

[0144] Next, in S416, the control device 170 executes processing for evaluating the safety of the usage environment of the user terminal 120. The control device 170 derives the values ​​of one or more indicators (sometimes referred to as index values) related to the safety of the user terminal 120 based on various data acquired by the environmental sensor 150 in accordance with the index settings determined in S414. The control device 170 evaluates the safety of the user terminal 120 based on the values ​​of the above indicators in accordance with the evaluation settings determined in S414. Specifically, the control device 170 derives an evaluation value indicating the degree of safety of the user terminal 120 based on the values ​​of the above indicators.

[0145] In one embodiment, the control device 170 derives an evaluation value for the current safety of the user terminal 120 based on the current installation position and current installation orientation of the display area 142. In another embodiment, the control device 170 determines a combination (sometimes referred to as a pattern) of the installation position and installation orientation of the display area 142 such that the evaluation of the safety of the user terminal 120 satisfies predetermined criteria. The installation position of the display area 142 may be represented by a numerical value or a numerical range. The installation orientation of the display area 142 may be represented by a numerical value or a numerical range.

[0146] For example, the control device 170 determines a pattern such that the degree of viewability described above is smaller than a predetermined value. As described above, when the display area 142 is housed in the housing 130, the installation position and installation direction of the display area 142 substantially coincide with the installation position and installation direction of the user terminal 120. The procedure for deriving the index value and the procedure for deriving the evaluation value will be described in detail below.

[0147] Next, in S420, the control device 170 determines whether the safety evaluation value of the user terminal 120 is within a predetermined numerical range (sometimes referred to as an acceptable range). The above numerical range may have either an upper limit or a lower limit, or both an upper limit and a lower limit.

[0148] The control device 170 may adjust the sensitivity or responsiveness of the above-mentioned determination with respect to fluctuations in the evaluation value of the safety of the user terminal 120, taking into consideration the user experience and system stability. As will be described later, if the evaluation value of the safety of the user terminal 120 is outside the allowable range, processing for restricting the use of the user terminal 120 (sometimes referred to as a use restriction processing) may be executed in S432. Furthermore, if the evaluation value of the safety of the user terminal 120 returns to within the allowable range, the above-mentioned use restriction may be lifted. If the sensitivity or responsiveness of the above-mentioned determination is too high, the restriction on the use of the user terminal 120 and the lifting of the restriction will be repeated frequently. As a result, the screen display will become unstable, and the user experience will be degraded.

[0149] According to this embodiment, the sensitivity or responsiveness of the determination is adjusted so that the frequency of switching between the use restriction and the restriction release is less than a predetermined value. This stabilizes the system and improves the user experience. The method for adjusting the sensitivity or responsiveness of the determination is not particularly limited, and any known method may be adopted.

[0150] In one embodiment, the control device 170 may determine that the rating value is outside the allowable range based on the integral of the amount of change in the rating value over a short period of time. The control device 170 may determine whether the rating value is outside the allowable range based on whether the degree corresponding to the integral of the amount of change in the rating value over a short period of time satisfies a predetermined standard. The length of the short period of time may be between 0.1 and 1 second, or may be around 0.5 seconds. This adjusts the balance between the responsiveness of the usage restriction and the stability of the system. As a result, the user's usage experience is improved.

[0151] The control device 170 may limit the use of the display area 142 of the user terminal in proportion to the degree corresponding to the integral of the change in the rating value within the short period of time. If the integral is greater than a predetermined value (sometimes referred to as an upper limit), the control device 170 may change the color, pattern, or brightness of at least a portion of the interior of one or more windows displayed in the display area 142 to reduce the readability of the information displayed in the window. The control device 170 may change the color, pattern, or brightness of the interior of the largest window among the one or more windows. The control device 170 may change the color of the interior of the window to gray, white, or black. The control device 170 may change the color of the interior of the window to gray (sometimes referred to as graying). A window may be a display area allocated to each program running on the user terminal 120.

[0152] If the integral value is smaller than a predetermined value (sometimes referred to as a lower limit value), the control device 170, for example, executes 100% source image output to the display area 142. This achieves maximum legibility. If the integral value is equal to or greater than the lower limit value and equal to or less than the upper limit value, the control device 170 may determine the above-described manner of changing the color, pattern, or brightness according to the integral value. For example, the control device 170 determines the degree of graying (semi-transparency) according to the integral value. This allows the degree of legibility to be limited according to the integral value.

[0153] The control device 170 may temporarily cancel the above-mentioned usage restriction process based on instructions from the user or other authorized person. The control device 170 may also relax the various thresholds described above (for example, the above-mentioned upper limit and / or lower limit) based on instructions from the user or other authorized person. The time during which the usage restriction process can be released or the time during which the thresholds can be relaxed is determined in advance by, for example, the user or other authorized person. Examples of the above-mentioned authorized person include the user's superior, a system administrator, etc.

[0154] For example, if a window assigned to a remote conference program gradually becomes grayed out while a user is speaking during the remote conference, the user experience will be degraded if the graying process is not stopped. According to the above embodiment, even in such a case, the graying process can be temporarily stopped or the graying can be canceled, thereby minimizing degradation of the user experience.

[0155] In another embodiment, the control device 170 may determine that the rating value is outside the allowable range if the duration of the state in which the rating value is outside the allowable range exceeds a predetermined value. This prevents the rating value from being determined to be outside the allowable range if the rating value suddenly becomes outside the allowable range. As a result, the user's experience is improved.

[0156] The control device 170 may adjust the sensitivity or responsiveness of the above-mentioned determination based on the evaluation result regarding the safety of the usage environment of the user terminal 120 obtained in S416. For example, when the safety of the usage environment satisfies a predetermined standard, the control device 170 adjusts the sensitivity or responsiveness of the above-mentioned determination so that the sensitivity or responsiveness of the determination is lower than when the safety of the usage environment does not satisfy the standard. For example, the control device 170 adjusts the sensitivity or responsiveness of the above-mentioned determination so that the sensitivity or responsiveness of the determination is lower as the level of safety of the usage environment increases. This improves the user's usage experience.

[0157] If the evaluation value of the safety of the user terminal 120 is within the acceptable range (Yes in S420), the process proceeds to S440. On the other hand, if the evaluation value of the safety of the user terminal 120 is outside the acceptable range (No in S420), in S430, the control device 170 determines whether the length of time that has elapsed since the user terminal 120 or the program was started based on a start-up instruction from the user 22 is greater than a predetermined second threshold. The above-mentioned elapsed time may be the time that has elapsed since the start-up instruction was accepted, or may be the time that has elapsed since the start-up process of the user terminal 120 or the program was completed, or may be the time that has elapsed since the control device 170 detected the start-up of the user terminal 120 or the program.

[0158] In S430, for example, if the elapsed time is greater than the second threshold (Yes in S430), in S432, the control device 170 executes the use restriction process described above. Examples of the use restriction process include a process for restricting acceptance of instructions from the user 22, a process for locking the screen displayed in the display area 142, a process for displaying a predetermined image in the display area 142, a process for displaying a warning in the display area 142, a process for adjusting the brightness of the display area 142 to darken the screen, and a process for narrowing the viewing angle of the display device 140.

[0159] In the process for displaying a predetermined image in display area 142, the image may be displayed over the entire screen or on a portion of the screen. In the process for displaying a predetermined image in display area 142, the image may be a semi-transparent image. The process for displaying a predetermined image in display area 142 may be a process for hiding at least a portion of the screen that would have been displayed if the use restriction process had not been executed, or may be a process for superimposing a semi-transparent image on at least a portion of the screen that would have been displayed if the use restriction process had not been executed.

[0160] The control device 170 may determine the type and / or level of usage restriction processing based on the evaluation result regarding the safety of the usage environment of the user terminal 120 obtained in S416. Examples of the evaluation result include (i) at least one index value of one or more indexes regarding the safety of the user terminal 120, (ii) a type of index value, among the one or more indexes regarding the safety of the user terminal 120, whose magnitude satisfies a predetermined condition, and (iii) an evaluation value indicating the level of safety of the user terminal 120. Examples of the predetermined condition include a condition that the index value is greater than a predetermined value, a condition that the index value is equal to or greater than a predetermined value, a condition that the index value is smaller than a predetermined value, and a condition that the index value is equal to or less than a predetermined value.

[0161] The control device 170 may determine the type and / or degree of the use restriction process so that the type and / or degree of the use restriction process differs between when the index value of a specific index satisfies the above-mentioned predetermined condition and when the index value of the index does not satisfy the condition. The control device 170 may determine the type and / or degree of the use restriction process so that the type and / or degree of the use restriction process differs between when the combination of indexes that satisfies the predetermined condition is a specific combination and when it does not.

[0162] For example, even if the safety evaluation value is outside the allowable range, if the safety of the usage environment satisfies a predetermined standard, the control device 170 determines, as the usage restriction process, a process other than at least one of a process for restricting the acceptance of instructions from the user 22, a process for locking the screen displayed in the display area 142, and a process for displaying a warning in the display area 142. In this case, the control device 170 may determine, as the usage restriction process, at least one of a process for displaying a predetermined image in the display area 142, a process for adjusting the brightness of the display area 142 to darken the screen, and a process for narrowing the viewing angle of the display device 140. This allows the user to continue using the user terminal 120, for example, if the usage environment is relatively safe. As a result, the user's usage experience is improved.

[0163] Examples of the level of usage restriction processing include the amount of change in brightness, the amount of change in viewing angle, the size of the image to be superimposed, and the transparency of the image to be superimposed. The control device 170 may determine the level of usage restriction processing so that the safer the usage environment, the lower the level of usage restriction. The control device 170 may determine the level of usage restriction processing so that the safer the usage environment, the lower the level of usage restriction. This allows the user to continue using the user terminal 120, for example, when the usage environment is relatively safe. As a result, the user's usage experience is improved.

[0164] While the use of the user terminal 120 is restricted, the control device 170 may or may not execute the process described in relation to S416. If the control device 170 executes the process described in relation to S416 during the period when the use of the user terminal 120 is restricted, the safety evaluation value may return to within the acceptable range during that period.

[0165] In this case, the control device 170 may cancel or suspend the execution of the process described in relation to S432 or the process described in relation to S434 and remove the above-mentioned restriction. For example, if the level of security of the user terminal 120 is higher than a predetermined level (i.e., if the degree of possibility that the output image will be viewed by a third party is lower than a predetermined level), the control device 170 may permit the output of the original output image.

[0166] The control device 170 may adjust the sensitivity or responsiveness of the determination as to whether to lift the restriction, in the same way as when the control device 170 determines whether to restrict the use of the user terminal 120. This reduces the frequency of switching between usage restriction and restriction lifting, resulting in an improved user experience.

[0167] Next, in S434, the control device 170 executes a process (sometimes referred to as a guidance process) to guide the user 22 to adjust the installation position and installation orientation of the user terminal 120 so as to improve the safety of the user terminal 120 or so as to satisfy a predetermined standard for the safety of the user terminal 120. In this way, the control device 170 can support the user 22 in adjusting the installation position and / or installation orientation of the user terminal 120.

[0168] An example of the guidance process is a process for guiding the user 22 to an appropriate installation position and / or installation direction (sometimes referred to as a guide process). In one embodiment, the control device 170 guides the user 22 by outputting a message such as, "Turn the display area 142 approximately 10 degrees to the right." The control device 170 may output the message in the display area 142, or may output the message by voice. In another embodiment, the control device 170 may superimpose on the display area 142 a current image of the interior of the workroom 300 captured by the peripheral data generating device 152 and / or the camera 154, an image providing a hint as to an installation position and installation direction that satisfies safety standards. Examples of the hint image include an image for highlighting an area where safety is insufficient, an image for highlighting an area where safety is sufficiently ensured, and an image for indicating the adjustment direction of the display area 142. The hint image may have a transparent or semi-transparent area.

[0169] As described above, according to one example of the present embodiment, the sensitivity or responsiveness in determining whether the safety evaluation value of the user terminal 120 is within the acceptable range is adjusted. This allows for the introduction of (i) a function that dynamically changes one or more thresholds or parameters that are the basis for determining safety in accordance with the usage environment or the display content of the display area 142, and (ii) a delay function that delays the time until a penalty such as a screen lock or usage restriction is imposed after the safety evaluation value exceeds a threshold or the evaluation value falls outside the acceptable range.

[0170] For example, an office may be deemed safe because there is little chance of intrusion by outsiders, and the threshold used as the standard for determining safety may be lowered, etc. The environment and display content may be automatically determined and the threshold may be changed dynamically, or the threshold may be changed by an operation by a superior, etc. Furthermore, when automatically determining whether it is an office, it may be determined that it is an office based on the presence or absence of a specific SID using the identification ID of a wireless connection (commonly called an SID), or it may be determined by checking a wired connection or its attributes, or the presence of a specific server or interface.

[0171] If the content displayed in the display area 142 is confidential information, the aforementioned threshold may be raised to strictly evaluate the security judgment. The sensitivity of the displayed content may be automatically evaluated based on the type of application used or the frequency of occurrence of words that are perceived as confidential information. The exceptional function described above may be implemented by, for example, introducing a boost button that lowers the judgment, and dynamically relaxing the threshold by pressing the boost button when desired or approved by the user. The boost state may be released over time, or automatically released when a significant change in the surrounding state of the display area 142 is detected (when a significant change in the surrounding state of the terminal, such as a change in terminal location or orientation, or a change in brightness, is detected).

[0172] Furthermore, the control device 170 may store some or all of the images and information acquired by the environmental sensor 150 during processing to lock the screen displayed in the display area 142, or under exceptional circumstances such as when the level of safety is low or when the safety determination is explicitly relaxed or strengthened. Furthermore, the target time range of the stored information may be a sufficient time period that includes the occurrence of the exceptional situation, or it may be a partial time period, may be discrete, or may target only slightly related information.

[0173] (An example of detailed guide processing) In this embodiment, the imaging direction of the peripheral data generating device 152 and / or the camera 154 substantially coincides with the normal direction of the display area 142. In this case, the control device 170 can assist the user 22 in adjusting the installation position and / or installation direction of the user terminal 120, for example, by the following procedure.

[0174] In one embodiment, the control device 170 controls the display device 140 to display, in the display area 142, an image (sometimes referred to as an adjustment image) in which (a) a current image of the interior of the work room 300 captured by the peripheral data generating device 152 and / or the camera 154 and (b) (i) an image of the interior of the work room 300 (sometimes referred to as a guide image) or (ii) an icon indicating the position of a feature point in the interior of the work room 300 (sometimes referred to as a guide icon) when the display area 142 is installed according to the combination of the installation position and installation direction determined in S416 are superimposed. The current image and the guide image or guide icon can be superimposed using a known method.

[0175] The control device 170 may control the display device 140 to display an icon (sometimes referred to as an adjustment icon) indicating the adjustment direction of the display area 142 in the display area 142. The control device 170 may control the display device 140 to display an image for highlighting an area where safety is insufficient, an image for highlighting an area where safety is sufficiently ensured, or the like in the display area 142.

[0176] The current image of the interior of the work room 300 is generated based on, for example, the current two-dimensional image or three-dimensional data captured in S434, while the guide image or guide icon is generated based on, for example, the past two-dimensional image or three-dimensional data captured in S416.

[0177] In S416, multiple two-dimensional images or three-dimensional data may be acquired at different imaging positions and / or imaging directions. Based on these two-dimensional images or three-dimensional data, the control device 170 generates two-dimensional images or three-dimensional data that would be captured by the surrounding data generation device 152 and / or the camera 154 if the display area 142 were installed according to the combination of the installation position and installation direction determined in S416. The control device 170 generates a guide image or a guide icon based on the above two-dimensional images or three-dimensional data.

[0178] For example, the control device 170 may determine to use a single two-dimensional image or a part thereof captured by the peripheral data generating device 152 and / or the camera 154 as a guide image. The control device 170 may determine the display position of a guide icon based on the single two-dimensional image. The control device 170 may determine to generate a combined image by combining multiple two-dimensional images captured by a single or multiple peripheral data generating devices 152 and / or cameras 154, and use the combined image or a part thereof as a guide image. The control device 170 may determine the display position of a guide icon based on the combined image.

[0179] In another example, the control device 170 may generate a guide image or a guide icon using the above-described three-dimensional virtual space. For example, first, the control device 170 generates a three-dimensional model of the structures constituting the work room 300 and / or the structures arranged inside the work room 300 based on the past two-dimensional images or three-dimensional data captured in S416. The control device 170 may generate the above three-dimensional model based on multiple two-dimensional images or three-dimensional data captured at different imaging positions and / or from different imaging directions.

[0180] This generates a three-dimensional virtual space of the workroom 300. For example, three-dimensional models of walls, pillars, doors, windows, furniture, etc. are generated based on the multiple AR markers captured in the multiple two-dimensional images. Furthermore, the generated three-dimensional models are placed at appropriate positions in the three-dimensional space corresponding to the workroom 300 based on the positions and orientations of the AR markers obtained by analyzing the two-dimensional images.

[0181] Next, the control device 170 determines the position and orientation (sometimes referred to as posture) of the peripheral data generating device 152 and / or the camera 154 when the display area 142 is installed according to the combination of the installation position and installation orientation determined in S416. The control device 170 generates an image of the interior of the workroom 300 captured by the peripheral data generating device 152 and / or the camera 154, assuming that the peripheral data generating device 152 and / or the camera 154 are placed at the above-mentioned position and with the above-mentioned posture in the three-dimensional virtual space of the workroom 300. This allows the control device 170 to generate a guide image or a guide icon.

[0182] By displaying the adjustment image or adjustment icon in the display area 142, the user 22 can adjust the installation position and installation orientation of the user terminal 120 while viewing the screen output in the display area 142. For example, the user 22 adjusts the installation position and / or installation orientation of the display area 142 so that the current image of the interior of the work room 300 and the guide image approximately match. For example, the user 22 adjusts the installation position and / or installation orientation of the display area 142 so that the positions of the feature points in the current image of the interior of the work room 300 approximately match the positions of the guide icons in the adjustment image.

[0183] According to this embodiment, the installation position and installation direction of the user terminal 120 can be adjusted even if the viewing angle of the display area 142 is larger than the viewing angle of the peripheral data generating device 152 and / or the camera 154. Note that the adjustment method according to this embodiment can also be applied when the viewing angle of the display area 142 is smaller than the viewing angle of the peripheral data generating device 152 and / or the camera 154.

[0184] When the user terminal 120 is equipped with a peripheral data generation device 152 and / or a camera 154, as in the user terminal 120 according to this embodiment, depending on the type of the peripheral data generation device 152 and / or the camera 154, when the user 22 moves the user terminal 120 to adjust the installation position and installation direction of the user terminal 120, a strong afterimage may appear in the image captured by the peripheral data generation device 152 and / or the camera 154. For example, when the peripheral data generation device 152 is a LiDAR using a laser, even if the user 22 moves the user terminal 120 relatively quickly, the amount of afterimage that appears is extremely small. On the other hand, when the peripheral data generation device 152 and / or the camera 154 is an image sensor such as a CMOS, when the user 22 moves the user terminal 120 relatively quickly, the peripheral data generation device 152 and / or the camera 154 cannot capture a clear image. Furthermore, when the peripheral data generating device 152 and / or the camera 154 captures the above-mentioned marker, if the degree of movement of the peripheral data generating device 152 and / or the camera 154 becomes large, the coordinates of the marker cannot be calculated accurately in the process for determining the camera coordinates of the marker.

[0185] Therefore, the control device 170 may output information to guide the user on how to move the user terminal 120. For example, the control device 170 outputs instructions indicating the moving speed of the user terminal 120, instructions indicating whether the moving speed of the user terminal 120 is appropriate or inappropriate, instructions to stop the user terminal 120, etc. The above instructions may be displayed in the display area 142 or output as audio.

[0186] While the guide process is being executed, the control device 170 repeatedly evaluates the safety of the user terminal 120. For example, the control device 170 determines whether the evaluation value of the safety of the user terminal 120 is within an acceptable range. The control device 170 may also determine whether the installation position and installation orientation of the display area 142 approximately match the installation position and installation orientation determined in S416.

[0187] If it is determined that the safety evaluation value of the user terminal 120 is within an acceptable range, or if it is determined that the installation position and installation orientation of the display area 142 substantially match the installation position and installation orientation determined in S416, the control device 170 outputs a message indicating that the user terminal 120 has been installed in a safe position. For example, the control device 170 controls the display device 140 to display the above message in the display area 142. The control device 170 may also output the above message by voice. This allows the user 22 to easily adjust the installation position and installation orientation of the user terminal 120.

[0188] (Another example of the induction process) In this embodiment, the control device 170 repeatedly evaluates the safety of the user terminal 120 constantly, periodically, or at any timing. Therefore, even if guide information regarding how to move the user terminal 120 is not output, the user 22 can install the user terminal 120 in a position and orientation that ensures safety. More specifically, the user 22 arbitrarily changes the installation position and / or installation orientation of the user terminal 120. Each time the installation position and / or installation orientation of the user terminal 120 is changed, the control device 170 evaluates the safety of the user terminal 120 at the installation position and installation orientation and presents the results of the evaluation to the user 22. This allows the user 22 to install the user terminal 120 in a position and orientation that ensures safety.

[0189] When the guide process ends in S434, the control device 170 repeats the processes from S416 onwards. This allows the control device 170 to continue monitoring the safety of the user terminal 120.

[0190] On the other hand, in S430, for example, if the elapsed time is less than the second threshold (No in S430), in S440 the control device 170 inquires of the user 22, for example, whether or not the user 22 wishes to be suggested a more appropriate usage environment. For example, if the user 22 inputs information indicating that the user 22 wishes to be suggested into the user terminal 120 (Yes in S440), the control device 170 accepts the input and executes the guide process described in relation to S434.

[0191] On the other hand, in S440, for example, if the user 22 does not input information indicating that he / she desires the above-mentioned proposal into the user terminal 120, or if he / she inputs information indicating that he / she does not desire the above-mentioned proposal into the user terminal 120 (No in S440), the control device 170 determines in S440, for example, whether or not to terminate the user terminal 120 or the program started in S412. Specifically, the control device 170 determines whether or not an instruction to terminate the user terminal 120 or the above-mentioned program (sometimes referred to as an end instruction) has been accepted.

[0192] If the termination instruction is accepted (Yes in S450), the control device 170 terminates the user terminal 120 or the program. This ends the processing. On the other hand, if the termination instruction is not accepted (No in S450), the control device 170 repeats the processing from S416 onwards. This allows the control device 170 to continue monitoring the safety of the user terminal 120.

[0193] (An example of a process for evaluating the safety of the usage environment) As described above, Fig. 5 schematically illustrates an example of the processing in step 416. As shown in Fig. 5, in this embodiment, the details of the processing for evaluating the safety of the user terminal 120 (which may be referred to as evaluation processing) will be described using as an example a case where the control device 170 evaluates the safety of the user terminal 120 by comprehensively determining the values ​​of multiple indexes, including a first index, a second index, a third index, and a fourth index. Note that even when the control device 170 evaluates the safety of the user terminal 120 by comprehensively determining the values ​​of multiple indexes, the safety of the user terminal 120 may ultimately be evaluated based on the value of a single index.

[0194] As described above, the evaluation process described in relation to S416 is periodically repeated while the user terminal 120 or the program is running. When the first evaluation process is executed, the types of the first, second, third, and fourth indices are determined based on, for example, the index settings determined in S414. In the first evaluation process, at least one of the first, second, third, and fourth indices may not be derived.

[0195] According to this embodiment, first, in S512, the control device 170 derives the value of a first index included in the multiple indexes. The control device 170 derives the value of the first index based on, for example, the index setting determined in S414. The first index may be an index indicating the degree to which object movement or light fluctuation is detected in the vicinity of the installation position of the user terminal 120.

[0196] In S514, the control device 170 derives the value of a second index included in the multiple indexes. The control device 170 derives the value of the second index based on, for example, the index setting determined in S414. The second index may be an index indicating the placement status of obstructions present around the installation position of the user terminal 120. The second index may be an index indicating the degree to which the obstructions block the line of sight of a third party to the display area 142.

[0197] In S516, the control device 170 derives the value of a third index included in the multiple indexes. The control device 170 derives the value of the third index based on, for example, the index setting determined in S414. The third index may be an index indicating the status of the surrounding environment of the installation location of the user terminal 120.

[0198] The third index may be an index indicating the illuminance around the installation location of the user terminal 120 or the distribution or fluctuation of the illuminance. The third index may be an index indicating the outdoor brightness in the area where the installation location of the user terminal 120 belongs. The third index may be an index indicating the degree of temperature distribution around the installation location of the user terminal 120. The third index may be an index indicating the degree of visibility around the installation location of the user terminal 120. The third index may be an index indicating the concentration of suspended matter, airborne matter, or odorous matter around the installation location of the user terminal 120 or the distribution or fluctuation of the concentration.

[0199] In S518, the control device 170 derives the value of a fourth index included in the multiple indexes. The control device 170 derives the value of the fourth index based on, for example, an object or an event observed in the vicinity of the installation location of the user terminal 120. The fourth index may be an index indicating the risk of information leakage depending on the type of object or event. For example, the control device 170 refers to a database in which, for each of one or more objects or events, the type and / or attribute of the object or event is associated with a value indicating the degree of likelihood that a third party will view the output image from the location where the object or event was observed, and determines the value of the fourth index based on the type and / or attribute of the object or event actually observed.

[0200] In one embodiment, at least two steps included in S512, S514, S516, and S518 are performed in parallel. In another embodiment, at least two steps included in S512, S514, S516, and S518 are performed sequentially.

[0201] Next, in S520, the control device 170 determines a procedure for deriving an evaluation value indicating the degree of safety of the user terminal 120 based on the values ​​of one or more indicators. In this embodiment, the control device 170 determines the above-mentioned evaluation setting based on various data acquired by the environmental sensor 150 during the execution periods of S512, S514, S516, and S518.

[0202] For example, the control device 170 determines the type of one or more indices to be used to derive the evaluation value from among the multiple indices based on (i) information acquired by the environmental sensor 150 and / or (ii) the current time or the time zone to which the current time belongs. The control device 170 may determine the type of one or more indices to be used to derive the evaluation value from among the multiple indices based on (i) information acquired by the environmental sensor 150, (ii) the current time or the time zone to which the current time belongs, and (iii) the latitude and / or longitude of the installation location of the user terminal 120.

[0203] For example, the control device 170 determines a weight value to be set for each of one or more indicators used to derive an evaluation value based on (i) information acquired by the environmental sensor 150 and / or (ii) the current time or the time zone to which the current time belongs. The control device 170 may determine a weight value to be set for each of one or more indicators used to derive an evaluation value from among the multiple indicators based on (i) information acquired by the environmental sensor 150, (ii) the current time or the time zone to which the current time belongs, and (iii) the latitude and / or longitude of the installation location of the user terminal 120.

[0204] Next, in S530, the control device 170 derives an evaluation value indicating the degree of safety of the user terminal 120 based on the values ​​of one or more indicators. The control device 170 derives the evaluation value based on the evaluation settings determined in S520. For example, the control device 170 derives the evaluation value based on the values ​​of each of the multiple indicators derived in S512, S514, and S516 and the weight values ​​set for each of the multiple indicators. This allows the control device 170 to evaluate the safety of the user terminal 120 or the degree of viewability described above.

[0205] The process for evaluating the safety of the user terminal 120 may be an example of a process for deriving the viewability.

[0206] (An example of another embodiment) In the present embodiment, the details of the information leakage prevention system 100 have been described using as an example a case where the control device 170 evaluates the safety of the user terminal 120 based on a plurality of indexes. However, the information leakage prevention system 100 is not limited to the present embodiment.

[0207] In another embodiment, the control device 170 may evaluate the security of the user terminal 120 based on at least one of the first index, the second index, the third index, and the fourth index. For example, the control device 170 may comprehensively determine the values ​​of one or more indexes to evaluate the security of the user terminal 120. In yet another embodiment, the control device 170 may evaluate the security of the user terminal 120 based on the value of a predetermined single index.

[0208] In the present embodiment, the details of the information leakage prevention system 100 have been described using as an example a case where the use restriction process is executed in S432 and then the guidance process is executed in S434. However, the information leakage prevention system 100 is not limited to the present embodiment.

[0209] In other embodiments, the procedure of S434 may be omitted. For example, the control device 170 may omit the procedure of S434 in accordance with an instruction or setting from the user.

[0210] 6 schematically illustrates an example of the internal configuration of the control device 170. In this embodiment, the control device 170 includes, for example, a surrounding state acquisition unit 610, an activation detection unit 622, a timing unit 624, a virtual space construction unit 630, a safety management unit 640, a placement determination unit 652, a placement support unit 654, an output control unit 660, and a storage unit 670. In this embodiment, the surrounding state acquisition unit 610 includes, for example, a structural information acquisition unit 612, a physical property information acquisition unit 614, an environmental information acquisition unit 616, and an attribute information acquisition unit 618. The elements that make up the control device 170 are configured to be able to send and receive information to and from each other.

[0211] In this embodiment, the surrounding state acquisition unit 610 acquires information (sometimes referred to as surrounding state information) indicating the state of the surroundings of the display area 142. The surrounding state acquisition unit 610 acquires, for example, the surrounding state information generated by the environmental sensor 150. The surrounding state acquisition unit 610 may output the surrounding state information to the safety management unit 640. In this embodiment, the surroundings of the display area 142 may include the installation position of the display area 142 and the vicinity of the installation position.

[0212] In this embodiment, the structural information acquisition unit 612 acquires information (sometimes referred to as peripheral data) indicating the shape, size, and / or location of one or more objects arranged around the display area 142. The structural information acquisition unit 612 outputs the peripheral data to, for example, the safety management unit 640. The peripheral data may be an example of peripheral state information.

[0213] The structural information acquisition unit 612 acquires, for example, image data of an image having one or more objects arranged around the display area 142 as its subject. The structural information acquisition unit 612 acquires the image data from, for example, the peripheral data generation device 152. The image may be a still image or a moving image. The image may be a stereo image. The image data may be an example of peripheral data.

[0214] The structural information acquisition unit 612 acquires, for example, three-dimensional point cloud data or distance data of one or more objects arranged around the display area 142. The structural information acquisition unit 612 acquires the three-dimensional point cloud data or distance data from, for example, the surrounding data generation device 152. The distance data may include information indicating a relative distance from a reference position and information indicating a distance ranking index. The reference position may be a predetermined specific position or a specific object. The three-dimensional point cloud data and / or distance data may be an example of surrounding data.

[0215] Examples of distance ranking indices include those derived using monocular camera depth estimation technology such as MiDaSnet. These indices are often not proportional to absolute distance and can be used as reference indices for ranking. For example, the absolute scale of the target can be inferred from the scale of the learned person information based on its similarity, and the distance to the target can then be inferred.

[0216] In this embodiment, the physical property information acquisition unit 614 acquires information (sometimes referred to as physical property information) indicating the physical properties of one or more objects arranged around the display area 142. The physical property information acquisition unit 614 outputs the physical property information to, for example, the safety management unit 640. The physical property information may be an example of surrounding state information.

[0217] The physical property information includes, for example, information indicating the optical properties of the surface of the object. The physical property information includes, for example, information indicating the characteristics of the electromagnetic waves traveling from the surface of the object toward the display area 142. The characteristics of the electromagnetic waves may be characteristics related to optical properties. The electromagnetic waves may be electromagnetic waves reflected by the surface of the object, electromagnetic waves transmitted through the object, or electromagnetic waves emitted by the object. Examples of optical properties include wavelength, amplitude, intensity, and brightness. Examples of light intensity or brightness include irradiance, photon density, and illuminance.

[0218] The physical property information acquisition unit 614 acquires, for example, image data of an image whose subject is one or more objects arranged around the display area 142. The physical property information acquisition unit 614 acquires, for example, image data output by the camera 154. The image data may be data of a still image or data of a moving image. As will be described later, in this embodiment, the first index described above is derived based on fluctuations in light intensity. Therefore, the image data is preferably data of a moving image.

[0219] This allows the physical property information acquisition unit 614 to acquire color information of one or more objects arranged around the display area 142. Color information may be an example of physical property information. Physical property information of a transparent or translucent object may include information indicating the characteristics of electromagnetic waves that have passed through the object. For example, color information of window glass may include color information of objects that can be seen through the window glass.

[0220] The physical property information acquisition unit 614 may acquire multiple pieces of video data captured in different imaging directions, which are video images of the periphery of a specific position in real space. The multiple pieces of video data may be multiple pieces of video data captured in multiple separate sessions, or multiple pieces of video data cut out from multiple positions in a single video.

[0221] For example, if the user 22 wishes to install the user terminal 120 at a specific location, the user 22 rotates the user terminal 120 around the desired installation location and captures images of the surrounding area of ​​the location using the camera 154 of the user terminal 120. This allows multiple pieces of video data captured from different imaging directions to be acquired.

[0222] The user 22 may capture an image of the surroundings of the above-mentioned position without the user 22 appearing in the image. For example, the user 22 captures an image of the surroundings of the user terminal 120 while positioned behind the camera 154. This omits the process of excluding the image of the user 22 from the image captured by the camera 154.

[0223] In this embodiment, the environmental information acquisition unit 616 acquires, for example, information indicating the status of the surrounding environment of the display area 142 (sometimes referred to as environmental information). The environmental information acquisition unit 616 acquires, for example, environmental information generated by the environmental sensor 150. The environmental information acquisition unit 616 may output the environmental information to the safety management unit 640. The environmental information may be an example of surrounding state information.

[0224] For example, the environmental information acquisition unit 616 acquires information (sometimes referred to as illuminance information) indicating the illuminance around the display area 142. For example, the environmental information acquisition unit 616 acquires, from the illuminance sensor 156, information indicating the measurement result of the illuminance by the illuminance sensor 156.

[0225] For example, the environmental information acquisition unit 616 acquires information (sometimes referred to as temperature information) indicating the temperature around the display area 142. For example, if the camera 154 is configured to be able to generate an infrared image, the environmental information acquisition unit 616 acquires image data of the thermal image generated by the camera 154.

[0226] For example, the environmental information acquisition unit 616 may acquire various types of information provided by other information providing systems via a communication network. For example, the environmental information acquisition unit 616 acquires weather information, time information, and the like from the other information providing systems.

[0227] In this embodiment, the attribute information acquisition unit 618 acquires, for example, information indicating the type or attribute of an object or event detected in the vicinity of the display area 142 (sometimes referred to as attribute information). The environmental information acquisition unit 616 may output the attribute information to the safety management unit 640. The attribute information may be an example of surrounding state information.

[0228] More specifically, the environmental information acquisition unit 616 acquires, for example, image data and / or audio data output by the environmental sensor 150. The attribute information acquisition unit 618 analyzes the image data and / or audio data to detect an object or event. The attribute information acquisition unit 618 determines the type or attribute of the detected object or event, for example, by referring to an appropriate database. This allows the attribute information acquisition unit 618 to acquire attribute information.

[0229] For example, the attribute information acquisition unit 618 analyzes image data of the periphery of the display area 142 to detect one or more objects located in the periphery of the display area 142. The attribute information acquisition unit 618 may refer to a database that stores features and / or attributes of various objects to identify the detected objects and determine the attributes of the objects. The method for detecting an object from an image and identifying the detected object is not particularly limited, and any known method may be applied.

[0230] For example, the attribute information acquisition unit 618 analyzes audio data in which audio around the display area 142 is recorded to detect an event that has occurred around the display area 142. The attribute information acquisition unit 618 may refer to a database that stores the characteristics and / or attributes of various events to identify the detected event or determine the attributes of the event.

[0231] For example, if a dog's bark or growl is detected, it is determined that a person is nearby. Also, if a person's voice or footsteps are detected, it is determined that a person is nearby. If approaching footsteps are detected, it is determined that the number of people nearby has increased. If receding footsteps are detected, it is determined that the number of people nearby has decreased.

[0232] In this embodiment, the startup detection unit 622 detects the startup of the user terminal 120, or the startup of a BIOS, OS, or application program running on the user terminal 120. When the startup of the user terminal 120, or the startup of a BIOS, OS, or application program running on the user terminal 120 is detected, the startup detection unit 622 may output information indicating that the above startup has been detected to the timer unit 624 and / or the safety management unit 640.

[0233] In this embodiment, the timer 624 measures the time that has elapsed since the activation detection unit 622 detected the activation. The timer 624 may output information indicating the measurement result to the safety management unit 640.

[0234] In this embodiment, the virtual space construction unit 630 constructs a three-dimensional virtual space corresponding to the periphery of the user terminal 120. The virtual space construction unit 630 constructs the three-dimensional virtual space based on, for example, information acquired by the structural information acquisition unit 612. Details of the virtual space construction unit 630 will be described later.

[0235] In this embodiment, the safety management unit 640 derives the degree of viewability of the output image by a third party other than the user 22, based on the state of the vicinity of the display area 142 indicated by the surrounding state information acquired by the surrounding state acquisition unit 610. The safety management unit 640 may evaluate the safety of the display area 142 based on the state of the vicinity of the display area 142.

[0236] For example, the safety management unit 640 determines that the safety of the display area 142 is higher when the risk of information displayed in the display area 142 being leaked is lower (i.e., the lower the degree of viewability). Examples of the safety of the display area 142 include the safety of the display area 142 installed in a specific position and direction, the safety of the installation position of the display area 142, and the safety of the installation position and installation direction of the display area 142. When the display area 142 is housed in the housing 130, the safety of the display area 142 may mean the safety of the user terminal 120.

[0237] The safety management unit 640, for example, comprehensively evaluates various pieces of information collected by the structural information acquisition unit 612 and derives the degree of viewability when the display area 142 is installed in a specific position and direction. More specifically, the safety management unit 640, for example, calculates the values ​​of one or more types of indicators indicating the degree of viewability. The safety management unit 640 derives an evaluation value indicating the degree of viewability or the degree of safety using a learning model or function for comprehensively evaluating the values ​​of the one or more indicators. The safety management unit 640 may derive an evaluation value for each of multiple patterns with different layouts of the display area 142. Details of the safety management unit 640 will be described later.

[0238] In one embodiment, the safety management unit 640 evaluates the safety of the display area 142 when a process for determining the placement of the display area 142 is executed. The process for determining the placement of the display area 142 is executed, for example, at least one of the following times: when the user 22 starts using the user terminal 120; when the user 22 starts using a specific application program; and when the lock or sleep mode is released. In another embodiment, the safety management unit 640 evaluates the safety of the display area 142 periodically or at any time while the user 22 is using the user terminal 120.

[0239] In this embodiment, the placement determination unit 652 determines the placement of the display area 142. The placement determination unit 652 may determine the placement of the display area 142 based on the evaluation result of the safety management unit 640. An example of the evaluation result of the safety management unit 640 is the degree of viewability. As described above, the degree of safety of the user terminal 120 is an example of the degree of viewability.

[0240] The arrangement of the display area 142 may be exemplified by the position of the display area 142 (sometimes referred to as the installation position) and / or the direction of a normal vector 242 of the display area 142 (sometimes referred to as the installation direction). The direction of the normal vector 242 may be exemplified by at least one of the direction of the vector in the horizontal direction and the direction of the vector in the vertical direction. When the display area 142 is housed in the housing 130, the arrangement of the display area 142 and the arrangement of the user terminal 120 substantially coincide with each other.

[0241] In one embodiment, the placement determination unit 652 acquires information indicating the placement of the display area 142 and information indicating the degree of viewability in the placement from the safety management unit 640. Examples of the information indicating the placement of the display area 142 include information indicating the installation position of the display area 142, and information indicating the installation position and installation direction of the display area 142.

[0242] If the degree of viewability satisfies a predetermined criterion, the layout determination unit 652 determines the above layout as the layout of the display area 142. An example of the predetermined criterion is that the degree of viewability is smaller than a predetermined level.

[0243] The predetermined degree may be different when display region 142 is located within an area having a predetermined characteristic and when display region 142 is located outside the area. The area may define the installation position of display region 142, or may define the installation position and installation direction of display region 142.

[0244] According to one embodiment of the information leakage prevention system 100, even in an area where the degree of viewability is relatively high, if the use of the user terminal 120 in that area has been approved in advance by a person (sometimes referred to as an approver) who has the authority to permit the user 22 to use the user terminal 120, the user 22 can use the user terminal 120. In this case, the criteria regarding the degree of viewability that are applied when it is confirmed that the display area 142 is located within the range of the area differ from the criteria regarding the degree of viewability that are applied when it is not confirmed that the display area 142 is located within the range of the area. Note that the approval may specify a start time when the approval becomes effective, and may also specify an expiration date for the approval.

[0245] The degree of viewability that is applied when it is confirmed that display area 142 is located within the range of the area may be greater than the degree of viewability that is applied when it is not confirmed that display area 142 is located within the range of the area. When it is confirmed that display area 142 is located within the range of the area, the determination of the degree of viewability may be omitted.

[0246] In this case, examples of the predetermined feature include features of the peripheral data acquired by the peripheral data generating device 152 of the user terminal 120 located in the area where the approver has approved the use of the user terminal 120. More specifically, examples include one or more feature points included in a two-dimensional image captured by the peripheral data generating device 152, and one or more feature points included in the point cloud data acquired by the peripheral data generating device 152. The predetermined feature may be a two-dimensional image or three-dimensional data captured by the peripheral data generating device 152 when the approver approves the use of the user terminal 120. Note that the approval may specify a start time when the approval becomes valid, and may also specify an expiration date for the approval.

[0247] According to this embodiment, the user terminal 120 stores information indicating the spatial structure at the time the approver approved the use of the user terminal 120. The user terminal 120 may include a database that stores information indicating the spatial structure at the time the approver approved the use of the user terminal 120 and information indicating the degree of browsability.

[0248] As a result, for example, the control device 170 of the user terminal 120 can compare the data acquired by the surrounding data generation device 152 at the current position and orientation of the user terminal 120 with the recorded spatial structure data. If the two match, the control device 170, for example, references the above-mentioned database and extracts information indicating the degree of viewability stored in association with the recorded spatial structure. The control device 170 may evaluate the safety of the user terminal 120 based on the extracted information indicating the degree of viewability. Note that if the two match, the control device 170 may determine that the degree of viewability is less than a predetermined level and that the installation position and / or orientation of the user terminal 120 is safe.

[0249] According to this embodiment, if the user terminal 120 or the display area 142 is installed in a similar manner to a space where an approver has previously approved the use of the user terminal 120, the user 22 can use the user terminal 120. For example, if an approver has approved the use of the user terminal 120 in a seat with specific characteristics on a Shinkansen train, the user 22 can use the user terminal 120 without having to apply for approval every time they use the Shinkansen.

[0250] The above approval may be set for each user 22, or for each space or feature of a space. When the above approval is set for each space or feature of a space, for example, if user A performs an approval operation for a seat with specific features on a Shinkansen train and approval is obtained for that seat, when another user B uses a seat with similar features on another Shinkansen train, user B can use the user terminal 120 without applying for approval.

[0251] According to one embodiment of the information leakage prevention system 100, even in an area where the degree of viewability is relatively high, for example, when the user 22 uses the user terminal 120 in a room where only persons with a confidentiality obligation can enter, the user 22 can use the user terminal 120. In this case, the standard regarding the degree of viewability that is applied when it is confirmed that the display area 142 is located within the range of the area is different from the standard regarding the degree of viewability that is applied when it is not confirmed that the display area 142 is located within the range of the area.

[0252] The degree of viewability that is applied when it is confirmed that display area 142 is located within the range of the area may be greater than the degree of viewability that is applied when it is not confirmed that display area 142 is located within the range of the area. When it is confirmed that display area 142 is located within the range of the area, determination of the degree of viewability may be omitted. Note that when it is confirmed that display area 142 is located within the range of the area, a function or system for determining the degree of viewability may not be activated.

[0253] In this case, examples of the predetermined characteristic include a wireless signal or sound wave receivable in the area, a check of the presence or connection to a specific OBJ via a wired LAN connection in the area, etc. Similarly, when the user terminal 120 acquires information indicating that the user 22 is located within the range of the area from an attendance management system or the like, the determination of the degree of viewability may be omitted. Furthermore, when the user terminal 120 acquires (i) information indicating that the approver or the user himself / herself has approved that the display region 142 is located within the range of the area, or (ii) information indicating that the approver or the user himself / herself has approved the use of the user terminal 120, the determination of the degree of viewability may be omitted.

[0254] On the other hand, if the degree of viewability does not satisfy a predetermined standard, the placement determination unit 652 determines to output a message to the user 22 to prompt the user 22 to change at least one of the installation position and installation orientation of the display area 142. The placement determination unit 652 controls, for example, the display device 140 to display the message in the display area 142. Note that the message may be output as audio.

[0255] In another embodiment, the placement determination unit 652 acquires information indicating the placement of the display area 142 and information indicating the degree of viewability in that placement for each of a plurality of cases in which the placement of the display area 142 is different from each other, from the safety management unit 640. Examples of the information indicating the placement of the display area 142 include information indicating the installation position of the display area 142, and information indicating the installation position and installation direction of the display area 142.

[0256] The layout determination unit 652 compares the degree of viewability of each of the multiple cases with a predetermined standard, and extracts cases whose degree of viewability satisfies the predetermined standard. The layout determination unit 652 determines one of the layouts indicated by the extracted cases as the layout of the display area 142.

[0257] For example, the placement determination unit 652 determines the placement indicated by the case with the best degree of viewability as the placement of the display area 142. For example, the placement determination unit 652 extracts multiple cases with particularly best degrees of viewability. The placement determination unit 652 may extract the above cases based on the degree of viewability, or may extract the above cases so that the number of extracted cases is equal to or less than a predetermined number. The placement determination unit 652 presents information indicating the placement of each of the extracted cases to the user 22, and determines the case selected by the user 22 as the placement of the display area 142.

[0258] In this embodiment, the placement assistance unit 654 assists the user 22 in installing the display region 142 or the user terminal 120. The placement assistance unit 654 may assist the user 22 in installing the display region 142 or the user terminal 120 by executing the guidance process or guide process in S434 described in relation to FIG.

[0259] In this embodiment, the output control unit 660 controls the output of images by the display device 140. The output control unit 660 may execute various use restriction processes in S432 described in relation to Fig. 4. The output control unit 660 may execute a process for removing the use restriction (sometimes referred to as a restriction removal process).

[0260] For example, if the degree of viewability derived by the safety management unit 640 is smaller than a predetermined degree, the output control unit 660 permits output of the output image. For example, if the evaluation value indicating the degree of viewability or the degree of safety described above is within the above-described allowable range, the output control unit 660 determines that the degree of viewability is smaller than the predetermined degree.

[0261] For example, if the degree of viewability derived by the safety management unit 640 exceeds a predetermined level, the output control unit 660 determines to execute the above-described use restriction process. For example, if the evaluation value indicating the above-described degree of viewability or degree of safety is outside the above-described allowable range, the output control unit 660 determines that the degree of viewability exceeds the predetermined level. The output control unit 660 may stop outputting the output image that was displayed before the degree of viewability exceeded the predetermined level, may adjust the brightness or contrast of the output image, or may display an image different from the above-described output image in the display area 142.

[0262] The output control unit 660 can white out or black out the screen displayed in the display area 142 by adjusting the brightness or contrast of the output image. This interferes with the output of the original output image. The output control unit 660 can display an image different from the original output image in the display area 142, thereby interfering with the output of part or all of the original output image. The image that is output to interfere with the output of the original output image may be a semi-transparent image or an opaque image.

[0263] For example, when the degree of viewability derived by the safety management unit 640 is lower than a predetermined degree while the use of the user terminal 120 is restricted, the output control unit 660 permits the output of the original output image. For example, the output control unit 660 determines to execute a restriction release process. When the restriction release process is executed, the output of the original output image is resumed. The output control unit 660 may determine that the degree of viewability is lower than the predetermined degree when the evaluation value indicating the above-described degree of viewability or degree of safety is within the above-described allowable range.

[0264] Examples of images different from the above output image include an image for outputting a screen that may be viewed by a third party, an image for outputting a warning, an image for preventing the viewing of the original output image, etc. These images may have the same configuration as the various images described in relation to the use restriction process.

[0265] If the output control unit 660 starts execution of the use restriction process the moment the degree of viewability exceeds a predetermined degree, or if the output control unit 660 starts execution of the restriction removal process the moment the degree of viewability falls below the predetermined degree, there is a possibility that the state of the user terminal 120 will frequently switch between a state in which use of the user terminal 120 is restricted and a state in which use of the user terminal 120 is not restricted. Frequent switching between a state in which use of the user terminal 120 is restricted and a state in which use of the user terminal 120 is not restricted will degrade the user experience.

[0266] Therefore, the output control unit 660 may adjust the sensitivity or responsiveness of the above-mentioned determination regarding the degree of browsability in response to fluctuations in the safety assessment value of the user terminal 120, taking into consideration the user experience and system stability. As described above, any known method may be used to adjust the sensitivity or responsiveness. This adjusts, for example, the time it takes for a fluctuation in the safety assessment value of the user terminal 120 to affect the determination result regarding the degree of browsability and the magnitude relationship between the predetermined degree. This reduces the frequency of switching between a state in which the use of the user terminal 120 is restricted and a state in which the use of the user terminal 120 is not restricted, even if the safety assessment value of the user terminal 120 suddenly fluctuates.

[0267] The output control unit 660 may adjust the sensitivity or responsiveness so that the length of time is greater than a predetermined value. The sensitivity or responsiveness of the determination may also be adjusted based on the safety evaluation value of the user terminal 120.

[0268] In the above embodiment, the details of the use restriction by the output control unit 660 have been described using as an example a case where the output control unit 660 restricts the use of the user terminal 120 by stopping the output of the original output image or interfering with the viewing of the original output image. However, the use restriction by the output control unit 660 is not limited to the above embodiment.

[0269] The output control unit 660 may restrict acceptance of operation input from the user 22 by controlling the output of images by the display device 140. Restricting acceptance of operation input may be an example of usage restriction.

[0270] For example, when a process for deriving the degree of viewability or a process for evaluating the safety of the installation position and / or installation direction of the display area 142 is executed, the output control unit 660 displays a transparent screen in front of the desktop screen output by the display device 140. At this stage, the user 22 can input various instructions by mouse operation, keyboard operation, touch operation, etc. On the other hand, if the degree of viewability derived by the safety management unit 640 exceeds a predetermined level, a screen that may be viewed by a third party, a warning screen, or the like, different from the output image described above, is displayed.

[0271] Most of these screens are not transparent, and even if the user 22 operates a mouse on these screens, the mouse operation cannot penetrate these screens and operate icons on the desktop located below these screens, which limits the acceptance of operational inputs from the user 22.

[0272] Note that the state of some pixels of the screen that may be viewed by a third party may be specified as transparent. This allows instructions input by the user 22 to be accepted in the area where the pixels are arranged. Furthermore, the screen that may be viewed by a third party and the transparent screen may be smaller than the desktop screen. This allows instructions from the user 22 to be accepted in a portion of the desktop screen.

[0273] In this embodiment, the storage unit 670 stores various types of information. In one embodiment, the storage unit 670 stores various types of information used in information processing executed in the user terminal 120. In another embodiment, the storage unit 670 stores various types of information generated by information processing executed in the user terminal 120. The user terminal 120 may transmit at least a portion of the information stored in the storage unit 670 to an external server periodically, or when a predetermined event is detected.

[0274] The safety management unit 640 may be an example of a viewability derivation unit. The safety management unit 640 may be an example of a peripheral data acquisition unit or an information processing device.

[0275] 7 schematically illustrates an example of the internal configuration of the structural information acquisition unit 612. In this embodiment, the structural information acquisition unit 612 includes a three-dimensional point cloud acquisition unit 712, a distance image acquisition unit 714, a stereo image acquisition unit 716, and a two-dimensional image acquisition unit 718.

[0276] In this embodiment, the three-dimensional point cloud acquisition unit 712 acquires three-dimensional point cloud data of one or more objects arranged around the display area 142 from the surrounding data generation device 152. The three-dimensional point cloud data has information indicating the three-dimensional coordinate values ​​of each of the one or more points. The three-dimensional point cloud data may include information indicating the three-dimensional coordinate values ​​of each of the one or more points and information indicating the color of each point.

[0277] The distance image acquisition unit 714 acquires data (sometimes referred to as distance data or depth data) of distance images (sometimes referred to as depth images, depth maps, etc.) of one or more objects arranged around the display area 142 from the surrounding data generation device 152. A distance image is a planar image that represents the depth or the distance to an object, and the value of each pixel in the planar image indicates the distance to the object corresponding to that pixel.

[0278] The stereo image acquisition section 716 acquires, from the peripheral data generation device 152, stereo image data of one or more objects arranged around the display area 142 as subjects. The stereo image acquisition section 716 may generate distance images of the one or more objects arranged around the display area 142 based on the stereo images.

[0279] The two-dimensional image acquisition unit 718 acquires, from the peripheral data generation device 152, data of a two-dimensional image whose subject is one or more objects arranged around the display area 142 (acquiring image data may be simply referred to as acquiring an image). The two-dimensional image acquisition unit 718 may acquire a two-dimensional image whose subject is one or more objects to which one or more markers are attached.

[0280] The three-dimensional point cloud acquisition unit 712 may be an example of a peripheral data acquisition unit. The distance image acquisition unit 714 may be an example of a peripheral data acquisition unit. The stereo image acquisition unit 716 may be an example of a peripheral data acquisition unit. The two-dimensional image acquisition unit 718 may be an example of a peripheral data acquisition unit.

[0281] 8 schematically illustrates an example of the internal configuration of the safety management unit 640. In this embodiment, the safety management unit 640 includes a data conversion unit 810 and an object position determination unit 820. In this embodiment, the data conversion unit 810 includes a marker position determination unit 812 and a point cloud generation unit 814. In this embodiment, the object position determination unit 820 includes a first calculation unit 830 and a second calculation unit 840. In this embodiment, the first calculation unit 830 includes a point cloud data acquisition unit 832, a distance calculation unit 834, and an angle calculation unit 836. The second calculation unit 840 includes an edge detection unit 842 and an occlusion angle calculation unit 844.

[0282] In this embodiment, the data conversion unit 810 converts two-dimensional data into three-dimensional data. The data conversion unit 810 generates three-dimensional data of at least a portion of one or more objects arranged around the periphery of the display area 142, for example, based on the stereo images acquired by the stereo image acquisition unit 716. The data conversion unit 810 generates three-dimensional data of at least a portion of one or more objects arranged around the periphery of the display area 142, for example, based on the two-dimensional images acquired by the two-dimensional image acquisition unit 718.

[0283] The data conversion unit 810 outputs the generated three-dimensional data to the object position determination unit 820. This allows the object position determination unit 820 to construct a virtual space including a three-dimensional model of at least a part of one or more objects arranged around the display area 142. Furthermore, the object position determination unit 820 can determine the relative positional relationship between the display area 142 and the one or more objects in the virtual space.

[0284] In this embodiment, the details of the data conversion unit 810 will be described using as an example a case where, when a predetermined type of marker appears in a two-dimensional image acquired by the two-dimensional image acquisition unit 718, the data conversion unit 810 generates three-dimensional data of at least a portion of one or more objects arranged around the display area 142 based on the two-dimensional image. Examples of the predetermined type of marker include the above-mentioned AR marker, QR code (registered trademark), and barcode. For the purpose of simplifying the explanation, in this embodiment, an example of the data conversion process in the data conversion unit 810 will be described using as an example a case where the display area 142 and the peripheral data generation device 152 are housed in the same housing 130, and the direction of the normal vector 242 of the display area 142 and the direction of the optical axis 252 of the peripheral data generation device 152 are substantially the same.

[0285] In this embodiment, the marker position determination unit 812 analyzes a two-dimensional image of one or more objects to which one or more markers are attached, and determines the positional relationship between at least a portion of the one or more markers and the display area 142. For example, the marker position determination unit 812 first analyzes the two-dimensional image acquired by the two-dimensional image acquisition unit 718 and executes processing to detect a predetermined type of marker. If one or more markers are not detected, the marker position determination unit 812 ends the processing. This ends, for example, the data conversion processing in the data conversion unit 810.

[0286] For example, the marker position determination unit 812 calculates three-dimensional coordinates in the camera coordinate system from the four vertices of each marker. For example, if the camera coordinate system moves, the distribution coordinates of the AR markers in the camera coordinate system at a past reference point in time may differ from the distribution coordinates of the AR markers in the camera coordinate system at the current point in time. Therefore, the marker position determination unit 812 may determine whether the camera coordinate system has moved. For example, the marker position determination unit 812 performs a process to exhaustively find pairs of markers (sometimes referred to as paired markers) that are the same instance between the camera coordinate system at a past reference point in time and the camera coordinate system at the current point in time. Furthermore, the marker position determination unit 812 determines whether the camera coordinate system has moved based on the positions of the paired markers.

[0287] Specifically, the marker position determination unit 812 first calculates the transformation matrices of the two coordinate systems in a brute-force manner at each of the multiple time points at which the above-mentioned determination is performed. This derives a transformation matrix that changes the marker coordinates. Next, the marker position determination unit 812 uses the derived transformation matrix to transform the actual current marker position into a past reference point, and evaluates whether it matches the displayed marker image or whether the difference is significant using Euclidean distance or the like. The marker position determination unit 812 repeatedly attempts the above process to determine a pair group with the smallest difference, or a pair group with a difference whose absolute value is smaller than a predetermined value. This allows the marker position determination unit 812 to find paired markers.

[0288] On the other hand, if one or more markers are detected by the image analysis, the marker position determination unit 812 extracts, for example, from the one or more detected markers, a marker whose distance from the display area 142 can be calculated. For example, if the dimensions of the marker are known, or if it is known that two or more markers of the same type are attached on approximately the same plane, the distance between the marker and the display area 142 can be calculated by analyzing a two-dimensional image in which the marker appears.

[0289] Next, for each or at least some of the markers whose distances to the display area 142 can be calculated, the marker position determination unit 812 determines the positional relationship between the marker and the display area 142. The positional relationship between the marker and the display area 142 may be the relative positional relationship between the marker and the display area 142 in the real world. The marker position determination unit 812 may calculate (i) the distance between the display area 142 and the representative point of each marker, and (ii) the angle between the normal vector 242 of the display area 142 and the direction from the representative point of the display area 142 toward the representative point of each marker.

[0290] In this embodiment, the point cloud generation unit 814 generates a point cloud around the position of the marker whose positional relationship with the display area 142 has been determined by the marker position determination unit 812. If information about the point cloud to be generated is embedded in the marker, the point cloud generation unit 814 generates the point cloud according to the information embedded in the marker. As described above, examples of the information about the point cloud include the dimensions of the generated point cloud in the world coordinate system.

[0291] If information about the point cloud to be generated is not embedded in the marker, the point cloud generation unit 814 may generate a point cloud in a predetermined range around the marker, for example. If information about the point cloud to be generated is not embedded in the marker but identification information of the marker is embedded in the marker, the point cloud generation unit 814 may refer to a database that stores the identification information of the marker and information about the point cloud to be generated around the marker in association with each other, and generate a point cloud according to the information stored in the database.

[0292] In this embodiment, the object position determination unit 820 analyzes the surrounding data acquired by the structural information acquisition unit 612 and determines the positional relationship between each of one or more objects and the display area 142. The positional relationship may be a relative positional relationship in the real world. Examples of procedures for determining the relative positional relationship include a procedure based on information processing by the first calculation unit 830 and a procedure based on information processing by the second calculation unit 840.

[0293] In this embodiment, the first calculation unit 830 determines the relative positional relationship between the display area 142 and the one or more objects by using three-dimensional data of the one or more objects arranged around the display area 142. The first calculation unit 830 may construct a virtual space including three-dimensional models of at least a portion of the one or more objects arranged around the display area 142. The first calculation unit 830 may determine the relative positional relationship between the display area 142 and the one or more objects in the virtual space.

[0294] In this embodiment, the point cloud data acquisition unit 832 acquires three-dimensional point cloud data for each of the one or more objects based on the peripheral data acquired by the structural information acquisition unit 612. In one embodiment, the point cloud data acquisition unit 832 acquires point cloud data acquired by the three-dimensional point cloud acquisition unit 712 or the distance image acquisition unit 714. In another embodiment, the point cloud data acquisition unit 832 acquires point cloud data generated by the data conversion unit 810.

[0295] In this embodiment, the distance calculation unit 834 calculates the distance between the representative point of the display area 142 and each of a plurality of points virtually arranged on the surface of each of one or more objects represented by the three-dimensional point cloud data acquired by the point cloud data acquisition unit 832. An example of the representative point of the display area 142 is the representative point 240 described in relation to Fig. 2. Note that the representative point of the display area 142 may be any point on the surface of the display area 142 on the side where the output image is displayed, and is not limited to the representative point 240.

[0296] In this embodiment, the angle calculation unit 836 calculates the absolute value of the angle between the normal direction at the representative point of the display area 142 and the direction from the representative point of the display area 142 to each of a plurality of points arranged on the surface of each of the one or more objects. An example of the normal direction at the representative point of the display area 142 is the direction of the normal vector 242 at the representative point 240.

[0297] According to this embodiment, the relative distance between each of the one or more objects and the display area 142 is determined by information processing in the distance calculation unit 834. Furthermore, the relative direction between each of the one or more objects and the display area 142 is determined by information processing in the angle calculation unit 836. As a result, the positional relationship between each of the one or more objects and the display area 142 is determined.

[0298] In this embodiment, the second calculation unit 840 determines the positional relationship between each of the one or more objects detected around the display area 142 by analyzing the surrounding data and the display area 142, without considering the distance between the object and the display area 142. Note that, in the process of detecting the object, information indicating the distance between the object and the display area 142 may be used.

[0299] In this embodiment, for the purpose of simplifying the explanation, an example of a procedure in which the second calculation unit 840 analyzes a two-dimensional image and determines the relative positional relationship between the display area 142 and one or more objects will be described, taking as an example a case in which each of one or more objects is provided with a set of markers indicating the ends of each object in a substantially horizontal direction, and the markers are captured in a two-dimensional image acquired by the two-dimensional image acquisition unit 718. Note that a person skilled in the art who has read the description of this specification will understand that the relative positional relationship between the display area 142 and one or more objects can be determined even if no markers are captured in the two-dimensional image, as long as the positions of both ends of the one or more objects in a substantially horizontal direction can be determined.

[0300] In this embodiment, the edge detection unit 842 detects a plurality of markers included in the two-dimensional image by analyzing the two-dimensional image acquired by the two-dimensional image acquisition unit 718. Based on a combination of the detected plurality of markers, the edge detection unit 842 detects one edge and the other edge of each of one or more objects in the approximately horizontal direction.

[0301] In this embodiment, the occlusion angle calculation unit 844 calculates, for each of the one or more objects, an occlusion angle, which is the angle between the direction from the representative point of the display area 142 toward one end of each object and the direction from the representative point of the display area 142 toward the other end of each object. Based on the occlusion angles of each of the one or more objects, the occlusion angle calculation unit 844 calculates the proportion of the viewing angle of the display area 142 that is occluded by the one or more objects. If the occlusion angle of a first object and the occlusion angle of a second object overlap, the occlusion angle calculation unit 844 calculates the proportion that is occluded by the one or more objects without counting the overlap.

[0302] The marker position determining unit 812 may be an example of a two-dimensional image acquiring unit or a marker position determining unit.

[0303] 9 schematically illustrates an example of information processing in the first calculation unit 830. The distance calculation unit 834 and the angle calculation unit 836, for example, refer to the point cloud data or the distance data to obtain information about the vectors pointing from the representative point 250 to each of n points P1 to Pn included in the point cloud 910 representing a single object.

[0304] 9, vector 924 indicates a vector directed from representative point 250 toward point P1. The magnitude of vector 924 is dp1, and the angle formed between vector 924 and optical axis 252 is θP1. Similarly, vector 926 indicates a vector directed from representative point 250 toward point Pn. The magnitude of vector 924 is dpn, and the angle formed between vector 926 and optical axis 252 is θPn.

[0305] 10 schematically illustrates an example of information processing in the second calculation unit 840. In this embodiment, a marker 1022 is attached to one end of the shielding object 1020, and a marker 1024 is attached to the other end of the shielding object 1020. Similarly, a marker 1042 is attached to one end of the shielding object 1040, and a marker 1044 is attached to the other end of the shielding object 1040.

[0306] The shielding angle calculation unit 844 can calculate the shielding angle θ A of the shielding object 1020 using the markers 1022 and 1024. The shielding angle calculation unit 844 can calculate the shielding angle θ B of the shielding object 1040 using the markers 1042 and 1044.

[0307] 11 schematically illustrates an example of the internal configuration of the safety management unit 640. In this embodiment, the safety management unit 640 includes, for example, a derivation procedure determination unit 1130, an index value derivation unit 1140, and an evaluation unit 1150. In this embodiment, the derivation procedure determination unit 1130 includes, for example, an index setting unit 1132 and an evaluation setting unit 1134. In this embodiment, the index value derivation unit 1140 includes, for example, a first index derivation unit 1142, a second index derivation unit 1144, a third index derivation unit 1146, and a fourth index derivation unit 1148.

[0308] In this embodiment, the derivation procedure determination unit 1130 determines a derivation procedure, which is a procedure for the safety management unit 640 to derive the degree of viewability or the degree of safety. An example of the derivation procedure is the setting related to the safety evaluation of the user terminal 120 described above. The derivation procedure determination unit 1130 may determine the derivation procedure such that the derivation procedure is different when the elapsed time since the activation detection unit 622 detected the activation satisfies a predetermined condition from when the elapsed time does not satisfy the predetermined condition. As described above, an example of the predetermined condition is that the length of the elapsed time is shorter than a predetermined first threshold.

[0309] In this embodiment, the index setting unit 1132 determines the content of the index setting described above. The index setting unit 1132 may determine the content of the index setting in the first setting described above, or may determine the content of the index setting in the second setting described above. Examples of the content of the index setting include the type of index used in the evaluation process, a function or learning model for deriving the value of the index, the type of explanatory variable of the function or learning model, and the parameter value of the function or learning model. Any known evaluation function can be used as the function. The function may be a weighted linear sum.

[0310] The index setting unit 1132 determines the type of one or more indices to be used in the evaluation process, for example, based on (i) the state of the surroundings of the display area 142 indicated by the surrounding state information acquired by the surrounding state acquisition unit 610, and / or (ii) the current time or the time zone to which the current time belongs. The index setting unit 1132 may determine a function or a learning model for deriving the value of the index depending on the type of index. The index setting unit 1132 may determine parameter values ​​of the function or learning model depending on the type of function or learning model.

[0311] In one embodiment, the index setting unit 1132 analyzes image data output by the camera 154 to determine the type of index to be used in the evaluation process. The index setting unit 1132 may analyze the image data output by the camera 154 to determine whether to correct the index value or the evaluation value in the evaluation process. The index setting unit 1132 may determine the manner of correction or the degree of correction based on the analysis result.

[0312] For example, the index setting unit 1132 acquires image data output by the camera 154 from the structural information acquisition unit 612 or the physical property information acquisition unit 614. For example, when sunlight directly strikes the imaging element of the camera 154, the camera 154 outputs an image in which at least a portion is whited out. However, it is difficult for the safety management unit 640 to accurately determine the surrounding situation based solely on the whited-out image. Even in such a case, the safety management unit 640 may derive an evaluation value based on the surrounding data acquired by the surrounding data generation device 152 and / or various pieces of environmental information acquired by the environmental information acquisition unit 616, thereby improving the accuracy of deriving the evaluation value. Furthermore, the safety management unit 640 may correct the evaluation value in the whited-out region, thereby improving the accuracy of deriving the evaluation value.

[0313] More specifically, the index setting unit 1132 determines whether or not a whiteout region is included in the image data output by the camera 154. If it is determined that a whiteout region is included in the image data output by the camera 154, the index setting unit 1132 may determine to calculate the evaluation value using, of one or more indexes used in the evaluation process, an index that can be calculated without using the image data output by the camera 154. The index setting unit 1132 may correct the index value so that the degree of viewability in the whiteout region is increased.

[0314] The index setting unit 1132 may distinguish between a whiteout caused by sunlight incident on the imaging element and a whiteout caused by strong light being irradiated onto the surface of a white object. For example, the index setting unit 1132 calculates the positional relationship between the camera 154 and the sun from GPS information and / or the date and time when the camera 154 outputs the image data. The index setting unit 1132 may determine whether the whiteout in the image data is caused by sunlight incident on the imaging element from the positional relationship between the camera 154 and the sun.

[0315] In another embodiment, the index setting unit 1132 determines the type of index to be used in the evaluation process based on whether the area around the installation location is dark. The index setting unit 1132 may determine whether to correct the index value or the evaluation value in the evaluation process based on whether the area around the installation location is dark. The index setting unit 1132 may determine the manner of correction or the degree of correction.

[0316] For example, the index setting unit 1132 determines whether the area around the installation position is dark based on the illuminance data output by the illuminance sensor 156. The index setting unit 1132 may determine whether the area around the installation position is dark by taking into consideration environmental information other than illuminance, such as latitude and longitude, date and time, instead of or in addition to illuminance.

[0317] The correction methods described in connection with these embodiments are not particularly limited. For example, the degree of misrecognition can be improved by using an average value of multiple depth estimation results. Examples of multiple depth estimation results include a depth estimation result using a front image (e.g., an image output by the camera 154) and a depth estimation result using a top-down or left-right flipped image. In an ideal system, the depth estimated using the front image, the depth estimated using the left-right flipped image, and the depth estimated using the top-down flipped image are expected to be the same value. However, if an abnormality occurs in the image output by the camera 154, the depth estimated using the front image, the depth estimated using the left-right flipped image, and the depth estimated using the top-down flipped image will differ. Therefore, by estimating depth using a statistical value (e.g., an average value) of these values, the estimation accuracy and the stability of the estimation process can be improved.

[0318] In this embodiment, the evaluation setting unit 1134 determines the content of the evaluation setting described above. The evaluation setting unit 1134 may determine the content of the evaluation setting in the first setting described above, or may determine the content of the evaluation setting in the second setting described above. Examples of the content of the evaluation setting include the type of indicator used as an explanatory variable for deriving the evaluation value, a function or learning model for deriving the evaluation value, and parameter values ​​of the function or learning model. Any known evaluation function can be used as the function. The evaluation function may be a function that uses index values ​​of one or more indicators as explanatory variables and the evaluation value as a response variable. The function may be a weighted linear sum.

[0319] The evaluation setting unit 1134 determines a function or a learning model for deriving an evaluation value, for example, according to the type of index set by the index setting unit 1132. The evaluation setting unit 1134 determines a function or a learning model for deriving an evaluation value, for example, based on (a) the type of index set by the index setting unit 1132 and (b) (i) the state of the surroundings of the display area 142 indicated by the surrounding state information acquired by the surrounding state acquisition unit 610, and / or (ii) the current time or the time zone to which the current time belongs. The above function may be a weighted linear sum. The evaluation setting unit 1134 determines a weight value to be set for each of one or more indexes used in the evaluation process, for example, based on (i) the state of the surroundings of the display area 142 indicated by the surrounding state information acquired by the surrounding state acquisition unit 610, and / or (ii) the current time or the time zone to which the current time belongs.

[0320] The evaluation setting unit 1134 may determine the weighting values ​​to be set for one or more indicators used in the evaluation process, based on the environmental information acquired by the environmental information acquisition unit 616. This allows the control device 170 to appropriately set the weighting values ​​according to the surrounding environment of the user terminal 120.

[0321] For example, the evaluation setting unit 1134 determines the weighting value to be set for each of one or more indicators used in the evaluation process based on (i) the illuminance indicated by the illuminance information acquired by the environmental information acquisition unit 616, and / or (ii) the current time or the time zone to which the current time belongs. As described above, the current time or the time zone to which the current time belongs can be used as an indicator of the brightness around the user terminal 120. This allows the control device 170 to derive the above evaluation value based on the illuminance around the user terminal 120.

[0322] For example, there may be a significant difference in the exposure quality of the image captured by an optical camera between daytime and nighttime environments. As a result, the accuracy of deriving the evaluation value may decrease in certain environments. For example, in extremely dark places, the accuracy of deriving the evaluation value may decrease. Therefore, when the accuracy of deriving the evaluation value is relatively likely to decrease, the risk of information leakage can be further reduced by deriving the evaluation value so that the above-described use restriction process is executed. Examples of cases where the accuracy of deriving the evaluation value is relatively likely to decrease include when the illuminance indicated by the illuminance information is lower than a predetermined value, or when the illuminance predicted based on the current time or the time zone to which the current time belongs is lower than a predetermined value.

[0323] In this embodiment, the index value derivation unit 1140 derives the values ​​(sometimes referred to as index values) of one or more indexes related to the degree of viewability described above. The index value derivation unit 1140 derives the index value of each index in accordance with the content of the index setting determined by the index setting unit 1132. For example, the index value derivation unit 1140 derives index values ​​for the types of indexes determined by the index setting unit 1132.

[0324] (Indicator value of the first indicator) In this embodiment, the first index derivation unit 1142 derives the value of the first index described above. As described above, the first index indicates the degree to which object movement or light fluctuation is detected around the installation position of the display region 142. For example, if there is a large fluctuation in the light intensity of pixels corresponding to a specific area in the real world in video data captured around the installation position of the display region 142, there is a high possibility that an object has moved, deformed, or deteriorated in that area. Furthermore, by analyzing the microscopic fluctuation in light intensity of the pixels corresponding to the specific area, it can be determined, for example, whether natural light is irradiated onto that area, whether natural light is passing through that area, etc.

[0325] In this embodiment, the first index derivation unit 1142 derives the fluctuation in light intensity between frames by, for example, analyzing multiple frames included in the video data acquired by the physical property information acquisition unit 614. The first index derivation unit 1142 derives the value of the first index based on the fluctuation in light intensity. The value of the first index is derived based on, for example, statistics of the fluctuation in light intensity.

[0326] The first index derivation unit 1142 may derive a first index for the same video data by changing the conditions for deriving the first index, thereby deriving multiple types of first indexes from the same video data.

[0327] Examples of conditions for deriving the first index include (i) the shape and / or size of the calculation unit when dividing each frame of the video into a plurality of predetermined regions (sometimes referred to as calculation units), (ii) the position in the video data of the plurality of frames used to derive the first index and the number of such frames, and (iii) the elements used to derive the first index when the color of each pixel is specified by a plurality of elements. When each of the plurality of frames constituting the video data is a color image, the color of each pixel constituting the color image is specified by the respective values ​​of a plurality of elements for specifying the color. For example, when each frame is a color image in RGB format, the color of each pixel in each frame is specified by an R value, which is the value of the R element, a G value, which is the value of the G element, and a B value, which is the value of the B element.

[0328] The first index derivation unit 1142 outputs, for example, data (sometimes referred to as variation data) in which identification information for identifying each of one or more calculation units is associated with the value of the first index of each calculation unit. The first index derivation unit 1142 may output data (sometimes referred to as variation image data) in which the identification information of each of multiple calculation units and the value of the first index of each calculation unit are stored in image format. In this case, the identification information of each calculation unit indicates, for example, the x coordinate and y coordinate in the image. The variation image data may be an example of variation data.

[0329] In one embodiment, the first index derivation unit 1142 outputs multiple pieces of variation data having different shapes and / or sizes of calculation units. For example, using the same group of frames extracted from the same video data, the first index derivation unit 1142 outputs (i) variation image data generated based on the value of the first index derived for each square calculation unit consisting of 9 pixels, and (ii) variation image data generated based on the value of the first index derived for each square calculation unit consisting of 1024 pixels. Note that the shape of the calculation unit and the number of pixels constituting the calculation unit are not limited to the above example.

[0330] This allows the evaluation unit 1150 to apply different evaluation criteria to each of the multiple dynamic image data to evaluate the degree of safety of the display area 142. As a result, the degree of safety of the display area 142 can be evaluated with greater accuracy.

[0331] For example, in a case where the value of the first index is determined so that the greater the risk of information leakage, the larger the value of the first index, and if either the evaluation criterion "if the value of the first index derived for each square calculation unit consisting of 9 pixels is greater than a threshold, reduce the evaluation of the security of display area 142" or the evaluation criterion "if the value of the first index derived for each square calculation unit consisting of 1024 pixels is greater than a threshold" is applied, it is difficult to set the threshold, and as a result, it is also difficult to improve the accuracy of the evaluation.

[0332] In contrast, when the evaluation criterion that "if the value of the first index derived for each square calculation unit consisting of 9 pixels is greater than the first threshold, or if the value of the first index derived for each square calculation unit consisting of 1024 pixels is greater than the second threshold" is applied, setting of the threshold becomes easier than when either of the above evaluation criteria is applied. Also, by appropriately setting the threshold, the accuracy of the evaluation improves.

[0333] In another embodiment, the first index derivation unit 1142 outputs multiple pieces of variation data that differ in the positions and / or number of frames used to derive the first index. For example, the first index derivation unit 1142 uses the same group of frames extracted from the same video data to output (i) one piece of variation image data generated based on the value of the first index derived using all frames included in the group of frames, and (ii) one or more pieces of variation image data generated based on the value of the first index derived using some of the frames included in the group of frames.

[0334] For example, the first index derivation unit 1142 extracts the 150th to 399th frames included in the video data as a frame group used to derive the first index. The first index derivation unit 1142 derives the value of the first index for each calculation unit using the 150th to 399th frames and outputs basic variation image data. The first index derivation unit 1142 also derives the value of the first index for each calculation unit using the 150th to 199th frames and outputs other variation image data. Similarly, the first index derivation unit 1142 derives the value of the first index for each calculation unit using the 200th to 249th frames, the 250th to 299th frames, the 300th to 349th frames, and the 350th to 399th frames, respectively, and outputs another nine pieces of variation image data.

[0335] The smaller the number of frames (sometimes referred to as target frames) used in the calculation, the greater the impact that a sudden event that occurs during the imaging period of the target frames has on the value of the first index derived using the target frames. As described above, the frequency distribution of artificial light fluctuations often has a sharper shape than the frequency distribution of natural light fluctuations. The frequency distribution of the above fluctuations corresponds to the frequency distribution of light fluctuations described above.

[0336] As a result, the evaluation unit 1150 can determine the proportion of natural light contained in the light detected in each pixel or each calculation unit, for example, by comparing the frequency distribution of fluctuations when the number of target frames is small with the frequency distribution of fluctuations when the number of target frames is large. For example, if (i) the absolute value of the difference between the variances, standard deviations, or coefficients of variation of the frequency distributions is larger than a predetermined value, or (ii) the ratio of the variance, standard deviation, or coefficient of variation of the frequency distribution of fluctuations when the number of target frames is small to the variance, standard deviation, or coefficient of variation of the frequency distribution of fluctuations when the number of target frames is large is smaller than a predetermined value, the evaluation unit 1150 determines that the proportion of natural light is high.

[0337] In yet another embodiment, the first index derivation unit 1142 outputs multiple pieces of variation data in which different color elements are used to derive the first index. For example, when each frame is a color image in RGB format, the first index derivation unit 1142 uses the same group of frames extracted from the same video data to output (a) variation image data generated based on the value of the first index derived using at least two of the R value, the G value, and the B value, and (b) other variation image data generated based on (i) a combination different from the above combination, or (ii) a value of the first index derived using any one of the R value, the G value, and the B value.

[0338] The inventors have discovered that the frequency at which light fluctuations are emphasized varies depending on the type of subject or the surface condition of the subject. For example, the R value of light reflected from the surface of human skin fluctuates more than the G or B value of that light. Furthermore, the R value of light reflected from the surface of human skin fluctuates more than the R value of light reflected from the surface of a plant. Furthermore, because the sensitivity characteristics of human vision differ from those of an imaging device, the display device 140 outputs an image that has been color-calibrated according to the human visual characteristics. Therefore, the balance of the R, G, and B values ​​differs between light from an object displayed on a display device such as a monitor or screen and light from a real-world object.

[0339] As a result, the first index derivation unit 1142 derives a first index based on, for example, one or two of the R value, the G value, and the B value, thereby improving the accuracy with which the evaluation unit 1150 detects a specific type of object (for example, a human). Furthermore, the evaluation unit 1150 can improve the accuracy with which the evaluation unit 1150 detects a specific type of object (for example, a human) by, for example, comparing a first index derived based on a first combination of the R element, the G element, and the B element with a first index derived based on a second combination of the R element, the G element, and the B element. The first and second combinations differ in the RGB elements that constitute each combination. The first combination may be composed of one of the R element, the G element, and the B element, and the second combination may be composed of one of the R element, the G element, and the B element.

[0340] The first index derivation unit 1142 may derive the first index by changing at least two of the conditions for deriving the first index described above. The first index derivation unit 1142 may execute processing for correcting movement of the camera 154 itself on the video data acquired by the physical property information acquisition unit 614, and then execute processing for deriving the first index. Details of the information processing in the first index derivation unit 1142 will be described later.

[0341] (Index value of the second index) In this embodiment, the second index derivation unit 1144 derives an index value of the second index described above. As described above, the second index indicates the arrangement of obstructions present around the installation position of the display area 142. For example, the second index indicates the degree to which the obstructions block the line of sight of a third party to the display area 142. For example, the second index derivation unit 1144 analyzes the surrounding data acquired by the structural information acquisition unit 612 to derive the degree to which each of the obstructions arranged around the display area 142 blocks the line of sight of a third party.

[0342] The second index derivation unit 1144 may derive the degree to which each of the obstructing objects arranged around the display region 142 blocks the line of sight of a third party, based on the point cloud data or distance data acquired by the structural information acquisition unit 612. The second index derivation unit 1144 may calculate the distance between the subject and the camera 154 from the image data acquired by the structural information acquisition unit 612 to generate distance data, and use the distance data to derive the degree to which each of the obstructing objects arranged around the display region 142 blocks the line of sight of a third party. The second index derivation unit 1144 may derive the degree to which each of the obstructing objects arranged around the display region 142 blocks the line of sight of a third party, using the three-dimensional virtual space constructed by the object position determination unit 820.

[0343] The second index derivation unit 1144 derives the value of the second index based on, for example, the distance between each of the multiple points calculated by the distance calculation unit 834 and a representative point of the display area 142. The second index derivation unit 1144 may (i) derive the value of the second index for each of the multiple points, or (ii) after identifying each of one or more obstructing objects, derive the value of the second index for each obstructing object based on the distance between the multiple points on each obstructing object or the representative point of each obstructing object and a representative point of the display area 142.

[0344] The second index derivation unit 1144 derives the value of the second index based on, for example, the absolute value of the angle for each of the multiple points calculated by the angle calculation unit 836. The second index derivation unit 1144 may (i) derive the value of the second index for each of the multiple points, or (ii) identify each of one or more obstructing objects and then derive the value of the second index for each of the obstructing objects.

[0345] The second index derivation unit 1144 derives the value of the second index based on, for example, the distance for each of the multiple points calculated by the distance calculation unit 834 and the absolute value of the angle for each of the multiple points calculated by the angle calculation unit 836. The second index derivation unit 1144 may (i) derive the value of the second index for each of the multiple points, or (ii) identify each of one or more obstructing objects and then derive the value of the second index for each of the obstructing objects.

[0346] (Indicator value of the third indicator) In this embodiment, the third index derivation unit 1146 derives the index value of the above-described third index. As described above, the third index indicates the status of the surrounding environment of the installation position of the display area 142. The third index derivation unit 1146 derives the value of the third index based on, for example, the status of the surrounding environment indicated by the environmental information acquired by the environmental information acquisition unit 616.

[0347] In one embodiment, the third index derivation unit 1146 derives the value of the third index based on the illuminance indicated by the illuminance information acquired by the environmental information acquisition unit 616. The third index derivation unit 1146 may derive the value of the third index such that the value of the third index increases as the illuminance increases.

[0348] In another embodiment, the third index derivation unit 1146 may determine the magnitude of the visibility at the installation position of the display region 142 based on the state of the surrounding environment indicated by the environmental information acquired by the environmental information acquisition unit 616. For example, the third index derivation unit 1146 refers to a database that stores the state of the surrounding environment and the magnitude of the visibility in association with each other, and determines the magnitude of the visibility based on the environmental information acquired by the environmental information acquisition unit 616. The third index derivation unit 1146 may derive the value of the third index based on the magnitude of the visibility.

[0349] For example, the greater the visibility, the easier it is for the user of the information processing terminal to notice the presence of a third party in the vicinity. On the other hand, the greater the visibility, the easier it is for a third party to view information output by the information processing terminal. Therefore, the third index derivation unit 1146 may derive the value of the third index such that, for example, the greater the visibility, the greater the value of the third index. The third index derivation unit 1146 may derive the value of the third index such that, the greater the visibility, the smaller the value of the third index.

[0350] (Fourth indicator value) In this embodiment, the fourth index derivation unit 1148 derives the index value of the above-described fourth index. As described above, the fourth index indicates the risk of information leakage depending on the type of object or event. The fourth index derivation unit 1148 derives the value of the fourth index based on, for example, attribute information acquired by the attribute information acquisition unit 618. The fourth index derivation unit 1148 determines the value of the fourth index based on the type and / or attribute of the object or event actually observed, for example, by referring to a database in which, for each of one or more objects or events, the type and / or attribute of the object or event is associated with a value indicating the degree of likelihood that a third party will view the output image from the position where the object or event was observed.

[0351] As described above, the attribute information acquisition unit 618 detects an object by analyzing the image data output by the environment sensor 150. For example, the attribute information acquisition unit 618 detects the contour of the object in the image and determines the type and / or attribute of the detected object. However, when detecting an object using contour detection technology, it is difficult to determine whether the detected object is displayed on a display device such as a display or screen, or whether the object actually exists in real space.

[0352] In one embodiment, the fourth index derivation unit 1148 may determine or change the type and / or attribute of a detected object based on the position where the object is detected in an image. For example, when a first object and a second object are detected, and the position of the second object is inside the outline of the first object and the type or attribute of the first object is a display device such as a display or a screen, the fourth index derivation unit 1148 determines that the type and / or attribute of the second object is an image displayed on the display device. Furthermore, the fourth index derivation unit 1148 determines the value of the fourth index based on the determination result that the type and / or attribute of the second object is an image displayed on the display device. This allows the fourth index derivation unit 1148 to appropriately determine the value of the fourth index.

[0353] In another embodiment, the fourth index derivation unit 1148 may determine or change the type and / or attribute of the detected object based on the color of the detected object. Because the sensitivity characteristics of human vision differ from those of an imaging device, the display device outputs an image whose color has been calibrated according to the human visual characteristics. Therefore, the color detected by the camera 154 differs between when an object is displayed on the display device and when the object actually exists in real space. This allows the fourth index derivation unit 1148 to determine or change the type or attribute of the detected object based on the color of the detected object. Furthermore, the fourth index derivation unit 1148 determines the value of the fourth index based on the determined or changed type or attribute of the object. This allows the fourth index derivation unit 1148 to appropriately determine the value of the fourth index.

[0354] In this case, the fourth index derivation unit 1148 may determine or change the type and / or attribute of the detected object in cooperation with the first index derivation unit 1142. As described above, the first index derivation unit 1142 can derive a first index based on any combination of the R element, the G element, and the B element. Furthermore, the first index derivation unit 1142 can output a first index derived based on a first combination of the R element, the G element, and the B element, and a first index derived based on a second combination of the R element, the G element, and the B element. The fourth index derivation unit 1148 determines the type and / or attribute of the detected object based on the first index output by the first index derivation unit 1142, for example, by referring to a database in which, for each of one or more objects or events, (a) the type and / or attribute of the object or event is associated with (b) (i) information indicating the value of a first index derived from a specific combination of an R element, a G element, and a B element, or (ii) information indicating a comparison result between the value of the first index derived based on a first combination of an R element, a G element, and a B element and the value of the first index derived based on a second combination of the R element, a G element, and a B element. The fourth index derivation unit 1148 determines the value of the fourth index based on the determined type or attribute of the object. This allows the fourth index derivation unit 1148 to appropriately determine the value of the fourth index.

[0355] (Evaluation of degree of accessibility) In this embodiment, the evaluation unit 1150 evaluates the degree of browsability based on the value of one or more indexes derived by the index value derivation unit 1140. The evaluation unit 1150 may derive an evaluation value indicating the degree of browsability.

[0356] (comprehensive evaluation) In this embodiment, the evaluation unit 1150 derives the above evaluation value in accordance with the content of the evaluation setting determined by the evaluation setting unit 1134. The evaluation unit 1150 may derive the above evaluation value by inputting the index values ​​of one or more types of indexes determined by the index setting unit 1132 into an arbitrary function or learning model.

[0357] For example, the evaluation unit 1150 derives a weighted linear sum of the index values ​​of one or more types of indexes determined by the index setting unit 1132 based on the index values ​​of the one or more types of indexes and the weight values ​​set for each of the one or more indexes. The evaluation unit 1150 outputs the weighted linear sum as the evaluation value. This allows the evaluation unit 1150 to comprehensively evaluate the degree of viewability.

[0358] The method for comprehensively evaluating the degree of viewability is not limited to the above embodiment. In other embodiments, the evaluation unit 1150 may comprehensively evaluate the degree of viewability using a script including weighted logic calculations, conditional branching, and repetitive processing. In still other embodiments, the evaluation unit 1150 may comprehensively evaluate the degree of viewability using neural network technology. For example, the evaluation unit 1150 derives one or more digest values ​​that express the characteristics of the various index values ​​based on the various index values ​​output by the index value derivation unit 1140. The evaluation unit 1150 may derive the digest values ​​using a general-purpose script including various function processes, convolution processes, and the like. The evaluation unit 1150 receives one or more digest values ​​as input, prepares training data that outputs the results of the comprehensive evaluation, and executes a training process for an arbitrary learning model. The evaluation unit 1150 may comprehensively evaluate the degree of viewability using a trained learning model.

[0359] (Evaluation based on the first indicator) In this embodiment, the evaluation unit 1150 evaluates the degree of viewability based on the value of the first index. For example, the evaluation unit 1150 evaluates the degree to which object movement or light fluctuation is detected around the installation location of the user terminal 120, thereby evaluating the safety of the user terminal 120. For example, the evaluation unit 1150 evaluates the safety of the user terminal 120 higher the degree to which object movement or light fluctuation is detected around the installation location of the user terminal 120. The procedure for evaluating the degree of viewability based on the value of the first index will be described in detail below.

[0360] (Evaluation based on the second indicator) In this embodiment, the evaluation unit 1150 evaluates the degree of viewability based on the value of the second index. For example, the evaluation unit 1150 derives the evaluation value based on the positional relationship determined by the object position determination unit 820.

[0361] As described above, the degree of viewability of the output image is used as an index of the degree of safety against information leakage. The greater the degree of viewability of the output image, the more difficult it is to block the view of a third party with an obstruction. In other words, the greater the degree of viewability of the output image, the lower the degree of safety against information leakage.

[0362] In one embodiment, when comparing a case where the distance between the display area 142 and the obstruction is large with a case where the distance between the display area 142 and the obstruction is small, it is easier for a third party to get in between the display area 142 and the obstruction when the distance between the display area 142 and the obstruction is large. Furthermore, when the distance between the display area 142 and the obstruction is large, it becomes more difficult for the user 22 of the user terminal 120 to detect a third party peeking at the display area 142. In other words, it can be considered that the greater the distance between the display area 142 and the obstruction, the greater the degree of viewability.

[0363] On the other hand, if a third party is present between the display area 142 and an obstruction and is peeking at the output image, the greater the distance between the third party and the display area 142, the more difficult it becomes for the third party to understand the content of the output image. In other words, the greater the distance between the display area 142 and the obstruction, the less viewable the output image becomes.

[0364] According to another embodiment, when a third party views the display area 142 from an oblique angle, compared to when the third party views the display area 142 from the front, the third party can more easily understand the content of the output image when the third party views the display area 142 from the front. On the other hand, the user 22 is present in front of the display area 142. Therefore, it is considered relatively difficult for a third party to peek at the display area 142 from behind the user 22 in front of the display area 142.

[0365] Therefore, the method of deriving the degree of viewability can be set appropriately depending on the purpose and the situation. In one embodiment, the evaluation unit 1150 derives an evaluation value indicating the degree of viewability such that (i) the degree of viewability increases as an object is positioned farther away from the display area 142, and / or (ii) the degree of viewability decreases as an object is positioned closer to the front of the display area 142. In another embodiment, the evaluation unit 1150 derives an evaluation value indicating the degree of viewability such that (i) the degree of viewability increases as an object is positioned closer to the display area 142, and / or (ii) the degree of viewability increases as an object is positioned closer to the front of the display area 142.

[0366] (An embodiment in which the degree of viewability is derived based on the first calculation unit 830) In one embodiment, the second index derivation unit 1144 calculates a suppression index value for each of the plurality of points, the suppression index value indicating the degree to which each point suppresses third parties from viewing the output image, based on the distance for each of the plurality of points calculated by the distance calculation unit 834. The evaluation unit 1150 derives an evaluation value indicating the degree of viewability by summing the suppression index values ​​for each of the plurality of points. The suppression index may be an example of a second index.

[0367] In another embodiment, the second index derivation unit 1144 calculates a suppression index value for each of the multiple points, which is an index indicating the degree to which each point suppresses third parties from viewing the output image, based on the absolute value of the angle for each of the multiple points calculated by the angle calculation unit 836. The evaluation unit 1150 derives an evaluation value indicating the degree of viewability by summing the suppression index values ​​for each of the multiple points. The suppression index may be an example of a second index.

[0368] In yet another embodiment, the second index derivation unit 1144 calculates a suppression index value for each of the multiple points based on the distance for each of the multiple points calculated by the distance calculation unit 834 and the absolute value of the angle for each of the multiple points calculated by the angle calculation unit 836. The evaluation unit 1150 derives an evaluation value indicating the degree of viewability by summing the suppression index values ​​for each of the multiple points. The suppression index may be an example of a second index.

[0369] (An embodiment in which the degree of viewability is derived based on the calculation result of the second calculation unit 840) The evaluation unit 1150 may derive an evaluation value indicating the degree of viewability based on the occlusion angles of each of the one or more objects calculated by the occlusion angle calculation unit 844. For example, the evaluation unit 1150 may derive an evaluation value indicating the degree of viewability based on the proportion of the viewing angle of the display region 142 that is occluded by one or more objects. The evaluation unit 1150 may derive an evaluation value indicating the degree of viewability such that the greater the occlusion proportion, the smaller the degree of viewability.

[0370] (Evaluation based on the third indicator) In this embodiment, the evaluation unit 1150 evaluates the degree of viewability based on the value of the third index. The evaluation unit 1150 derives an evaluation value indicating the degree of viewability based on, for example, the value of the third index. The evaluation unit 1150 may derive an evaluation value indicating the degree of viewability based on the status of the surrounding environment indicated by the environmental information acquired by the environmental information acquisition unit 616. The evaluation unit 1150 may derive an evaluation value indicating the degree of viewability such that the greater the visibility, the smaller the degree of viewability. The evaluation unit 1150 may derive an evaluation value indicating the degree of viewability such that the smaller the visibility, the smaller the degree of viewability.

[0371] For example, the evaluation unit 1150 may derive an evaluation value indicating the degree of viewability such that the degree of viewability decreases as the illuminance increases. The evaluation unit 1150 may also derive an evaluation value indicating the degree of viewability such that the degree of viewability decreases as the illuminance decreases.

[0372] (Evaluation based on the fourth indicator) In this embodiment, the evaluation unit 1150 evaluates the degree of viewability based on the value of the fourth index. The evaluation unit 1150 derives an evaluation value indicating the degree of viewability, for example, based on the value of the fourth index. The evaluation unit 1150 may derive the evaluation value indicating the degree of viewability based on the type or attribute of an object or event indicated by the attribute information acquired by the attribute information acquisition unit 618. The evaluation unit 1150 may output the value of the fourth index as the evaluation value indicating the degree of viewability.

[0373] 12 schematically shows an example of information processing in the safety management unit 640. In this embodiment, an example of processing for evaluating the safety of the user terminal 120 will be described, taking as an example a case where the viewing angle of the display area 142 is smaller than the viewing angle of the peripheral data generation device 152 or the camera 154. More specifically, an example of processing for the safety management unit 640 to evaluate the safety of the user terminal 120 using an image 1200 will be described. The image 1200 shows an example of an image captured by the peripheral data generation device 152 or the camera 154 of the user terminal 120 placed at position A in the work room 300.

[0374] As described above, in this embodiment, the viewing angle of the display area 142 is smaller than the viewing angle of the peripheral data generating device 152 or the camera 154. Therefore, the size of the area corresponding to the viewing angle of the display area 142 (represented as a unit image 1240 in the drawing) is smaller than the size of the image 1200. The safety management unit 640 may evaluate safety by shifting the center 1242 of the image (represented as a unit image 1240 in the drawing) having a size corresponding to the viewing angle of the display area 142 in the left-right or up-down direction along a predetermined dotted line 1250. This allows the safety management unit 640 to evaluate safety in multiple installation directions using a single image 1200.

[0375] The position of the center 1242 of each unit image 1240 in the image 1200 corresponds to the installation orientation of the display region 142. In one embodiment, the shape and size of each unit image 1240 are adjusted according to the position of the unit image 1240 in the image 1200. For example, the shape and size of each unit image 1240 are adjusted so that the larger the distance between the position of the center 1242 of each unit image 1240 and the center of the image 1200, the larger the size of the unit image 1240 and the greater the degree of deformation of the shape. This allows the safety management unit 640 to evaluate the safety of the user terminal 120 in a plurality of patterns in which the installation orientation of the display region 142 is different, based on a single image data or a single three-dimensional data generated by the surrounding data generation device 152 or the camera 154.

[0376] In another embodiment, the safety management unit 640 may evaluate the safety of the user terminal 120 in a plurality of patterns in which the installation direction of the display area 142 is different, assuming that the shape and size of the area corresponding to the viewing angle of the display area 142 are the same regardless of the installation direction of the display area 142. This reduces the amount of calculation.

[0377] (An example of another embodiment) In the present embodiment, an example of the image 1200 and the safety management unit 640 has been described using the case where the viewing angle of the display area 142 is smaller than the viewing angle of the peripheral data generating device 152 or the camera 154 as an example. However, as described above, in other embodiments, the viewing angle of the display area 142 may be larger than the viewing angle of the peripheral data generating device 152 or the camera 154.

[0378] For example, the safety management unit 640 can acquire information beyond the viewing angle of the peripheral data generation device 152 or the camera 154 by combining multiple pieces of information about the surrounding environment of the installation position of the display area 142 acquired by the peripheral data generation device 152 or the camera 154. For example, the safety management unit 640 acquires multiple images of the surroundings of the installation position of the display area 142, with at least one of the imaging positions and imaging directions being different. The multiple images may be images captured by a single peripheral data generation device 152, or may be images captured by multiple peripheral data generation devices 152. The safety management unit 640 synthesizes the multiple images to generate a single image (sometimes referred to as a combined image). As a result, a combined image in which multiple images are synthesized may be generated as another example of the image 1200.

[0379] The imaging direction may be expressed in two dimensions or three dimensions. The safety management unit 640 may calculate the position, shape, and size of an area corresponding to the viewing angle of the display area 142 when the installation mode of the display area 142 is changed by analyzing the combined image using a procedure similar to that for the image 1200 in the above embodiment. Examples of the installation mode include the installation position, installation direction, and roll direction.

[0380] There are no particular limitations on the specific procedure for synthesizing a plurality of images to generate a combined image. The combined image may be, for example, a panoramic image obtained by synthesizing a plurality of images.

[0381] The safety management unit 640 may generate a three-dimensional virtual space (described below) based on multiple two-dimensional images or a combined image. However, if the overlapping portion of the two images to be combined is less than approximately 20% to 30% of each image, the number of features required for accurately combining the two images is small, making it difficult to accurately combine the two images. In particular, when generating a three-dimensional virtual space based on multiple two-dimensional images or a combined image, a misalignment of a few pixels in the two-dimensional image may lead to a large misalignment in the three-dimensional coordinates. For example, the above-mentioned misalignment may occur in the two-dimensional image if the image is captured out of focus, in a dark place, or in backlight.

[0382] Therefore, when the safety management unit 640 generates a three-dimensional virtual space based on multiple two-dimensional images or a combined image, the control device 170 preferably executes a process for reducing or correcting the combination error. Examples of the process for reducing or correcting the combination error include (a) a process for estimating the amount of movement of the peripheral data generation device 152 or the camera 154 while the peripheral data generation device 152 or the camera 154 is continuously capturing multiple images (e.g., while capturing a moving image) and sequentially combining the images while taking into account the amount of movement, and (b) (i) a process for combining the coordinate systems of the spatial structure in the three-dimensional virtual space, and then calculating the two-dimensional coordinates of each feature point when the three-dimensional model or three-dimensional point cloud is displayed in two dimensions, (ii) a process for determining whether the two-dimensional coordinates of each feature point displayed in two dimensions match the positions of each feature point in the two-dimensional image before combination, and (iii) a process for correcting the combination position so as to reduce the error between each feature point when it is determined that all or part of the feature points do not match.

[0383] The specific procedure for determining whether the two-dimensional coordinates of each feature point displayed in two dimensions match the position of each feature point in the two-dimensional image before merging is not particularly limited. For example, for structural data collected in two coordinate systems, a three-dimensional transformation matrix is ​​calculated so that the coordinates match using some transformation matrix. Then, the collected information is merged into one coordinate system using the three-dimensional transformation matrix. To calculate this transformation matrix, it is necessary to find multiple identical feature points in the two coordinate systems. Therefore, for example, a correct answer is inferred by inductive brute force for the pairing of the feature points. For example, a probabilistic method is used for the above inference.

[0384] As will be described later, when multiple markers are arranged in real space, the above pairing becomes easier by making the values ​​of each marker unique. Note that even if the values ​​of multiple markers are the same, the above-mentioned brute force inference process can be performed by finding unique features from the correlation between the roll directions and / or positions of paired markers.

[0385] In the present embodiment, the details of the information leakage prevention system 100 have been described using as an example a case where light fluctuation is detected to determine whether or not a third party is present around the installation position of the display area 142. However, the information leakage prevention system 100 is not limited to the present embodiment.

[0386] In another embodiment, the control device 170 may derive an index value indicating the degree of clutter of objects in the vicinity using the degree of detected light fluctuation as a precaution against hidden cameras. The control device 170 may determine safety based on the degree of clutter of objects. An example of a high degree of clutter is a state in which large light fluctuations are observed in the vicinity. A high degree of clutter makes it difficult to detect the presence of a hidden camera, for example. Therefore, for example, the control device 170 may calculate an index value such that the higher the degree of clutter, the lower the level of safety. By introducing a determination based on the degree of clutter, preventative determination becomes possible, and the number of locations with a high level of safety increases. On the other hand, for example, the mere presence of books, miscellaneous items, etc. scattered in the installation position of the display area 142 may restrict the location where the terminal can be used. Therefore, when the safety evaluation value of the user terminal 120 is derived based on the index value indicating the degree of clutter, the above-mentioned function for adjusting the sensitivity and / or responsiveness of the determination may be used.

[0387] Fig. 13 schematically illustrates an example of information processing in the control device 170. An example of a procedure in which the evaluation unit 1150 evaluates the degree of viewability based on the value of the first index will be described using Fig. 13. As described above, in this embodiment, the evaluation unit 1150 evaluates the degree to which object movement or light fluctuation is detected in the vicinity of the installation location of the user terminal 120, thereby evaluating the safety of the user terminal 120. In this embodiment, an example of a process in which the control device 170 acquires imaging condition data 1310 and video data 1320 from the camera 154 and generates dynamic image data 1340 will be described.

[0388] In this embodiment, the control device 170 acquires video data generated by the camera 154. The control device 170 analyzes the pixel values ​​of each pixel constituting each of the multiple frames included in the video data to derive fluctuations in the intensity of light reaching the camera 154 over a specific period of time.

[0389] Fluctuations in the intensity of light from a subject captured by camera 154 reflect (i) fluctuations in the refractive index of the space between the light source and the subject, (ii) fluctuations in objects disposed in the space between the light source and the subject, (iii) fluctuations in the refractive index of the space between the subject and camera 154, and / or (iv) fluctuations in objects disposed in the space between the subject and camera 154. The light from the subject may be reflected light reflected from the surface of the subject, transmitted light transmitted through the subject, or emitted light emitted by the subject. Examples of fluctuations in the object include fluctuations in at least one of the position, size, shape, and optical properties of the object. Examples of optical properties include refractive index, transmittance, reflectance, etc.

[0390] Therefore, the magnitude of the above-mentioned fluctuations in light intensity can be used as an index indicating the degree to which object movement or light fluctuations are detected around camera 154. The magnitude of the above-mentioned fluctuations in light intensity can also be used as an index indicating the degree of viewability described above. In other words, if the fluctuations in light intensity in a specific area in a frame are small, this indicates that a small object, such as a moving object, transmitting light, or reflecting light, is located at a position corresponding to that area in the real world.

[0391] For example, the control device 170 (a) divides each frame of the video into a plurality of predetermined regions (sometimes referred to as calculation units), and (b) for each of the plurality of calculation units, derives the value of an index (sometimes referred to as a fluctuation index) that indicates the fluctuation of the color and / or brightness of the region between a plurality of frames based on the pixel values ​​of the pixels included in each calculation unit. Examples of the fluctuation index include at least one of the variance, standard deviation, and coefficient of variation, and values ​​obtained by dividing these by a reference value. The fluctuation index may be an example of a first index.

[0392] Each of the multiple calculation units includes one or more pixels. The larger the number of pixels included in each calculation unit, the smaller the load for calculating the light intensity fluctuations and the faster the calculation speed. On the other hand, the smaller the number of pixels included in each calculation unit, the better the accuracy or resolution.

[0393] At least two computational units included in a single frame may differ in at least one of size and shape. For example, the size of the computational unit displayed near the center of the screen may differ from the size of the computational unit displayed on the periphery of the screen. For example, the shape of the computational unit displayed near the center of the screen may differ from the shape of the computational unit displayed on the periphery of the screen.

[0394] (Configuration of imaging condition data 1310) In this embodiment, the image capturing condition data 1310 includes information indicating an image capturing position. The image capturing condition data 1310 may further include information indicating at least one of an image capturing direction and a focal length of the camera 154.

[0395] If the focal length f of camera 154 is known, control device 170 can calculate the angle between the direction of the optical axis of camera 154 and the direction from representative point 250 of camera 154 toward position Q in the real world that corresponds to position q, based on the positional relationship between a specific position q in the image and position o of the optical axis in the image. In the camera coordinate system of camera 154, the coordinates of position o are expressed as (0, 0, f), for example, the coordinates of position q are expressed as (x, y, f), and the coordinates of position Q are expressed as (X, Y, Z), for example.

[0396] Note that, when the installation position and installation direction of the display area 142 have been determined, the imaging condition data 1310 does not need to include information indicating the imaging position and information indicating the imaging direction. For example, when the user 22 installs the user terminal 120 in a specific position and direction and the degree of viewability at that position and direction is verified, the imaging condition data 1310 does not need to include information indicating the imaging position and information indicating the imaging direction. Furthermore, when the focal length f of the camera 154 is known, the imaging condition data 1310 does not need to include information indicating the focal length of the camera 154.

[0397] (Video data 1320 configuration) In this embodiment, the video data 1320 includes a plurality of frames, including a frame 1322, a frame 1324, a frame 1332, and a frame 1334. Each frame included in the video data 1320 may be a two-dimensional image.

[0398] The data structure of a frame that constitutes the video data 1320 includes, for example, identification information for each pixel that constitutes the frame, information indicating the position of each pixel in the frame, and information indicating the brightness of each pixel. i The data includes, for example, xi indicating the position of the frame in the x-direction on the image plane, yi indicating the position of the frame in the y-direction on the image plane, and a pixel value indicating brightness. Brightness may be expressed as luminance or value. If the frame is a color image expressed in RGB format, the pixel value indicating brightness includes, for example, information indicating the brightness value of each RGB. In other words, the color of each pixel can be identified by the information indicating the brightness of each pixel.

[0399] (Configuration of the dynamic image data 1340) In this embodiment, the dynamic image data 1340 has multiple frames, including frame 1342 and frame 1344. Each of the multiple frames constituting the dynamic image data 1340 is generated based on the data of multiple frames included in the moving image data 1320. As described above, in this embodiment, each frame of the moving image is divided into multiple predetermined calculation units, and a dynamic index value is derived for each calculation unit. Therefore, each calculation unit of the frames constituting the moving image data 1320 corresponds to each pixel of the frames constituting the dynamic image data 1340.

[0400] The data structure of a frame that constitutes the variation image data 1340 includes, for example, identification information of each pixel that constitutes the frame, information indicating the position of each pixel in the frame, and information indicating the degree of variation in brightness of each pixel. For example, the i-th pixel VP iThe data includes xi indicating the position in the x direction on the image plane of the frame, yi indicating the position in the y direction on the image plane, and a pixel value indicating the degree of brightness fluctuation at the pixel.

[0401] The pixel value may be the value of the fluctuation index described above. If the frame is a color image represented in RGB format, three values ​​indicating the degree of fluctuation in the brightness of each of RGB may be stored as the pixel value, or a single value indicating the degree of fluctuation in the total brightness of each of RGB may be stored.

[0402] In one embodiment, when a general camera 154 captures an image in an extremely dark place, the evaluation accuracy is improved by using the B luminance data of the luminance of each of RGB, or by setting the weight of the B luminance data to be greater than the weight of the R and G luminance data. In another embodiment, when the subject of camera 154 is human skin, the evaluation accuracy is improved by using the R luminance data, or by setting the weight of the R luminance data to be greater than the weight of the B and G luminance data. In yet another embodiment, when the subject of camera 154 is a plant, the evaluation accuracy is improved by using the G luminance data, or by setting the weight of the G luminance data to be greater than the weight of the B and R luminance data.

[0403] (Configuration of control device 170) In this embodiment, the control device 170 includes a physical property information acquisition unit 614, a fluctuation index derivation unit 1374, an image generation unit 1376, and a safety management unit 1378. The fluctuation index derivation unit 1374 is disposed in the first index derivation unit 1142. The safety management unit 1378 may be another example of the safety management unit 640. Details of the safety management unit 1378 will be described later.

[0404] The fluctuation index derivation unit 1374 derives the value of the above-described fluctuation index, thereby deriving the degree of fluctuation in the intensity of light reaching the camera 154 during a specific period. The fluctuation index derivation unit 1374 derives the value of the fluctuation index, for example, by the following procedure.

[0405] According to this embodiment, the fluctuation index derivation unit 1374 first extracts, in accordance with a predetermined rule, a plurality of frames from the video data acquired by the physical property information acquisition unit 614. The fluctuation index derivation unit 1374 may extract a plurality of frames from the video data acquired by the physical property information acquisition unit 614 that was captured while the camera 154 was stationary.

[0406] Examples of the predetermined rule include (i) a rule that a predetermined number of consecutive frames are extracted, (ii) a rule that a predetermined number of frames are extracted from frames corresponding to a period of a predetermined length (sometimes referred to as a unit period), and (iii) a rule that a predetermined second number of frames (the second number being less than the first number) are extracted from a predetermined first number of consecutive frames. For example, according to rule (i), 10 consecutive frames are extracted. On the other hand, according to rule (ii) or (iii), for example, one frame is extracted every three frames, for a total of 10 frames are extracted.

[0407] Next, the fluctuation index derivation unit 1374 divides each of the extracted frames into a plurality of calculation units. The fluctuation index derivation unit 1374 may divide each of the extracted frames into a plurality of calculation units according to a predetermined rule. The shapes of the plurality of calculation units may all be the same, some may be different, or all may be different. The sizes of the plurality of calculation units may all be the same, some may be different, or all may be different.

[0408] Next, the fluctuation index derivation unit 1374 derives a fluctuation index value for each of the plurality of calculation units. The fluctuation index value for each calculation unit is determined based on, for example, the pixel values ​​of the pixels included in each calculation unit.

[0409] For example, if camera 154 was stationary during the period in which the extracted frames were captured, the first calculation unit of the first frame and the first calculation unit of the second frame correspond to the same position in the real world. In this case, the first calculation unit of the first frame and the first calculation unit of the second frame may be said to correspond to each other. This relationship may also be referred to as a correspondence relationship between calculation units between frames.

[0410] Therefore, the fluctuation index derivation unit 1374 first sums the pixel values ​​of the pixels included in each calculation unit in each frame. This obtains the sum of the pixel values ​​for each calculation unit in each of the multiple frames. Next, the fluctuation index derivation unit 1374 calculates an index indicating the fluctuation of the above sum between the multiple frames for each calculation unit (sometimes referred to as performing statistical processing). As described above, examples of the index indicating the fluctuation include at least one of the variance, standard deviation, and coefficient of variation, and values ​​obtained by dividing these by a reference value. The fluctuation index derivation unit 1374 associates information indicating the value of the fluctuation index for each calculation unit with identification information for each calculation unit, and outputs the information to the image generation unit 1376.

[0411] The fluctuation index derivation unit 1374 may analyze the extracted multiple frames to determine whether or not the camera 154 moved during the period in which the multiple frames were captured. If it is determined that the camera 154 moved during the period, the fluctuation index derivation unit 1374 first determines the correspondence between the calculation units among the multiple frames.

[0412] For example, the fluctuation index derivation unit 1374 first determines a first frame of multiple frames as a reference frame. Next, the fluctuation index derivation unit 1374 identifies a region corresponding to the first region of the first frame from multiple regions of the second frame. The same process is repeated for the other frames. The same process is also repeated for the second region of the first frame. In this way, the correspondence between the calculation units between frames is determined. Next, the fluctuation index derivation unit 1374 performs the above-mentioned statistical process between corresponding calculation units, taking into account the correspondence between the calculation units between frames. In this way, the value of the fluctuation index for each calculation unit is derived.

[0413] In this embodiment, the image generation unit 1376 generates each frame (sometimes referred to as a fluctuation image) of the fluctuation image data 1340. The image generation unit 1376 may generate each frame of the fluctuation image data 1340 based on the value of each fluctuation index of the multiple calculation units derived by the fluctuation index derivation unit 1374 and a predetermined number of gradations.

[0414] In one embodiment, the image generation unit 1376 generates a variation image such that each of the plurality of computational units constitutes a pixel of the variation image. For example, the image generation unit 1376 first determines the position of each computational unit in the variation image based on the positional relationship of the plurality of computational units in a single frame (sometimes referred to as a normal image) of the video data 1320. The variation index derivation unit 1374 may determine the position of each computational unit in the variation image so that the positional relationship of the plurality of computational units in the normal image is reflected.

[0415] Next, the variation index derivation unit 1374 associates identification information of each calculation unit, information indicating the position of each calculation unit in the variation image, and information indicating the value of the variation index for each calculation unit, to generate each frame of the variation image data 1340. The information indicating the value of the variation index may be information indicating a continuous numerical value or information indicating a gradational division. If the number of gradations of the variation image is predetermined, the variation index derivation unit 1374 may determine the pixel value of each pixel based on the value of the variation index for each pixel of the variation image and the number of gradations.

[0416] The fluctuation index derivation unit 1374, for example, directly derives a fluctuation index from various degrees of fluctuation (sometimes referred to as primary derivation). The fluctuation index derivation unit 1374 may generate a new fluctuation index based on several derived fluctuation indexes using a narrowing filter operation according to the characteristics and purpose (sometimes referred to as secondary derivation). Examples of the filter operation include (i) logical calculations such as sum, product, and logical sum, and (ii) conditional branching. The content of the filter operation is determined, for example, based on the basis for extracting each of the multiple fluctuation indexes. Examples of the basis for extracting the multiple fluctuation indexes include the type of sensor used to calculate the fluctuation index, the aggregation conditions used to calculate the fluctuation index, and the RGB elements or frequencies used to calculate the fluctuation index. A secondary derivation may be performed based on the result of an already calculated primary derivation, or a further secondary derivation may be performed based on the result of an already calculated secondary derivation.

[0417] As an example of secondary derivation, in one embodiment, when camera 154 captures an image of an extremely dark place, the extremely dark area in the image can be detected by extracting pixels that satisfy the condition 130>B>G>R. The extremely dark area may be the entire image or a part of the image. Since the calculated fluctuation index tends to be small in an extremely dark area, a filter calculation process is performed to correct the value of the fluctuation index in the extremely dark area. This can improve the accuracy of deriving the fluctuation index.

[0418] In another embodiment, when the camera 154 captures images of a light source and the sun, pixels in the exposed whiteout area can be detected by extracting pixels that satisfy the condition, for example, color = ALL 255 AND deviation = 0. In this case, accuracy can be improved by substituting a variation index with another variation index by weighting it.

[0419] In yet another embodiment, for example, if slight vibrations occur in the user terminal 120 due to typing on the keyboard, vibrations in the camera 154 will also appear in the standard deviation. In this case, for example, as an example of the filter operation described above, a fluctuation index is generated by calculating the product or AND of the fluctuation indexes of two deviations. Examples of the two deviations are 0.5 seconds and 0.1 seconds. This can prevent a decrease in evaluation accuracy due to unnecessary vibrations.

[0420] In yet another embodiment, the image generation unit 1376 generates the variation image such that each of the plurality of pixels included in each calculation unit constitutes a pixel of the variation image. For example, the image generation unit 1376 first determines the pixel value of each of the one or more pixels included in each of the plurality of calculation units. The image generation unit 1376 may determine the value of the variation index for each calculation unit as the value of the variation index for each pixel included in the calculation unit.

[0421] Next, the fluctuation index derivation unit 1374 associates the identification information of each pixel with information indicating the position of each pixel in the fluctuation image and information indicating the value of the fluctuation index for each pixel, thereby generating each frame of the fluctuation image data 1340. The information indicating the value of the fluctuation index may be information indicating a continuous numerical value or information indicating a gradational division. If the number of gradations of the fluctuation image is predetermined, the fluctuation index derivation unit 1374 may determine the pixel value of each pixel based on the value of the fluctuation index for each pixel of the fluctuation image and the number of gradations.

[0422] As described above, there are cases where the physical property information acquisition unit 614 acquires multiple pieces of moving image data 1320 captured in different directions. In this case, the fluctuation index derivation unit 1374 may generate fluctuation image data 1340 corresponding to each of the multiple pieces of moving image data 1320.

[0423] In this embodiment, the safety management unit 1378 evaluates the safety of the display area 142. The safety management unit 1378 may evaluate the safety of the display area 142 by deriving the degree of viewability of the output image by a third party different from the user 22.

[0424] The safety management unit 1378 evaluates the safety of the display area 142, for example, when the display area 142 is arranged at the imaging position of the camera 154. The safety management unit 1378 may evaluate the safety of the display area 142 when the display area 142 is arranged at the imaging position of the camera 154 so that the installation direction of the display area 142 substantially coincides with the imaging direction of the camera 154. For example, the safety management unit 1378 determines that the safety of the display area 142 is higher the lower the risk of information displayed in the display area 142 being leaked. Examples of the safety of the display area 142 include the safety of the installation position of the display area 142 and the safety of the installation position and installation direction of the display area 142. When the display area 142 is housed in the housing 130, the safety of the display area 142 may mean the safety of the user terminal 120.

[0425] In this embodiment, the safety management unit 1378 differs from the safety management unit 640 in that the safety management unit 1378 evaluates the safety of the display area 142 using the fluctuation index derived by the fluctuation index derivation unit 1374 or the fluctuation image generated by the image generation unit 1376. Except for the above differences, the safety management unit 1378 may have the same configuration as the safety management unit 640. Details of the safety management unit 1378 will be described later.

[0426] Each frame of the dynamic image data 1340 may be an example of a dynamic image. The image generation unit 1376 may be an example of a dynamic image generation unit. The safety management unit 1378 may be an example of a viewability derivation unit.

[0427] 14 schematically illustrates an example of the generation process of the dynamic image data 1340. In this embodiment, the physical property information acquisition unit 614 includes a ring buffer 1472, and data of each frame included in the video data 1320 is stored in each of a plurality of queues of the ring buffer 1472 in order.

[0428] In this embodiment, for example, data of frames 1322 to 1332, which are normal images, is input to the fluctuation index derivation unit 1374, and the fluctuation index derivation unit 1374 derives the value of an index (sometimes referred to as a fluctuation index, as described above) that indicates a fluctuation in color and / or brightness between the input frames. Furthermore, the image generation unit 1376 generates data of frame 1342, which is a fluctuation image, based on the value of the fluctuation index derived by the fluctuation index derivation unit 1374. Similarly, data of frames 1324 to 1334 is input to the fluctuation index derivation unit 1374, and the image generation unit 1376 generates frame 1344. The fluctuation index derivation unit 1374 and the image generation unit 1376 repeat similar processes to generate fluctuation image data 1340.

[0429] For the sake of simplicity, the present embodiment has described an example of the process for generating the variation image data 1340, taking as an example a case where pixel values ​​of pixels in the entire region of each frame from frame 1322 to frame 1332 are input to the variation index derivation unit 1374, and the image generation unit 1376 generates frame 1342 corresponding to the entire region. However, the process for generating the variation image data 1340 is not limited to this embodiment.

[0430] In another embodiment, pixel values ​​of pixels in a partial region of each frame are input to the fluctuation index derivation unit 1374, and the image generation unit 1376 generates a frame corresponding to the partial region. Examples of normal images (sometimes referred to as target images) that are input to the fluctuation index derivation unit 1374 and that are the target of the fluctuation image generation process include each of the multiple unit images 1240 described in relation to FIG. 12 .

[0431] Fig. 15 schematically illustrates an example of the internal configuration of fluctuation index derivation unit 1374. An example of correction processing when camera 154 moves during a period in which multiple frames input to fluctuation index derivation unit 1374 are captured will be described using Fig. 15. In this embodiment, fluctuation index derivation unit 1374 includes fluctuation index calculation unit 1522, movement degree determination unit 1524, and correction unit 1526.

[0432] In this embodiment, for example, data of multiple frames that constitute a target image is input to the fluctuation index calculation unit 1522, and the fluctuation index calculation unit 1522 calculates the fluctuation index value for each of multiple calculation units included in the target image. The fluctuation index calculation unit 1522 may calculate the fluctuation index value according to a procedure similar to the procedure described in relation to Figures 13 and 14.

[0433] In this embodiment, the movement degree determination unit 1524 determines the degree of movement (sometimes referred to as movement degree) of the camera 154 during the period in which the above-mentioned multiple frames were captured. Examples of the movement degree include the direction and amount of movement. For example, the movement degree determination unit 1524 analyzes the video data 1320 and determines the movement degree of the camera 154 that captured the video data 1320 based on the movement degree of feature points in the images between the above-mentioned multiple frames.

[0434] In this embodiment, the correction unit 1526 corrects the value of the fluctuation index of each of the multiple calculation units based on the degree of movement of the camera 154 determined by the movement degree determination unit 1524. For example, the correction unit 1526 determines whether the camera 154 has moved during the above-mentioned period based on the degree of movement determined by the movement degree determination unit 1524.

[0435] If it is determined that camera 154 has not moved during the above period, correction unit 1526 determines that correction is unnecessary. In this case, fluctuation index derivation unit 1374 outputs the value calculated by fluctuation index calculation unit 1522 as the value of the fluctuation index.

[0436] On the other hand, if it is determined that camera 154 has moved during the above period, correction unit 1526 determines that correction is necessary. For example, correction unit 1526 first determines the correspondence between calculation units among the multiple frames input to fluctuation index derivation unit 1374. Next, correction unit 1526 performs the statistical processing described above between corresponding calculation units, taking into account the correspondence between the calculation units among the frames, and calculates the value of the fluctuation index for each calculation unit. In this case, fluctuation index derivation unit 1374 outputs the value calculated by correction unit 1526 as the value of the fluctuation index.

[0437] (An example of another embodiment) In the present embodiment, an example of the correction process has been described using as an example a case where the fluctuation index calculation unit 1522 and the correction unit 1526 calculate the value of the fluctuation index. However, the correction process is not limited to this embodiment.

[0438] In another embodiment, the fluctuation index calculation unit 1522 may determine the calculation unit for each frame to be subjected to statistical processing depending on whether or not the camera 154 has moved. For example, before the fluctuation index calculation unit 1522 calculates the fluctuation index, the correction unit 1526 determines whether or not correction is necessary. If it is determined that correction is not necessary, the fluctuation index calculation unit 1522 performs statistical processing between multiple calculation units arranged at the same position in each frame. On the other hand, if it is determined that correction is necessary, the fluctuation index calculation unit 1522 determines the correspondence relationship between the calculation units among multiple frames, taking into account the degree of movement of the camera 154. The fluctuation index calculation unit 1522 performs statistical processing between multiple corresponding calculation units.

[0439] The safety management unit 1378 will be described in detail with reference to Figures 16, 17, 18, and 19. Figure 16 schematically shows an example of the internal configuration of the safety management unit 1378. Figure 17 schematically shows an example of information processing in the safety management unit 1378. Figure 18 schematically shows an example of a two-dimensional image 1800 that is a normal image in which a marker is captured. Figure 19 schematically shows an example of a panoramic image 1900 in which a plurality of dynamic images are combined.

[0440] 16 schematically illustrates an example of the internal configuration of the safety management unit 1378. In this embodiment, the safety management unit 1378 includes, for example, a derivation procedure determination unit 1130, an index value derivation unit 1140, an evaluation unit 1150, and an evaluation target determination unit 1620. In this embodiment, the evaluation target determination unit 1620 includes, for example, a variation image acquisition unit 1622, a combined image generation unit 1624, a unit image extraction unit 1626, a marker detection unit 1632, a pixel value modification unit 1634, an imaging position determination unit 1636, and an imaging direction determination unit 1638.

[0441] The safety management unit 1378 differs from the safety management unit 640 in that it includes an evaluation target determination unit 1620. Except for the above differences, the safety management unit 1378 may have the same configuration as the safety management unit 640.

[0442] As described above, when the display area 142 and the camera 154 are housed in the same housing 130, the imaging position of the camera 154 approximately coincides with the installation position of the display area 142 or the user terminal 120. Similarly, the imaging direction of the camera 154 approximately coincides with the installation direction of the display area 142 or the user terminal 120.

[0443] In this embodiment, the evaluation target determination unit 1620 determines a target image to be evaluated for safety from the dynamic image data 1340. As described above, the target image may be an entire frame included in the dynamic image data 1340, or a part of the frame.

[0444] In this embodiment, the variation image acquisition unit 1622 acquires one or more variation images generated by the image generation unit 1376. When the camera 154 captures multiple videos with different imaging directions, the variation image acquisition unit 1622 may acquire multiple variation images for each of the multiple videos.

[0445] In this embodiment, the combined image generation unit 1624 generates a combined image by panoramic synthesis of the plurality of variation images generated by the image generation unit 1376. The panoramic image 1900 described in relation to Fig. 19 may be an example of a combined image.

[0446] In this embodiment, the unit image extracting unit 1626 extracts one or more unit images to be evaluated from the varying image or the combined image. Each unit image corresponds to the viewing angle of the display area 142 when the display area 142 is arranged at a specific position with a specific orientation. The unit image extracting unit 1626 may extract multiple unit images with different specific orientations.

[0447] In this embodiment, the marker detection unit 1632 detects a marker reflected in a video that is the source of the dynamic image. As described above, the marker may include information indicating the degree of viewability. The marker may include information indicating the degree of viewability and information indicating the size of an area to which the degree of viewability applies. As described above, examples of the marker include an AR marker, a QR code (registered trademark), and a barcode. Examples of the AR marker include ArUco and Chameleon Code, but the AR marker is not limited to these. For example, a QR code (registered trademark) and a barcode may be used as the AR marker.

[0448] In this embodiment, the pixel value modification unit 1634 modifies pixel values ​​of the dynamic image when the marker detection unit 1632 detects a marker. The pixel value modification unit 1634 may modify the pixel values ​​of the dynamic image based on information indicated by the detected marker. When the marker is a QR code (registered trademark) or a barcode, the pixel value modification unit 1634 may acquire the information indicated by the detected marker by referring to a database that stores identification information of the marker and the content of information processing when the marker is detected in association with each other.

[0449] For example, the pixel value modification unit 1634 determines the value of the variation index corresponding to an area in which a marker is captured among multiple areas of the normal image, based on information indicating the degree of viewability indicated by the marker. When the detected marker indicates the size of an area to which the degree of viewability applies, the pixel value modification unit 1634 may identify the area to which the degree of viewability applies among multiple areas of the normal image, based on the information indicated by the marker. The pixel value modification unit 1634 determines the pixel values ​​of pixels included in an area in the variation image or unit image corresponding to the above area, based on the information indicating the degree of viewability indicated by the detected marker. For example, the pixel value modification unit 1634 changes the pixel values ​​of the above pixels in the variation image or unit image to a value corresponding to the degree of viewability indicated by the detected marker.

[0450] In this embodiment, the imaging position determination unit 1636 determines the imaging position of the target image. The imaging position determination unit 1636 determines the imaging position of the target image based on the imaging condition data 1310, for example.

[0451] In this embodiment, the imaging direction determination unit 1638 determines the imaging direction of the target image. When the center point of the target image coincides with the position of the optical axis of the camera 154 in the dynamic image from which the target image is extracted, the imaging direction determination unit 1638 determines the imaging direction of the target image, for example, based on the imaging condition data 1310. On the other hand, when the center point of the target image does not coincide with the position of the optical axis of the camera 154 in the dynamic image from which the target image is extracted, the imaging position determination unit 1636 determines the imaging direction of the target image, for example, based on the position of the optical axis of the camera 154 in the dynamic image from which the target image is extracted, the position of the center point of the target image in the dynamic image from which the target image is extracted, the focal length of the camera 154, and the imaging condition data 1310.

[0452] In this embodiment, the evaluation unit 1150 derives the degree of viewability of the output image by a third party other than the user of the camera 154 when the display area 142 is placed at a position where the video was captured by the camera 154. The degree of viewability may be an example of an index indicating safety.

[0453] In this embodiment, the evaluation unit 1150 derives an evaluation value indicating the degree of viewability based on the values ​​of the variation indices for each of the multiple regions derived by the variation index derivation unit 1374. The pixel values ​​of the target image determined as the evaluation target by the evaluation target determination unit 1620 indicate the values ​​of the variation indices for each of the multiple regions derived by the variation index derivation unit 1374.

[0454] Therefore, in this embodiment, the evaluation unit 1150 derives the degree of viewability based on the target image. The evaluation unit 1150 may derive an evaluation value indicating the degree of viewability for each of the multiple target images.

[0455] In one embodiment, as described above, the target image may be a plurality of variable images or unit images captured in different imaging directions, thereby deriving an evaluation value indicating the degree of viewability in each of the plurality of imaging directions.

[0456] In another embodiment, the target image may be a fluctuating image or a unit image at multiple points in time while the user 22 is continuously using the camera 154. This allows the evaluation unit 1150 to monitor the safety of the user terminal 120 while the user 22 is using the user terminal 120.

[0457] The evaluation unit 1150 may calculate, for each of the multiple regions, a viewing index value that indicates the degree of likelihood that the output image will be viewed from a position in real space corresponding to each of the multiple regions, based on the value of the variation index for each of the multiple regions. The evaluation unit 1150 may derive an evaluation value that indicates the degree of viewability, based on the value of the viewing index for each of the multiple regions. In one embodiment, the evaluation unit 1150 derives the evaluation value that indicates the degree of viewability by summing the values ​​of the viewing index for each of the multiple regions. In another embodiment, the evaluation unit 1150 derives the evaluation value that indicates the degree of viewability by calculating the average value of the viewing index for each of the multiple regions adjacent to a specific region (the value obtained by dividing the sum of the values ​​of the viewing index for each region by the number of regions).

[0458] In one embodiment, the evaluation unit 1150 calculates the value of the viewing index such that the value of the viewing index in an area with a large variation index value is larger than the value of the viewing index in an area with a small variation index value. In another embodiment, the evaluation unit 1150 calculates the value of the viewing index such that the value of the viewing index in an area close to the position of the optical axis of the camera 154 in the video is larger than the value of the viewing index in an area far from the position of the optical axis of the camera 154 in the video. In yet another embodiment, the evaluation unit 1150 calculates the value of the viewing index such that the value of the viewing index in an area close to the horizon is larger than the value of the viewing index in an area far from the horizon.

[0459] 17 is a schematic diagram illustrating an example of information processing in the safety management unit 1378. An example of a procedure in which the safety management unit 1378 evaluates the safety of the display area 142 will be described with reference to FIG. 17. As described above, the safety management unit 1378 evaluates the safety of the display area 142 using the fluctuation index derived by the fluctuation index derivation unit 1374 or the fluctuation image generated by the image generation unit 1376.

[0460] According to this embodiment, first, in S1722, the safety management unit 1378 acquires one or more variation images and the imaging positions and imaging directions corresponding to each variation image. For example, the safety management unit 1378 acquires variation image data 1340 and imaging condition data 1310 indicating the imaging conditions when the video data 1320, which is the source of the variation image data 1340, was captured.

[0461] Next, in S1724, the safety management unit 1378 determines whether or not there is an event for changing the pixel value of the variation image. If an event for changing the pixel value of the variation image is detected, the safety management unit 1378 changes the pixel value of the variation image according to the detected event.

[0462] An example of an event for changing the pixel values ​​of the dynamic image is when a predetermined type of marker is detected in a frame of the video data 1320 that is the source of a frame included in the dynamic image data 1340. For example, the safety management unit 1378 analyzes the frame of the video data 1320 to determine whether or not a predetermined type of marker is present. Examples of the marker include an AR marker, a QR code (registered trademark), and a barcode. Examples of the AR marker include ArUco and Chameleon Code, but the AR marker is not limited to these. As described above, for example, a QR code (registered trademark) and a barcode may be used as the AR marker.

[0463] According to this embodiment, the marker includes information indicating the degree of safety of the position where the marker is attached or the object where the marker is attached, or includes information associated with the information indicating the degree of safety (for example, identification information of the marker).The marker may include information regarding the range of effect of the marker, or may include information associated with the information regarding the range of effect of the marker (for example, identification information of the marker).

[0464] This allows the safety management unit 1378 to acquire information indicating the degree of safety of a position to which a marker is attached or an object to which the marker is attached that is detected in a frame of the video data 1320. The safety management unit 1378 may acquire various types of information by referring to a database in which identification information of a marker, information indicating the position to which the marker is attached or the degree of safety of an object to which the marker is attached, and information regarding the range of effect of the marker are associated with each other.

[0465] The safety management unit 1378 changes the pixel value of the corresponding pixel in the dynamic image based on the position of the calculation unit or pixel where the marker is detected in the normal image and the degree of safety indicated by the marker. This allows the user 22 to install the user terminal 120 in an appropriate location that is in line with reality.

[0466] For example, since a glass window transmits and reflects light, the value of the variation index of a pixel corresponding to the glass window in the normal image will be relatively large in the variation image. However, if there is little possibility that the room will be peeked into from the outside, a marker indicating high safety is attached to the glass window in the real world. If a marker is not attached to the glass window, the user 22 cannot use the user terminal 120 in a position with their back to the glass window. On the other hand, if a marker is attached to the glass window, the user 22 can use the user terminal 120 in a position with their back to the glass window.

[0467] Next, in S1726, the safety management unit 1378 extracts a target image to be evaluated for safety from the one or more dynamic images acquired in S1722. In one embodiment, the safety management unit 1378 uses the entire region of the single dynamic image generated by the image generation unit 1376 as the target image and evaluates the safety of the position and / or direction corresponding to the target image. In another embodiment, the safety management unit 1378 uses a partial region of the single dynamic image generated by the image generation unit 1376 as the target image and evaluates the safety of the position and / or direction corresponding to the target image. An example of the partial region of the dynamic image is a region corresponding to the viewing angle of the display region 142.

[0468] Next, in S1728, the safety management unit 1378 determines the imaging position and imaging direction corresponding to the extracted target image. For example, the safety management unit 1378 determines the imaging position and imaging direction based on the imaging condition data 1310 acquired in S1722.

[0469] Next, in S1730, the safety management unit 1378 determines an evaluation value regarding the safety of the target image. The safety management unit 1378 also outputs the evaluation value regarding the safety of the target image in association with the imaging position and / or imaging direction of the target image.

[0470] Fig. 18 schematically illustrates an example of a two-dimensional image 1800. In Fig. 18, a marker 1822 indicates an example of the marker described in relation to S1724 in Fig. 17. Furthermore, a region 1824 indicates an example of the range over which the effect of the marker 1822 extends.

[0471] Fig. 19 schematically illustrates an example of a panoramic image 1900. As described in relation to S1726 in Fig. 17, the safety management unit 1378 extracts a target image to be evaluated for safety from one or more variation images acquired in S1722. In this case, the safety management unit 1378 may generate a single composite image by combining multiple variation images generated from normal images captured in different directions.

[0472] For example, the safety management unit 1378 synthesizes the above-mentioned multiple variable images to generate a panoramic image 1900. According to the example shown in Fig. 19 , a variable image 1920, a variable image 1922, a variable image 1924, a variable image 1926, and a variable image 1928 are synthesized to form a single panoramic image 1900.

[0473] The safety management unit 1378 may extract an image (sometimes referred to as a unit image) showing an area corresponding to the viewing angle of the display area 142 from the panoramic image 1900, and determine the unit image as the target image. The safety management unit 1378 may extract a plurality of unit images 1940 as the target image by shifting the position of a center 1942 of the extracted unit image 1940 in the left-right direction or the up-down direction along a predetermined dotted line 1950.

[0474] (An example of another embodiment) In another embodiment, the panoramic image 1900 may be periodically refreshed. For example, a new panoramic image 1900 may be generated at predetermined time intervals. In this case, a guide process for generating the new panoramic image 1900 may be executed. This allows the latest information about the real world to be reflected in the panoramic image 1900. As a result, the accuracy of the safety assessment is improved.

[0475] 19, an example of a technique for virtually expanding the angle of view of an imaging device (sometimes referred to as an angle of view expansion technique) has been described, taking as an example a case where the control device 170 executes a process (sometimes referred to as panoramic processing) in which multiple variable images generated from normal images captured in different directions are combined to generate a single composite image (the panoramic image described above). Each of the multiple normal images or variable images described above may be referred to as a current image.

[0476] However, the angle-of-view extension technology is not limited to this embodiment. Another example of the angle-of-view extension technology for the current image is detecting n-th order reflections (n ​​is an integer greater than or equal to 2) of electromagnetic waves and / or sound waves from the current image to detect fluctuations, dynamic changes, and the like of objects outside the angle of view. For example, if the surface of a subject in an image is prone to reflect light, an object behind the imaging device that captured the image will be reflected on the surface of the subject. The control device 170 can detect the movement of an object located outside the angle of view of the imaging device by analyzing the fluctuations of light or shadow reflected on the surface of the subject. Examples of objects with surfaces that are prone to reflect electromagnetic waves and / or sound waves include mirrors, glass windows, whiteboards, walls with high directional reflection characteristics, TV screens that are turned off, and refrigerators. The control device 170 may analyze the image to detect the boundaries of the subject and / or identify the type of the detected subject or determine whether the subject has a reflective surface.

[0477] For example, the control device 170 pays attention to the degree of fluctuation of n-th order reflection of light or shadow reflected on the surface of an object existing within the angle of view of the imaging device. If the change in the degree of fluctuation of n-th order reflected light is large, the control device 170 may determine that a moving object exists around the imaging device (for example, outside the angle of view of the...

Claims

1. a surrounding state acquisition unit that acquires surrounding state information indicating the arrangement of objects around a display area where an output image output by an image output device is displayed; a viewability deriving unit that derives a degree of viewability of the output image by a third party different from the user of the image output device based on an arrangement of objects around the display area indicated by the surrounding state information acquired by the surrounding state acquisition unit; Equipped with The object around the display area is an obstructing object that can block the line of sight of the third person. Information processing device.

2. A peripheral state acquisition unit that acquires peripheral state information indicating the arrangement of objects around a display area where an output image output by an image output device is displayed; a viewability deriving unit that derives a degree of viewability of the output image by a third party different from the user of the image output device based on an arrangement of objects around the display area indicated by the surrounding state information acquired by the surrounding state acquisition unit; Equipped with the viewability deriving unit derives the degree of viewability based on an arrangement of objects that satisfy predetermined conditions related to dimensions, which are indicated by the surrounding state information acquired by the surrounding state acquisition unit; The predetermined condition regarding the dimensions includes a condition that at least one of width, height, and depth is larger than a predetermined value indicating that the object has a size large enough to block the line of sight of the third person. Information processing device.

3. The viewability deriving unit derives the degree of viewability based on a distance between the display area and an object in a periphery of the display area.

3. The information processing device according to claim 1.

4. A peripheral state acquisition unit that acquires peripheral state information indicating the arrangement of objects around a display area where an output image output by an image output device is displayed; a viewability deriving unit that derives a degree of viewability of the output image by a third party different from the user of the image output device based on an arrangement of objects around the display area indicated by the surrounding state information acquired by the surrounding state acquisition unit; Equipped with The viewability deriving unit derives a larger degree of viewability as the distance between the display area and an object in the periphery of the display area increases. Information processing device.

5. The viewability deriving unit derives the degree of viewability based on an arrangement of objects that satisfy predetermined conditions regarding dimensions, which are indicated by the peripheral state information acquired by the peripheral state acquiring unit. The information processing device according to claim 1 .

6. A peripheral state acquisition unit that acquires peripheral state information indicating the arrangement of objects around a display area where an output image output by an image output device is displayed; a viewability deriving unit that derives a degree of viewability of the output image by a third party different from the user of the image output device based on an arrangement of objects around the display area indicated by the surrounding state information acquired by the surrounding state acquisition unit; Equipped with the viewability deriving unit derives the degree of viewability based on an arrangement of objects that satisfy predetermined conditions related to dimensions, which are indicated by the surrounding state information acquired by the surrounding state acquisition unit; the predetermined condition regarding the dimensions includes a condition that at least one of width, height, and depth is greater than a predetermined value; The viewability deriving unit derives a smaller degree of viewability as the distance between the display area and an object that satisfies a predetermined condition regarding the dimensions indicated by the peripheral state information acquired by the peripheral state acquisition unit becomes smaller. Information processing device.

7. A peripheral state acquisition unit that acquires peripheral state information indicating the arrangement of objects around a display area where an output image output by an image output device is displayed; a viewability deriving unit that derives a degree of viewability of the output image by a third party different from the user of the image output device based on an arrangement of objects around the display area indicated by the surrounding state information acquired by the surrounding state acquisition unit; Equipped with the surrounding state information includes a video obtained by capturing an image of the surroundings of the installation position of the image output device, The viewability deriving unit evaluates the degree to which light fluctuation is detected around the installation position by analyzing the video, derives a clutter level of a plurality of objects around the display area indicated by the surrounding state information using the degree to which light fluctuation is detected, and derives the viewability level based on the clutter level. Information processing device.

8. The browsability deriving unit derives a larger degree of browsability as the degree of clutter increases. The information processing device according to claim 7 .

9. A program for causing a computer to function as the information processing device according to any one of claims 1, 2, and 6 to 8.

Images