Information processing device, information processing method, and information processing program

Abstract

To abstract an object that a user does not want to see in a view field image of a goggle-type wearable device.SOLUTION: An information processing device according to the present invention comprises: an acquisition unit that acquires a visual field image of a goggle-type wearable device; an identification unit that identifies an object that a user cannot permit from the visual field image; a conversion unit that converts the object into an abstracted content that can be permitted by the user; and a display control unit that displays the abstracted content at a position of the object in a superimposed manner in the visual field image of the goggle-type wearable device.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] There is disclosed a technique for preferentially displaying virtual objects to be presented to a user.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] <00​​​​​​​​​​​The information processing device according to the present application includes an acquisition unit that acquires a field of view image of a glasses-type wearable device, an identification unit that identifies objects unacceptable to the user from the field of view image and further identifies the category of such objects, and an abstract content that is acceptable to the user according to the identified category of the object. , the name or category name of the object A conversion unit that converts to the abstract content of the object in the field of view of the glasses-type wearable device 、 The system includes a display control unit that superimposes the name or category name of the object, and the identification unit identifies the category of the object by further subdividing each of the following into major, medium, and minor classifications: insects / animals, garbage / dust, pictures / photographs / videos, characters / texts, and expressions based on specific thoughts / beliefs, and if the object is an insect, the major classification is living organisms, the medium classification is insects, and the minor classification is the type / name of the insect, and the conversion unit identifies the object of The aforementioned abstract content When converting, The aforementioned object, Same category other Object In the image conversion Instead of doing that, The aforementioned object To an abstract icon or pictogram conversion The display control unit then displays the object at the position where the object is located. The aforementioned object The system is characterized by displaying an abstract icon or pictogram superimposed on it. [Effects of the Invention]

[0007] According to one embodiment, in the field of view image of a glasses-type wearable device, it is possible to abstract away things that the user does not want to see. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is an explanatory diagram showing an overview of the information processing method according to the embodiment. [Figure 2] Figure 2 shows an example of the configuration of an information processing system according to the embodiment. [Figure 3] Figure 3 shows an example of the configuration of a terminal device according to an embodiment. [Figure 4]Figure 4 shows an example of an abstract information database. [Figure 5] Figure 5 shows an example of the configuration of a server device according to this embodiment. [Figure 6] Figure 6 shows an example of a user information database. [Figure 7] Figure 7 shows an example of a historical information database. [Figure 8] Figure 8 shows an example of a model information database. [Figure 9] Figure 9 is a flowchart showing the processing procedure according to the embodiment. [Figure 10] Figure 10 shows an example of a hardware configuration. [Modes for carrying out the invention]

[0009] The following describes in detail, with reference to the drawings, embodiments for implementing the information processing device, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the information processing device, information processing method, and information processing program according to the present application. Furthermore, the same parts are denoted by the same reference numerals in the following embodiments, and redundant descriptions are omitted.

[0010] [1. Overview of Information Processing Methods] First, with reference to Figure 1, an overview of the information processing method performed by the information processing device according to the embodiment will be described. Figure 1 is an explanatory diagram showing an overview of the information processing method according to the embodiment. In Figure 1, an example is given in which objects that the user does not want to see are abstracted in the field of view image of a glasses-type wearable device.

[0011] As shown in FIG. 1, the information processing system 1 includes a terminal device 10, a server device 100, and an AR goggle 200 (AR glasses). The terminal device 10, the server device 100, and the AR goggle 200 are connected to each other via a network N (see FIG. 2) so as to be communicable with each other by wire or wirelessly. In the present embodiment, the terminal device 10 and the AR goggle 200 cooperate with the server device 100. The terminal device 10 and the AR goggle 200 cooperate with each other. The AR goggle 200 may be communicable with the server device 100 via the terminal device 10.

[0012] The terminal device 10 is a smart device such as a smartphone or a tablet terminal used by a user U, and is a portable terminal device capable of communicating with an arbitrary server device via a wireless communication network such as 4G (Generation) or LTE (Long Term Evolution). Further, the terminal device 10 has a screen such as a liquid crystal display and has a screen having a touch panel function, and accepts various operations on display data such as content, such as a tap operation, a slide operation, and a scroll operation, by a finger or a stylus from the user U. Note that, among the screens, an operation performed on a region where content is displayed may be regarded as an operation on the content. Further, the terminal device 10 may be not only a smart device but also an information processing device such as a desktop PC (Personal Computer) or a notebook PC.

[0013] The server device 100 is an information processing device that cooperates with the terminal device 10 of each user U and provides API (Application Programming Interface) services and various data for various applications (hereinafter referred to as apps) and the like to the terminal device of each user U, and is realized by a computer, a cloud system, or the like.

[0014] Further, the server device 100 may be an information processing device that provides some kind of web service online to each terminal device 10 of each user U. For example, as a web service, the server device 100 may provide services such as Internet connection, search service, SNS (Social Networking Service), e-commerce (EC: Electronic Commerce), electronic payment, online game, online banking, online trading, accommodation and ticket reservation, video and music distribution, news, map, route search, route guidance, route information, operation information, weather forecast, etc. In fact, the server device 100 may cooperate with various servers that provide the above web services and mediate the web services, or be responsible for the processing of the web services.

[0015] In addition, the server device 100 can acquire user information regarding the user U. For example, the server device 100 acquires information regarding the attributes of the user U, such as the gender, age, and residential area of the user U. Then, the server device 100 stores and manages the information regarding the attributes of the user U together with the identification information (such as user ID) indicating the user U.

[0016] Furthermore, the server device 100 acquires various historical information (log data) indicating user U's actions from user U's terminal device 10, or from various servers based on the user ID, etc. For example, the server device 100 acquires location history, which is the history of user U's location and date and time, from the terminal device 10. The server device 100 also acquires search history, which is the history of search queries entered by user U, from the search server (search engine). The server device 100 also acquires browsing history, which is the history of content viewed by user U, from the content server. The server device 100 also acquires purchase history (payment history), which is the history of user U's product purchases and payment processing, from the e-commerce server or payment processing server. The server device 100 may also acquire listing history and sales history, which are the history of user U's listings on the marketplace, from the e-commerce server or payment processing server. The server device 100 also acquires posting history, which is the history of user U's posts, from posting servers that provide word-of-mouth posting services or SNS servers. The various servers mentioned above may also be the server device 100 itself. In other words, the server device 100 may function as the various servers mentioned above.

[0017] AR Goggles 200 (AR Glasses) are glasses-type wearable devices with AR (Augmented Reality) functionality. Therefore, AR Goggles 200 have camera and near-field communication functions. Furthermore, they may also have audio functions, microphone functions, image recognition functions, and voice recognition functions. AR Goggles 200 can also support virtual spaces and the metaverse. In practice, AR Goggles 200 may have not only AR functionality but also VR (Virtual Reality) and MR (Mixed Reality) functionality. In addition, AR Goggles 200 may be smart glasses equipped with AR functionality, an AR / VR headset, or a head-mounted display (HMD) that serves as both VR and AR goggles. Furthermore, AR Goggles 200 may be a see-through type AR device like HoloLens, not just a non-transparent display device.

[0018] [1-1. Abstraction AR] In this embodiment, user U is wearing AR goggles 200. The AR goggles 200 captures a field of view image with its built-in camera. At this time, a predetermined object that user U cannot tolerate or does not want to see may be displayed in the field of view image. When user U is wearing AR goggles 200, the field of view image becomes user U's field of view, and because user U is close to the screen, user U cannot look away or avert their eyes, and is forced to see what is displayed in the field of view image, even if it is something they do not want to see.

[0019] However, if objects are completely removed (made invisible) from the field of view of the AR Goggles 200, a situation may arise where the object is actually present but invisible to user U, potentially leading to undesirable consequences. Furthermore, simply applying a mosaic effect (concealment) to objects within the field of view is insufficient. If the object being mosaicked is not uniquely identified, user U will not know why it was mosaicked or what is actually there, resulting in a lack of information for user U from the field of view.

[0020] In this embodiment, the user U's terminal device 10 works in conjunction with the AR goggles 200 to acquire the field of view image of the AR goggles 200 (the field of view image of user U wearing the AR goggles 200), detects a predetermined object OBJ from the acquired field of view image, and displays an abstracted content ABS (abstracted icon, etc.) which is an abstraction of the detected predetermined object OBJ, superimposed on the position where the predetermined object OBJ is located in the field of view image.

[0021] For example, as shown in Figure 1, the user U's terminal device 10 acquires a field of view image captured by the camera function of the AR goggles 200 via short-range wireless communication (or via a communication cable) (step S1).

[0022] Next, the user U's terminal device 10 identifies and extracts a predetermined target object OBJ from the acquired field of view image (step S2).

[0023] Examples of predetermined objects (OBJ) include things that user U does not want to look at directly, things that evoke feelings of disgust, or things that cause mental distress. The predetermined objects (OBJ) can be set in various ways, such as during initial setup when installing the app, through automatic updates by the app or software, or manually by user U using the terminal device 10. The terminal device 10 may also use machine learning or similar methods to estimate predetermined objects (OBJ) for each user based on user attributes (user segments or user personas), past history, current settings, etc.

[0024] In this case, user U's terminal device 10 may identify and extract target objects OBJ for each category. Examples of target object OBJ categories include insects / animals, garbage / dust, pictures / photographs / videos, text / documents, and expressions based on specific thoughts / beliefs. Furthermore, the target object OBJ categories may be subdivided into major, medium, and minor classifications. For example, the major classification could be "living things," the medium classification "insects," and the minor classification "types / names of insects."

[0025] Next, the user U's terminal device 10 determines the user U's tolerance level for a predetermined object OBJ and the level of abstraction when abstracting the object OBJ (step S3).

[0026] In this case, the terminal device 10 may accept the user U to manually set a predetermined object OBJ along with the tolerance and abstraction levels, or it may estimate the tolerance and abstraction levels for each user using machine learning or the like based on the user U's user attributes, past history, current settings, etc.

[0027] Next, the terminal device 10 of user U selects or generates an abstracted content ABS, which is an abstracted version of a predetermined object OBJ, in order to abstract the predetermined object OBJ according to the tolerance level and abstraction level of each user U (step S4).

[0028] In this case, the terminal device 10 may gradually abstract a predetermined object OBJ according to the tolerance level and abstraction level of each user U. For example, the terminal device 10 may store pairs of predetermined object OBJs and abstracted content ABS in advance for each tolerance level and abstraction level, and select the abstracted content ABS corresponding to the predetermined object OBJ according to the tolerance level and abstraction level, or it may automatically generate the abstracted content ABS from the predetermined object OBJ according to the tolerance level and abstraction level. In addition, the terminal device 10 may gradually blur or change the transparency / transparency of the predetermined object OBJ according to the tolerance level and abstraction level.

[0029] Next, the user U's terminal device 10 converts a predetermined object OBJ included in the field of view image of the AR goggles 200 into abstract content ABS (step S5).

[0030] At this time, the terminal device 10 provides the AR goggles 200 with information regarding abstracted content ABS, which is an abstraction of a predetermined object OBJ, via short-range wireless communication (or via a communication cable). For example, the terminal device 10 provides information regarding the correspondence between the predetermined object OBJ and the abstracted content ABS, as well as information regarding the display position and display instructions of the abstracted content ABS. Furthermore, if the abstracted content ABS is displayed in a special display mode that differs from the usual, the terminal device 10 also provides information regarding that display mode.

[0031] Next, the AR goggles 200 superimpose and display an abstracted content ABS, which is an abstraction of the predetermined object OBJ, at the location where the predetermined object OBJ is located within the field of view image (step S6).

[0032] In practice, the user U's terminal device 10 may be controlled to display an abstracted content ABS, which is an abstraction of a predetermined object OBJ, superimposed on the position where the predetermined object OBJ is located within the field of view image of the AR goggles 200.

[0033] Next, the user U's terminal device 10 communicates with the server device 100 via the network N (see Figure 2) and provides the server device 100 with information regarding the abstraction of a predetermined object OBJ (step S7).

[0034] In this case, the server device 100 collects information regarding the abstraction of a predetermined object OBJ from the terminal devices 10 of an unspecified number of users U via the network N (see Figure 2). Alternatively, if the server device 100 performs machine learning using Federated Learning, without aggregating the data, it may obtain the learning results (difference data and features) from the terminal devices 10 of the users U via the communication unit 110.

[0035] Next, the server device 100 learns about the abstraction of a predetermined object OBJ for each user attribute (which can also be a user segment or user persona) of user U using machine learning, and generates / updates an estimation model based on the learning results (step S8).

[0036] For example, the server device 100 learns user attributes, a predetermined object OBJ, tolerance levels, abstraction levels, and abstracted content ABS as training data, and generates / updates an estimation model that outputs tolerance levels, abstraction levels, and abstracted content ABS when user attributes and a predetermined object OBJ are input. Alternatively, the server device 100 may generate / update an estimation model by integrating the learning results obtained from the user U's terminal device 10 through federated learning.

[0037] Next, the server device 100 provides the generated / updated estimation model to each user U's terminal device 10 via the network N (see Figure 2) (step S9).

[0038] As a result, in step S2, when user U's terminal device 10 identifies and extracts a predetermined object OBJ from the acquired field of view image, it can estimate the predetermined object OBJ based on the estimation model provided by the server device 100, and then identify and extract the estimated predetermined object OBJ.

[0039] Furthermore, the server device 100 may provide not only the generated / updated estimation model itself, but also the parameter differences between the global model (central model) on the server device 100 side corresponding to the estimation model and the local model (child model) on the user U's terminal device 10 side. For example, if the server device 100 performs machine learning in a distributed state without aggregating data using federated learning, it may provide the user U's terminal device 10 with information regarding modifications to the shared / integrated model on the server device 100 side.

[0040] [1-2.Additional and supplementary matters] In this embodiment, the user U's terminal device 10 converts a predetermined object OBJ contained within the field of view image of the AR goggles 200 into abstract content ABS. The user U's terminal device 10 then displays the abstract content ABS, which is an abstraction of the extracted object OBJ, within the field of view image. For example, the user U's terminal device 10 displays the abstract content ABS superimposed on the location where the object OBJ is located.

[0041] The abstracted content (ABS) may be a two-dimensional (2D) image, a three-dimensional (3D) image, or a 3D object / 3D model (3D shape). Furthermore, the user U's terminal device 10 may automatically generate the abstracted content (ABS) from the object's OBJ file.

[0042] Furthermore, when extracting a predetermined object OBJ, the user U's terminal device 10 may decide whether or not to extract the predetermined object OBJ based on its size, distance, etc., within the field of view image. Specifically, the user U's terminal device 10 may extract the predetermined object OBJ as an object OBJ to be abstracted if it is of a size and distance that allows the user U to recognize (visually perceive) the predetermined object OBJ within the field of view image. For example, the user U's terminal device 10 may extract the predetermined object OBJ as an object OBJ to be abstracted if it is located within a predetermined range from the user U within the field of view image (for example, within a distance of 2m) and is displayed at a predetermined size (for example, a total length of 1cm or more).

[0043] Furthermore, user U's terminal device 10 may, depending on the tolerance level and abstraction level, change the extracted predetermined object OBJ into "text" and display the "text" superimposed on the location of the object OBJ. For example, user U's terminal device 10 may, depending on the tolerance level and abstraction level, superimpose the "name" or "category name" of the extracted predetermined object OBJ onto the location of the object OBJ.

[0044] Alternatively, the user U's terminal device 10 may, depending on the tolerance level and abstraction level, change the extracted predetermined object OBJ into a "category-indicating image" and superimpose the "category-indicating image" onto the location where the object OBJ is located. The category-indicating images may be pre-set for each category.

[0045] Furthermore, if the extracted predetermined object OBJ is an insect, the user U's terminal device 10 may change the object OBJ to an "abstract insect icon / pictogram" and overlay the "abstract insect icon / pictogram" at the location where the object OBJ is located. Basically, since it is presumed that people who dislike a particular insect dislike insects in general, it is preferable to change it to some kind of pictogram that abstracts insects rather than changing it to an image of a different insect. In this case, if not all insects are object OBJs, the user U's terminal device 10 may change only the insects that are set as object OBJs to "abstract insect icons / pictograms" and overlay the "abstract insect icons / pictograms" at the location where the object OBJs are located, rather than uniformly changing all insects to "abstract insect icons / pictograms". Furthermore, when user U's terminal device 10 overlays and displays an "abstract insect icon / pictogram," it may display an unnatural image of an "abstract insect icon / pictogram" that is clearly altered, rather than the usual image of an "abstract insect icon / pictogram" that exists in nature. For example, it may display an "abstract insect icon / pictogram" that is several times larger than normal, an "abstract insect icon / pictogram" with a color or shape that does not exist in the environment of the current location, or an "abstract insect icon / pictogram" that glows or flashes.

[0046] Alternatively, user U's terminal device 10 may, depending on the tolerance level and abstraction level, change the extracted predetermined object OBJ into a "caricature illustration" and display the "caricature illustration" superimposed on the location of the object OBJ. The caricature illustrations may be pre-set for each object OBJ or category, or they may be automatically converted from the object OBJ by image conversion or machine learning. Furthermore, user U's terminal device 10 may, depending on the tolerance level and abstraction level, change the extracted predetermined object OBJ into a "pictogram" and display the "pictogram" superimposed on the location of the object OBJ.

[0047] Furthermore, the terminal device 10 of user U may accept settings from user U regarding the level of abstraction of the abstracted content ABS when abstracting the object OBJ. For example, the terminal device 10 of user U may accept settings from user U regarding how high the level of abstraction should be. In addition, the terminal device 10 of user U may select or automatically generate the abstracted content ABS according to the level of abstraction.

[0048] Furthermore, the terminal device 10 of user U may ask user U how much they dislike the object OBJ, and change the level of abstraction according to the answer. Also, the terminal device 10 of user U may ask user U whether they like or dislike the abstracted content ABS, and further change the level of abstraction according to the answer. At this time, the terminal device 10 of user U may learn from the answers from user U and change the level of abstraction based on the learning results.

[0049] Alternatively, user U's terminal device 10 may estimate, calculate, and automatically set the level of abstraction of the abstracted content ABS. For example, user U's terminal device 10 may estimate user U's preferences, etc., from user U's user attributes, and calculate an acceptable score for each category according to the estimated preferences, etc. In this case, user U's terminal device 10 lowers the level of abstraction as the acceptable score increases.

[0050] Furthermore, the user U's terminal device 10 may estimate the user U's context and convert a predetermined object OBJ into abstract content ABS corresponding to the estimated context. For example, the user U's terminal device 10 may set and change the abstract content ABS according to the context of the user U's location (whereabouts), such as at home, in the office, on the street, in a public place, or inside a car. Also, the user U's terminal device 10 may set and change the abstract content ABS according to the context of the user U's surroundings, such as when the user is alone (no one around) or when there are many people around.

[0051] Alternatively, user U's terminal device 10 may set and change the abstract content ABS according to the context of user U's time (time period), such as before going to work, while traveling, during working hours, while eating, during a meeting, or after leaving work. Furthermore, user U's terminal device 10 may set and change the abstract content ABS according to the context of the day of the week, such as weekdays, weekends, or public holidays.

[0052] Furthermore, user U's terminal device 10 may set and change the abstraction content ABS depending on whether the extracted object OBJ is a "real object" or a "virtual object" displayed on the screen by AR functionality, etc. In other words, the object OBJ is not limited to a real object, but may also be data displayed by AR (augmented reality), etc. If user U really dislikes the object OBJ, user U's terminal device 10 will change it into abstraction content ABS even if it is a virtual object. If user U doesn't dislike it that much, or depending on the degree of like or dislike, real objects will be changed into abstraction content ABS, and virtual objects will be displayed as they are. Whether user U really dislikes it can be determined by directly asking user U, or by inferring it from the level of abstraction setting of the abstraction content ABS.

[0053] The processing performed by the terminal device 10 may be carried out by the server device 100 which is in cooperation with the terminal device 10, or by the AR goggles 200. Furthermore, the AR goggles 200 may cooperate with the terminal device 10 via the network N (see Figure 2), or it may cooperate directly with the server device 100 via the network N (see Figure 2) without going through the terminal device 10.

[0054] [2. Example of an information processing system configuration] Next, the configuration of the information processing system 1, which includes the server device 100 according to the embodiment, will be described using Figure 2. Figure 2 is a diagram showing an example of the configuration of the information processing system 1 according to the embodiment. As shown in Figure 2, the information processing system 1 according to the embodiment includes a terminal device 10, a server device 100, and AR goggles 200. These various devices are connected to each other via a network N, either by wire or wireless communication. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

[0055] Furthermore, the number of devices included in the information processing system 1 shown in Figure 2 is not limited to those illustrated. For example, in Figure 2, for the sake of simplicity, one terminal device 10 and one AR goggle 200 are shown, but this is merely an example and not limiting; there may be two or more of each.

[0056] Terminal device 10 is an information processing device used by user U. For example, terminal device 10 may be a smart device such as a smartphone or tablet, a feature phone, a PC (Personal Computer), a PDA (Personal Digital Assistant), a game console or AV equipment with communication functions, a car navigation system, a smartwatch, etc. Alternatively, terminal device 10 may be a house or building compatible with IoT (Internet of Things), a car, home appliances, electronic devices, etc.

[0057] Furthermore, the AR Goggles 200 is not limited to AR glasses; it may also be smart glasses, head-mounted displays, or other glasses-type wearable devices.

[0058] Furthermore, the terminal device 10 and AR goggles 200 can connect to the network N via wireless communication networks such as LTE (Long Term Evolution), 4G (4th Generation), and 5G (5th Generation), or via short-range wireless communication such as Bluetooth (registered trademark) and Wi-Fi (Local Area Network), and communicate with the server device 100.

[0059] The server device 100 is, for example, a computer such as a PC or blade server, or a mainframe or workstation. The server device 100 may also be implemented through cloud computing.

[0060] [3. Example of terminal device configuration] Next, the configuration of the terminal device 10 will be explained using Figure 3. Figure 3 is a diagram showing an example of the configuration of the terminal device 10. As shown in Figure 3, the terminal device 10 comprises a communication unit 11, a display unit 12, an input unit 13, a positioning unit 14, a sensor unit 20, a control unit 30 (controller), and a storage unit 40.

[0061] (Communications Section 11) The communication unit 11 is connected to the network N (see Figure 2) by wire or wireless connection and transmits and receives information to and from the server device 100 via the network N. For example, the communication unit 11 can be implemented using a NIC (Network Interface Card) or an antenna.

[0062] (Display section 12) The display unit 12 is a display device that displays various information such as location information. For example, the display unit 12 may be a liquid crystal display (LCD) or an organic electro-luminescent display (OLED). The display unit 12 may also be a touch panel display, but is not limited to this.

[0063] (Input section 13) The input unit 13 is an input device that receives various operations from the user U. For example, the input unit 13 has buttons for inputting characters, numbers, etc. The input unit 13 may also be an input / output port (I / O port) or a USB (Universal Serial Bus) port. If the display unit 12 is a touch panel display, a part of the display unit 12 functions as the input unit 13. The input unit 13 may also be a microphone that receives voice input from the user U. The microphone may be wireless.

[0064] (Positioning unit 14) The positioning unit 14 receives signals (radio waves) transmitted from GPS (Global Positioning System) satellites and, based on the received signals, acquires position information (e.g., latitude and longitude) indicating the current position of the terminal device 10. In other words, the positioning unit 14 determines the position of the terminal device 10. Note that GPS is just one example of a GNSS (Global Navigation Satellite System).

[0065] Furthermore, the positioning unit 14 can determine its position using various methods other than GPS. For example, the positioning unit 14 may use various communication functions of the terminal device 10 to determine its position as an auxiliary positioning means for position correction, etc., as described below.

[0066] (Wi-Fi positioning) For example, the positioning unit 14 determines the location of the terminal device 10 by utilizing the Wi-Fi® communication function of the terminal device 10 and the communication network provided by each telecommunications company. Specifically, the positioning unit 14 determines the location of the terminal device 10 by performing Wi-Fi communication, etc., and determining the distance to nearby base stations and access points.

[0067] (Beacon positioning) Furthermore, the positioning unit 14 may determine the location using the Bluetooth® function of the terminal device 10. For example, the positioning unit 14 determines the location of the terminal device 10 by connecting to a beacon transmitter connected via the Bluetooth® function.

[0068] (Geomagnetic positioning) Furthermore, the positioning unit 14 determines the position of the terminal device 10 based on the geomagnetic pattern of the structure, which has been measured in advance, and the geomagnetic sensor provided by the terminal device 10.

[0069] (RFID positioning) Furthermore, if, for example, the terminal device 10 is equipped with an RFID (Radio Frequency Identification) tag function equivalent to that of a contactless IC card used at a train station ticket gate or in a store, or if it is equipped with a function to read RFID tags, the location where it was used will be recorded along with the information on the payment or other transactions made by the terminal device 10. The positioning unit 14 may determine the location of the terminal device 10 by acquiring such information. Alternatively, the location may be determined by an optical sensor or infrared sensor equipped in the terminal device 10.

[0070] The positioning unit 14 may, if necessary, determine the position of the terminal device 10 using one or a combination of the positioning means described above.

[0071] (Sensor unit 20) The sensor unit 20 includes various sensors mounted on or connected to the terminal device 10. The connection can be wired or wireless. For example, the sensors may be detection devices other than the terminal device 10, such as wearable devices or wireless devices. In the example shown in Figure 3, the sensor unit 20 includes an acceleration sensor 21, a gyro sensor 22, a barometric pressure sensor 23, a temperature sensor 24, a sound sensor 25, a light sensor 26, a magnetic sensor 27, and an image sensor (camera) 28.

[0072] The sensors 21-28 described above are merely examples and not limiting. In other words, the sensor unit 20 may be configured to include some of the sensors 21-28, or it may include other sensors such as humidity sensors in addition to or instead of the sensors 21-28.

[0073] The acceleration sensor 21 is, for example, a 3-axis acceleration sensor and detects the physical movement of the terminal device 10, such as its direction of movement, velocity, and acceleration. The gyro sensor 22 detects the physical movement of the terminal device 10, such as its tilt in the three axes, based on its angular velocity. The barometric pressure sensor 23 detects the atmospheric pressure around the terminal device 10, for example.

[0074] Since the terminal device 10 is equipped with the acceleration sensor 21, gyroscope 22, barometric pressure sensor 23, etc., it becomes possible to determine the position of the terminal device 10 using technologies such as pedestrian dead-reckoning (PDR) that utilize these sensors 21 to 23. This makes it possible to obtain indoor location information that is difficult to obtain with positioning systems such as GPS.

[0075] For example, a pedometer using an accelerometer 21 can calculate the number of steps, walking speed, and distance walked. Additionally, a gyroscope 22 can be used to determine the user U's direction of movement, gaze direction, and body tilt. Furthermore, the barometric pressure detected by the barometric pressure sensor 23 can be used to determine the altitude and floor number of the user U's terminal device 10.

[0076] The temperature sensor 24 detects, for example, the ambient temperature around the terminal device 10. The sound sensor 25 detects, for example, the ambient sound around the terminal device 10. The light sensor 26 detects the ambient illumination around the terminal device 10. The magnetic sensor 27 detects, for example, the Earth's magnetic field around the terminal device 10. The image sensor 28 captures an image of the area around the terminal device 10.

[0077] The aforementioned pressure sensor 23, temperature sensor 24, sound sensor 25, light sensor 26, and image sensor 28 can detect the surrounding environment and conditions of the terminal device 10 by detecting atmospheric pressure, temperature, sound, and illuminance, respectively, and by capturing images of the surroundings. Furthermore, it becomes possible to improve the accuracy of the location information of the terminal device 10 based on the surrounding environment and conditions.

[0078] (Control Unit 30) The control unit 30 includes, for example, a microcomputer having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM, input / output ports, and various circuits. Alternatively, the control unit 30 may be composed of hardware such as an integrated circuit (ASIC) or FPGA (Field Programmable Gate Array). The control unit 30 includes a transmission unit 31, a reception unit 32, and a processing unit 33.

[0079] (Transmitter 31) The transmission unit 31 can transmit various information, such as information input by the user U using the input unit 13, various information detected by sensors 21-28 mounted on or connected to the terminal device 10, and location information of the terminal device 10 determined by the positioning unit 14, to the server device 100 via the communication unit 11.

[0080] (Receiving unit 32) The receiving unit 32 can receive various information provided by the server device 100, as well as requests for various information from the server device 100, via the communication unit 11.

[0081] (Processing 33) The processing unit 33 controls the entire terminal device 10, including the display unit 12. For example, the processing unit 33 can output and display various information transmitted by the transmission unit 31 and various information received from the server device 100 by the reception unit 32 to the display unit 12.

[0082] Furthermore, the processing unit 33 includes an acquisition unit 33A, a setting unit 33B, an estimation unit 33C, a specific unit 33D, a generation unit 33E, a conversion unit 33F, and a display control unit 33G.

[0083] (Acquisition part 33A) The acquisition unit 33A acquires field of view images from the glasses-type wearable device. For example, the acquisition unit 33A acquires field of view images captured by the camera function of the glasses-type wearable device.

[0084] (Setting section 33B) The setting unit 33B accepts settings for object OBJs that are unacceptable to the user U. The setting unit 33B also accepts settings regarding the user U's tolerance level for the object OBJs. Furthermore, the setting unit 33B accepts settings regarding the level of abstraction when abstracting the object OBJs.

[0085] Furthermore, the setting unit 33B inquires with user U about the degree of preference for the object OBJ and receives a response from user U regarding the degree of preference for the object OBJ. In addition, the setting unit 33B inquires with user U about the degree of preference for the converted abstracted content ABS and receives a response from user U regarding the degree of preference for the converted abstracted content ABS. Then, the setting unit 33B changes the level of abstraction when abstracting the object OBJ according to each response.

[0086] (Estimation part 33C) The estimation unit 33C estimates object OBJs that user U cannot accept, based on various information (user information, history information, etc.), including user U's attribute information. The estimation unit 33C also estimates user U's tolerance level for object OBJs, based on various information (user information, history information, etc.), including user U's attribute information. Furthermore, the estimation unit 33C estimates the level of abstraction required when abstracting object OBJs, based on various information (user information, history information, etc.), including user U's attribute information.

[0087] (Specific part 33D) The specific unit 33D identifies and extracts objects OBJ that the user U cannot tolerate from the field of view image of the glasses-type wearable device.

[0088] (Generation section 33E) The generation unit 33E automatically generates abstracted content ABS from the object OBJ. For example, the generation unit 33E automatically generates abstracted content ABS from the object OBJ according to the user U's tolerance settings for the object OBJ. The generation unit 33E also automatically generates abstracted content ABS from the object OBJ according to the settings for the level of abstraction when abstracting the object OBJ.

[0089] (Conversion unit 33F) The conversion unit 33F converts the object OBJ into an abstract content ABS that is acceptable to the user U. For example, if an acceptable level is set, the conversion unit 33F converts the object OBJ into an abstract content ABS selected or generated according to the acceptable level. Also, if a level of abstraction is set, the conversion unit 33F converts the object OBJ into an abstract content ABS selected or generated according to the level of abstraction.

[0090] (Display Control Unit 33G) The display control unit 33G overlays and displays abstract content ABS at the location of the object OBJ within the field of view image of the glasses-type wearable device.

[0091] In this case, the display control unit 33G may automatically determine the display mode of the abstracted content ABS according to the user U's settings regarding tolerance and level of abstraction.

[0092] (Storage unit 40) The storage unit 40 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as HDD (Hard Disk Drive), SSD (Solid State Drive), and optical discs. Various programs and various data are stored in this storage unit 40.

[0093] For example, the storage unit 40 stores pairs of object OBJ and abstracted content ABS. The storage unit 40 also stores various settings such as tolerance and level of abstraction. As shown in Figure 4, the storage unit 40 has an abstracted information database 40A.

[0094] (Abstract Information Database 40A) The abstraction information database 40A stores various information related to the abstraction of a given object. Figure 4 shows an example of the abstraction information database 40A. In the example shown in Figure 4, the abstraction information database 40A has items such as "object," "acceptability," "level of abstraction," "abstracted content," and "display mode."

[0095] "Object" refers to identification information for identifying a specified object OBJ. "Tolerance" refers to the user U's tolerance level for the specified object OBJ. "Abstraction level" refers to the level of abstraction when abstracting the object OBJ. "Abstracted content" refers to identification information for identifying the abstracted content ABS (abstract icon, etc.) obtained by abstracting the specified object OBJ, or the abstracted content ABS itself (image data, etc.). "Display mode" refers to the display mode of the abstracted content ABS within the field of view image of the AR goggles 200.

[0096] For example, in the example shown in Figure 4, a predetermined object OBJ identified by the object "OBJ#1" is converted into abstracted content "ABS#1" which is an abstraction of the predetermined object OBJ, according to "tolerance #1" and "level of abstraction #1", and is displayed in the field of view image of the AR goggles 200 according to "display mode #1".

[0097] In the example shown in Figure 4, abstract values ​​such as "OBJ#1", "Tolerance#1", "AbstractionLevel#1", "ABS#1", and "DisplayMode#1" are used for illustration, but it is assumed that "OBJ#1", "Tolerance#1", "AbstractionLevel#1", "ABS#1", and "DisplayMode#1" will store specific strings, numbers, and other information.

[0098] Furthermore, the abstraction information database 40A is not limited to the above and may store various types of information depending on the purpose. For example, the abstraction information database 40A may store information regarding the category (classification) of a given object OBJ. The abstraction information database 40A may also store information regarding the context in which a given object OBJ is abstracted. The abstraction information database 40A may also store information regarding the position, size, and distance of a given object OBJ within the field of view image of the AR goggles 200.

[0099] [4. Example of Server Device Configuration] Next, the configuration of the server device 100 according to the embodiment will be described using Figure 5. Figure 5 is a diagram showing an example of the configuration of the server device 100 according to the embodiment. As shown in Figure 5, the server device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0100] (Communications Department 110) The communication unit 110 is implemented, for example, by a NIC (Network Interface Card). The communication unit 110 is connected to the network N (see Figure 2) by wire or wireless connection.

[0101] (Storage unit 120) The storage unit 120 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as HDDs, SSDs, and optical discs. As shown in Figure 5, the storage unit 120 has a user information database 121, a history information database 122, and a model information database 123.

[0102] (User Information Database 121) The user information database 121 stores user information about user U. For example, the user information database 121 stores various information such as user U's attributes. Figure 6 shows an example of the user information database 121. In the example shown in Figure 6, the user information database 121 has items such as "User ID (Identifier)", "Age", "Gender", "Home", "Workplace", and "Interests".

[0103] "User ID" refers to identification information used to identify user U. Note that "User ID" may be user U's contact information (telephone number, email address, etc.) or identification information used to identify user U's terminal device 10.

[0104] Furthermore, "Age" indicates the age of user U, identified by the user ID. Note that "Age" may be information indicating user U's specific age (e.g., 35 years old), or information indicating user U's age group (e.g., 30s), or "Age" may be information indicating user U's date of birth, or information indicating user U's generation (e.g., born in the 1980s). Furthermore, "Gender" indicates the gender of user U, identified by the user ID.

[0105] Furthermore, "Home" indicates the location information of user U's home, which is identified by the user ID. In the example shown in Figure 6, "Home" is represented by an abstract code such as "LC11," but it could also be latitude and longitude information, etc. Also, for example, "Home" could be a regional name or address.

[0106] Furthermore, "Workplace" indicates the location information of the workplace (or school in the case of a student) of user U, identified by the user ID. In the example shown in Figure 6, "Workplace" is illustrated with an abstract code such as "LC12," but it may also be latitude and longitude information, etc. Also, for example, "Workplace" may be a regional name or address.

[0107] Furthermore, "Interests" indicate the interests of user U, who is identified by their user ID. In other words, "Interests" indicate the subjects of high interest to user U, who is identified by their user ID. For example, "Interests" may be search queries (keywords) that user U enters into a search engine. In the example shown in Figure 6, one "Interest" is shown for each user U, but there may be multiple interests.

[0108] For example, in the example shown in Figure 6, user U, identified by user ID "U1", is in their 20s and is male. Also, for example, user U, identified by user ID "U1", has their home address at "LC11". Furthermore, for example, user U, identified by user ID "U1", has their workplace at "LC12". Finally, for example, user U, identified by user ID "U1", is interested in "sports".

[0109] In the example shown in Figure 6, abstract values ​​such as "U1," "LC11," and "LC12" are used to illustrate the information, but it is assumed that "U1," "LC11," and "LC12" actually store specific strings, numbers, or other information. In the following diagrams relating to other information, abstract values ​​may also be used to illustrate the information.

[0110] The user information database 121 is not limited to the above and may store various types of information depending on the purpose. For example, the user information database 121 may store various types of information about user U's terminal device 10. In addition, the user information database 121 may store information about user U's demographic, psychographic, geographic, and behavioral attributes. For example, the user information database 121 may store information such as name, family structure, place of origin (hometown), occupation, job title, income, qualifications, type of residence (detached house, apartment, etc.), whether or not a car is owned, commuting time, commuting route, commuter pass section (station, line, etc.), frequently used stations (other than the nearest station to home / workplace), lessons / classes (location, time, etc.), hobbies, interests, and lifestyle.

[0111] (History Information Database 122) The history information database 122 stores various information related to the history information (log data) that shows the user U's actions. Figure 7 shows an example of the history information database 122. In the example shown in Figure 7, the history information database 122 has items such as "User ID", "Location History", "Search History", "Browsing History", "Purchase History", and "Posting History".

[0112] "User ID" indicates identification information used to identify user U. "Location History" indicates the location history, which is the history of user U's location and movements. "Search History" indicates the search history, which is the history of search queries entered by user U. "Browsing History" indicates the browsing history, which is the history of content viewed by user U. "Purchase History" indicates the purchase history, which is the history of purchases made by user U. "Posting History" indicates the posting history, which is the history of posts made by user U. Note that "Posting History" may include questions about user U's possessions.

[0113] For example, in the example shown in Figure 7, user U, identified by user ID "U1", moves as described in "Location History #1", searches as described in "Search History #1", views content as described in "Browsing History #1", purchases specified goods at specified stores as described in "Purchase History #1", and posts as described in "Posting History #1".

[0114] In the example shown in Figure 7, abstract values ​​such as "U1", "Location History #1", "Search History #1", "Browsing History #1", "Purchase History #1", and "Posting History #1" are used for illustration. However, it is assumed that "U1", "Location History #1", "Search History #1", "Browsing History #1", "Purchase History #1", and "Posting History #1" will actually store specific strings, numbers, and other information.

[0115] The history information database 122 is not limited to the above and may store various types of information depending on the purpose. For example, the history information database 122 may store the usage history of user U for a specified service. The history information database 122 may also store the visit history of user U to a physical store or a facility. The history information database 122 may also store the payment history of user U using the terminal device 10 for payments (electronic payments).

[0116] (Model Information Database 123) The model information database 123 stores various information about the estimation model that estimates abstracted content ABS, which is an abstraction of a predetermined object OBJ. Figure 8 shows an example of the model information database 123. In the example shown in Figure 8, the model information database 123 has items such as "user attributes", "model", "object", "tolerance", "level of abstraction", "centralized content", and "display mode".

[0117] "User Attributes" indicates the user attributes of user U (can also be a user segment or user persona). "Model" indicates identification information for estimating an estimation model that estimates an abstracted content ABS (Abstracted Content) that abstracts a given object OBJ for each user attribute, or the estimation model itself. "Object" indicates identification information for estimating a given object OBJ. "Tolerance" indicates user U's tolerance for the given object OBJ. "Abstraction Level" indicates the level of abstraction when abstracting the object OBJ. "Abstracted Content" indicates identification information for estimating the abstracted content ABS (abstracted icon, etc.) that abstracts a given object OBJ, or the abstracted content ABS itself (image data, etc.). "Display Mode" indicates the display mode of the abstracted content ABS within the field of view image of the AR goggles 200.

[0118] For example, in the example shown in Figure 8, a user U with the user attribute "Attribute #A" is provided with an estimated model "Model #A". When a predetermined object OBJ identified by the object "OBJ #A1" is input to this estimated model "Model #A", "Tolerance #A1", "Abstraction Level #A1", abstracted content "ABS #A1", and "Display Mode #A1" are output.

[0119] In the example shown in Figure 8, abstract values ​​such as "Attribute #A", "Model #A", "OBJ#A1", "Tolerance #A1", "Abstraction Level #A1", "ABS#A1", and "Display Mode #A1" are used to illustrate the concept. However, it is assumed that "Attribute #A", "Model #A", "OBJ#A1", "Tolerance #A1", "Abstraction Level #A1", "ABS#A1", and "Display Mode #A1" will actually store specific strings, numbers, and other information.

[0120] Furthermore, the model information database 123 may store various types of information depending on the purpose, not limited to those described above. For example, the model information database 123 may store identification information to identify individual users U, rather than user attributes of user U. In other words, estimation models may be prepared for each individual user, not limited to each user attribute. For example, the model information database 123 may store estimation models customized for each user. The model information database 123 may also store information regarding the category (classification) of a given object OBJ. The model information database 123 may also store information regarding the context when abstracting a given object OBJ.

[0121] (Control unit 130) Returning to Figure 5, let's continue the explanation. The control unit 130 is a controller, and is realized by various programs (corresponding to an example of an information processing program) stored in the internal memory of the server device 100, such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), ASIC (Application Specific Integrated Circuit), or FPGA (Field Programmable Gate Array), executing them using a memory area such as RAM as the working area. In the example shown in Figure 5, the control unit 130 has an acquisition unit 131, a determination unit 132, a learning unit 133, and a provision unit 134.

[0122] (Acquisition part 131) The acquisition unit 131 acquires the search query entered by user U. For example, when user U enters a search query into a search engine or the like and performs a keyword search, the acquisition unit 131 acquires the search query via the communication unit 110. In other words, the acquisition unit 131 acquires the keyword entered by user U into the search box of a search engine, website, or app via the communication unit 110.

[0123] Furthermore, the acquisition unit 131 acquires user information about user U via the communication unit 110. For example, the acquisition unit 131 acquires identification information (such as user ID), location information, and attribute information of user U from user U's terminal device 10. The acquisition unit 131 may also acquire identification information and attribute information of user U when user U is registered. The acquisition unit 131 then registers the user information in the user information database 121 of the storage unit 120.

[0124] Furthermore, the acquisition unit 131 acquires various historical information (log data) indicating the user U's actions via the communication unit 110. For example, the acquisition unit 131 acquires various historical information indicating the user U's actions from the user U's terminal device 10, or from various servers based on the user ID, etc. The acquisition unit 131 then registers the various historical information in the history information database 122 of the storage unit 120.

[0125] Furthermore, the acquisition unit 131 may acquire a set of user U (or user attributes of user U), a predetermined object OBJ, tolerance level, abstraction level, abstracted content ABS, and display mode from the user U's terminal device 10 via the communication unit 110. The acquisition unit 131 may also register the acquired information in the model information database 123 of the storage unit 120.

[0126] Furthermore, if the acquisition unit 131 performs machine learning using Federated Learning, where the data is distributed without aggregation, it may acquire the learning results (difference data and features) from the user U's terminal device 10 via the communication unit 110.

[0127] (Judgment unit 132) The determination unit 132 determines the user attributes (user segment, user persona, etc.) of user U from identification information (user ID, etc.) that identifies user U. For example, the determination unit 132 determines the user attributes of user U based on various information (user information, history information, etc.) linked to a user ID common to various services.

[0128] (Learning Section 133) The learning unit 133 performs machine learning using user attributes of user U, a predetermined object OBJ, tolerance, level of abstraction, abstracted content ABS, and display mode as training data. When user attributes of user U and a predetermined object OBJ are input, the learning unit 133 generates / updates an estimation model that outputs tolerance, level of abstraction, abstracted content ABS, and display mode as estimation results. The learning unit 133 then registers the generated / updated estimation model in the model information database 123 of the storage unit 120.

[0129] In practice, the learning unit 133 may generate / update an estimation model that outputs an abstracted content ABS and a display mode as estimation results, when it receives user attributes of user U, a predetermined object OBJ, tolerance level, and abstraction level as input. In other words, the combination of input data and output data is arbitrary.

[0130] Furthermore, the learning unit 133 may integrate the learning results obtained from the user U's terminal device 10 through associative learning to generate / update an estimation model.

[0131] (Provider 134) The provisioning unit 134 provides the generated / updated estimation model to user U who has the target user attributes. That is, the provisioning unit 134 distributes the generated / updated estimation model to the terminal device 10 of user U according to the user attributes via the communication unit 110.

[0132] Furthermore, the provisioning unit 134 may, via the communication unit 110, on behalf of the user U's terminal device 10, estimate abstracted content ABS corresponding to a predetermined object OBJ using the generated / updated estimation model, and provide the estimated abstracted content ABS to the user U's terminal device 10.

[0133] Furthermore, when performing machine learning in a distributed state without aggregating data through federated learning, the provision unit 134 may, via the communication unit 110, provide information not only about the generated / updated estimation model itself, but also about the modifications made to the shared / integrated model on the server device 100 side, to the user U's terminal device 10.

[0134] [5. Processing Procedure] Next, the processing procedure by the terminal device 10 according to this embodiment will be described using Figure 9. Figure 9 is a flowchart of the processing procedure according to this embodiment. Note that the processing procedure shown below is repeatedly executed by the control unit 30 of the terminal device 10.

[0135] For example, as shown in Figure 9, the setting unit 33B of the terminal device 10 accepts settings regarding the object OBJ that the user U (user) cannot tolerate, the degree to which the user U tolerates the object OBJ, and the degree of abstraction when abstracting the object OBJ (step S101).

[0136] Alternatively, the estimation unit 33C of the terminal device 10 may estimate, from various information (user information, history information, etc.), the object OBJ that user U cannot tolerate, the degree to which user U tolerates the object OBJ, and the degree of abstraction when abstracting the object OBJ.

[0137] Next, the acquisition unit 33A of the terminal device 10 acquires the field of view image captured by the camera function of the AR goggles 200, which is a glasses-type wearable device, via the short-range wireless communication function or via the communication unit 11 (step S102).

[0138] Next, the identification unit 33D of the terminal device 10 identifies and extracts objects OBJ that the user U cannot tolerate from the field of view image of the AR goggles 200, which are glasses-type wearable devices (step S103).

[0139] Next, the generation unit 33E of the terminal device 10 automatically generates abstracted content ABS from the object OBJ according to the user U's settings regarding the tolerance and level of abstraction for the object OBJ (step S104).

[0140] In this case, the generation unit 33E may automatically generate abstract content ABS from the object OBJ using the estimation model described later.

[0141] Next, the conversion unit 33F of the terminal device 10 converts the object OBJ into an abstract content ABS that is acceptable to the user U (step S105).

[0142] Next, the display control unit 33G of the terminal device 10 displays abstract content ABS superimposed on the position of the object OBJ in the field of view image of the AR goggles 200, which is a glasses-type wearable device, either via the short-range wireless communication function or via the communication unit 11 (step S106).

[0143] Next, the transmission unit 31 of the terminal device 10 transmits (provides) information (learning data) regarding the combination of user U, a predetermined object OBJ, tolerance level, abstraction level, abstracted content ABS, and display mode to the server device 100 via the communication unit 11 (step S107).

[0144] Next, the receiving unit 32 of the terminal device 10 receives an estimation model from the server device 100 via the communication unit 11, which, as a response, outputs an abstracted content ABS and a display mode when it receives a predetermined object OBJ, tolerance, and abstraction level as an estimation result (step S108).

[0145] Next, the processing unit 33 of the terminal device 10 stores the received estimated model in the storage unit 40 and reflects it so that the generation unit 33E can automatically generate abstracted content ABS from a predetermined object OBJ using the estimated model (step S109).

[0146] [6. Variant Example] The terminal device 10 and server device 100 described above may be implemented in various other forms besides those of the embodiment described above. Therefore, the following describes modifications of the embodiment.

[0147] In the above embodiment, some or all of the processing performed by the user U's terminal device 10 may actually be performed by the AR goggles 200. For example, the AR goggles 200 may complete the processing in a standalone manner (by itself). In this case, the AR goggles 200 is assumed to be equipped with the functions of the terminal device 10 in the above embodiment.

[0148] Furthermore, in the above embodiment, some or all of the processing performed by the user U's terminal device 10 may actually be performed by the server device 100. For example, data may be aggregated and processed by the server device 100 from each user U's terminal device 10. In this case, the server device 100 is assumed to have the same functions as the terminal device 10 in the above embodiment. Also, in the above embodiment, since the server device 100 is in cooperation with the terminal device 10, from the user U's perspective, it does not matter whether the server device 100 is performing the processing of the terminal device 10.

[0149] Furthermore, in this embodiment, some or all of the processing performed by the server device 100 may actually be performed by the terminal device 10. For example, the terminal device 10 may generate an estimation model through on-device learning. That is, processing may be completed on the terminal device 10 without communication between the terminal device 10 and the server device 100. In this case, the terminal device 10 is assumed to have the functions of the server device 100 in the above embodiment. Also, in the above embodiment, since the terminal device 10 is in cooperation with the server device 100, from the perspective of the user U, it appears as if the processing of the server device 100 is also being performed by the terminal device 10. In other words, from another perspective, it can be said that the terminal device 10 is equipped with the server device 100.

[0150] Furthermore, in this embodiment, the user U's terminal device 10 may be able to set predetermined objects that it does not want other users to see. For example, predetermined objects may be things that it does not want children to see, or confidential company information that it does not want anyone other than its own employees to see. In this case, the user U uses the terminal device 10 to set predetermined objects that it does not want other users other than those it has authorized (authorized users) to see. The terminal device 10 provides and registers the authorized users and the settings for the predetermined objects to the server device 100. The server device 100 reflects the settings for the predetermined objects on the terminal devices 10 of users other than the authorized users. The settings for these predetermined objects may be changeable only by the user U (and authorized users) who set them, and may not be changeable by other users. For example, when an outsider wears AR goggles 200 to take a company tour or factory tour, confidential company objects and objects subject to confidentiality obligations may be displayed in an abstracted form according to the tolerance level and abstraction level set by the user U.

[0151] [7. Effects] As described above, the information processing device (terminal device 10, server device 100, or AR goggles 200) according to the present invention comprises: an acquisition unit 33A that acquires a field of view image of a glasses-type wearable device; an identification unit 33D that identifies an object OBJ that is unacceptable to the user U from the field of view image; a conversion unit 33F that converts the object OBJ into abstract content ABS that is acceptable to the user U; and a display control unit 33G that superimposes and displays the abstract content ABS at the position of the object OBJ in the field of view image of the glasses-type wearable device.

[0152] The information processing device according to the present invention further comprises a setting unit 33B that accepts the setting of an object OBJ that is unacceptable to the user U, and a storage unit that stores a pair of object OBJ and abstract content ABS.

[0153] The information processing device according to the present invention further includes an estimation unit 33C that estimates object OBJs that are unacceptable to user U from the attribute information of user U. The conversion unit 33F converts the object OBJs into abstract content ABS that are acceptable to user U.

[0154] The information processing device according to the present invention further comprises a generation unit 33E that automatically generates abstract content ABS from an object OBJ. A conversion unit 33F converts the object OBJ into abstract content ABS.

[0155] The information processing device according to the present invention further includes a setting unit 33B that receives a setting from a user U regarding the tolerance level for the object OBJ. The conversion unit 33F converts the object OBJ into an abstract content ABS according to the tolerance level.

[0156] The information processing device according to the present invention further includes an estimation unit 33C that estimates the user U's tolerance level for the object OBJ from the user U's attribute information. The conversion unit 33F converts the object OBJ into abstract content ABS according to the tolerance level.

[0157] The information processing device according to the present invention further includes a setting unit 33B that accepts settings regarding the level of abstraction when abstracting an object OBJ. The conversion unit 33F converts the object OBJ into an abstracted content ABS according to the level of abstraction.

[0158] The information processing device according to the present invention further includes an estimation unit 33C that estimates the level of abstraction to be used when abstracting an object OBJ from the attribute information of user U. The conversion unit 33F converts the object OBJ into an abstracted content ABS according to the level of abstraction.

[0159] The setting unit 33B queries user U about their preference level for the object OBJ, receives a response from user U regarding their preference level for the object OBJ, and further queries user U about their preference level for the converted abstracted content ABS, receives a response from user U regarding their preference level for the converted abstracted content ABS, and changes the level of abstraction when abstracting the object OBJ according to each response. The conversion unit 33F converts the object OBJ into abstracted content ABS according to the level of abstraction.

[0160] By any or a combination of the above-described processes, the information processing device according to the present invention can abstract away elements that the user does not want to see in the field of view image of a glasses-type wearable device.

[0161] [8. Hardware Configuration] Furthermore, the terminal device 10 and server device 100 according to the above-described embodiment are realized by a computer 1000 having a configuration such as that shown in Figure 10. The following explanation will use the terminal device 10 as an example. Figure 10 is a diagram showing an example of the hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output interface 1060, an input interface 1070, and a network interface 1080 are connected by a bus 1090.

[0162] The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, as well as programs read from the input device 1020, and executes various processes. The arithmetic unit 1030 can be implemented using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

[0163] The primary storage device 1040 is a memory device, such as RAM (Random Access Memory), that temporarily stores data used by the arithmetic unit 1030 for various calculations. The secondary storage device 1050 is a storage device where data used by the arithmetic unit 1030 for various calculations and various databases are registered, and can be implemented using ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. The secondary storage device 1050 may be internal storage or external storage. The secondary storage device 1050 may also be a removable storage medium such as USB (Universal Serial Bus) memory or SD (Secure Digital) memory card. The secondary storage device 1050 may also be cloud storage (online storage), NAS (Network Attached Storage), file server, etc.

[0164] The output I / F 1060 is an interface for transmitting information to be output to output devices 1010, such as displays, projectors, and printers, and is implemented using connectors of standards such as USB (Universal Serial Bus), DVI (Digital Visual Interface), and HDMI (High Definition Multimedia Interface). The input I / F 1070 is an interface for receiving information from various input devices 1020, such as mice, keyboards, keypads, buttons, and scanners, and is implemented using, for example, USB.

[0165] Furthermore, the output interface 1060 and input interface 1070 may be wirelessly connected to the output device 1010 and input device 1020, respectively. In other words, the output device 1010 and input device 1020 may be wireless devices.

[0166] Furthermore, the output device 1010 and the input device 1020 may be integrated as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated as an input / output I / F.

[0167] The input device 1020 may also be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0168] The network interface 1080 receives data from other devices via network N and sends it to the computing unit 1030, and also transmits data generated by the computing unit 1030 to other devices via network N.

[0169] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output interface 1060 and the input interface 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0170] For example, when computer 1000 functions as terminal device 10, the arithmetic unit 1030 of computer 1000 realizes the functions of control unit 30 by executing a program loaded onto primary storage device 1040. Alternatively, computer 1000's arithmetic unit 1030 may load a program obtained from another device via network interface 1080 onto primary storage device 1040 and execute the loaded program. Furthermore, computer 1000's arithmetic unit 1030 may cooperate with other devices via network interface 1080 and call and use program functions, data, etc., from other programs on other devices.

[0171] [9. Other] Although embodiments of the present invention have been described above, the present invention is not limited by the content of these embodiments. Furthermore, the aforementioned components include those that can be easily conceived by those skilled in the art, those that are substantially the same, and those that fall within the so-called equivalent range. Moreover, the aforementioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the embodiments described above.

[0172] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0173] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0174] For example, the server device 100 described above may be implemented using multiple server computers, and the configuration can be flexibly changed, such as by calling external platforms via APIs (Application Programming Interfaces) or network computing depending on the function.

[0175] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent.

[0176] Furthermore, the terms "section, module, unit" mentioned above can be replaced with "means" or "circuit," etc. For example, the acquisition unit can be replaced with acquisition means or acquisition circuit. [Explanation of Symbols]

[0177] 1. Information Processing System 10 Terminal devices 33 Processing Unit 33A Acquisition Department 33B Setting section 33C Estimation part 33D specific part 33E Generator 33F Conversion Unit 33G Display Control Unit 40 Storage section 40A Abstract Information Database 100 Server Devices 110 Communications Department 120 Storage section 121 User Information Database 122 History Information Database 123 Model Information Database 130 Control Unit 131 Acquisition Department 132 Judgment section 133 Learning Department 134 Provision Department

Claims

1. An acquisition unit that acquires visual images from a glasses-type wearable device, A unit that identifies objects that the user cannot tolerate from the aforementioned visual image and further identifies the category of such objects, A conversion unit that converts the object into abstract content, the name of the object, or a category name acceptable to the user, according to the category of the identified object, A display control unit that overlays the abstract content, the name of the object, or the category name of the object onto the position of the object in the field of view image of the glasses-type wearable device. Equipped with, The aforementioned identification unit subdivides the categories of the subject matter into major, medium, and minor classifications, respectively, for insects / animals, garbage / dust, pictures / photographs / videos, text / documents, and expressions based on specific thoughts / beliefs. If the subject matter is an insect, the major classification is a living organism, the medium classification is an insect, and the minor classification is the type / name of the insect. When the conversion unit converts the object into the abstract content, it does not convert the object into an image of another object in the same category, but rather converts the object into an abstract icon or pictogram. The display control unit overlays an abstract icon or pictogram of the object at the location where the object is located. An information processing device characterized by the following:

2. A setting unit that accepts settings for objects that the user cannot accept due to initial setup during app installation or automatic updates by the app, and also accepts manual settings by the user, A storage unit that stores pairs of objects set by the application or the user and the abstract content. The information processing apparatus according to claim 1, further comprising:

3. The system further comprises an estimation unit that uses a machine learning-based estimation model that outputs objects that the user cannot tolerate when given information on at least one of the user's attributes, past history, and current settings, and estimates objects that the user cannot tolerate based on the information on at least one of the user's attributes, past history, and current settings. The aforementioned identification unit identifies objects that the user deems unacceptable from the visual field image, The conversion unit uses an estimation model that, when inputting information about at least one of the user's attributes, past history, and current settings, along with the object, via machine learning, outputs abstract content acceptable to the user, thereby converting the identified object into the abstract content acceptable to the user. The information processing apparatus according to feature 1.

4. The system further comprises a generation unit that automatically generates caricatures of the object as abstract content from the object using image conversion or machine learning, The conversion unit converts the object into the caricatured illustration as the abstract content. The information processing apparatus according to feature 1.

5. The system further includes a setting unit that accepts settings regarding the user's tolerance level for the object, The conversion unit converts the object into abstract content, which is an abstraction of the object in stages according to the tolerance level. The information processing apparatus according to feature 1.

6. The system further includes an estimation unit that estimates the user's tolerance level for the object based on the user's attribute information. The conversion unit converts the object into abstract content, which is an abstraction of the object in stages according to the tolerance level. The information processing apparatus according to feature 1.

7. The system further includes a setting unit that accepts settings regarding the level of abstraction when abstracting the aforementioned object for each user, The conversion unit converts the object into abstracted content, which is an abstraction obtained by progressively abstracting the object according to the level of abstraction. The information processing apparatus according to feature 1.

8. The aforementioned setting unit is, The user is asked about the degree to which they like the object, and the user is given a response to the inquiry about the degree to which they like the object. Furthermore, the system queries the user for their preference regarding the converted abstracted content, and receives a response from the user regarding their preference regarding the converted abstracted content. Depending on the answer, the level of abstraction used when abstracting the aforementioned object will be changed. The conversion unit converts the object into abstracted content, which is an abstraction obtained by progressively abstracting the object according to the level of abstraction. The information processing apparatus according to feature 7.

9. The system further comprises an estimation unit that estimates the level of abstraction when abstracting the object based on the user's attribute information, The conversion unit converts the object into abstracted content, which is an abstraction obtained by progressively abstracting the object according to the level of abstraction. The information processing apparatus according to feature 1.

10. An information processing method performed by an information processing device, The process of acquiring visual images from a glasses-type wearable device, The process involves identifying objects that the user cannot tolerate from the aforementioned visual image, and further identifying the category of such objects. A conversion step of converting the object into abstract content, the name of the object, or the category name that is acceptable to the user, according to the category of the identified object, A display control step of superimposing the abstract content, the name of the object, or the category name of the object onto the position of the object in the field of view image of the glasses-type wearable device. Includes, In the aforementioned identification process, the categories of the object are subdivided into major, medium, and minor classifications, and if the object is an insect, the major classification is an organism, the medium classification is an insect, and the minor classification is the type or name of the insect. In the conversion step, when converting the object into the abstract content, the object is not converted into an image of another object in the same category, but rather into an abstract icon or pictogram. In the display control step, an icon or pictogram representing the object is superimposed and displayed at the location where the object is located. An information processing method characterized by the following:

11. Procedure for acquiring visual images from a glasses-type wearable device, A procedure for identifying objects that the user cannot tolerate from the aforementioned visual image, and further identifying the category of such objects, A conversion procedure that converts the object into abstract content, the name of the object, or the category name that is acceptable to the user, according to the category of the identified object, A display control procedure for displaying the abstract content, the name of the object, or the category name of the object superimposed on the position of the object in the field of view image of the glasses-type wearable device. An information processing program that causes a computer to execute, In the aforementioned identification procedure, the categories of the object are subdivided into major, medium, and minor classifications, and if the object is an insect, the major classification is the organism, the medium classification is the insect, and the minor classification is the type / name of the insect. In the conversion procedure described above, when converting the object into the abstract content, the object is not converted into an image of another object in the same category, but rather into an abstract icon or pictogram. In the display control procedure described above, an icon or pictogram representing the object is superimposed on the location where the object is located. An information processing program characterized by the following features.

Images