Information processing device

Abstract

According to the present invention, a mobile appliance includes a first acquiring unit, a first generating unit, and an image processing unit. The first acquiring unit acquires movement information relating to movement of a user wearing augmented reality (AR) glasses on the head, and image information representing a captured image captured by means of a first imaging device mounted on the AR glasses. The first generating unit generates a partial image cut out from the captured image by controlling a position at which a portion is to be cut out from the captured image, in accordance with the movement information. The image processing unit performs image processing with respect to the partial image.

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] XR glasses that apply XR technologies such as AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality) have become widespread. One example of the use of XR glasses is to assist workers in performing their work by displaying information about the work on the XR glasses. For example, Patent Document 1 listed below discloses a maintenance support system that assists with work when a part in electronic device equipment breaks down. The maintenance support system notifies the AR glasses used by the maintenance worker of information about a replacement part for the broken part and notifies the worker that the broken part can be replaced. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-170249 Summary of the Invention [Problem to be solved by the invention]

[0004] Services using XR glasses require recognition of the state of the real space based on images captured by a camera mounted on the XR glasses, and various processes according to the state of the real space (for example, presenting information according to the situation in the real space). Because the state of the real space changes from moment to moment, high-speed processing is required to provide services that reflect the real-time state. However, hardware computational resources are limited. To perform high-speed processing, it is desirable to reduce the amount of calculation as much as possible.

[0005] An object of the present invention is to provide an information processing device that performs image-based processing more efficiently. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present invention includes an acquisition unit that acquires motion information relating to the motion of a user wearing an imaging device on their head and image information indicating an image captured by the imaging device, a generation unit that generates a partial image cut out from the captured image by controlling the position at which a portion is cut out from the captured image in accordance with the motion information, and an image processing unit that performs image processing on the partial image. [Effects of the Invention]

[0007] According to one aspect of the present invention, image-based processing can be performed more efficiently than when the entire captured image is the processing target. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an overview of an information processing system 1 according to a first embodiment. [Figure 2] 1 is a block diagram showing a configuration of an information processing system 1 according to a first embodiment. [Figure 3] FIG. 1 is an explanatory diagram showing the appearance of AR glasses 10A. [Figure 4] FIG. 1 is a block diagram showing the configuration of AR glasses 10A. [Figure 5] FIG. 2 is a block diagram showing the configuration of a mobile device 20A. [Figure 6] FIG. 2 is a front view of the device DV1. [Figure 7] FIG. 10 is a diagram showing the relationship between a device DV1 and an XY coordinate system in real space. [Figure 8] 10 is a diagram showing the relationship between a captured image PC showing a device DV1 and an xy coordinate system. FIG. [Figure 9] 10 is a flowchart showing the operation of the processing device 206. [Figure 10]FIG. 2 is a block diagram showing a configuration of an information processing system 2 according to a second embodiment. [Figure 11] FIG. 2 is a block diagram showing the configuration of the AR glasses 10B. [Figure 12] FIG. 2 is a block diagram showing the configuration of a mobile device 20B. [Figure 13] 1 is a diagram schematically illustrating the visual field range of a user U. FIG. [Figure 14] 1 is a diagram schematically illustrating the visual field range of a user U. FIG. [Figure 15] FIG. 10 is a front view of the device DV2. [Figure 16] 10 is a diagram showing an example of the positional relationship between a captured image PC and the visual field range of a user U. FIG. [Figure 17] 10 is a diagram showing an example of the positional relationship between a captured image PC and the visual field range of a user U. FIG. [Figure 18] 10 is a flowchart showing the operation of the processing device 206. DETAILED DESCRIPTION OF THE INVENTION

[0009] A. First embodiment Hereinafter, the configuration of an information processing system 1 including an information processing device according to a first embodiment of the present invention will be described with reference to FIGS.

[0010] A-1. System Configuration FIG. 1 is a diagram illustrating an overview of an information processing system 1 according to the first embodiment. FIG. 2 is a block diagram illustrating the configuration of the information processing system 1 according to the first embodiment. The information processing system 1 includes AR glasses 10A worn on the head of a user U, a mobile device 20A held by the user U, and an inertial measurement unit 30 that measures the movement of the head of the user U. As will be described later, the AR glasses 10A are equipped with a first imaging device 124A. Therefore, it can be said that the user U is wearing the first imaging device 124A on his or her head. The mobile device 20A is an example of an information processing device.

[0011] In this embodiment, the information processing system 1 assists the user U in performing work using image processing with AI (artificial intelligence). For example, the user U performs wiring work between multiple devices DV stored in a rack RA. For example, if the user U inserts a connector into the wrong port while performing the wiring work, it is expected that the value of the indicator IN or the lighting state of the lamp LP will differ from the normal state. For this reason, the information processing system 1 monitors the value of the indicator IN of the device DV, the lighting state of the lamp LP, and the like using image processing with AI. Hereinafter, components monitored by the information processing system 1, such as the indicator IN and the lamp LP, will be referred to as "monitored objects." In this embodiment, the monitored objects are components that display the operating status of the devices DV. If the display state of the monitored objects differs from the normal state, the information processing system 1 notifies the user U using the AR glasses 10A. This allows the user U to pay less attention to the monitored objects and concentrate on the wiring work.

[0012] Furthermore, the multiple devices DV may be different types of devices. Therefore, the arrangement of the indicator IN and lamp LP on the operation surface of each device DV, as well as the value of the indicator IN and the color of the lamp LP in a normal state, are also different. By using AI, even in an environment where different types of devices DV coexist, it is possible to identify the monitored object from the image and determine whether the monitored object is in a normal state.

[0013] A-2. AR Glasses 10A The AR glasses 10A are a see-through wearable display worn on the head of a user U. The AR glasses 10A display virtual objects on display panels provided in each of the binocular lenses 110A, 110B under the control of a portable device 20A. The AR glasses 10A are an example of a device equipped with a first image capture device 124A. Note that, for example, a goggle-shaped see-through head-mounted display having the same functions as the AR glasses 10A may be used as the device equipped with the first image capture device 124A.

[0014] 3 is an explanatory diagram showing the external appearance of the AR glasses 10A. The AR glasses 10A have temples 101 and 102, a bridge 103, body parts 104 and 105, rims 106 and 107, lenses 110A and 110B, and an imaging lens LEN that can be seen from the outside.

[0015] The bridge 103 is provided with an imaging lens LEN that constitutes the first imaging device 124A shown in FIG.

[0016] The body 104 is provided with a display panel for the left eye and an optical member for the left eye. The display panel is, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The display panel for the left eye displays an image based on control from, for example, a mobile device 20A described later. The optical member for the left eye is an optical member that guides light emitted from the display panel for the left eye to the lens 110A. The body 104 is also provided with a sound emitting device 122 described later.

[0017] A display panel for the right eye and an optical member for the right eye are provided on the body 105. The display panel for the right eye displays an image, for example, under control of the mobile device 20A. The optical member for the right eye is an optical member that guides light emitted from the display panel for the right eye to 110B. The body 105 is also provided with a sound emitting device 122, which will be described later.

[0018] Rim 106 holds lens 110A, and rim 107 holds lens 110B.

[0019] Each of lenses 110A and 110B has a half mirror. The half mirror of lens 110A transmits light representing real space, thereby guiding the light representing real space to the left eye of user U. The half mirror of lens 110A also reflects light guided by an optical member for the left eye to the left eye of user U. The half mirror of lens 110B transmits light representing real space, thereby guiding the light representing real space to the right eye of user U. The half mirror of lens 110B also reflects light guided by an optical member for the right eye to the right eye of user U.

[0020] When the user U wears the AR glasses 10A, the lenses 110A and 110B are positioned in front of the left and right eyes of the user U. The user U wearing the AR glasses 10A can visually recognize a real space represented by light transmitted through the lenses 110A and 110B superimposed on an image projected onto a display panel by the projection device 121.

[0021] FIG. 4 is a block diagram showing the configuration of the AR glasses 10A. In addition to the temples 101 and 102, bridge 103, body parts 104 and 105, rims 106 and 107, lenses 110A and 110B, and imaging lens LEN described above, the AR glasses 10A also include a projection device 121, a sound emitting device 122, a communication device 123, a first imaging device 124A, a storage device 125, a processing device 126, and a bus 127. The components shown in FIG. 4 are stored in the body parts 104 and 105, for example. The projection device 121, the sound emitting device 122, the communication device 123, the first imaging device 124A, the storage device 125, and the processing device 126 are connected to one another by a bus 127 for communicating information. The bus 127 may be configured using a single bus, or may be configured using different buses between elements such as devices.

[0022] The projection device 121 includes a lens 110A, a display panel for the left eye, an optical member for the left eye, a lens 110B, a display panel for the right eye, and an optical member for the right eye. Light representing real space passes through the projection device 121. The projection device 121 displays an image based on control from the mobile device 20A. In this embodiment, the image displayed by the projection device 121 is, for example, a warning message notified by a notification unit 233 (described later).

[0023] The sound emitting device 122 is located on each of the torsos 104 and 105. The sound emitting device 122 may not be located on each of the torsos 104 and 105, but may be located on, for example, one of the torsos 104 and 105, at least one of the temples 101 and 102, or the bridge 103. The sound emitting device 122 is, for example, a speaker. The sound emitting device 122 is controlled by the mobile device 20A directly or via a processing device 126 of the AR glasses 10A. The sound emitting device 122 outputs a work auxiliary sound, such as an alarm sound to alert the user U while working. The sound emitting device 122 may not be included in the AR glasses 10A and may be separate from the AR glasses 10A.

[0024] The communication device 123 communicates with the communication device 203 (see FIG. 4) of the mobile device 20A using wireless communication or wired communication. In this embodiment, the communication device 123 communicates with the communication device 203 of the mobile device 20A using short-range wireless communication such as Bluetooth (registered trademark).

[0025] The first imaging device 124A captures an image of a subject and outputs image information indicating the captured image (hereinafter referred to as a "captured image PC"). In this embodiment, the imaging direction of the first imaging device 124A is positioned so as to match the direction of the head of the user U. Therefore, the captured image PC captures objects and the like located in front of the user U (in the field of view). For example, while the user U is working, a captured image PC is captured that shows a device DV stored in the rack RA. The captured image PC generated by the first imaging device 124A is transmitted as image information to the mobile device 20A via the communication device 123. The first imaging device 124A repeats imaging at a predetermined imaging interval and transmits the generated image information to the mobile device 20A each time an image is captured.

[0026] The first imaging device 124A has, for example, an imaging optical system and an imaging element. The imaging optical system is an optical system including at least one imaging lens LEN (see FIG. 3). For example, the imaging optical system may have various optical elements such as a prism, or may have a zoom lens or a focus lens. The imaging element is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor.

[0027] The storage device 125 is a recording medium readable by the processing device 126. The storage device 125 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), and an EEPROM (Electrically Erasable Programmable Read Only Memory). The volatile memory is, for example, a RAM (Random Access Memory). The storage device 125 stores the program PG1.

[0028] The processing device 126 includes one or more central processing units (CPUs). The one or more CPUs are examples of one or more processors. Each of the processor and the CPU is an example of a computer.

[0029] The processing device 126 reads the program PG1 from the storage device 125. The processing device 126 executes the program PG1 to function as the operation control unit 130. The operation control unit 130 may be configured by circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array).

[0030] The operation control unit 130 controls the operation of the AR glasses 10A. For example, the operation control unit 130 provides the projection device 121 with a control signal for image display that the communication device 123 has received from the mobile device 20A. The projection device 121 displays an image indicated by the control signal for image display. Furthermore, the operation control unit 130 provides the sound emitting device 122 with a control signal for audio output that the communication device 123 has received from the mobile device 20A. The sound emitting device 122 emits a sound indicated by the control signal for audio output. Furthermore, the operation control unit 130 transmits image information indicating the captured image PC captured by the first imaging device 124A to the mobile device 20A.

[0031] A-3. Portable device 20A The mobile device 20A monitors the monitoring target using a captured image PC captured by the first imaging device 124A of the AR glasses 10A. Furthermore, when an abnormality in the monitoring target is detected, the mobile device 20A notifies the user U using the AR glasses 10A. The mobile device 20A is preferably, for example, a smartphone, a tablet, or the like.

[0032] 5 is a block diagram showing the configuration of mobile device 20A. Mobile device 20A includes touch panel 201, communication device 203, storage device 205, processing device 206, and bus 207. Touch panel 201, communication device 203, storage device 205, and processing device 206 are interconnected by bus 207 for communicating information. Bus 207 may be configured using a single bus, or may be configured using different buses for each device.

[0033] The touch panel 201 displays various information to the user U and detects touch operations by the user U. The touch panel 201 serves as both an input device and an output device. For example, the touch panel 201 is configured by bonding a touch sensor unit capable of detecting touch operations between a cover glass and various display panels such as a liquid crystal display panel or an organic EL display panel. For example, when the user U's finger is in contact with the touch panel 201, the touch panel 201 periodically detects the contact position of the user U's finger on the touch panel 201 and transmits touch information indicating the detected contact position to the processing device 206.

[0034] The communication device 203 communicates with the communication device 123 (see FIG. 4) of the AR glasses 10A using wireless communication or wired communication. In this embodiment, the communication device 203 communicates with the communication device 123 using the same type of short-range wireless communication as the communication device 123 of the AR glasses 10A. The communication device 203 also communicates with the inertial measurement unit 30 (see FIGS. 1 and 2) using wireless communication or wired communication. In this embodiment, the communication device 203 communicates with the inertial measurement unit 30 using short-range wireless communication.

[0035] The storage device 205 is a recording medium readable by the processing device 206. The storage device 205 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM. The storage device 205 stores a program PG2 and a trained model LM.

[0036] The trained model LM is a learning model that has learned the state of the monitored object. More specifically, the trained model LM is a model that has learned the normal and abnormal states of the monitored object using, for example, deep learning using a convolutional neural network. When an image of the monitored object's appearance is input to the trained model LM, it outputs whether the display of the monitored object is normal. As described above, the monitored object is a component that displays the operating status of the device DV. Therefore, if the display of the monitored object is abnormal, the operating status of the device DV may be abnormal. In other words, the trained model LM can be used to monitor whether the operating status of the device DV is normal. The method for generating the trained model LM is a well-known technique, and therefore a detailed description thereof will be omitted. The image processing unit 232, which will be described later, detects abnormalities in the monitored object using the trained model LM.

[0037] Processing unit 206 includes one or more CPUs. The one or more CPUs are examples of one or more processors. Each of the processors and CPUs is an example of a computer.

[0038] The processing device 206 reads the program PG2 from the storage device 205. By executing the program PG2, the processing device 206 functions as a first acquisition unit 230A, a first generation unit 231A, an image processing unit 232, and a notification unit 233. At least one of the first acquisition unit 230A, the first generation unit 231A, the image processing unit 232, and the notification unit 233 may be configured by a circuit such as a DSP, an ASIC, a PLD, or an FPGA.

[0039] The inertial measurement unit 30 measures, for example, the acceleration of the user U's head on each of three axes representing three-dimensional space, and the angular velocity of the user U's head when each of these three axes is used as an axis of rotation. The inertial measurement unit 30 is attached to a hat worn by the user U. Thus, the inertial measurement unit 30 measures the acceleration and angular velocity each time the user U's head moves. The user U wears AR glasses 10A on their head, and the AR glasses 10A have a first image capture device 124A built in. Therefore, the amount of movement of the first image capture device 124A can be measured using the measurement values ​​of the inertial measurement unit 30.

[0040] In this embodiment, the inertial measurement unit 30 is attached to a hat worn by the user U, but the inertial measurement unit 30 may be built into the AR glasses 10A, for example. In this case, the first acquisition unit 230A acquires the measurement values ​​transmitted from the communication device 123 of the AR glasses 10A via the communication device 203. The inertial measurement unit 30 is not limited to being attached to a hat worn by the user U, and may be attached anywhere that moves in conjunction with the movement of the user U's head.

[0041] Furthermore, in this embodiment, the inertial measurement unit 30 is used to acquire information about the movement of the user U's head. However, instead of the inertial measurement unit 30, for example, a geomagnetic sensor may be used. The geomagnetic sensor detects the geomagnetism surrounding the Earth. The geomagnetic sensor detects the values ​​of magnetic force in the three axial directions of X, Y, and Z. The movement of the user U's head is estimated based on the changes in the magnetic force.

[0042] Furthermore, the first acquisition unit 230A acquires image information indicating a captured image PC captured by the first imaging device 124A mounted on the AR glasses 10A. The first acquisition unit 230A acquires image information indicating the captured image PC captured by the first imaging device 124A received by the communication device 203. As described above, the captured image PC includes an object or the like located in front of (in the field of view of) the user U. The first acquisition unit 230A sequentially acquires image information and information related to the movement of the user U's head while the user U is working.

[0043] The first generating unit 231A generates a partial image PS cut out from the captured image PC by controlling the position at which a portion is cut out from the captured image PC according to the movement information. As described above, while the user U is working, a captured image PC showing the device DV stored in the rack RA is captured. The first generating unit 231A cuts out a portion showing the monitored object from the captured image PC captured by the first imaging device 124A to generate a partial image PS.

[0044] The generation of the partial image PS performed by the first generation unit 231A will be described in more detail with reference to FIGS. 6 to 8. FIG. 6 is a front view of a device DV1, which is an example of the device DV. The device DV1 includes an indicator IN1, a lamp LP1, and multiple ports PT. The monitored objects of the device DV1 are the indicator IN1 and the lamp LP1. For convenience of explanation, the following focuses on the indicator IN1 among the monitored objects. For example, as shown in FIG. 7, an XY coordinate system having an X axis and a Y axis is defined in real space. For example, the reference time is set to time T1, and the imaging range Rt1 of the first imaging device 124A at time T1 is assumed to be an area surrounded by (X0, Y0), (Xe, Y0), (Xe, Ye), and (X0, Ye). The indicator IN1 is assumed to be an area surrounded by (X1, Y1), (X2, Y1), (X2, Y2), and (X1, Y2) in the coordinate system in real space.

[0045] FIG. 8 is a diagram showing a captured image PC. The captured image PC captured at time T1 in the imaging range Rt1 is defined as captured image PC1. An xy coordinate system having an x-axis and a y-axis is defined on the captured image PC. The captured image PC has coordinates indicated by (x0, y0) to (xe, Ye). In the captured image PC1, the indicator IN1 is defined as an area surrounded by (x1, y1), (x2, y1), (x2, y2), and (x1, y2). Hereinafter, the "position of the monitored object in the captured image PC" is defined as a set of coordinates that specifies the area in which the monitored object is captured in the captured image PC.

[0046] The position of the indicator IN1 in the captured image PC1 may be specified by the user U tracing the outer edge of the indicator IN1 on the captured image PC1 displayed on the touch panel 201. Alternatively, the position of the indicator IN1 in the captured image PC1 may be identified, for example, by performing image recognition using the learned model LM in the processing device 206. Hereinafter, an image in which the position of the monitored object in the captured image PC is specified or identified will be referred to as a "reference image." The captured image PC1 is assumed to be the reference image. The first generation unit 231A generates an image of an area surrounded by hatched (x1, y1), (x2, y1), (x2, y2), and (x1, y2) as a partial image PS corresponding to the indicator IN1.

[0047] Here, it is assumed that the user U moves from time T1 to time T2 (time T2 is after time T1), and the position of the first imaging device 124A changes. The amount of movement of the first imaging device 124A from time T1 to time T2 is M1(α, β) using XY coordinate values. α and β are positive numbers. The amount of movement M1 can be calculated based on the measurement values ​​of the inertial measurement unit 30. In this case, the imaging range Rt2 at time T2 is the area surrounded by (X0+α, Y0+β), (Xe+α, Y0+β), (Xe+α, Ye+β), and (X0+α, Ye+β). Meanwhile, the coordinates in real space of the indicator IN1 are the same as those at time T1.

[0048] As shown in FIG. 8, a captured image PC captured at time T2 of an imaging range Rt2 is designated as a captured image PC2. Like the captured image PC1, the captured image PC2 has coordinates represented by (x0, y0) to (xe, ye). However, due to a change in the position of the imaging range Rt in real space from time T1 to time T2, the coordinates of the indicator IN1 in the captured image PC2 differ from the coordinates of the indicator IN1 in the captured image PC1. Specifically, using m1(γ, δ), which is a conversion of the movement amount M1(α, β) of the first imaging device 124A into a movement amount on the captured image PC, the indicator IN1 in the captured image PC2 is defined as a region surrounded by (x1-γ, y1-δ), (x2-γ, y1-δ), (x2-γ, y2-δ), and (x1-γ, y2-δ). γ and δ are positive numbers. That is, the position of the indicator IN1 in the captured image PC2 changes by −m1 compared to the captured image PC1, which is the reference image. In this case, the first generation unit 231A generates an image of the area surrounded by (x1-γ, y1-δ), (x2-γ, y1-δ), (x2-γ, y2-δ), and (x1-γ, y2-δ) as the partial image PS corresponding to the indicator IN1.

[0049] Thereafter, the first generation unit 231A calculates the amount of movement Mx (x is an integer equal to or greater than 1) of the first imaging device 124A from time Tx to time Tx+1 based on the measurement values ​​of the inertial measurement unit 30. The first generation unit 231A also converts the amount of movement Mx of the first imaging device 124A into an amount of movement mx on the captured image PC. The first generation unit 231A regards a position obtained by moving the amount of movement (-mx) from the position (coordinates) of the indicator IN1 on the captured image PCx at time Tx as the position of the indicator IN1 on the captured image PCx+1 at time Tx+1, and generates a partial image PS.

[0050] In this way, the first generation unit 231A identifies the position of the monitored object (e.g., indicator IN1) in the captured image PC at each time using the measurement values ​​of the inertial measurement unit 30. In other words, the first generation unit 231A changes the coordinates of the area of ​​the captured image PC to be the partial image PS based on the measurement values ​​of the inertial measurement unit 30. Therefore, compared to tracking the position of the monitored object in the captured image PC using an image processing technique such as background subtraction, it is possible to reduce the processing load on the processing device 206 and increase the processing speed of the processing device 206.

[0051] In the above description, a two-dimensional XY coordinate system has been used for convenience, but the first generating unit 231A may generate the partial image PS in consideration of the amount of movement of the user U in three-dimensional coordinates.

[0052] The image processing unit 232 performs image processing on the partial image PS cut out by the first generation unit 231A. In this embodiment, the image processing is state monitoring of the monitored object using AI. The image processing unit 232 uses the trained model LM stored in the storage device 205 to determine whether the state of the monitored object shown in the partial image PS generated by the first generation unit 231A is normal or not.

[0053] The image to be processed by the image processing unit 232 is not the captured image PC of the first imaging device 124A itself, but the partial image PS generated by the first generation unit 231A. Therefore, in this embodiment, the size of the image to be processed is smaller than when the captured image PC of the first imaging device 124A itself is the processing target. This reduces the processing load of the processing device 206, and increases the processing speed of the processing device 206.

[0054] The image processing unit 232 may monitor the monitoring target using other methods instead of using AI. For example, the image processing unit 232 may monitor the monitoring target using a method such as reading the value of the indicator IN in the partial image PS using an OCR (Optical Character Reader) and determining whether the read value is within a predetermined threshold range. Even in this case, the size of the image to be processed is smaller than the captured image PC. This reduces the processing load of the processing device 206 and increases the processing speed of the processing device 206.

[0055] The notification unit 233 notifies the user U when the image processing unit 232 determines that there is an abnormality in the state of the monitored object. The notification unit 233 generates, for example, a control signal (a control signal for image display) for causing the projection device 121 of the AR glasses 10A to display a warning message, and transmits the control signal to the AR glasses 10A via the communication device 203. The notification unit 233 also generates, for example, a control signal (a control signal for sound output) for causing the sound emission device 122 of the AR glasses 10A to output a warning sound, and transmits the control signal to the AR glasses 10A via the communication device 203. Both visual notification, such as displaying a warning message, and auditory notification, such as outputting a warning sound, may be performed, or only one of them may be performed.

[0056] When the user U receives the warning message or the warning sound, the user U can realize that there is a possibility that the content or procedure of his / her work is incorrect. In this case, the user U can quickly respond to the work error by checking the content or procedure of the work. This improves the efficiency and accuracy of the work.

[0057] A-4. Operation of the processing unit 206 9 is a flowchart showing the operation of the processing device 206. The processing device 206 functions as the first acquisition unit 230A and acquires a reference image, which is an image PC captured by the first imaging device 124A at a reference time (step S101). The processing device 206 identifies the position of the monitored object in the reference image (step S102). As described above, the position of the monitored object in the reference image may be specified by the user U or may be identified by the processing device 206.

[0058] The processing device 206 functions as the first generation unit 231A, and cuts out an area including the monitored object from the reference image to generate a partial image PS (step S103). The processing device 206 also functions as the image processing unit 232, and performs image processing on the partial image PS generated in step S103 (step S104). More specifically, the processing device 206 applies the learned model LM to the partial image PS to determine whether or not there is an abnormality in the state of the monitored object.

[0059] If there is an abnormality in the state of the monitored object (step S105: YES), the processing device 206 functions as the notification unit 233, generates a control signal for causing the AR glasses 10A to output a warning message or a warning sound, and transmits the control signal to the AR glasses 10A. That is, the processing device 206 functions as the notification unit 233, notifies the user U of the abnormality (step S106), and ends the processing of this flowchart.

[0060] If there is no abnormality in the state of the monitored object (step S105: NO), the processing device 206 functions as the first acquisition unit 230A and acquires the measurement values ​​of the inertial measurement unit 30 (step S107). The processing device 206 functions as the first generation unit 231A and determines whether the head of the user U has moved based on the measurement values ​​of the inertial measurement unit 30 (step S108).

[0061] If the head of the user U has moved (step S108: YES), the processing device 206 functions as the first generating unit 231A and changes the position of the captured image PC to be cut out as the partial image PS (step S109). If the head of the user U has not moved (step S108: NO), the processing device 206 shifts the process to step S110.

[0062] The processing device 206 functions as the first acquisition unit 230A until monitoring of the monitored object is terminated (step S110: NO), acquires the captured image PC of the first imaging device 124A (step S111), returns to step S103, and repeats the subsequent processes. The end of monitoring corresponds to, for example, when the user U has finished his work and moved away from the monitored object. Then, when monitoring of the monitored object is terminated (step S110: YES), the processing device 206 terminates the processing of this flowchart.

[0063] A-5. Summary of the first embodiment As described above, according to the first embodiment, in the mobile device 20A, the first generation unit 231A cuts out a part of the captured image PC as a partial image PS, and the image processing unit 232 performs image processing on the partial image PS. Therefore, according to the first embodiment, the processing load on the processing device 206 is reduced compared to when image processing is performed on the entire captured image.

[0064] Furthermore, according to the first embodiment, a partial image PS is generated by cutting out an area corresponding to a pre-specified object from the captured image PC in accordance with the movement of the head of the user U. Therefore, according to the first embodiment, the processing load on the processing device 206 is reduced compared to tracking a specified portion in an image using image analysis.

[0065] Moreover, according to the first embodiment, the first acquisition unit 230A acquires information regarding the movement of the head of the user U using the inertial measurement unit 30. Therefore, according to the first embodiment, the movement of the head of the user U, i.e., the change in the imaging direction of the first imaging device 124A, is detected with high accuracy. Furthermore, according to the first embodiment, the processing load on the processing device 206 is reduced compared to tracking the movement of the head of the user U using image analysis.

[0066] Furthermore, according to the first embodiment, the state of the monitored object is monitored while the user U is working, so the user U can reduce the level of attention he or she pays to the monitored object. This allows the user U to concentrate more on the work, improving work efficiency.

[0067] B. Second embodiment The configuration of an information processing system 2 including an information processing device according to a second embodiment of the present invention will be described below with reference to Figures 10 to 18. In the following description, for the sake of simplicity, the same components as those in the first embodiment will be denoted by the same reference numerals, and a description of their functions may be omitted. In addition, in the following description, for the sake of simplicity, differences between the second embodiment and the first embodiment will be mainly described.

[0068] B-1. System configuration of information processing system 2 10 is a block diagram showing the configuration of an information processing system 2 according to the second embodiment. The information processing system 2 includes AR glasses 10B worn on the head of a user U and a mobile device 20B held by the user U.

[0069] B-2.AR Glasses 10B 11 is a block diagram showing the configuration of the AR glasses 10B. The AR glasses 10B include an infrared light emitting device 128 in addition to the configuration of the AR glasses 10A shown in FIG. 4. The infrared light emitting device 128 irradiates infrared light onto the eyes (e.g., onto the cornea) of a user U wearing the AR glasses 10B. The infrared light emitting device 128 includes an irradiator on the surfaces of the rims 106 and 107 that face the eyes of the user U, for example.

[0070] Furthermore, the AR glasses 10B include a second imaging device 124B in addition to the first imaging device 124A. As described above, the first imaging device 124A has an imaging lens LEN on the bridge 103 of the AR glasses 10B, and captures an image of an object located in front of (in the field of view of) the user U. As in the first embodiment, the image captured by the first imaging device 124A is referred to as a captured image PC.

[0071] On the other hand, the second imaging device 124B has an imaging lens LEN (not shown) on the surface of the rims 106 and 107 that faces the eyes of the user U when the user U is wearing the AR glasses 10B. The second imaging device 124B captures an image including the eyes of the user U. As described above, the eyes of the user U are irradiated with infrared light by the infrared light emitting device 128. Therefore, the image captured by the second imaging device 124B shows the eyes of the user U irradiated with infrared light. The image captured by the second imaging device 124B is referred to as an eye-tracking image PE.

[0072] B-3. ​​Mobile Device 20B FIG. 12 is a block diagram showing the configuration of the mobile device 20B. The processing device 206 of the mobile device 20B functions as an eye gaze tracking unit 234 in addition to the functions shown in FIG. 5. The eye gaze tracking unit 234 tracks the gaze movement of the user U and calculates gaze information related to the gaze movement of the user U. In this embodiment, the eye gaze tracking unit 234 tracks the gaze movement of the user U using the corneal reflex method. As described above, the infrared light emitting device 128 of the AR glasses 10B emits infrared light, generating light reflection points on the cornea of ​​the user U's eye. The eye gaze tracking unit 234 identifies the light reflection points on the cornea and the pupil from the gaze tracking image PE captured by the second image capturing device 124B. Then, the eye gaze tracking unit 234 calculates the direction of the user U's eyeball, i.e., the direction of the user U's gaze, based on the light reflection points and other geometric features. The eye gaze tracking unit 234 continuously calculates the gaze direction of the user U and calculates gaze information related to the gaze movement of the user U.

[0073] 5. Moreover, the processing device 206 functions as a second acquisition unit 230B in place of the first acquisition unit 230A shown in Fig. 5. Moreover, the processing device 206 functions as a second generation unit 231B in place of the first generation unit 231A shown in Fig. 5.

[0074] The second acquisition unit 230B acquires movement information related to the movement of the user U wearing the AR glasses 10A on his / her head. In the second embodiment, the second acquisition unit 230B acquires gaze information related to the movement of the user U's line of sight as the movement information. The second acquisition unit 230B acquires the gaze information calculated by the gaze tracking unit 234. The second acquisition unit 230B successively acquires the gaze information while the user U is working.

[0075] Furthermore, the second acquisition unit 230B acquires image information of a captured image PC captured by the first imaging device 124A mounted on the AR glasses 10B. The second acquisition unit 230B acquires image information indicating the captured image PC of the first imaging device 124A received by the communication device 203. As described above, the captured image PC of the first imaging device 124A includes an object or the like located in front of (in the field of view of) the user U. The second acquisition unit 230B sequentially acquires image information while the user U is working.

[0076] Furthermore, the second acquisition unit 230B acquires image information of an image PE for gaze tracking captured by the second imaging device 124B mounted on the AR glasses 10B. The image PE for gaze tracking acquired by the second acquisition unit 230B is used for gaze tracking performed by the gaze tracking unit 234.

[0077] The second generation unit 231B generates a partial image PS cut out from the captured image PC by controlling the position at which a portion is cut out from the captured image PC according to the movement information. As described above, while the user U is working, a captured image PC showing the device DV stored in the rack RA is captured. The second generation unit 231B generates a partial image PS by cutting out an area outside the area visually recognized by the user U from the captured image PC captured by the first imaging device 124A based on the line of sight information.

[0078] The generation of the partial image PS performed by the second generation unit 231B will be described in more detail with reference to Fig. 13 to Fig. 17. Fig. 13 and Fig. 14 are diagrams schematically showing the field of view of the user U. More specifically, Fig. 13 is a diagram showing the field of view in the direction of the user U's line of sight. Fig. 14 is a diagram showing the field of view as seen from above the user U.

[0079] The visual field of the user U is mainly divided into a central visual field V1, an effective visual field V2, and a peripheral visual field V3. Outside the peripheral visual field V3, there exists an outside visual field VX, which is the outside world of the visual field.

[0080] The central visual field V1 is the area where the user U's ability to discriminate visual information is most pronounced. For convenience, the center point of the central visual field V1 is defined as the viewpoint VP. The direction of the user U's line of sight L is defined as the direction from the user U to the viewpoint VP. If the plane parallel to the direction of separation between the user U's eyes is defined as the horizontal plane, the central visual field V1 in the horizontal plane has a range of up to approximately 1° with respect to the line of sight direction L. The angle of the outer edge of each visual field range with respect to the line of sight direction L is called the "field of view angle." For example, the field of view angle of the central visual field V1 is approximately 1°.

[0081] Although the user U's discrimination ability for the effective visual field V2 is lower than that for the central visual field V1, he or she is able to recognize simple characters such as numbers as visual information. In other words, within a range closer to the viewpoint VP than the effective visual field V2, the user U can recognize character information. The effective visual field V2 on the horizontal plane is in the range of approximately 1° to 10° with respect to the line of sight L. In other words, the field of view angle of the effective visual field V2 is approximately 10°.

[0082] The user U's discrimination ability in the peripheral field V3 is required to be at least able to distinguish between the presence and absence of an object. The peripheral field V3 is divided into multiple ranges depending on the level of the user U's discrimination ability. Specifically, the peripheral field V3 is divided into a first peripheral field V3A that can recognize shapes (symbols), a second peripheral field V3B that can distinguish changing colors, and a third peripheral field V3C that is a field of view (auxiliary field) that can recognize the presence of visual information. The first peripheral field V3A in the horizontal plane ranges from approximately 10° to 30° with respect to the line of sight L. That is, the field of view of the first peripheral field V3A is approximately 30°. The second peripheral field V3B in the horizontal plane ranges from approximately 30° to 60° with respect to the line of sight L. That is, the field of view of the second peripheral field V3B is approximately 60°. The third peripheral field V3C in the horizontal plane ranges from approximately 60° to 100° with respect to the line of sight L. That is, the field of view angle of the third peripheral field V3C is approximately 100°.

[0083] The outside field of view VX is an area where the user U is unaware of visual information, i.e., cannot see.

[0084] Thus, the user U's discrimination ability increases the closer to the central visual field V1 and decreases the farther from the central visual field V1. Note that the width of these visual field ranges varies from person to person. Also, Figures 13 and 14 are schematic illustrations of the positional relationship of each visual field range, and the ratio of the width of each visual field range and the angle with the line of sight L, etc., may differ from the actual situation.

[0085] 15 is a front view of device DV2, an example of device DV. Device DV2 includes multiple switches SW1 to SW14 and a lamp LP2. Each of switches SW1 to SW14 can be in an on or off state. In FIG. 12, all of switches SW1 to SW14 are in an off state. Furthermore, lamp LP2 can be in an off state or an on state, for example.

[0086] In the first embodiment, for example, when switches SW1 and SW2 of switches SW1 to SW14 are designated as the monitoring object, the first generating unit 231A identifies the positions of switches SW1 and SW2 in the captured image PC based on the movement of the head of the user U, and generates a partial image PS. That is, in the first embodiment, the monitoring object was fixed.

[0087] In contrast to this, in the second embodiment, the monitored object is not fixed, but is changed based on the field of view range of the user U. More specifically, the second generating unit 231B generates the partial image PS by cutting out an area outside the area where the user U can recognize predetermined information from the captured image PC based on the line of sight information.

[0088] As described above, the user U does not have the ability to discriminate between all areas visible to the eye, but rather the farther an area is from the viewpoint VP, the lower the discrimination ability. For this reason, in the second embodiment, the second generation unit 231B cuts out areas away from the user U's viewpoint VP as partial images PS, and subjects these to image processing using AI performed by the image processing unit 232. On the other hand, areas close to the user U's viewpoint VP are areas where the user U has a high discrimination range, as described above. Therefore, for areas close to the viewpoint VP, the image processing unit 232 does not perform image processing, but the user U himself determines the state.

[0089] In this embodiment, the second generation unit 231B determines the range to be cut out as the partial image PS based on the above-mentioned field of view range. For example, the second generation unit 231B cuts out, as the partial image PS, a portion of the captured image PC that corresponds to the peripheral field of view V3 and the outside field of view VX. In this case, the area outside the area in which the predetermined information can be recognized is the peripheral field of view V3 and the outside field of view VX. The predetermined information is text information. Note that, although it depends on the angle of view of the first imaging device 124A, the outside field of view VX is generally not captured in the captured image PC.

[0090] At this time, the second generation unit 231B identifies the position of the viewpoint VP of the user U based on the line-of-sight information and extracts a portion that is a predetermined distance or more from the viewpoint VP as the partial image PS. The predetermined distance can be calculated geometrically, for example, from the above-mentioned field of view angle. For example, if the peripheral field of view V3 and the outside field of view VX are used as the partial image PS, the distance from the viewpoint VP to the peripheral field of view V3 can be calculated by calculating D × tan θ, where D is the distance between the imaging object, such as a device DV, and the user U (first imaging device 124A), and θ is the field of view angle of the effective field of view V2 adjacent to the peripheral field of view V3. Alternatively, for example, the visual characteristics of the user U may be measured in advance, and the predetermined distance may be changed to match the visual characteristics of the user U.

[0091] 16 and 17 are diagrams illustrating an example of the positional relationship between the captured image PC and the visual field of the user U. For example, as shown in FIG. 16, when the user U's viewpoint VP is located at the center of the device DV2, the central visual field V1 and the effective visual field V2 are located in the range from the viewpoint VP to a predetermined horizontal distance LX. Specifically, the central visual field V1 and the effective visual field V2 are ranges that include the lamp LP2 and the switches SW1 to SW7 and SW9 to SW13. In this case, the second generation unit 231B cuts out the range of the captured image PC excluding the central visual field V1 and the effective visual field V2, i.e., the image including the switches SW8 and SW14, shown shaded, as the partial image PS. The object reflected in the cut-out partial image PS becomes the processing target of the image processing unit 232.

[0092] 17, for example, when the viewpoint VP of the user U is located to the left of the device DV2, the range including the lamp LP2 and the switches SW1 to SW3 and SW9 is located in the central visual field V1 and the effective visual field V2. In this case, the second generation unit 231B cuts out the range of the captured image PC excluding the central visual field V1 and the effective visual field V2, that is, the image including the switches SW4 to SW6 and SW10 to SW12 shown in hatching, as the partial image PS.

[0093] As in the first embodiment, the image processing unit 232 performs image processing on the partial image PS cut out by the second generation unit 231B. As described above, image processing is state monitoring of the monitored object using AI. The image processing unit 232 uses the trained model LM stored in the storage device 205 to determine whether the state of the monitored object shown in the partial image PS generated by the second generation unit 231B is normal.

[0094] In the second embodiment, too, the image to be processed by the image processing unit 232 is not the captured image PC of the first imaging device 124A itself, but the partial image PS generated by the second generation unit 231B. Therefore, in this embodiment, the size of the image to be processed is smaller than when the captured image PC of the first imaging device 124A itself is the processing target. This reduces the processing load of the processing device 206, and increases the processing speed of the processing device 206.

[0095] B-4. Operation of the processing unit 206 18 is a flowchart showing the operation of the processing device 206. The processing device 206 functions as the second acquisition unit 230B and acquires the captured image PC captured by the first imaging device 124A and the gaze tracking image PE captured by the second imaging device 124B (step S201). The processing device 206 functions as the gaze tracking unit 234 and calculates gaze information related to the movement of the gaze of the user U using the gaze tracking image PE (step S202).

[0096] The processing device 206 functions as the second generation unit 231B and generates an image as a partial image PS by excluding the portions located in the central visual field V1 and the effective visual field V2 of the user U from the captured image PC (step S203). The processing device 206 functions as the image processing unit 232 and performs image processing on the partial image PS generated in step S203 (step S204). More specifically, the processing device 206 applies the learned model LM to the partial image PS and determines whether or not there is an abnormality in the state of the monitored object included in the partial image PS.

[0097] If there is an abnormality in the state of the monitored object (step S205: YES), the processing device 206 functions as the notification unit 233, generates a control signal for causing the AR glasses 10A to output a warning message or a warning sound, and transmits the control signal to the AR glasses 10A. That is, the processing device 206 functions as the notification unit 233, notifies the user U of the abnormality (step S206), and ends the processing of this flowchart.

[0098] Furthermore, if there is no abnormality in the state of the monitored object (step S205: NO), the processing device 206 returns to step S201 and repeats the subsequent processes until monitoring of the monitored object is terminated (step S207: NO). End of monitoring corresponds, for example, to when the user U has finished his work and moved away from the monitored object. Then, when monitoring of the monitored object is terminated (step S207: YES), the processing device 206 terminates the processing according to this flowchart.

[0099] B-5. Summary of the second embodiment As described above, according to the second embodiment, the second generation unit 231B generates the partial image PS by cutting out an area outside the area visually recognized by the user U from the captured image PC. Therefore, the area not visually recognized by the user U becomes the processing target in the image processing unit 232. This reduces the load on the user U.

[0100] Furthermore, according to the second embodiment, the second generation unit 231B cuts out, as the partial image PS, a portion that is a predetermined distance or more away from the viewpoint VP of the user U. Therefore, an area outside the area visually recognized by the user U is cut out by simple processing.

[0101] C: Modified Example The following are variations of the above-described embodiment. Two or more variations arbitrarily selected from the following variations may be combined as appropriate within the scope of not mutually contradicting each other.

[0102] C1: First modified example In the second embodiment, the partial image PS is generated by cutting out an area outside the area visually recognized by the user U. At this time, the partial image PS may be divided into a plurality of areas based on the distance from the viewpoint VP, and the content of the image processing performed by the image processing unit 232 may be changed.

[0103] For example, in the description of FIGS. 16 and 17, a portion of the captured image PC corresponding to the peripheral visual field V3 and the outside visual field VX is cropped as the partial image PS. Here, the peripheral visual field V3 includes a first peripheral visual field V3A and a second peripheral visual field V3B. The image processing unit 232 may perform different image processing for the portion corresponding to the first peripheral visual field V3A and the portion corresponding to the second peripheral visual field V3B. Specifically, image processing with a relatively low load is performed on the portion corresponding to the first peripheral visual field V3A, which is relatively close to the central visual field V1. This is because the first peripheral visual field V3A is an area close to the effective visual field V2 and is an area where the user U can recognize to a certain extent. On the other hand, processing with a relatively high load is performed on the portion corresponding to the second peripheral visual field V3B in order to strengthen monitoring. This is because the second peripheral visual field V3B is an area where the user U's recognition ability is relatively low.

[0104] For example, if the object to be monitored is a lamp LP, monitoring whether the lamp is lit or identifying the color of the lamp light places a greater burden on the processing device 206. Therefore, for example, the image processing unit 232 only monitors whether the lamp is lit or not for the portion corresponding to the first peripheral visual field V3A, and monitors whether the lamp is lit or not and identifies the color of the lamp light for the portion corresponding to the second peripheral visual field V3B.

[0105] That is, the second generation unit 231B identifies the position of the viewpoint VP of the user U based on the line-of-sight information, and extracts a partial image PS corresponding to the first peripheral field of view V3A and a partial image PS corresponding to the second peripheral field of view V3B based on the distance from the position of the viewpoint VP. The degree to which the user U gazes at the partial image corresponding to the first peripheral field of view V3A is different from the degree to which the user U gazes at the partial image corresponding to the second peripheral field of view V3B. "The degree to which the user U gazes is different" can be rephrased as, for example, "The user U's discrimination ability is different." In this case, the user U's discrimination ability for the partial image corresponding to the first peripheral field of view V3A is different from the discrimination ability for the partial image corresponding to the second peripheral field of view V3B.

[0106] The image processing performed by image processing unit 232 on partial image PS corresponding to first peripheral visual field V3A is different from the image processing performed by image processing unit 232 on partial image PS corresponding to second peripheral visual field V3B. Partial image PS corresponding to first peripheral visual field V3A is an example of a first partial image, and partial image PS corresponding to second peripheral visual field V3B is an example of a second partial image.

[0107] According to the first modification, the partial image PS is divided into multiple parts based on the distance from the viewpoint, and different image processing is performed on each part, thereby improving the usefulness of the image processing and making more effective use of the resources of the processing device 206.

[0108] C2: Second modified example In the first and second embodiments, the AR glasses 10A and the portable device 20A, or the AR glasses 10A and the portable device 20B, are separate entities. However, this is not limiting, and for example, the AR glasses 10A may have the functions of the portable device 20A, or the AR glasses 10A may have the functions of the portable device 20B. That is, the first acquisition unit 230A, the second acquisition unit 230B, the first generation unit 231A, the second generation unit 231B, the image processing unit 232, the notification unit 233, and the gaze tracking unit 234 may be executed by the processing device 126 of the AR glasses 10A or 10B.

[0109] According to the second modification, for example, it is possible to monitor the monitoring target while the user U is working, without using the mobile devices 20A and 20B.

[0110] C3: Third modified example In the first and second embodiments, image processing of the partial image PS is performed by the mobile device 20A or 20B. However, the present invention is not limited to this. For example, image processing of the partial image PS may be performed in an image processing server connected to the mobile device 20A or 20B via a network. In this case, the mobile device 20A or 20B transmits the partial image PS generated by the first generation unit 231A or the second generation unit 231B to the image processing server. The image processing server performs image processing on the partial image PS. When an abnormality is detected in the monitored object, the image processing server transmits a control signal to the mobile device 20A or 20B to notify the user U using the AR glasses 10A or 10B.

[0111] According to the third modification, even if the mobile devices 20A and 20B do not have a program for implementing the image processing unit 232 or do not have the processing power to execute a program for implementing the image processing unit 232, it is possible to monitor the monitoring target while the user U is working. Furthermore, according to the third modification, the image transmitted from the mobile device 20A or 20B to the image processing server is not the captured image PC itself, but a partial image PS obtained by clipping a part of the captured image PC. This reduces the communication load between the mobile device 20A or 20B and the image processing server, and the image processing load on the image processing server, thereby increasing the processing speed of the entire system.

[0112] C4: Fourth variant In the first and second embodiments, the first imaging device 124A was mounted on the AR glasses 10A and 10B. However, this is not limiting, and for example, only an imaging device corresponding to the first imaging device 124A may be worn on the head of the user U. Furthermore, the device mounted with the first imaging device 124A is not limited to a display device such as the AR glasses 10A and 10B, and may be, for example, an audio output device that outputs audio.

[0113] C5: Fifth variant In the first and second embodiments, the AR glasses 10A and 10B feed back (notify) the user U with the results of image processing performed on a portion (partial image) of an image captured by the first imaging device 124A mounted on the AR glasses 10A and 10B. This is not a limitation, and the results of image processing may be fed back by a device other than the AR glasses 10A and 10B. For example, the results of image processing may be fed back to the mobile device 20A or 20B, or another information processing device held by the user U. Furthermore, the results of image processing may be fed back to a person other than the user U (for example, a work supervisor who supervises work performed by the user U), or to an information processing device not held by the user U (such as a work management server).

[0114] D:Other (1) Each function illustrated in Figure 3, Figure 4, Figure 11, or Figure 12 is realized by any combination of hardware and software. There are no particular limitations on how each function is realized. Each function may be realized using a single device that is physically or logically coupled, or may be realized using a device that is configured by connecting two or more physically or logically separated devices directly or indirectly (for example, using wires, wirelessly, etc.). Each function may be realized by combining software with the single device or the multiple devices.

[0115] (2) In this specification, the term "apparatus" may be replaced with other terms such as circuit, device, or unit.

[0116] (3) In each of the first embodiment, the second embodiment, and the first to third modifications, storage device 125 and storage device 205 may be configured by at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Also, the program may be transmitted from a network via a telecommunications line.

[0117] (4) Each of the first embodiment, the second embodiment, and the first to third modifications may be implemented using any of the following standards: LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (where x is, for example, an integer or a decimal point), FRA (Future Radio Access), NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE The present invention may be applied to at least one of systems using 802.20, UWB (Ultra-Wideband), Bluetooth (registered trademark), or other appropriate systems, and next-generation systems that are extended, modified, created, or defined based on these systems. The present invention may also be applied to a combination of multiple systems (e.g., a combination of at least one of LTE and LTE-A with 5G).

[0118] (5) The order of the exemplary processing procedures, sequences, or flowcharts shown in the first embodiment, second embodiment, and first to third modifications may be changed as long as there is no contradiction. For example, the methods described herein present various step elements in an exemplary order and are not limited to the particular order presented.

[0119] (6) In each of the first embodiment, the second embodiment, and the first to third modifications, input and output information, etc. may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information, etc. may be overwritten, updated, or added. Output information, etc. may be deleted. Input information, etc. may be transmitted to another device.

[0120] (7) In each of the first embodiment, the second embodiment, and the first to third variants, the determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a comparison of numerical values ​​(e.g., a comparison with a predetermined value).

[0121] (8) The programs exemplified in the first embodiment, the second embodiment, and the first to third modifications should be broadly construed to mean instructions, instruction sets, code, code segments, program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, or functions, regardless of whether they are called software, firmware, middleware, microcode, hardware description languages, or by other names. Furthermore, software, instructions, or the like, may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, optical fiber cable, twisted pair, and digital subscriber line (DSL)) and wireless technology (such as infrared and microwave), at least one of these wired and wireless technologies is included within the definition of a transmission medium.

[0122] (9) The information and the like described in each of the first embodiment, the second embodiment, and the first to third modifications may be represented using any of a variety of different technologies. For example, data, information, and the like that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields, magnetic particles, optical fields, photons, or any combination thereof. Note that terms described in this specification and terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

[0123] (10) In each of the first embodiment, the second embodiment, and the first to third modifications, the terms "system" and "network" are used interchangeably.

[0124] (11) In each of the first embodiment, the second embodiment, and the first to third modifications, the mobile device 20A or 20B may be a mobile station, which may also be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other appropriate term.

[0125] (12) A mobile station may be referred to as a transmitting device, a receiving device, a communication device, or the like. A mobile station may be a device mounted on a mobile object, or the mobile object itself. A mobile object refers to an object that can move. A mobile object can move at any speed. A mobile object can be stopped. Examples of mobile objects include, but are not limited to, vehicles, transport vehicles, automobiles, motorcycles, bicycles, connected cars, excavators, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, handcars, rickshaws, ships and other watercraft, airplanes, rockets, satellites, drones (registered trademark), multicopters, quadcopters, balloons, and objects mounted thereon. A mobile object may also be a mobile object that moves autonomously based on an operation command. A mobile object may be a vehicle (e.g., a car, an airplane, etc.), an unmanned mobile object (e.g., a drone, an autonomous vehicle, etc.), or a robot (manned or unmanned). A mobile station also includes devices that do not necessarily move during communication operations. For example, the mobile station may be an IoT (Internet of Things) device such as a sensor.

[0126] (13) In each of the first embodiment, the second embodiment, and the first to third modifications, the term "determining" may encompass a wide variety of actions. "Determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching a table, database, or other data structure), ascertaining, and regarding that as a "determination." Also, "determining" may include regarding receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and accessing (e.g., accessing data in memory) as a "judgment" or "decision." Furthermore, "decision" can include the act of considering something to be "decided" such as resolving, selecting, choosing, establishing, or comparing. In other words, "decision" can include the act of considering something to be "decided" to be an action. "Decision" can also be interpreted as "assuming," "expecting," or "considering," among others.

[0127] (14) In each of the first embodiment, the second embodiment, and the first to third modifications, the term "connected," or any variation thereof, refers to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more electric wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0128] (15) In each of the first embodiment, the second embodiment, and the first to third modifications, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0129] (16) As used herein, any reference to elements using designations such as "first" and "second" does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

[0130] (17) When the words "include," "including," and variations thereof are used in the first embodiment, second embodiment, and first to third modifications in this specification or claims, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" used in this specification or claims is not intended to mean an exclusive logical OR.

[0131] (18) Throughout this application, where articles are added by translation, such as a, an, and the in English, the disclosure may include the plural form of the noun following these articles.

[0132] (19) It is clear to those skilled in the art that the present invention is not limited to the embodiments described in this specification. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description in this specification is intended as an illustrative explanation and does not have any limiting meaning on the present invention. Furthermore, multiple aspects selected from the aspects exemplified in this specification may be combined. [Explanation of symbols]

[0133] 1,2...information processing system, 10A,10B...AR glasses, 20A,20B...portable device, 30...inertial measurement unit, 121...projection device, 122...sound emission device, 123,203...communication device, 124A...first imaging device, 124B...second imaging device, 125,205...storage device, 126,206...processing device, 127,207...bus, 128...infrared light emitting device, 130...operation control unit, 201...touch panel, 230A...first acquisition unit, 230B...second acquisition unit, 231A...first generation unit, 231B...second generation unit, 232...image processing unit, 233...notification unit, 234...gaze tracking unit, DV (DV1, DV2)...device, LEN...imaging lens, LM...trained model, PC...captured image, PS...partial image.

Claims

1. an acquisition unit that acquires motion information related to motion of a user wearing an imaging device on their head and image information indicating an image captured by the imaging device; a generating unit that generates a partial image cut out from the captured image by controlling a position at which a part is cut out from the captured image in accordance with the motion information; an image processing unit that performs image processing on the partial image, the acquisition unit acquires, as the movement information, information regarding movement of the head of the user; the generating unit generates the partial image by cutting out a region corresponding to a pre-specified object from the captured image in accordance with information related to the movement of the head. Information processing device.

2. the acquisition unit acquires information about the movement of the head from an inertial measurement unit or a geomagnetic sensor attached to the head of the user.

2. The information processing device according to claim 1.

3. an acquisition unit that acquires motion information related to motion of a user wearing an imaging device on their head and image information indicating an image captured by the imaging device; a generating unit that generates a partial image cut out from the captured image by controlling a position at which a part is cut out from the captured image in accordance with the motion information; an image processing unit that performs image processing on the partial image, the acquisition unit acquires, as the movement information, gaze information relating to a gaze movement of the user; the generating unit generates the partial image by cutting out, from the captured image, an area outside an area in which the user can recognize predetermined information, based on the line-of-sight information; the partial images include a first partial image and a second partial image; a degree to which the user gazes at the first partial image and a degree to which the user gazes at the second partial image are different from each other; The generation unit Identifying a position of the user's viewpoint based on the line-of-sight information; cutting out the first partial image and the second partial image based on the distance from the viewpoint position; the image processing performed by the image processing unit on the first partial image and the image processing performed by the image processing unit on the second partial image are different from each other; Information processing device.

Images