Information sharing system

Abstract

To provide an information sharing system which can improve usability of a user.SOLUTION: An information sharing system 1 sharing information in a virtual space VS includes a processor 101 and a memory 102 used in the processor, wherein the processor includes a virtual space management part 10 for executing a predetermined computer program stored in the memory, and thereby displaying a three-dimensional object 5 in the virtual space, a user operation analysis part 11 for analyzing the user operation in the virtual space, a target region setting part 12 for setting a target region in an arbitrary place of the object, on the basis of a relation between a selection range indicated by track information included in the analyzed user operation and the object, and a relevant information management part 13 for managing relevant information associated with the target region, and the user operation analysis part displays the image of the selected object as a plane for operation in the virtual space between the object and a user viewpoint.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information sharing system. [Background technology]

[0002] There is known a technique for having a user wear a user terminal such as an HMD (Head Mounted Display) to select and operate objects in a virtual space (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-009552 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 discloses one solution to the problem that when an object contains other objects, it is not possible to uniquely identify whether one is selecting a larger object or one of the other objects inside it.

[0005] However, there are cases where an object is not recognized as a separate object from other objects and cannot be clearly separated. For example, if a real object is scanned with a 3D scanner along with surrounding objects, and the objects corresponding to those objects are represented as point cloud data (PointCloud) in a virtual space, it is difficult to clearly separate the object from the surrounding objects. It is also difficult to clearly distinguish between an object and the objects (sub-objects) contained within it. This is because the physical structure of the object is not clearly reflected in the point cloud data obtained by a 3D scanner.

[0006] Therefore, when sharing information through an object in a virtual space, the information may be associated with a location different from the location intended by the user, and the information may not be shared smoothly with other users. For example, even if a user embeds information about a phenomenon occurring in a real object in a corresponding location of an object in a virtual space and wishes to share the information with other users, if the embedding location is misaligned, the information the user is trying to convey may not be conveyed accurately to other users, resulting in poor usability for users.

[0007] Therefore, the present disclosure provides an information sharing system that can improve usability for users. [Means for solving the problem]

[0008] In order to solve the above problems, an information sharing system for sharing information in a virtual space according to one aspect of the present invention includes a processor and a memory used by the processor, and the processor executes a predetermined computer program stored in the memory to realize a virtual space management unit that displays three-dimensional objects in the virtual space, a user operation analysis unit that analyzes user operations in the virtual space, a target area setting unit that sets a target area at any location on the object based on the relationship between the object and a selection range indicated by trajectory information included in the analyzed user operation, and a related information management unit that associates related information with the set target area and manages it, and the user operation analysis unit displays an image of the selected object in the virtual space between the object and the user viewpoint as an operation plane. Effect of the Invention

[0009] According to the present invention, a target area can be set at any location on an object, and related information can be managed in association with the target area. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is an overall configuration diagram of an information processing system including an information sharing system. [Diagram 2]FIG. 2 is a hardware configuration diagram of a user terminal. [Diagram 3] FIG. 4 is an explanatory diagram showing how a user operates an object. [Figure 4] An example of a table for managing objects. [Diagram 5] An example of a table that manages the correspondence between objects and colors. [Figure 6] An example of a table that manages sub-objects. [Figure 7] An example of a table that manages the correspondence between sub-objects and colors. [Figure 8] 1 is an example of a table for managing media information. [Figure 9] 11 is a flowchart showing a process of registering media information in an object. [Figure 10] FIG. 13 is an explanatory diagram showing how a target object is selected from a group of objects in a virtual space. [Figure 11] An example of the three-dimensional surface of a selected object. [Figure 12] An example of an unfolded map that unfolds a 3D object into 2D. [Figure 13] FIG. 13 is an explanatory diagram showing how the value of red color information as the first channel is allocated in the horizontal direction of the expansion map. [Figure 14] FIG. 13 is an explanatory diagram showing how blue color information values ​​as a second channel are assigned in the vertical direction of the unfolded map. [Figure 15] An example of a colored development map that combines red and blue information values. [Figure 16] An example of a colored development map displayed as a 3D image. [Figure 17] FIG. 13 is an explanatory diagram of a case where a user sets a target region on the surface of an object. [Figure 18] 11 is a flowchart showing a process of calling up media information associated with an object. [Figure 19] FIG. 1 is an explanatory diagram showing a state in which a user selects a desired area on the surface of an object with a fingertip. [Figure 20] FIG. 11 is an overall configuration diagram of an information sharing system according to a second embodiment. [Figure 21] FIG. 11 is an explanatory diagram showing how an image name is estimated from an image in a selected area. [Figure 22] FIG. 11 is an explanatory diagram showing how another region related to a target region selected by a user is extracted. [Figure 23] FIG. 11 is an explanatory diagram showing how a related region and a target region are associated with each other. [Figure 24] An example of a table that manages sub-objects. [Diagram 25] 11 is a flowchart showing a process of registering a related area in association with a new area specified by a user. [Figure 26] 13 is an explanatory diagram showing how an operation plane is created from an image of the surface of an object that the user wishes to operate and is provided to the user in the third embodiment. [Figure 27] FIG. 11 is an explanatory diagram showing the relationship between the operation plane and the object from another direction. [Figure 28] 11 is an explanatory diagram showing how operation modes are arranged on an operation plane, and selection of a mode and selection of a target area are performed seamlessly in a series of operations. [Figure 29] FIG. 11 is an explanatory diagram showing an example of a user operation using an operation plane. [Diagram 30] 11 is an explanatory diagram showing how the display position of an object is changed from the front to the rear by moving a fingertip forward and backward relative to the operation plane. FIG. [Diagram 31] 11 is a flowchart showing a process of analyzing a user operation. [Diagram 32] FIG. 13 is a diagram showing the overall configuration of a system including an information sharing system according to a fourth embodiment. [Diagram 33] FIG. 13 is an overall diagram of a system including an information sharing system according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In this embodiment, a system is disclosed that allows a user to specify any part of a three-dimensional object with a relatively simple operation, associate related information with the object, and share the information among multiple users.

[0012] In recent years, a so-called digital twin has been proposed, in which information collected from real-world objects by sensors is reflected in virtual-world objects, reproducing the real-world and virtual-world objects as if they were twins. By using digital twins, it becomes relatively easy to increase the efficiency of development and design, improve quality, and reduce costs.

[0013] On the other hand, in the real world, various phenomena and events occur that cannot be predicted in virtual space alone. Therefore, in order to accurately associate information about a phenomenon or event that occurred in the real world with an object (three-dimensional object) in virtual space, it is necessary to be able to flexibly set the position to which related information is associated. Therefore, in the present disclosure, it is possible to set an area of ​​an appropriate size at an appropriate position on an object, regardless of the part configuration of the object.

[0014] Furthermore, in the present disclosure, the name of the image of the area of ​​the object specified by the user (for example, the area of ​​the object surface) is automatically determined, and whether or not it is a new target area is determined and managed based on the image name. This makes it possible to identify the specified area and determine whether or not it is a new target area without having to check the object management table using the position and size of the specified object.

[0015] Furthermore, in the present disclosure, an operation plane is generated using an image of an object selected by a user, and is displayed in a virtual space between the object and a user's viewpoint. The user can operate the operation plane by a plane operation unit for operating the operation plane, such as a fingertip or a stylus pen. EXAMPLES

[0016] A first embodiment will be described with reference to Figures 1 to 19. An information sharing system 1 of this embodiment can be used for so-called digital twins, for example, but the applicable field of the information sharing system 1 is not limited to digital twins.

[0017] 1 is a diagram showing the overall configuration of an information processing system including an information sharing system 1. The information sharing system 1 is configured using a computer including, for example, a processor 101, a memory 102, an interface unit 03, etc. The memory 102 here includes a main storage device and an auxiliary storage device.

[0018] The processor 101 executes a predetermined computer program stored in the memory 102 to realize functions 10 to 14 of the information sharing system 1 described below. The processor 101 is not limited to a CPU (Central Processing Unit) and may include a GPU (Graphics Processing Unit) or an ASIC (Application Specific Integrated Circuit), etc.

[0019] The interface (abbreviated as IF in the drawing) unit 103 includes, for example, a communication interface device (not shown) that connects the information sharing system 1 to the communication network CN, and an input / output interface device (not shown) that connects the information sharing system 1 to a user interface device (not shown). The user interface device is a device for exchanging information between an administrator of the information sharing system 1 and the information sharing system 1, and includes, for example, a monitor display, a keyboard, a touch panel, a synthetic voice device, etc.

[0020] The information sharing system 1 can be connected to a storage medium MM via an interface unit 103. The storage medium MM is configured as, for example, a memory device, a hard disk device, an optical disk device, a magneto-optical disk device, a magnetic tape device, or the like, and non-temporarily stores computer programs and data. The storage medium MM can transfer computer programs and data to the memory 102 of the information sharing system 1 and store them therein. Computer programs and data can also be transferred from the memory 102 to the storage medium MM and stored therein. A specific computer program that realizes each function of the information sharing system 1 is stored in the storage medium MM, the storage medium MM is connected to another computer, and the computer program stored in the storage medium MM is installed in the other computer, thereby making the other computer function as the information sharing system 1.

[0021] The communication network CN may be a public communication network such as the Internet, or a private communication network. The communication network CN may be either a wireless communication network or a wired communication network, or may be a combination of a wireless communication network and a wired communication network.

[0022] The information sharing system 1 can be connected to an external device 3 via a communication network CN. The external device 3 is, for example, a file server, a network storage, or the like.

[0023] The information sharing system 1 is connected to at least one (usually a plurality of) user terminals 2 via a communications network CN. The configuration of the user terminal 2 will be described later.

[0024] The information sharing system 1 includes, for example, a virtual space management unit 10, a user operation analysis unit 11, a target area setting unit 12, a related information management unit 13, and a database (abbreviated as DB in the drawing) 14.

[0025] The virtual space management unit 10 manages the virtual space VS. The virtual space management unit 10 renders one or more specified objects 5 and places them in the virtual space VS, and changes the display of the objects 5 in accordance with the operation of the viewpoint.

[0026] The user operation analysis unit 11 detects and analyzes operations of the user 4 on the virtual space VS via the user terminal 2. The user operation analysis unit 11 includes, for example, an operation planar display unit 111, an operation detection unit 112, and a display depth control unit 113.

[0027] The operation plane display unit 111 displays an operation plane 1110 between an object 5 to be operated and the user's viewpoint in the virtual space VS. The user 4 uses his / her fingertip, a stylus pen, or the like as a plane operation unit for operating the operation plane 1110 described below, and selects a desired area of ​​the object 5. The plane operation unit is not limited to the user's fingertip or a stylus pen held by the fingertip, but may also include the wrist of the user's hand connected to the fingertip.

[0028] The user's operation on the operation plane 1110 is detected by the operation detection unit 112. As described later, after the user selects an operation mode displayed on the operation plane 1110, the user can select (also called designation) a desired area without removing the planar operation unit from the operation plane 1110. The selection of an operation mode by the user is also detected by the operation detection unit 112.

[0029] The display depth control unit 113 controls the display depth of the object 5 to be displayed in the virtual space VS, as described below. The display depth here refers to the surface of the object that the user can see. The display depth control unit 113 can continuously (or stepwise) change the surface provided to the user from the foreground of the object through the inside of the object to the rear side of the object.

[0030] As described below, the target area setting unit 12 sets a target area 53 at an arbitrary location of the object 5 based on the relationship between the selected range indicated by trajectory information 51 included in the analyzed user operation and the object 5. The target area setting unit 12 can set a target area across multiple objects (also called sub-objects) included in the object 5. Furthermore, the target area setting unit 12 can also identify an area specified by the user and notify the related information management unit 13 of the area.

[0031] The target region setting unit 12 includes, for example, an object specifying unit 121, an unfolded map generating unit 122, and a new region setting unit 123.

[0032] The object identification unit 121 identifies the original three-dimensional object 5 to be targeted based on the analyzed user operation. The unfolded map generation unit 122 generates an unfolded map G3 in which the target object 5 is unfolded on a two-dimensional plane, as described later. The unfolded map G3 is given coordinate identification information that uniquely identifies each point of the unfolded map on the unfolded map. The coordinate identification information is generated using information of any two colors out of three color information (R, G, B) that can be set to each point of the unfolded map. In the present disclosure, each point of the unfolded map is identified by a value of red color information (R value) and a value of blue color information (B value). The unfolded map in which color information is set as coordinate identification information is also called a colored unfolded map G4. The colored unfolded map G4 is information used in the target area setting unit 12 and is not presented to the user. What is presented to the user is the unfolded map G3 in which the three-dimensional object 5 is unfolded on a two-dimensional plane. The unfolded map G3 presented to the user does not have color information as coordinate identification information, but has normal color information for color display in the virtual space VS.

[0033] As described below, when the area 53 specified by the user (selected area 53) is different from the existing areas registered in the database 14, the new area setting unit 123 sets the new area 53N. The set new area 53N is registered in the database 14 as one of the existing areas. Here, the new area can also be called a new target area.

[0034] The target area setting unit 12 can set a target area across a part of a certain object and a part of another object. Therefore, the user can freely specify a target area as necessary. If the area specified by the user is new, the new area setting unit 123 sets a new area 53N.

[0035] When setting a new target area, the new area setting unit 123 can extract other areas related to the new target area and associate the extracted other areas as related areas. The related areas and the new target area may be discontinuous on the surface of the object. As described later, for example, when a user specifies one of four tires of an automobile, the new area setting unit 123 can extract the other three tires as related areas, associate the tire specified by the user with the other three extracted tires, and set it as a new target area. Furthermore, the new area setting unit 123 can also manage the area behind the area specified by the user in association with the area specified by the user. In the previous example, it is the back side of the tire specified by the user, that is, the range that is not visible from the user's current viewpoint. In this case, the viewpoint of the virtual space when selecting a new target area is different from the viewpoint when extracting the related area, and the related area and the new target area are continuous on the surface of the object.

[0036] The related information management unit 13 manages related information in association with the target area set by the target area setting unit 12. When a user selects related information associated with an area specified by the user, the related information management unit 13 reads the selected related information from the database 14 and plays it back.

[0037] Here, related information refers to information related to the target area. Examples of related information include text memos, voice memos, moving images, still images, sketches, blueprints, and measurement data. Related information is also called media information. A single target area may be associated with multiple pieces of related information. A user can extract desired related information based on, for example, creation date and time, update date and time, media type (file type), file size, creator name, updater name, keywords, and the like, and play it in the virtual space.

[0038] The database 14 manages data used in the information sharing system 1. As described below, the database 14 manages, for example, a table for managing objects, a table for managing sub-objects, a table for managing the relationship between objects and colors, a table for managing sub-objects and colors, related information (media information), etc. The substance of the data managed by the database 14 may be stored in the memory 102 or in the external device 2.

[0039] The user operation analysis unit 11 and the target area setting unit 12 can be called, for example, an "object operation control unit" that controls the display and operation of an object in response to a user operation.

[0040] 2 shows a hardware configuration of the user terminal 2. The user terminal 2 is configured as, for example, a goggle-type terminal or a glasses-type terminal, and is worn on the body of the user. The user terminal 2 may be configured as, for example, an HMD (Head Mounted Display).

[0041] The user terminal 2 includes, for example, a processor 201, a memory 202, a camera 203, a sensor 204, a microphone (abbreviated as "mic" in the figure) 205, a display device 206, a user input / output port unit 207, and a communication interface unit 208.

[0042] The processor 201 executes a predetermined computer program stored in the memory 202 to realize the functions of the user terminal 2. The functions of the user terminal 2 include, for example, a function of providing a virtual space to a user based on an instruction from the information sharing system 1, and a function of transmitting a user's operation to the information sharing system 1.

[0043] The processor 201 is not limited to a CPU, and may include a GPU, etc. The memory 202 includes a main storage device and an auxiliary storage device.

[0044] The camera 203 is an external camera that captures an image in front of the user. The camera 203 may include an internal camera that captures an image of the user's face. The camera 203 may be a stereo camera, or may have a function for measuring distances using infrared laser light or the like.

[0045] The sensor 204 is, for example, an acceleration sensor, a motion sensor, etc. It may also include a sensor that measures vital data of the user.

[0046] The microphone 205 detects the voice uttered by the user. Although not shown in the figure, the user terminal 2 may also be equipped with earphones for transmitting sound to the user.

[0047] The user input / output port unit 207 transmits and receives signals to and from, for example, a controller held by the user. For example, the user terminal 2 can detect the movement of a stylus pen held by the user and button operations on the stylus pen. The user terminal 2 can also detect user operations in the virtual space from the movement of the user's fingertips captured by the camera 203 and transmit the detected operations to the information sharing system 1.

[0048] The communication interface unit 208 transmits and receives information to and from the information sharing system 1 via the communication network CN.

[0049] FIG. 3 shows how a user 4 operates an object 5 in a virtual space VS. The user 4 can operate the object 5 in the virtual space VS displayed on the display device 206 by moving the user's fingertip 41. The user's fingertip corresponds to an example of a planar operation unit. The movement of the user's fingertip 41 in the real space is photographed by the camera 203, and detected and analyzed by the user operation detection unit 112. The analysis result of the user operation is reflected in the virtual space VS managed by the virtual space management unit 10, and provided to the user from the display device 206.

[0050] 4 is an example of a table T1 for managing objects. The object management table T1 manages, for example, an object ID, an object name, a nickname, tag information, a link to a 3D mesh, a 3D position, a 3D angle, a link to an unfolded map, a list of sub-objects, a list to media information, and the like.

[0051] The object ID is information that identifies the object 5. The object name is the name of the object 5. The nickname is the nickname of the object 5. The tag information is a search tag set for the object 5. The link to the 3D mesh is a link to data of a three-dimensional mesh object. The 3D position indicates the position of the object 5 in the virtual space VS. The 3D angle is the angle of the object 5 placed in the virtual space VS. The link to the unfolded map is a link to a map in which the object 5 is unfolded. The list to sub-objects indicates a list of sub-objects included in the object 5. The list to media information is a list of media information associated with the object 5.

[0052] 5 is an example of table T2 that manages the correspondence between objects and colors. This table T2 manages, for example, color information and object IDs in association with each other. This makes it possible to immediately identify an object selected by a user from an image containing a mixture of multiple objects 5, as described below. This is because it is possible to determine that an object ID having color information of a point specified by the user is the object specified by the user.

[0053] 6 is an example of table T3 for managing sub-objects. Sub-object management table T3 manages, for example, sub-object IDs, sub-object names, links to parent objects, images of areas represented in an unfolded map, tag information, lists to media information, and the like.

[0054] The sub-object ID is information that identifies the sub-object. The sub-object name is the name of the sub-object. The link to parent object is information that identifies the object to which the sub-object directly belongs. The image of the area represented in the unfolded map is a link to image data that shows the sub-object on the unfolded map. The tag information is a search tag associated with the sub-object. The list to media information is a list of media information associated with the sub-object.

[0055] 7 is an example of table T4 that manages the correspondence between sub-objects and colors. This table T4 manages, for example, color information and sub-object ID in association with each other. This makes it possible to determine that a sub-object having color information that matches the color information of an area specified by the user is the sub-object specified by the user.

[0056] 8 is an example of table T3 for managing media information. Media information management table T5 manages media information as "related information." Each piece of media information includes, for example, a media information ID, a type, a storage destination, tag information, and a related emphasis distribution complexity index.

[0057] The media information ID is information for identifying the media information. The type is the type of the media information. The storage destination is the location where the media information is stored. The tag information is a search tag associated with the media information. The related highlighting portion indicates a portion where the relatedness to the media information is emphasized.

[0058] The process of registering media information in an object will be described with reference to the flowchart of Fig. 9. Fig. 10 shows an example of an object selection screen G1 for selecting a target object from among a group of objects in the virtual space.

[0059] The user operation analysis unit 11 analyzes a user operation (also called a gesture) in the virtual space (S101). The object identification unit 121 identifies the ID of an object selected by a user from the object selection screen G1 shown in Fig. 10 (S102). As described above, as an internal process of the information sharing system 1, each object is filled with predetermined color information, so that an object having the same color information as the color information of the area selected by the user can be determined to be the object selected by the user.

[0060] Referring to Fig. 10, an object selection screen G1 includes images G11 to G14 of a plurality of three-dimensional objects. Each of the objects G11 to G14 is filled with different color information. The screen G1 shown in Fig. 10 is control information used to quickly identify an object selected by the user, and is not presented to the user. Each object is presented to the user as if it were a normal color photograph.

[0061] Returning to Fig. 9, the object identification unit 121 renders the three-dimensional object selected by the user (S103). Fig. 11 shows an example of an image G2 obtained by rendering the three-dimensional object.

[0062] Returning to FIG. 9, the unfolded map generating unit 122 generates a colored unfolded map G4 of the three-dimensional object (S104). FIG. 12 is an example of an unfolded map G3 obtained by unfolding a three-dimensional object into two dimensions. FIG. 13 shows a first channel unfolded map G4R generated by allocating the value of red information as the first channel to the horizontal direction of the unfolded map. FIG. 14 is an example of a second channel unfolded map G4B generated by allocating the value of blue information as the second channel to the vertical direction of the unfolded map. FIG. 15 is an example of a colored unfolded map G4 obtained by combining the first channel unfolded map G4R and the second channel unfolded map G4B.

[0063] In the unfolded map G4R shown in FIG. 13, the red information as the first channel increases by a constant value toward the right side with the left side of FIG. 13 as a reference. In the unfolded map G4B shown in FIG. 14, the blue information as the second channel increases by a constant value toward the bottom side with the upper side of FIG. 14 as a reference. When the unfolded map G4R of FIG. 13 and the unfolded map G4B of FIG. 14 are combined, an unfolded map G4 having two types of color information for each point of the unfolded map is generated. As described above, this colored unfolded map G4 is information used to identify the area specified by the user, and is not presented to the user. However, if the user wishes, the colored unfolded map G4 may be presented to the user.

[0064] FIG. 16 is an example of rendering a colored unfolded map as a three-dimensional image G5. FIG. 17 shows an example in which a user sets a target area on the surface of an object. As shown in FIG. 17, a user specifies a desired area in an image G2 of a three-dimensional object such as a normal color photograph. The specified area is given a symbol P1. The target area setting unit 12 identifies the area P2 specified by the user in an image G5 (also called a colored three-dimensional model image) obtained by converting the colored unfolded map into a three-dimensional model. Furthermore, the target area setting unit 12 identifies the area P3 specified by the user in the colored unfolded map G4.

[0065] Refer to step S105 in Fig. 9. The target area setting unit 12 detects an area (e.g., area P2 in Fig. 17) designated by a user's gesture in the colored development map G4 (S105). The target area setting unit 12 determines whether the detected area is an area registered in the database 14 (S106).

[0066] The target area setting unit 12, for example, refers to a map (image G1) of object IDs and determines which part of each object has been specified by the user. If the area specified by the user's gesture matches a specific object, the target area setting unit 12 determines that the object has been selected. If the area specified by the user's gesture matches an existing sub-object in the specific object, the target area setting unit 12 determines that the sub-object has been selected.

[0067] If it is determined that the user has selected an entire existing object, or if it is determined that the user has selected an existing sub-object (registered sub-object), the result in step S16 is "YES" and the process proceeds to step S112.

[0068] If the area specified by the user's gesture is not an existing sub-object (S106: NO), the target area setting unit 12 determines that a new target area has been selected, and sets the new target area on the unfolded map (S107).

[0069] The target area setting unit 12 presents the new target area to the user and accepts the user's correction request (S108). The user can adjust the position and shape of the new target area on the unfolded map. The new target area is presented to the user, for example, as mask information filled in on the unfolded map. Furthermore, the target area setting unit 12 sets the name input by the user as the name of the new target area (S109). The new area setting unit 123 may automatically generate a temporary name for the new target area and propose it to the user, and wait for the user's approval of the temporary name or an instruction to correct it.

[0070] The new area setting unit 123 registers information about the new target area, in which the sub-object ID, the sub-object name, the image data, etc. have been set, in the database 14 (S110).

[0071] The related information management unit 13 manages the media information selected by the user in association with the target area set by the target area setting unit 12 (S111).

[0072] The process of calling up the media information associated with an object will be described with reference to the flowchart in FIG.

[0073] The target area setting unit 12 analyzes a user operation by a user gesture and determines that a call for media information has been instructed (S201). The target area setting unit 12 generates a map G1 including objects that are candidates for selection by the user, and presents the map G1 to the user (S202).

[0074] The target area setting unit 12 detects an area specified by the user (S203), selects an object (sub-object) corresponding to the area specified by the user (S204), and notifies the user of the selected object (S205).

[0075] The target area setting unit 12 determines whether the selected object has a sub-object (S206). If the selected object does not have a sub-object (S206: NO), the related information management unit 13 searches the database 14 (S210), and reads and plays back media information as related information associated with the object selected by the user (S211).

[0076] If the object selected by the user has a sub-object (S206: YES), the target area setting unit 12 selects candidates for the sub-object included in the area specified by the user (S207) and notifies the user of the selected sub-object candidate (S208). The user selects a desired sub-object from the notified sub-object candidates and transmits the result to the target area setting unit 12. If the target area setting unit 12 confirms the user's approval (S209: YES), the related information management unit 13 searches the database 14 (S210), reads out media information as related information associated with the sub-object approved by the user, and plays it (S211). If the user's approval is not obtained for the sub-object candidate (S209: NO), the process returns to step S207.

[0077] A manner in which a user designates a desired area on the surface of an object by a gesture using a fingertip will be described with reference to Fig. 19. A stylus pen or the like may be used instead of a fingertip.

[0078] The user moves the fingertip 41 so as to surround a target area 53 including a desired sub-object 52. A starting point 511 of trajectory information 51 of the fingertip 41 is inside the target area 53, and the fingertip 41 moves counterclockwise outside the target area 53. An end point 512 of the trajectory information 51 is located outside the target area 53. More specifically, when the user stops the fingertip 41 for a short time while keeping it on the sub-object 52, a process of setting the target area 53 is started, and trajectory information 51, which is the movement trajectory of the fingertip 41, is detected. The area surrounded by this trajectory information 51 is extracted as the target area 53.

[0079] According to the present embodiment configured as described above, based on the relationship between the selected range indicated by the trajectory information included in the user operation and the object 5, the target area 53 can be set at any location of the object, and media information can be associated with and managed in relation to the set target area 53. According to the present embodiment, a new target area other than the registered area can also be set at any location. Therefore, according to the present embodiment, the user can associate media information with a desired location regardless of the part configuration of the object. Therefore, in the information sharing system 1 of the present embodiment, information sharing through a virtual space with other users can be smoothly performed, improving the usability for the user.

[0080] When the information sharing system 1 is applied to so-called digital twins, media information about phenomena or events occurring in real-world objects can be associated with objects in virtual space at locations that accurately correspond to the real world, thereby improving information sharing among users involved in the digital twins.

[0081] According to this embodiment, the relationship between the selection range and the object is grasped on the unfolded map that unfolds the three-dimensional object into two dimensions, and the target area of ​​the object is specified. Furthermore, in this embodiment, the unfolded map is given coordinate identification information that uniquely identifies each point on the unfolded map. Therefore, by comparing and matching the area specified by the three-dimensional object with the unfolded map, the area (target area) specified by the user can be easily and accurately specified.

[0082] In this embodiment, two colors of three colors that can be set for each point are used as coordinate identification information, so that it is possible to immediately identify the correspondence between the area on the colored development map G4 and each point on the image G5 obtained by three-dimensionally converting the colored development map G4. The color information used for the colored development map is not limited to red and blue information. Green information may also be used. Furthermore, it is not limited to two colors, and three or more colors may also be used as coordinate identification information.

[0083] In this embodiment, since the colored development map G4 is used, the target area can be freely set in units of each point on the development map G4. Therefore, according to this embodiment, the target area can be set at any location in an irregular shape other than a circle, rectangle, or triangle, or in a free shape. EXAMPLES

[0084] Example 2 will be described with reference to Figures 20 to 25. In the following examples including this example, the differences from Example 1 will be mainly described. In information sharing system 1A of this example, new area setting unit 123A is provided with image recognition unit 1231 that distinguishes an image name from an image.

[0085] Fig. 20 is an overall configuration diagram of an information sharing system 1A. The information sharing system 1 shown in Fig. 1 and the information sharing system 1A shown in Fig. 20 are expressed in different ways, but have the same configuration except for an image recognition unit 1231.

[0086] As shown in the explanatory diagram of FIG. 21, the image recognition unit 1231 converts an input image into an object name (the name of the object indicated by the image) using, for example, artificial intelligence. Hereinafter, the object name is also referred to as an image name. In the example of FIG. 21, when an area corresponding to a tire is selected from an object 5 of an automobile and an image of the area is input to the image recognition unit 1231, "tire" is output as the image name. The text image recognition unit 1231 that converts an input image into an object name (image name) can be realized by so-called machine learning, deep learning, or the like.

[0087] A method for extracting other regions related to a target region selected by a user will be described with reference to FIG. 22. Within the range visible from the user's viewpoint VP1, a region 54(1) directly specified by the user is part of a target region 53(1). When the user's viewpoint is changed from viewpoint VP1 to viewpoint VP2, a region 54(2) that the user can specify appears. This is, for example, the relationship between a certain surface of a tire and another surface that is continuous with that surface. The regions 54(1) and 54(2) simply indicate different regions of the same object, and the two regions 54(1) and 54(2) are continuous.

[0088] Furthermore, the target region can be expanded to include regions 54(3) that are not contiguous with the directly specified region. For example, if the initially specified region is the "left front wheel," the target region can be expanded to include other tires, i.e., the "right front wheel," "left rear wheel," and "right rear wheel," as relevant regions. The relevant regions can also be called expanded regions.

[0089] 23 shows how an associated region 53(2) is associated with a target region 53(1) initially specified by a user, and the target region is enlarged. In this embodiment, a region corresponding to an object similar to the target region 53(1) initially specified by a user is extracted as a region of interest 53(2) and associated with the initially specified region 53(1). As a result, the initial target region 53(1) directly specified by the user is enlarged.

[0090] Fig. 24 is an example of a table T3A for managing sub-objects. Comparing the sub-object management table T3A shown in Fig. 24 with the sub-object management table T3 described in Fig. 6, an "image of the extended area represented in the unfolded map" has been added to the sub-object management table T3A of this embodiment. The "image of the extended area represented in the unfolded map" indicates the storage destination of image data of the extended area (associated area) in the unfolded map.

[0091] The process of registering a related area (extended area) in association with a new area designated by a user will be described with reference to the flowchart in FIG.

[0092] When a new area (new target area) is specified on the unfolded map (S301), the target area setting unit 12A inputs an image corresponding to the specified new area to the image recognition unit 1231 (S302), and acquires the name of the input image (name of the image object) from the image recognition unit 1231 (S303). The first area directly specified by the user is called the first area.

[0093] The target area setting unit 12A generates an area corresponding to the image name as a first area (S304), and compares the first area with the database 14 to see if it is an area already registered (S305).

[0094] If the first region has already been registered in the database 14 (S305: YES), the viewpoint VP is changed from the initial viewpoint VP1 to another viewpoint VP2, and an image seen from the viewpoint VP2 is generated (S306). This is to suggest to the user another region that complements the first region.

[0095] The target area setting unit 12A acquires the name of another area seen from another viewpoint VP2 from the image recognition unit 1231, and generates an area corresponding to the image name as a second area (S307). The target area setting unit 12A determines whether the first area and the second area are continuous, and temporarily registers them in the database 14 (S308).

[0096] The target area setting unit 12A presents to the user that the target area should be expanded by adding the second area to the first area (S309). If approval is obtained from the user, the target area setting unit 12A adds the second area to the first area to generate a new area (S310) and stores the generated new area in the database 14 (S311).

[0097] The present embodiment thus configured also has the same effect as that of the first embodiment. Furthermore, in this embodiment, the name of the area first specified directly by the user is judged by the image recognition unit 1231, and it is judged whether the same or similar object is stored in the database 14 based on the image name (object name). Therefore, it is possible to extract the same kind of object located away from the first specified area (first area) based on the image name and add it to the target area as an area (second area) related to the first area. As a result, according to this embodiment, the area first specified directly by the user can be expanded based on the meaning of the image name, improving the usability for the user. In contrast, if this embodiment is not adopted, the user needs to search for and register related areas one by one, which is time-consuming. EXAMPLES

[0098] Third embodiment will be described with reference to Figures 26 to 31. In this embodiment, an example of the configuration and operation of the user operation analysis unit 11 will be described.

[0099] Fig. 26 shows how an operation plane 1110 is created from an image of the surface of an object 5 that the user wishes to operate, and is provided to the user. Fig. 27 shows the relationship between the operation plane and the object from another angle.

[0100] An image of a predetermined range including the place on the three-dimensional object 5 where the user stops the fingertip 41 for a predetermined time is copied, and an operation plane 1110 is generated as shown in Fig. 29. The operation plane 1110 is displayed at a position between the user viewpoint VP and the object 5 in the virtual space VS.

[0101] The user moves the fingertip 41 on the operation plane 1110 to surround the outside of the target area 53 with the trajectory information 51. Reference numeral 42 denotes the wrist of the user 4.

[0102] FIG. 28 shows how operation modes M1 to M4 are arranged on an operation plane 1110, and selection of an operation mode and selection of a target area are performed seamlessly in a series of actions.

[0103] On the operation plane 1110, the position where the user presses the fingertip 41 for a predetermined period of time (indicated by a star in the drawing) is set as the center, and operation modes M1 to M4 are arranged around this position.

[0104] An operation mode is a mode that specifies the intention of a user operation, and is a virtual button for selecting an operation menu. At the bottom of Fig. 28, an overview of a mode management table T6 that manages the contents of the operation mode (abbreviated as "mode" in the figure) is shown. The operation mode includes, for example, modes that instruct the registration and playback of media information, such as "document registration," "search," "audio recording," and "viewpoint saving," and modes that instruct the display method of an object in a virtual space.

[0105] When the user presses fingertip 41 against a desired object (sub-object) for a short time, operation modes are displayed around the area where fingertip 41 is pressed. The user selects the desired operation mode by moving fingertip 41 toward the desired operation mode while keeping the fingertip 41 pressed. The operation mode that fingertip 41 virtually touches for a predetermined period of time or more is automatically selected. The user rotates fingertip 41 counterclockwise at least once while keeping the fingertip 41 pressed. As a result, the range surrounded by trajectory information 51 is detected as target area 53.

[0106] FIG. 30 shows how the user moves wrist 42 forward and backward relative to operation plane 1110 to change the display position of the object from the front to the rear. The object can be displayed in the virtual space as a semi-transparent object. The display position may be changed by moving fingertip 41 forward and backward. In that case, however, fingertip 41 is also used for operations other than the display position operation, so it is necessary to prevent the display surface from changing during other operations. Note that a part other than the wrist may be used as the planar operation unit.

[0107] When the user 4 moves the wrist 42 forward or backward relative to the operation plane 1110, the depth L of the surface displayed for the object 5 changes stepwise between depths L1 to L6 in accordance with the movement of the wrist 42. Of the surfaces of the object 5, the surface facing the user's viewpoint VP is the foreground, and has the smallest depth L1. As the display surface moves from the foreground toward the back of the object, the depth increases.

[0108] A point 55 where the line of sight from the user viewpoint VP intersects with the surface of the object moves in accordance with the movement of wrist 42. A depth code is added to the target area set at each depth. For example, target area 53 (L3) indicates that the target area 53 is displayed at depth L3 and can be specified. If there is another object behind the object, the surface of the other object appears at depth L6, penetrating through the object in front.

[0109] The process of analyzing a user operation will be described with reference to the flowchart in Fig. 31. The process in Fig. 31 specifies the operation mode and the target area based on the user operation, and also determines whether or not it is a new area.

[0110] The user operation analysis unit 11 detects the start of a user gesture (S401). For example, when the user stops the position of the fingertip 41 for a predetermined time, makes a specific movement, or utters "start operation" or the like, the user operation analysis unit 11 recognizes that a user operation by a gesture has started.

[0111] The user rotates the fingertip 41 in a predetermined direction, for example, to specify the angle of view for performing the user operation. The user operation analysis unit 11 determines the angle of view from the movement trajectory of the fingertip 41 (S402). The user operation analysis unit 11 acquires an image of the surface of the object based on the determined angle of view, generates an operation plane 1110 on which the acquired image is arranged, and displays it in the virtual space VS (S403).

[0112] The user operation analysis unit 11 determines whether the user operation has ended (S404). The user operation may be ended, for example, by stopping the fingertip 41 for a predetermined period of time, by tapping the operation plane 1110 multiple times with the fingertip 41, or by uttering "end of operation."

[0113] If the user operation is completed (S404: YES), this process ends. If the user operation is not completed (S404: NO), the user operation analysis unit 11 determines whether it matches the depth desired by the user (S405). The user can adjust the depth to a desired depth by moving the wrist 42 forward or backward relative to the operation plane 1110. The user operation analysis unit 11 detects the depth desired by the user from the forward or backward movement of the wrist 42, and displays the object at the detected depth (S406).

[0114] The user operation analysis unit 11 detects which operation mode has been selected based on the trajectory information 51 of the movement of the user's fingertip 41 (S407). Furthermore, the user operation analysis unit 11 specifies a target area 53 from the trajectory information 51 of the movement of the fingertip 41 (S408). The target area setting unit 12 searches the database 14 for an area having an image similar to the image of the target area specified by the user (S409). The target area setting unit 12 can search for a similar area by comparing image data with each other. As described in the other embodiments, the target area setting unit 12 may search for a similar area based on the image name from among the registered areas using the image recognition unit 1231 that determines the image name from the image.

[0115] If no registered area similar to the object area specified by the user in step S408 is found (S410: NO), the object area setting unit 12 defines the area specified in step S408 as a new area (new object area) (S411).

[0116] In response to this, if the target area setting unit 12 finds an area among the registered areas that is similar to the target area by a predetermined value or more (S410: YES), it presents the registered area (existing area) to the user and asks the user whether it is OK to specify the registered area as the target area (S412).

[0117] When the user's approval is obtained (S413: YES), the user operation analysis unit 11 ends this process, and in a step not shown, registers the media information in the database 14, or plays back the media information read from the database 14.

[0118] The present embodiment thus configured also provides the same operational effects as those of embodiment 1. This embodiment can be combined with either embodiment 1 or embodiment 2.

[0119] According to this embodiment, the user can select an operation mode and specify a target area, improving usability, simply by moving the fingertip 41. According to this embodiment, the three operations of identifying a target object, selecting an operation mode, and specifying a target area can be seamlessly performed in a series of movements, further improving usability.

[0120] According to this embodiment, the display position can be changed from the front side of the object to the rear side of the object according to the positional relationship between the wrist 42 that operates the operation plane 1110 and the operation plane 1110. Therefore, in this embodiment, even if a sub-object is included inside an object or another object is hidden behind an object, the target area can be appropriately set at a desired position and in a desired range, improving usability for the user. EXAMPLES

[0121] A fourth embodiment will be described with reference to Fig. 32. In the first embodiment, a cloud system was described in which a server on a communication network CN is used as the information sharing system 1, and each user terminal 2 shares the information sharing system 1. Instead of this, the information sharing system 1B of this embodiment is configured as a local computer directly connected to the user terminal 2B. The information sharing system 1B for each user terminal 2B is connected via the communication network CN.

[0122] The database (not shown) may be stored in the external device 3, or each information sharing system 1B may hold its own database. The databases held by each information sharing system 1B may be synchronized at a predetermined timing.

[0123] This embodiment thus configured can be combined with any of the first, second and third embodiments. EXAMPLES

[0124] A fifth embodiment will be described with reference to Fig. 33. In this embodiment, an information sharing system 1C is provided in a user terminal 2C. A database (not shown) may be stored in an external device 3 or provided in the information sharing system 1C. When a database is provided in each information sharing system 1C, they may be synchronized at a predetermined timing.

[0125] This embodiment thus configured can also be combined with any of the first, second and third embodiments.

[0126] The present invention is not limited to the above-described embodiments, and includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0127] In addition, each of the above configurations may be configured in part or in whole by hardware, or may be configured to be realized by executing a program on a processor. In addition, the control lines and information lines are shown as those considered necessary for the explanation, and not all control lines and information lines are shown in the product. In reality, it may be considered that almost all of the configurations are connected to each other.

[0128] The above-described embodiment includes at least the following configurations.

[0129] (Representation 1) An information sharing system for sharing information in a virtual space, comprising a processor and a memory used by the processor, the processor executing a predetermined computer program stored in the memory to realize a virtual space management unit that displays three-dimensional objects in the virtual space, a user operation analysis unit that analyzes user operations in the virtual space, a target area setting unit that sets a target area at any location of the object based on the relationship between the object and a selection range indicated by trajectory information included in the analyzed user operation, and a related information management unit that associates related information with the set target area and manages it, and the user operation analysis unit displays an image of the selected object in the virtual space as an operation plane between the object and the user viewpoint.

[0130] (Representation 2) An information sharing system as described in Representation 1, wherein the user operation analysis unit changes the display position from the front of the object to the rear of the object depending on the positional relationship between the operation plane and a plane operation unit that operates the operation plane.

[0131] (Representation 3) The information sharing system described in Representation 2, wherein the user operation analysis unit displays at least one mode display unit for selecting an operation mode around the contact point on the operation plane where the planar operation unit makes contact, and the selection range is determined based on trajectory information of the movement of the planar operation unit from the contact point through the operation mode.

[0132] (Representation 4) An information sharing system according to Representation 3, wherein the target area setting unit is capable of setting the target area across multiple sub-objects included in the object.

[0133] (Representation 5) An information sharing system described in any one of Representations 1-4, wherein the target area setting unit is capable of setting the target area across a portion of a sub-object included in the plurality of sub-objects and a portion of another sub-object included in the plurality of sub-objects.

[0134] (Representation 6) An information sharing system described in any one of Representations 1-5, wherein the target area setting unit, when the area corresponding to the selected range is an mismatched area that does not match a registered target area, registers the mismatched area as a new target area.

[0135] (Representation 7) An information sharing system described in any one of Representations 1-6, wherein the target identification unit grasps the relationship between the selection range and the object on an unfolded map that unfolds the three-dimensional object into two dimensions, and identifies the target area of ​​the object.

[0136] (Representation 8) An information sharing system according to any one of Representations 1-7, wherein the unfolded map is provided with coordinate identification information that uniquely identifies each point on the unfolded map.

[0137] (Representation 9) An information sharing system described in any one of Representations 1-7, wherein the coordinate identification information is generated using information of two colors out of three colors that can be set for each point.

[0138] (Representation 10) An information sharing system described in any one of Representations 1-9, wherein when the target area setting unit sets the new target area, other areas related to the new target area are extracted and the extracted other areas are associated as related areas.

[0139] (Representation 11) An information sharing system described in any one of Representations 1-10, wherein the relevant region and the new target region are discontinuous on the surface of the object.

[0140] (Representation 12) An information sharing system described in any one of Representations 1-11, wherein the viewpoint in the virtual space when selecting the new target area is different from the viewpoint when extracting the related area, and the related area and the new target area are continuous on the surface of the object.

[0141] (Representation 13) An information sharing system described in any one of Representations 1-12, wherein the target area setting unit further includes an image recognition unit that recognizes an input image and outputs the image name, and the target area setting unit causes the image of the area corresponding to the selected range to be recognized by the image recognition unit, and if the area corresponding to the selected range is an mismatching area that does not match the registered target area based on the image name recognized by the image recognition unit, registers the mismatching area as the new target area.

[0142] (Representation 14) An information sharing system according to any one of Representations 1-13, wherein the user operation analysis unit analyzes a user operation on one location of the object.

[0143] (Representation 15) The specified computer program executed in any one of the information sharing systems of Representations 1-14.

[0144] (Representation 16) A storage medium that non-temporarily stores the specified computer program executed in any one of the information sharing systems of Representations 1-15.

[0145] (Representation 17) An information sharing method for sharing information in a virtual space by a computer, the computer having a processor and a memory used by the processor, the processor executing a predetermined computer program stored in the memory to execute a virtual space management step of displaying a three-dimensional object in the virtual space, a user operation analysis step of analyzing user operations in the virtual space, a target area setting step of setting a target area of ​​the object based on a relationship between the object and a selection range indicated by trajectory information included in the analyzed user operation, and a related information management step of associating related information with the set target area and managing the related information, wherein the user operation analysis step displays an image of the selected object in the virtual space as an operation plane between the object and the user viewpoint.

[0146] (Representation 18) An information sharing system for sharing information in a virtual space, comprising a processor and a memory used by the processor, wherein the processor executes a predetermined computer program stored in the memory to display three-dimensional objects in the virtual space, and displays an image of a selected object in the virtual space as an operation plane between the object and a user's viewpoint. [Explanation of symbols]

[0147] 1, 1A, 1B, 1C: information sharing system, 2, 2B, 2C: user terminal, 5: object, 10: virtual space management unit, VS: virtual space, 41: fingertip, 42: wrist

Claims

1. An information sharing system for sharing information in a virtual space, A processor and a memory for use by the processor, The processor executes a predetermined computer program stored in the memory, a virtual space management unit for displaying three-dimensional objects in the virtual space; A user operation analysis unit that analyzes a user operation in the virtual space; a target area setting unit that sets a target area at an arbitrary location of the object based on a relationship between the object and a selection range indicated by trajectory information included in the analyzed user operation; a related information management unit that manages related information by associating it with the set target area; Realize this, The user operation analysis unit displays an image of a selected object as an operation plane in the virtual space between the object and a user's viewpoint. Information sharing system.

2. The user operation analysis unit changes a display position from the front side of the object to the rear side of the object according to a positional relationship between a plane operation unit that operates the operation plane and the operation plane. The information sharing system according to claim 1 .

3. The user operation analysis unit displaying at least one mode display section for selecting an operation mode around a contact point on the operation plane where the flat operation section makes contact; The selection range is determined based on trajectory information of the movement of the flat panel operation unit from the contact point through the operation mode. The information sharing system according to claim 2 .

4. The target area setting unit is capable of setting the target area across a plurality of sub-objects included in the object. The information sharing system according to claim 3 .

5. The target area setting unit is capable of setting the target area across a part of a sub-object included in the plurality of sub-objects and a part of another sub-object included in the plurality of sub-objects. The information sharing system according to claim 4.

6. When the area corresponding to the selected range is a mismatched area that does not match a registered object area, the object area setting unit registers the mismatched area as a new object area. The information sharing system according to claim 5 .

7. The target area setting unit grasps a relationship between the selection range and the object on a development map obtained by developing the three-dimensional object into two dimensions, and identifies the target area of ​​the object. The information sharing system according to claim 6.

8. The unfolded map is provided with coordinate identification information that uniquely identifies each point on the unfolded map. The information sharing system according to claim 7.

9. The coordinate identification information is generated using two colors of three colors that can be set for each point. The information sharing system according to claim 8.

10. When the target region setting unit sets the new target region, another region related to the new target region is extracted, and the extracted other region is associated with the new target region as an associated region. The information sharing system according to claim 6.

11. The region of interest and the new region of interest are discontinuous on the surface of the object. The information sharing system according to claim 10.

12. A viewpoint in the virtual space when the new target region is selected is different from a viewpoint when the related region is extracted; The region of interest and the new region of interest are contiguous on the surface of the object. The information sharing system according to claim 10.

13. The target area setting unit further includes an image recognition unit that recognizes an input image and outputs a name of the image. The target area setting unit causes the image recognition unit to recognize an image of an area corresponding to the selected area, and, if the area corresponding to the selected area is a mismatch area that does not match the registered target area based on an image name recognized by the image recognition unit, causes the mismatch area to be registered as the new target area. The information sharing system according to claim 6.

Images