Method for processing graphically displayable object by at least two users, system for processing graphically displayable object by user, computer program and storage medium

Abstract

A method for processing a graphically displayable object by at least two users, a system for processing a graphically displayable object by a user, a computer program and a storage medium. The invention relates to a method for processing a graphically displayable object (10) by at least two users. Further, the invention relates to a system, a computer program and a storage medium.

Description

[0001] The invention relates to a method for processing a graphically representable object by at least two users according to claim 1. Furthermore, the invention relates to a system for processing a graphically representable object by at least two users according to claim 9. Finally, the invention relates to a computer program according to claim 10 and a storage medium according to claim 11.

[0002] It may be efficient if several users or colleagues can work on a document together. For example, there is the concept of jointly processing a table or text document, where all participants see the same view and additional views are not available to each individual participant. Such a representation is advantageous for the processing of text documents, tables or whiteboards with text and pictures. However, such a representation is not helpful for improving the knowledge graph, especially if additional human knowledge is to be used. Therefore, such technical objects can currently only be performed by technical professionals with specialized knowledge, which limits the potential for true collaborative work and collective intelligence.

[0003] In the case of the above documents, individual objects cannot usually be locked for participants during collaborative work, which can lead to conflicts in collaboration and make it inefficient. In addition, some applications allow locking of sections for participants, but this results in other participants not being able to see their colleagues' editing contributions. In this case, the locking mechanism is not flexible because it is started and stopped by the participants themselves, which can sometimes lead to long locks because it is difficult for other participants to unlock the required sections.

[0004] The object of the present invention is to provide a method for processing graphically representable objects, as well as a system, a computer program and a storage medium for processing graphically representable objects, by which the above-mentioned problems of the prior art do not occur and by which advantageous processing of multiple graphically representable objects by several users is made possible.

[0005] According to the invention, this object is solved by the subject matter of the independent patent claims. Advantageous embodiments and further embodiments of the invention are indicated in the dependent patent claims as well as in the description and the drawings.

[0006] A first aspect of the invention relates to a method for processing a graphically displayable object by at least two users, the method comprising the steps of:

[0007] - importing, by means of a computing device, a file containing a graphically displayable object;

[0008] - displaying at least corresponding areas of the object on at least two separate display devices;

[0009] - detecting, via a first input device (16), a user input performed by a first user of the at least two users, wherein a portion of the object is selected by the first user;

[0010] - in at least one further display device of at least one further user, coloring the portion and / or its surroundings in a color associated with the first user; and

[0011] - blocking the area from processing by at least one second user.

[0012] The import can be effected, for example, from a memory of the computing device and / or via a communication network containing graphically displayable objects by a control device, in particular configured as a computing device. The display devices are each configured individually and can in particular be located at different locations, for example in different rooms separated from one another. The display devices are in particular controlled by the control device and / or a further control device connected thereto for displaying the corresponding area.

[0013] The detection is performed based on a user input by means of a first input device, wherein an area on the object is touched and / or selected by a first user. In other words, the respective user, who in particular wants to process the object, has an input device available by means of which he can interact with the object, for example he can select areas of the object by means of a search function or touch them, for example, depending on the type of input device.

[0014] In a fourth step, the object is displayed in at least one further display device in such a way that the part selected by the first user is marked, in particular with a color, based on the user input in such a way that this can be displayed by the display device. The part first marked by the first user with the aid of the input device is reserved for processing, so that the method effectively prevents the part of the object from being processed simultaneously by several users.

[0015] The method thus creates the possibility for several persons or users, in particular working from different physical and / or logical locations, to collaboratively access a common object via a VR (Virtual Reality) or AR (Augmented Reality) environment, which common object is available, for example, as an input file or repository and includes various digital artifacts, such as a knowledge graph.

[0016] The present invention is based on the recognition that several people may want to process and / or update a common model, such as a knowledge graph or other graphical representation, simultaneously from different locations. This is the essence of collaborative work, where the synergy of collective intelligence can be exploited. However, if two or more people want to modify the same object or the same part of an object, conflicts may occur, which are advantageously prevented by the method according to the present invention.

[0017] Furthermore, one problem with graphical representations of knowledge graphs is that they are often created automatically by AI algorithms and are rarely easily understandable by humans. As a result, especially for graphical representations of knowledge graphs, some parts of IT (information technology) are too specific or have too few inputs to properly train AI (artificial intelligence) algorithms. Therefore, some parts still need to be manually updated or processed.

[0018] However, in order for such updates to be effective, the graphical representation of the object, in particular a two-dimensional or three-dimensional representation, should be understandable to humans. This is often a challenge because different people have different backgrounds, which affects their understanding. In order for the contributions of the multiple people involved to be effective, the representation must be advantageously suitable for each user, for which a corresponding display device can be provided. Furthermore, for parallel collaboration, users want to know who else is involved in updating each part of the model or object and how long the processing may take accordingly. The automatic coloring and locking of objects according to the present invention can solve the mentioned problems.

[0019] Customization can include the visual appearance of the item, which each user can configure, color, design and / or arrange according to his or her own ideas and preferences. Thus, the visual appearance of an object for each participant can be the same or different from the visual appearance of an object for another participant. Of course, these appearances are viewed or experienced simultaneously.

[0020] Thus, for each participant, the visual appearance of an object may be the same or different from the visual appearance of the object for another participant. Of course, these different appearances are experienced simultaneously by different users.

[0021] In one embodiment of the invention, the color and / or the lock disappears, in particular slowly, after a predetermined time and / or depending on a further user input by means of the first input device and thus in particular by the first user. In other words, the coloring can be considered similar to marking an object in the environment with pheromones, e.g. the pheromones stay on the object and become less and disappear over time. Thus, the color may fade over time, making it visible to at least one other user that the lock is approaching its end, and thus when the first user has finished his processing, processing by other users can proceed. This leads to the advantage that the method can be used particularly effectively for processing graphical displayable objects, e.g. for three-dimensional displayable objects.

[0022] According to one embodiment, the intensity or darkness of the coloring of the locked elements of an object will indicate to the participant how long these elements will be locked and approximately when they will be available again. Preferably, the darkness of the locked elements varies over time, like a shading with or without discrete intensity steps.

[0023] In another embodiment of the present invention, at least two display devices are configured as VR head-mounted devices and / or AR head-mounted devices, and thus the object is displayed in a virtual space and / or multiple virtual spaces and / or at least one environment of the display device. In other words, the display device is used to generate a virtual image in a virtual environment and / or an augmented environment. In this regard, the user can additionally be represented as an avatar. Additionally or alternatively, for example, the object can be displayed in multiple virtual spaces so that each user can be in a separate virtual space. Furthermore, if there are more than two users, a working group can be formed in this way. When the object is displayed in the environment of the display device, augmented reality is used to display the object, whereby the environment can be located at the corresponding display devices separated from each other. This leads to the advantage that the method can be used particularly flexibly.

[0024] In a further embodiment of the invention, the first input device of the first user and / or at least one further input device of another of the at least two users are each configured as a controller and / or a data glove. In other words, one of the respective input devices is provided with sensors which are in particular able to detect positions in space, thereby enabling objects to be manipulated, for example, by means of gestures. This leads to the advantage that the method can be implemented in a particularly user-friendly manner.

[0025] In a further embodiment of the invention, the selection of a part and / or at least one further part of the object is performed by a gesture of the respective user using a respective input device. In other words, gesture control is made possible, by means of which the respective user can mark at least one part or several parts of the object, in particular simultaneously, in order to block them for processing. This leads to the advantage that the user can advantageously interact with a plurality of parts.

[0026] In another embodiment of the present invention, the object is a graph, in particular a knowledge graph, and / or a mind map, and / or is written in a unified modeling language, and / or can be represented by at least two alternative representations. In other words, different structural data can be graphically prepared in different ways. This leads to the advantage that the respective user can select from one representation. Additionally or alternatively, another advantage is that the object can be particularly advantageously used to display information.

[0027] In a further embodiment of the invention, a representation form and / or a color and / or a block time and / or an avatar and / or a virtual room are selected for one of the users with the aid of an artificial intelligence or a self-learning algorithm and / or a neural network. In other words, when the method is carried out, the self-learning algorithm observes the users and processes the personal selections made by the respective user. Additionally or alternatively, the AI ​​accesses, for example, the social media profiles of the user. For example, different user profiles can be accessed so that, for example, the corresponding display form is adjusted to the background knowledge of the user. The information obtained therefrom can be used to save the user input. This has the advantage that the process can be used particularly conveniently for the user.

[0028] In another embodiment of the present invention, the second user and / or another user can mark the area colored by the first user, thereby transmitting a request signal to the first user. In particular, the request signal can be represented as an icon or mark on an object, for example. Additionally or alternatively, the request signal can also be output tactilely via an input device or the like. In this way, the second user or another user can notify the first user of a processing request or cooperation. This leads to the advantage that the method can be efficiently used by multiple users.

[0029] A second aspect of the present invention relates to a system configured for processing graphically displayable objects by at least two users, and the system comprises a computing device, at least two display devices separated from each other, and at least two input devices, wherein the system is configured for executing the method according to the first aspect of the present invention.

[0030] In this respect, advantages and advantageous embodiments of the first aspect of the invention are to be regarded as advantages and advantageous embodiments of the second aspect of the invention, and vice versa.

[0031] A third aspect of the invention relates to a computer program. The computer program can be loaded into a memory of a computing device of a mapping device, for example, and comprises program components for executing the steps of the method when the computer program is executed in the computing device.

[0032] In this respect, advantages and advantageous embodiments of the third aspect of the invention are to be regarded as advantages and advantageous embodiments of both the second aspect and the first aspect of the invention, and vice versa in each case.

[0033] A fourth aspect of the present invention relates to a storage medium. The storage medium comprises electronically readable control information stored thereon, the control information comprising at least one computer program according to the second aspect of the present invention, and is configured to, when the storage medium is used in a computing device, perform the method presented herein according to the first aspect of the present invention.

[0034] In this respect, advantages and advantageous embodiments of the fourth aspect of the invention are to be regarded as advantages and advantageous embodiments of the third, second and first aspects of the invention, and vice versa in each case.

[0035] The present invention may take the form of a computer program including a program module accessible from a computer-available or computer-readable medium storing program code, for use by or in combination with one or more computers, processors, or instruction execution systems. For the purposes of this specification, a computer-available or computer-readable medium may be any device capable of containing, storing, transmitting, propagating, or transmitting a program, for use by or in combination with an instruction execution system, device, or apparatus. The medium may be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or device or apparatus) or a propagation medium, which itself as a signal carrier is not included in the definition of a physical computer-readable medium, which includes semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), read-only memory (ROM), hard disk, and optical disk, such as a compact disk read-only memory (CD-ROM), read / write optical disk, and DVD. As known to those skilled in the art, both the processor and program code for implementing each aspect of the technology may be centralized or distributed (or a combination thereof).

[0036] For use cases or usage scenarios that may occur when carrying out the method and are not explicitly described here, it can be provided that, according to the method, error messages and / or prompts for user feedback are output and / or default settings and / or predetermined initial states are set.

[0037] Although the present invention has been described in detail with reference to certain embodiments, it will be appreciated that the present invention is not limited to those embodiments. In view of this disclosure, many modifications and variations will exist for those skilled in the art without departing from the scope of the various embodiments of the present invention as described herein. Therefore, the scope of the present invention is indicated by the following claims rather than by the preceding description. All changes, modifications and variations within the meaning and scope of the equivalents of the claims are considered to be within their scope. All advantageous embodiments claimed in the method claims may also be applied to the system / device claims.

[0038] The accompanying drawings show:

[0039] Figure 1 is a schematic flow chart of a method for processing a graphically displayable object by at least two users; and

[0040] Figure 2 is a schematic diagram of a system in which at least two users process graphically displayable objects.

[0041] Figure 1 A schematic flow chart of a method for processing a graphically representable object by at least two users is shown, comprising several steps:

[0042] In a first step S1, a file containing a graphically representable object 10 is imported by the computing device 12. Figure 2 . In a second step S2, at least one corresponding area of ​​the object is displayed on at least two separate display devices 14. In a third step S3, a user input is detected by a first user of the at least two users by means of a first input device 16, wherein a portion on the object 10 is marked by the first user. In a fourth step S4, in at least one further display device 14 of at least one further user, the portion and / or the environment is colored with the color assigned to the first user. Furthermore, in a fifth step S5, the portion is blocked for processing by at least one second user.

[0043] By using the method, several users, in particular users from different physical locations, can access the object 10 or the data available therefor, in particular those imported in step S1 , via a virtual and / or augmented reality environment.

[0044] For this purpose, it has been shown to be advantageous if at least two display devices 14 are configured as VR head-mounted devices and / or AR head-mounted devices and thus (one or more) objects 10 are displayed in the virtual space of the display devices 14 and / or in several virtual spaces or in at least one environment.

[0045] In a virtual space, a user may be virtually presented via an avatar in one or more interconnected public spaces. Further, this applies to environments in which a virtual image of the user may also be displayed. Further, if an AR view is used instead of a VR view, the method enables a user in a room or environment to be viewed as an AR hologram.

[0046] The method can thus create the impression that several participants are almost physically working together. This allows them to see each other and capture what the others are doing, which improves remote communication. This is particularly valuable for educational purposes, for example.

[0047] Depending on the preferences of a particular user, a graph, particularly a knowledge graph, can be represented in a shared virtual space as a series of, for example, 3D spheres and / or other two-dimensional or three-dimensional elements. The connections between the spheres can represent the relationships between the elements. The representation of the elements of the graph can be selected by the user from a set of available views and / or automatically calculated based on the user's profile if the user allows access to, for example, a social profile, thereby enhancing the displayability of the entire graph to the current user. Each participant can see the entire scene in the occasion of their choice. For example, when a user searches for a specific part of the graph by entering a search term, the related events of the search are displayed as larger objects 10 in the virtual space.

[0048] The object 10 is a graph, in particular a knowledge graph, and / or a mind map, and / or is written in a unified modeling language, and / or can be represented by at least two alternative representation forms.

[0049] Furthermore, it is advantageous if after a predetermined time and / or depending on a further user input by means of the first input device 16 the blocking of the color and / or the colored area and / or the object area disappears, in particular slowly.

[0050] If a participant or user starts processing a part of the atlas or object 10, this part is marked by the participant's color, that is, each element of the part is surrounded, for example, by a transparent colored cloud having properties that mimic pheromones occurring in nature.

[0051] Such a color cloud or pheromone cloud can fade over time if the corresponding element or part has not been marked and / or processed by the owner of this color with the aid of an input device. That is, the intensity of the pheromone cloud shows other users the current processing focus of this particular first user. Thus, other users can see whether a certain area has become free for other participants. The pheromone cloud protects the processed part from competing accesses, and in particular its automatic disappearance or hiding protects the element from being locked for an unnecessary long time. For example, after a certain time without user input or contact with the first user, the pheromone becomes weaker and therefore the intensity of the color decreases, which can lead to automatic unlocking. Additionally or alternatively, the user can select the locking time. The user can be notified when the part or element is about to be released for processing.

[0052] Advantageously, the first input device 16 of the first user and / or at least one further input device 18 of the further user of the at least two users are configured as a controller and / or a data glove, respectively. The latter are thereby advantageously suitable for marking a portion of the object 10 and / or at least one further portion by gestures of the respective user using the respective input device 16, 18. In other words, for example, a gesture similar to casting a net can provide a method for selecting multiple portions of the atlas at once. For example, these can also be selected from a menu. Additionally or alternatively, specific gestures can be recognized by a camera or tracker of the head mounted device or the controller, respectively.

[0053] For example, the user's hands can be moved around several parts to select them and color them accordingly. Another alternative to casting a net on a subset of atlas elements is eye tracking, for example by means of a camera sensor of a VR headset or an AR headset, respectively. This method can be used, for example, after an initial search, when some relevant parts have been pre-selected by other search methods and are displayed near the user. This allows the user to select a subset of the parts currently being processed.

[0054] Advantageously, an artificial intelligence is also provided, which selects the representation form and / or color, the block time and / or the avatar and / or the corresponding virtual space for one of the users. Additionally, it is advantageous to use one virtual room and / or multiple virtual rooms, depending on the situation. Therefore, the virtual space can be used for grouping. For example, the object 10 can be displayed in multiple virtual spaces, each space containing parts of the object 10 that can be assigned to, for example, a specific theme and / or a technical and / or organizational unit and / or other grouping criteria. Therefore, the user can virtually walk around the rooms, search for relevant content and mark sub-areas for processing. After such a tour, the corresponding user can, for example, pull the parts he marked towards him by gestures, so that each part can be advantageously processed at the display device 14.

[0055] Furthermore, by hiding the colors or making them disappear, it is not necessary to deselect irrelevant parts, since they are automatically released after a period of time. The respective user can define a virtual room by selecting a combination of criteria and reserve the entire room for himself. The respective user can further define the corresponding distribution of the parts of the object 10. The respective parts of the object 10 or the atlas can be displayed in a space for each user, respectively, and these spaces can be entered by other users or their avatars, respectively. Additionally or alternatively, the parts can further be displayed in other virtual spaces.

[0056] Furthermore, it is advantageous for the method if the second user and / or another user can mark the area colored by the first user, whereby a query signal can be transmitted to the first user. In other words, the user can request and / or access the view of a colleague. In doing so, the user can subscribe to the elements occupied and colored by his colleague and suggest changes to them. In addition to the coloring, these areas contain small signs or marks, for example in the form of shading, which have the color of the other user. After marking the request signal, the first user who handles the area is shown, for example, a suggestion for a change. Otherwise, if there is no suggestion, only the user who wants to handle the section is displayed. If desired, the respective user can lend his color to the colleague for a certain time.

[0057] Figure 2 A system 20 is shown in a schematic view, which system 20 can be used for processing a graphically displayable object 10 by at least two users and which includes at least one computing device 12, at least two display devices 14 separated from each other, and at least two input devices 16 and 18. In this regard, the system 20 shown is configured to perform the presented method.

[0058] Both thus serve a particularly advantageous efficiency increase when processing an object 10. This enables efficiency increase, in particular, through different views of the same object 10 by different users. This makes it easier for the respective user to understand or grasp the diagram, which makes updating the graphical representation faster and less prone to errors.

[0059] For example, AI's algorithmic approach identifies user preferences and / or user profiles and can thereby change settings for the user, such as preferred display style, etc., which can also improve efficiency.

[0060] When a user touches or marks the part, for example by pointing to the corresponding area with a controller configured as a remote control, an element of the graphical display can be marked with a visual or virtual pheromone or color, respectively. These pheromones or colors can have different intensities depending on how often the user handles the corresponding part. Furthermore, it can be observed how much time has passed since the last touch or handling. This type of locking element makes it possible to avoid conflicts during collaborative work by at least two users. This can make the work more efficient and stable. In particular, the temporal nature of the colors or pheromones makes the collaborative work efficient.

[0061] Copying parts of the respective object 10 may be particularly relevant to the user, since gesture control in the virtual space allows for a quick visual adjustment of the focus or view of the respective user. All other participants can see their own grouping without affecting the focus of others. This is done, for example, in the user interface and thus in the display device 14 of the respective user.

[0062] The graph (object 10) or the information or data on which it is based is stored, for example in an abstract form, on a server or server infrastructure. Thus, the method and system 20 provide collaboration by using cloud computing and therefore by using a computing infrastructure.

[0063] The method provides a particularly advantageous approach of allowing collaborative processing by concurrent users through visualization and navigation of graphical representations. This enables the type of collaborative processing that may be useful, for example, during lectures and / or seminars and / or distance learning courses.

[0064] In this specification and claims, the terms "lock" and "block" are used synonymously, and the specific selection does not enter any limitation in any sense.

[0065] Independent of the use of the grammatical term, for example, "user" individuals with male and female identities are included within the term.

[0066] Reference Symbols

[0067] S1 First step

[0068] S2 Step 2

[0069] S3 Step 3

[0070] S4 Step 4

[0071] S5 Step 10 Object

[0072] 12 Computing Devices

[0073] 14 Display Devices

[0074] 16 First input device

[0075] 18 Second input device

[0076] 20 systems.

Claims

1. A method for processing a graphically representable object (10) by at least two users, comprising the steps of: - importing a file containing a graphically displayable object via a computing device (12); (S1) - displaying at least corresponding areas of the object (10) on at least two separate display devices (14); (S2) - detecting, via a first input device (16), a user input performed by a first user of the at least two users, wherein a portion of the object (10) is selected by the first user; (S3) - in at least one further display device (14) of at least one further user, coloring the portion and / or its surroundings in a color associated with the first user; (S4) and - blocking the area from processing by at least one second user. (S5) 2. The method according to claim 1, characterized in that After a predetermined time and / or depending on a further user input via the first input device (16), the color and / or the blocking disappears.

3. The method according to claim 1 or 2, characterized in that The at least two display devices (14) are configured as VR head mounted devices and / or AR head mounted devices, and thus the object (10) is displayed in a virtual space and / or several virtual spaces and / or at least one environment of the display device (14).

4. The method according to claim 1, characterized in that The first input device (14) of the first user and / or at least one further input device (18) of another of the at least two users are each designed as a controller and / or as a data glove.

5. The method according to claim 4, characterized in that The selection of the part and / or at least one further part of the object (10) is performed by a gesture of the respective user using the respective input device (16, 18).

6. The method according to claim 1, characterized in that The object (10) is a graph, in particular a knowledge graph, and / or a mind map, and / or is written in a unified modeling language, and / or can be represented by at least two alternative representation forms.

7. The method according to claim 6, characterized in that A representation and / or a color and / or a block time and / or an avatar and / or a virtual space is selected for one of the users by an AI method.

8. The method according to claim 1, characterized in that The second user and / or another user may mark the portion colored by the first user, thereby transmitting a request signal to the first user.

9. A system (20) for processing a graphically displayable object (10) by at least two users, the system comprising at least one computing device (12), at least two display devices (14) separated from each other, and at least two input devices (16, 18), the system being configured to execute a method according to one of claims 1 to 8.

10. Computer program directly loadable into a memory of a computing device, the computer program comprising program means for performing the steps of the method according to any one of claims 1 to 8 when the program is executed in an electronic computing device.

11. Storage medium having electronically readable control information stored thereon, the storage medium comprising at least one computer program according to claim 10 and being configured to perform a method according to one of claims 1 to 8 when the storage medium is used in a computing device.

Images