System and method for producing a digital personality clone, and use thereof

Abstract

The invention relates to a system and a method for producing a digital personality clone and to the use of a digital personality clone produced in this way. The technique disclosed here for the first time allows the generation of a digital personality clone of any person who makes themself available. A digital personality clone can assist in all aspects of life and work. In the field of work, a digital personality clone can instantaneously make experts from abroad available on site, and allows 24 / 7 participation in meetings, multiple times and all around the world. In addition, it can demonstrate empathy and emotions where normally any reactions are those of a machine. Finally, as people age, a digital personality clone can virtually display past physical, psychological or other characteristics and cognitive capabilities that would otherwise remain to only a limited extent or be lost altogether.

Description

[0001] Description

[0002] System and method for producing a digital personality clone and use therefor

[0003] Technical area

[0004] The invention relates to a system and a method for producing a digital personality clone and the use of a digital personality clone produced in this way.

[0005] There are already systems that provide an infrastructure with which individual, personal information and / or data is collected and processed into a user profile - also using artificial intelligence "Kl" - especially in the virtual world, for example in the industrial metaverse and for improved human-machine communication.

[0006] Despite constant further development of AI, it has not yet been possible to digitally create a machine with a genuine, human-like individuality and / or with individual knowledge and / or taste, style, e.g., speaking style, writing style, negotiating style, or opinion. In particular, adapting a machine to an individual human counterpart is not yet an issue in the further development of human-machine communication systems such as chatbots. However, it plays an important role, for example, in the smooth flow of human-machine communication. With regard to the efficiency of human-machine communication, this accommodation of the machine in the communication style is a decisive factor in saving time, incorrect solutions, and detours. These savings, in turn, have a direct and immediate impact on energy consumption and capacity.The difficulty in establishing a specific, individually tailored communication style, in turn, arises, among other things, from the continued lack of comprehensive solutions for generating a person's digital individuality, personal intellectual, cultural, and / or other characteristic background, knowledge, and communication style. Aside from the efficiency deficit caused by misunderstandings and misinterpretations, existing AI systems also do not effectively capture the breadth and depth of a person's experiences, preferences, skills, and / or behavioral nuances and / or do not do so in a way that would allow for the continuation of an intellectual legacy, customer service, and / or the training of successors in the workplace by machines or similar means.

[0007] Apart from that, there is currently no communication system that imitates an absent acquaintance or even a loved one and / or authentically supports a disabled human user with memory impairment, such as dementia, etc. Especially in the medical field, it is an important component of digitalization that the personal, individually recorded inputs coming from a human user are correctly processed and interpreted.

[0008] However, in many respects, there is a need to digitally capture a person, an individual, with their personal attitudes, opinions, style, idiosyncrasies, etc., and then, in a second step, to digitally reproduce them. Experts, specialists, decision-makers, and managers, in particular, are often needed in two or more positions simultaneously. In some of these positions, a digital clone of the person in question might be sufficient or at least help fill the gap.

[0009] International meetings often involve scheduling conflicts and / or time zone differences, meaning not all participants can connect at the same time. Nevertheless, there may be a need to consider the perspectives of all invited participants in the discussion. A digital clone trained with the perceptions, opinions, viewpoints, and perspectives of the person on whom the digital clone is based on the relevant topics would be a useful tool for this.

[0010] On the other hand, AI assistance is offered for all types of technical problems, especially when learning a new program, a new tool, a new technology, in the patient area, and / or when using an analytical-diagnostic device. Bot assistance is not always sufficient; sometimes, an additional expert, specialist, or expert is required. This expert, specialist, or expert may be busy elsewhere, for example, and it would be important to make their knowledge digitally accessible, independent of their location, and thus accessible everywhere.

[0011] The creation of a digital personality clone can bring decisive energy and cost advantages in terms of efficiency, cost-effectiveness, sustainability in terms of the required resources, etc. in machine-human communication for many technical digital use cases, such as in the industrial metaverse, in medicine, in dealing with chatbots, in industrial manufacturing and / or in the workplace.

[0012] The energy consumption of existing communication systems is not only limited to the end device, but also to the large central servers that are constantly running, processing vast amounts of data, and requiring cooling. Therefore, reducing the amount of data to be processed, shortening the use of digital devices, and shortening the processing time by a computer and an industrial plant, manufacturing, and / or communication, results in a significantly greater energy input and energetic impact beyond the energy savings at the user's end device, resulting in a significantly greater energy input and energy impact beyond the energy consumption of central servers, including cooling. Estimates now assume that data centers in Germany alone consume 10 to 15 billion kWh of electricity per year.

[0013] In order to ensure the smoothest and most efficient use of digital systems such as virtual reality and / or industrial metaverse, there is a need for digital technology to create a digital personality clone.

[0014] Technical background

[0015] Attempts have already been made to create virtual contacts that are as human-authentic as possible. For example, Google® Duplex® has developed a service that mimics human conversation, but this only works over the phone - see "Leviathan, Y., & Matias, Y. (2018). Google Duplex: An AI system for accomplishing real-world tasks over the phone".

[0016] Replika® is also a platform that allows you to create a Kl-friend, but this is limited to a friendly conversation partner - see www.replika.ai.

[0017] In "MyLifeBits," Microsoft engineer Gordon Bell attempted to store his own life in data form in order to capture it completely and make it quickly retrievable at specific points (see "Gemmell, J., Bell, G., & Lueder, R. (2006). MyLifeBits: a personal database for everything. Communications of the ACM, 49(1), 88-95). To this end, he attempted to store and catalog his life as completely as possible in the form of data.

[0018] Communication systems with at least one human participant are also called conversational systems. A prominent example is the chatbot ChatGPT® in Python®. It uses artificial intelligence (AI) that is continuously trained using data from previous communications with a human participant, the same human participant, or the same group of human participants, particularly with regard to individual characteristics. The goal is to establish the most seamless communication between humans and machines possible. This training is based on text input from a human user and additionally on digitally accessible training data, which may include the entire world literature.

[0019] In industrial development and manufacturing, data is generated from reality—i.e., the product at stage x of production—using all kinds of images, measurements, and / or sensors, and used to predict the product's quality, service life, completion date, etc. On the other hand, data from a digital image of the product or the precursor stage is increasingly being used. This data is generated in the so-called metaverse, e.g., using a "digital twin."

[0020] The representation of a technical object, or its precursor, in the metaverse is therefore possible – for example, through a "digital twin." Humans are represented in the metaverse by proxies, such as avatars. The quality, reliability, and / or efficiency of the avatars in the metaverse depends on the type of training of their underlying class. Digital personality clones in the sense of comprehensive digital personality duplicates – comparable to a "digital twin" for humans instead of inanimate objects – do not yet exist.

[0021] Summary of the invention

[0022] In many virtual reality applications, such as the industrial metaverse, human users are represented in the form of their avatars. There is a need to individualize avatars, particularly to bring them closer to personality profiles, i.e., to create personalized, individual avatars and not just—as is currently the case—categories and / or genres of avatars that correspond to a certain type but are not a digital twin of an individual.

[0023] The key to this is generally believed to be training data. Firstly, this data is subject to personal data protection; secondly, even if the data subject is interested in the most complete possible recording of themselves, there is no technology that enables comprehensive recording of the person, ideally including their character, background, cultural, educational, familial, in short, their very personal characteristics and individual assets. To date, the generation of training data for CLs connected to communication systems has been based on human communication participants in general terms, such as "how does a human behave in contrast to a machine," or, for example, how does a European, an athlete, a manager, a mother, etc., behave, without considering a specific human individual.To date, there is no communication system that replicates a specific partner based on a real human model and convincingly reproduces their characteristic profile, because the orientation of the avatars or digital representatives in virtual space imitates people with certain characteristic profiles “as such” and not individuals.

[0024] However, this leads to a situation where, in the virtual world, such as the metaverse, technical arrangements and tradable products can be perfectly replicated, but users and people in general cannot yet be digitally created in a convincingly individual way. A typical example of this is the "uncanny valley" phenomenon. The "uncanny valley" effect describes a phenomenon in which human-like robots or computer-generated characters that look almost, but not perfectly, human trigger feelings of discomfort or disgust in observers. This "uncanny valley" effect presumably occurs because small imperfections or inconsistencies in human-like characteristics are more noticeable, which could possibly be an evolutionary response to detect potential threats or sick individuals.

[0025] Therefore, the object of the present invention is to create a digital system and a method for generating a digital personality clone that surpasses previous digital systems in terms of human-individual authenticity, e.g., of machine communication participants, and thus enables the creation of a digital personality clone. Creating a seamless, natural-looking experience for the user in interaction with the avatar is crucial to avoid, for example, an "uncanny valley" effect.

[0026] In addition to creating an authentic population of the metaverse, this also raises the efficiency and understanding of machine-human communication to the next level. Furthermore, it creates the possibility of representation through an avatar. Thus, among other things, a digital personality clone significantly improves the performance of existing communication systems without additional energy input and / or additional operating costs and / or usage time. Above all, it saves computing capacity, the user's cognitive capacity, and / or storage space. Using a digital personality clone, both software and hardware products can be used more effectively, both in terms of energy consumption and the reliability of interpreting a response generated by AI.Such technology conserves all resources, is thus a sustainable solution, and counteracts disruptions caused by misunderstandings and / or errors in human-machine communication, which is already ubiquitous. It should be noted, however, that human-machine misunderstandings can be funny in minor cases but can be quite devastating under certain circumstances.

[0027] This object is achieved by the subject matter of the present invention as disclosed in the claims, the description and the figures.

[0028] Accordingly, the subject of the present invention is a system for generating a digital personality clone, which comprises two or more recording devices, one or more computer-aided units that process the data from the recordings into editable data, one or more interfaces to neural networks for processing the input by means of artificial intelligence, the latter with or without access to the IoT, wherein it is provided that the at least two recording devices are arranged with respect to the human user in such a way that a first recording device for the most complete recording of the cognitively perceived environment of the user, wherein the recording device - following the user and / or imitating the user - follows the line of sight and records as much as possible based on the available sensors of what is perceived and mentally processed by the human user with his 7 senses,generates data from it and forwards this data in real time to a neural network with a Kl, and a second recording device is focused on the human user and records the human user's reaction to what he perceives from his cognitive environment in real time, generates data from it and forwards this data - like the first recording device - to the neural network with the Kl, and the processing by the Kl calculates a personality from this data, makes the data package(s) with the calculated personality available for the creation of an avatar, and as a result of the processing of the data from the at least two recording devices, a data package results on which a digital personality clone of the human user is based.

[0029] Furthermore, the present invention relates to a method for generating a digital personality clone, comprising the following method steps: a) recording and data collection of the cognitively perceived environment as well as the digital and analogue activities and experiences of a human user, b) synchronous recording of the human user, how he experiences this cognitive environment and actions, how he reacts and / or what he perceives and feels during the digital and analogue activities and / or experiences, c) processing, intermediate storage and forwarding of the data obtained by a) and b) by one or more computer-aided units, d) generation of one or more results of the computer-aided processed input by a unit to create a data packet representing a digital personality clone,e) Saving the result in the form of data packets and f) optionally refining and supplementing the data packets and / or results of the AI ​​processing with new training data from further recordings, recordings and / or inputs by repeating the first two process steps a) and b) and further processing by means of process steps c), d) and e).

[0030] Furthermore, the subject matter of the invention is a computer program product with program code means which cause a correspondingly configured processor to carry out a method according to one of the preceding claims when the program code means are processed by the processor, as well as a computer-readable storage medium therefor.

[0031] Finally, the subject of the invention is the use of the digital personality clone in human-machine communication such as a chatbot or to represent the human user in the virtual world.

[0032] Humans generally have five main senses plus two "additional" senses: sight, hearing, smell, touch, and taste as the primary senses, and the sense of balance and proprioception as "additional" senses. In addition to the primary senses, there are already known sensory systems that capture sensory perceptions, convert them into data, and prepare them for processing by AI. This also includes smell and / or taste sensations, which are captured as multimodal stimuli.

[0033] Perceptions of the latter two senses cannot be generated by one organ alone; this perception is the result of a complex integration of multiple perceptions of the primary senses. However, the technology for capturing the additional senses is highly advanced and will surely lead to devices during the term of the patent that provide sensor technology capable of capturing perceptions via these senses and generating data from them.

[0034] The first recording device for capturing the user's cognitively perceived environment as completely as possible includes all types of sensors that mimic the human senses, such as hearing, sight, touch, smell, etc., for example, a video camera. This is coupled to the user's orientation, for example, in the form of glasses that follow their eyes and thus selectively capture only what the user's eyes can see. This is important for the purposes of the invention because, firstly,

[0035] What the user did not perceive – and was not in their field of vision, for example – cannot and could not have influenced their reaction. Secondly, a fraction of the data volume results if not all visual data in the 360° environment is recorded and processed, but rather the relevant selection of data from the visually perceived environment is made straight away. This is comparable to audio data, temperature data, recording of the composition of the ambient air, etc. For this purpose, the system comprises one or more aligned sensors, such as microphones, which automatically align themselves with the user. These preferably align themselves according to the direction of the user's head and thus generate a sound pattern that the user also perceives due to their two ears.For example, an acoustic deficit, in the form of poor hearing by the user, can also be defined in the system, for example that the user no longer hears well or not at all above a certain frequency.

[0036] The other senses available to a human user for cognitive perception of the environment, such as smell (sense of smell), taste (sense of taste), touch, sense of balance and / or proprioception, are also recorded via appropriate sensors, depending on the quality and features of the system, and the corresponding data is generated and forwarded to a neural network with a class A sensor. For this purpose, devices are equipped with the respective sensors directly on the user or in their immediate vicinity via a corresponding, possibly movable, device and networked. For example, sensors for detecting the smell of the environment, the perception of movement, position and gravity (balance), as well as position and movement (proprioception) can be integrated on and into appropriately equipped headphones.

[0037] The second recording device focuses on the human user and is part of the same system as the first. Both recording devices are connected to the same computer-based technology and linked to the same AI, so that the AI ​​can receive and process data from both recording devices in real time. Only in this way can usable training data be obtained.

[0038] The second recording device is directed at the human user and their reactions to the cognitively perceived environment. For this purpose, the second recording device primarily comprises a video camera with microphones that records gestures, facial expressions, micro-expressions, posture, possibly even escape behavior, increased heart rate, blood pressure, blinking, movement, trembling, body tension, etc. The second recording device also includes, for example, an "object tracking" device that is set to the user as a whole or parts of the user, such as the eyes, ears, nose, as the "object." There are various tracking variants, such as eye tracking, ear tracking, and head tracking. The second recording device generates data from both the optical and / or acoustic observation and other tracking and transmits it in real time, i.e., simultaneously with the data from the first recording device, to the CLs used jointly by recording device 1 and recording device 2.

[0039] The system can also record the entire body of the human user. Using a so-called "locomotion function," a mobile system with a first and second recording device—or parts thereof—follows the entire body of the human user and simultaneously records the cause of the user's reaction. Several robots 10, which are coupled, for example, with sensors housed in devices worn by the user and also equipped with a microphone and / or camera, follow the human user and detect where the user is walking. If necessary, through interaction with the other robots 10, they also detect where the user is looking. One robot 10 sees the user from behind, another from the side, and the third from the front, thus ensuring that no reaction or movement of the user and / or their body and / or their facial microexpressions is missed.

[0040] Both virtual reality, e.g. metaverse, and communication between humans and machines are now present in all areas of life. From medical technology to transport, infrastructure, mobility, culture and industrial manufacturing, data and data packets are exchanged everywhere; data is generated, processed, inputted, processed using generative AI, models further developed and finally outputted with new meaning. Typically, a human is involved at an interface and / or checks the results, so that input and / or output can be expected from this side. This is where the invention comes in, because it predicts the inputs and / or information that the digitally cloned human would provide and provides the appropriate input, which at least partially eliminates the need for the digitally cloned human to actually connect.

[0041] In the metaverse, for example, it is possible for a worker and / or expert to be needed and / or should be working on multiple tasks, requests, and / or "construction sites" simultaneously. The invention is intended to enable the digital personality clone of the human expert to answer questions and provide instructions, instead of communicating with the human expert in real life. The digital personality clone can work, answer, and / or make decisions "1,000 or more" times simultaneously in different parts of the metaverse. This invention is useful, among other things, when an expert is working and / or should be available in different locations and / or 24 / 7 and / or on one or more problems simultaneously. As a digital personality clone, this is because it exists as a data package and is not, for example, in a movie theater.

[0042] For example, a specialist, i.e., an expert on a shop floor, is supposed to guide workers who have a specific technical problem. Previously, workers would call a hotline or other help center and only receive assistance when a member of staff was available. The invention proposes creating digital personality clones of technical specialists to provide assistance. The result would be that experts could offer assistance on a much greater scale via their digital personality clones.

[0043] The situation is similar with the onboarding of a new employee: instead of a real colleague sitting next to the new employee or being connected to them virtually, the digital personality clone can take over parts or all of the onboarding by guiding the new employee through the digitally accessible working environment of the new job.

[0044] Even if an expert is requested from an industrial plant in the metaverse, that expert can only respond once unless a digital personality clone exists. With a digital personality clone, the same expert can offer technical assistance, process consulting support, and / or strategy consulting support anywhere in the world, 24 / 7.

[0045] A further embodiment of the invention would be the creation of basic characters, i.e., digital personalities that are cataloged, for example, in databases and / or a library. This would allow the basic bodies of the digital personality clones to be standardized and accessible worldwide, and the individualization of the digital personality clone could be simplified and / or shortened.

[0046] According to another advantageous embodiment, several experts, connected in the form of their digital personality clones, can be brought together—for example, to solve a technical problem arising in the industrial metaverse. The digital personality clones merge seamlessly because they are digital. On the other hand, instead of merging, one can also compare. Merging results in a decision; with multiple decisions, one can compare, weight, and select.

[0047] According to another embodiment, decisions can be simulated by assigning appropriate tasks to the digital personality clones. This also makes it possible to compare the decisions of various "similar" experts from a digital personality library. This allows the problem to be viewed in a differentiated manner without the human even having to give it a second thought. This differentiation is then available to the human at the same time as the problem, which can significantly improve decision-making.

[0048] In addition to their technical applications, these applications can also be used in administration, among other areas. Particularly in government offices, healthcare, organizational structures, bureaucracy, and general operations, they are incalculable. They can also be used to optimally prepare court decisions, potentially reducing or even completely avoiding errors such as judicial errors.

[0049] Embodiments of the invention

[0050] The system for generating a digital personality clone advantageously comprises several subsystems, each of which has hardware and software elements, as well as corresponding memory and interfaces.

[0051] Advantageously, the system for creating a digital personality clone collects data retrieved by the user whose digital personality clone is to be created via a correspondingly configured and set-up computer-based device with the aid of a computer program product. The user's digital activities are collected, secured, and / or stored by the system for creating a digital personality clone, partially or entirely, for example, sorted by topic, and transmitted to the corresponding CL, which is programmed and suitable for creating a data package as the basis for a digital personality clone.

[0052] The system for generating a digital personality clone therefore advantageously comprises one or more interfaces for the recording device(s) for capturing the digital and / or non-digital, virtual and / or real activities and / or experiences of the user. The analog real experiences, such as a missing screw on an airplane's wing, are converted into data packets that are forwarded to the neural network(s) running the system for generating a digital personality clone.

[0053] Preferably, a recording device is designed as a “wearable electronic”, i.e. as a portable device, in particular one that can be worn on the body, on or at the head and / or on clothing.

[0054] For example, one or more of the recording devices preferably comprise one or more cameras, one or more microphones, and / or fitness monitoring devices, which—again preferably—are worn on the user's body and record their current body markers. The camera is, for example, a normal camera or a 360° camera. The microphone is, for example, a wearable microphone. A wearable recording device is, for example, a smartwatch; all types of fitness trackers such as Fitbit Charge, VR headsets, smart jewelry, internet-enabled glasses, and / or Bluetooth headsets are first and / or second recording devices or parts thereof.

[0055] Furthermore, the system for generating a digital personality clone advantageously comprises one or more interfaces for generating the outputs and displaying the results of the method for generating a digital personality clone.

[0056] The subsystems can be distributed on a physical device, such as a small robot, or on several smaller devices, with different or the same types of interfaces.

[0057] According to an advantageous embodiment of the system for generating a digital personality clone, the device(s) of one or more subsystems are implemented as wearable electronic devices. "Wearable electronic devices" are electronic technology, devices, and sensors that can be comfortably worn on the body and / or are part of clothing. For example, there are motion sensors that can detect and store the user's movements in real time. Various body markers can also be collected and stored in this way, for example, via wristwatches such as fitness watches. According to an advantageous embodiment of the method, the respective subsystems of the system for generating a digital personality clone perform one or more of the following actions:

[0058] In this case, "users" are defined as human users, i.e., persons. a) Data collection:

[0059] First, it is advantageous to collect the digital data from a human user by means of a digital data collection to create their digital personality clones:

[0060] Typically, the system for creating a digital personality clone initially collects data about the user's standard handling of digital devices over a specific period of time and / or in a specific situation – e.g., only at work, especially if the user is willing to have their data collected. The system for creating a digital personality clone collects data about the user's digital activities on their devices, such as computers, smartphones, and / or tablets, as well as the user's capabilities with regard to text in emails and / or chats, websites visited, files accessed, streams watched, and games played. This data is also collected via the devices' built-in microphones and cameras to capture audio and video data.

[0061] The following are data collection tools that may be used all or only a selection of them simultaneously, for example, in one embodiment of the method and / or system for generating a digital personality clone:

[0062] Tools for process step "a)" of collecting data from the user's digital activities and experiences include, for example: a) browser extensions for collecting web browsing data; b) screen capture software for collecting visual data from the user's digital activities. This allows videos of the user's screen to be recorded and / or screenshots to be taken regularly; c) audio / video recording using, for example, built-in microphones and cameras on the user's devices; d) software for data capture and / or data conversion and data forwarding to neural networks; e) software for data forwarding to neural networks; f) software for encryption and secure transmission techniques for encrypted transmission to the central processing server and / or cloud storage.

[0063] The tools a) to f) can, on the one hand, be integrated into the system for generating a digital personality clone via the IoT and / or in the form of a computer program product with program code means which cause a correspondingly configured processor, when the program code means are processed by the processor, to carry out a process step for digital data collection.

[0064] In process step “a)”, data is also collected from the user’s analogue activities and experiences, i.e. the visual and acoustic information that the user perceives:

[0065] For example, the system for creating a digital personality clone can capture the visual and / or acoustic information from a corresponding device, such as a robot, with a camera and microphone over a certain period of time and / or in a certain situation - e.g. only at work when the user is willing to have his data collected.

[0066] For example, the robot (or a series of robots distributed in the room) has a series of sensors that detect the user's posture in order to calculate the user's head pose and / or gaze direction (e.g., a second camera with a computer vision algorithm for pose detection can be used as a sensor).

[0067] For example, according to another advantageous embodiment of the system for generating a digital personality clone, a robot has a movable "head" that can roll to follow the user's line of sight and capture the user's perspective or line of sight and / or observed field of vision. Another camera then acts as a sensor and captures the user's reaction to the events, processes, and / or realities perceived by the user. Capturing the reactions occurs simultaneously and is therefore so insightful regarding the user's "soft skills." For this purpose, microphones can be used, for example, to capture the announcements the user receives from third parties and, at the same time, the user's reaction to these announcements or any announcements from this "third party."

[0068] Once data collection is complete, data storage must be organized: Data can be collected and stored from various devices. At least two options are conceivable: a) All devices upload their collected data once per time unit, for example once a day, to a central storage facility and / or cloud infrastructure. Access to this data is preferably clearly regulated, for example, only possible by the owner. b) All devices keep the collected data decentralized and encrypted. The devices synchronize once a day to distribute copies of new information packets, ensuring that this data remains reliable, permanent, and / or persistent even if some devices are down and / or shut down.According to an advantageous embodiment, the devices and applications on these devices that collect the personal data are certified by the user for permission to collect the data.

[0069] For example, a private blockchain infrastructure is used that connects all participating devices, including personal laptops and / or desktops. Copies of the information blocks are stored on different devices with their IDs. The metadata of these information blocks contains not only their IDs but also additional information such as the source of the copy and the category (primary view of the person vs. external view of the person). All data is preferably stored encrypted so that only authorized users can access it.

[0070] After consolidating the data structure, a corresponding application accesses the consolidated data to use it for feature extraction and / or model training, with which the neural network for the digital personality clone is further developed with the new data.

[0071] Data feature extraction: After merging and consolidating data from multiple channels into a coherent representation, various features are extracted from the combined data. For visual input, object detection and tracking algorithms identify and track objects of interest. For audio input, speech recognition algorithms transcribe spoken words.

[0072] In addition, the system for generating a digital personality clone using AI and appropriate devices with computer vision and NLP techniques recognizes natural language, gestures, emotions and the conversation context (gesture recognition, emotion recognition and / or transformer-based language models to capture the context).

[0073] Computer vision, for example, is a technology that attempts to identify and understand objects and people in images and videos. NLP stands for Natural Language Processing and is an interdisciplinary technological field combining computer science and linguistics.

[0074] The gestures, facial emotions, and verbal context are all linked by comparing their data acquisition timestamps. This is done to connect the user's emotional response to the external signal / information to update the neural network of the user's digital personality clone, allowing it to behave more accurately and more authentically reflect the user's behavior.

[0075] For example, a large dataset combining all integrated data is fed into a large language model so that the system for generating a digital personality clone can build a comprehensive understanding of the user's behavior, preferences, and communication style. According to a particularly advantageous embodiment of the method, incremental learning techniques are used to train the AI ​​model with new data without retraining it from scratch. Behavioral adaptation: The system for generating a digital personality clone will, for example, use reinforcement learning to adapt its behavior based on the user's preferences and reactions in different situations.

[0076] From all collected data and / or input information, e.g., visual information seen by the user, audio heard by the user, and / or requests from the output, the system generates initial outputs to create a digital personality clone, e.g., in the form of text context and the emotional response of the personality clone, and compares these generated outputs with the user's actual response to these inputs or requests. In this way, the digital personality clone is continuously refined during operation.

[0077] Alternatively, the cloning system user can ask the system to process the request itself and, for example, compare the outputs to manually correct them by providing comprehensive feedback on what they don't like as a person and should therefore be changed. To simplify the process, this can also be done simply by using a very short reward / punishment cycle, such as pressing a green / red button or saying a rewarding / punishing keyword.

[0078] By providing personal, "manual" feedback and / or rewards from the user, the AI ​​of the digital personality clone generation system learns to adapt the digital personality clone's actions, such as tone of voice, gestures, or responses, contextually and / or emotionally to better match the user's personality and personal experiences / knowledge. The user will explicitly provide ratings for the digital personality clone's behavior, for example, during the deployment and presentation of the cloning system in a physical environment, and the algorithm, the AI, will attempt to maximize these ratings over the course of its performance.

[0079] Output of the result of the method and the system for generating a digital personality clone: ​​According to an advantageous embodiment, there can be, for example, a robot that carries out the method as an output device for the result of the system for creating the digital personality clone.

[0080] This robot includes, for example, a device for data acquisition, storage, and processing with an update or a generative process for improving the neural network, whereby it generates the output of the digital personality clone upon request. The output can also be visualized on a screen, for example, if the device is equipped with a screen. Any user's PC can be used to generate the output and / or display it visually on its screen if it has been certified by the user or, respectively, the "digital personality clone owner" (in short, "clone owner").

[0081] Alternatively, the results can be generated as audio data and output to any certified audio device. The results can be generated as files in any format and stored on the cloud infrastructure and / or sent via email to the clone owner or another recipient certified by the clone owner to receive the respective output type via email.

[0082] The results are responses to the external signals and / or information input into the digital personality clone generation system. These responses should correspond to the clone owner's response to these inputs. For example, if the digital personality clone generation system recognizes a request for information such as "writing code in a defined programming language," "translating something," "answering a question," "writing a paper," "painting a picture," "singing a song," etc., it generates the corresponding information in visual form (text, image, film) and / or audio format. The clone owner decides when the digital personality clone generation system is activated and when it is muted.

[0083] In the "silent" state, the system for creating a digital personality clone does not generate the output, etc., and / or it generates some results but does not output them externally (e.g., it uses them only for internal training to compare them with the user's actual output and to correct the underlying neural network).

[0084] According to a further advantageous embodiment, the clone owner can test their clone in a game or similar and then, if necessary manually, generate further training data by manually executing how they would behave in the game and thus correcting and / or supplementing their digital personality clone. External use of the outputs: According to one embodiment of the system, the clone owner can decide at any time whether or not to make a virtual interface of their system for generating a digital personality clone available to another user, either temporarily for a specific time slot (e.g., for a conference presentation or consultation session) or for an unlimited period. For this purpose, the system for generating a digital personality clone offers special access, which is also secured and managed and controlled by the clone owner.

[0085] The clone owner selects different areas of use of his clone (e.g. for family members, for a company XXX, for customers, etc.) and / or sets the time limit, volume limits (number of requests or amount of data processed and sent) and the topics / data / experiences that can be disclosed, and special areas of data / experiences that are prohibited for exposure for greater security.

[0086] The clone owner can set the price for the use of his clone per unit of time and / or data volume sent because, according to an advantageous embodiment, the system comprises a device for detecting the amount of data sent.

[0087] Comprehensive data collection: While some existing solutions, such as MyLifeBits®, have experimented with extensive personal data collection, this invention proposes a more integrated approach.

[0088] Not only is data collected from a user's digital interactions, but data from their analog, physical environment and daily activities is also generated and collected using built-in microphones and cameras. Automatic data collection captures two different types of input: (a) the user's personal experience / knowledge; (b) the user's response(s) to the signals and information perceived from the environment. This comprehensive data collection leads to a richer and highly nuanced understanding of the user and their response to external stimuli, with a more precise generation of expected outcomes, including the generation of emotional responses for authentic communication with the digital personality clone.

[0089] A digital personality clone has a whole range of applications.

[0090] Apart from the necessary creation of a digital personality clone for all applications in the metaverse, especially in the industrial metaverse, which is still practically devoid of distinctive personalities, digital personality clones can be used in all kinds of virtual meetings, in many service professions, in education, in culture, and especially in healthcare.

[0091] For example, when it comes to personalized learning and adapting technology to a new user, current AI systems such as Google® Duplex® and Replika.ai® can assume a broad user base or imitate human conversation at a general level, but they do not provide individuals.

[0092] In contrast, a digital personality clone according to an embodiment of the present invention is trained exclusively by a single individual, which allows it to tailor its behavior, reactions, and actions to a specific person.

[0093] The system for generating a digital personality clone, presented here for the first time, learns various perspectives of human behavior from various devices in different formats (visual recognition, audio recognition, emotional recognition, verbal / context understanding). The system for generating a digital personality clone allows for the combination of thorough user feedback with a very fast and simple reward / punishment system (e.g., via red / green button clicks) at any time convenient for the human user. It also allows for the combination of inputs from specialized devices and general-purpose devices such as laptops, webcams, fitness trackers, etc., which are individually certified by the user for the task of data collection, storage, synchronization, and processing.In contrast to other systems where the learning process is carried out centrally on a company premises, for example, according to the present invention all data, models (the neural network) and activities (data processing and system training / fine-tuning) are stored under the full control of the user, stored encrypted on his personal devices and trained only with his personal data and his personal neural network.

[0094] Practical use: Some AI systems, such as Google Duplex®, are designed for specific real-world tasks, but the system described here for generating a digital personality clone could learn from the user's complete and complex behavior and, after training, mimic and replace the user. The digital personality clone could perform various activities that a human might typically perform, such as filling out tax forms, participating in conferences where machine-generated results are presented, and writing specific and personalized statements and / or evaluations.

[0095] Current and existing AI assistants are generally unable to perform this task from a personal perspective. The system described for creating a digital personality clone is particularly useful for experts in high demand, when their specialized personal skills are frequently required, but the person cannot always participate in all requests 24 / 7.

[0096] Other special personal skills, such as emotional support, can also be requested from family members (for example, you could have your parents "on call" in your pocket every time you need them, regardless of their actual location, existence, and / or well-being).

[0097] A digital personality clone may also serve the purpose of continuity and legacy: One of the most striking aspects of this invention is its potential to preserve a user's personality and knowledge even after they are offboarded from the company, die, and / or in the event of cognitive decline, such as Alzheimer's disease, or simply when the person is physically unavailable for other reasons.

[0098] It could provide a form of digital continuity and legacy that current solutions do not offer, allowing loved ones to interact with the digital replica of the estranged user, or colleagues to access the retired user's professional knowledge and skills.

[0099] To reliably store knowledge and prevent data loss, the invention proposes storing copies of various information blocks duplicated on different devices, all of which are synchronized with each other and certified by the clone owner for data usage and processing. The use of multiple devices within the personality clone system with regular data synchronization also provides continuity of data collection throughout the day and across locations. Finally, the digital personality clone can also be used profitably in job shadowing and mentoring: The detailed data collected can be extremely useful in job shadowing or mentoring contexts, saving considerable travel time and working hours with a corresponding reduction in the resulting environmental impact. Job shadowing is a relatively expensive process due to the extensive personnel workload and is also energy-intensive.It requires synchronizing personal time, location, and other preferences between two or more individuals. On the other hand, personal mentoring is highly effective for training new and junior employees (or students, or simply specific community members). It could help successors understand the work style, decision-making process, and expertise, offering insights beyond what current solutions offer, as it can encompass the use of intuition and learning, one of the soft skills that is still very difficult to define, measure, and thus digitize in other ways. One particular aspect of why one-on-one mentoring is so effective is that it involves the emotional interaction between mentor and mentee in highly individualized settings.Our proposed system for creating a digital personality clone can therefore contribute to the supervision and guidance of many trainees by a single expert mentor, namely through his or her digital personality clone, who constantly and continuously conducts "job shadowing" and records all of the expert mentor's experiences during his or her normal work activities—especially regardless of language barriers.

[0100] Such a digital personality clone, which exists in the form of a data packet and is a piece of software, can be easily copied and distributed over the internet for use by multiple trainees at their own location and time. This also means, for example, parallel access to an expert-mentor clone via the internet – access that is managed by the clone owner. The current solution for this is the use of conventional online chatbots. However, these are impersonal and carry general and average knowledge, thus overlooking the personal relationships that make such communication particularly sustainable and effective.

[0101] Regardless of the grammatical gender of a particular term, persons with male, female or other gender identities are included.

[0102] Exemplary embodiments of the drawing Fig. 1 shows an exemplary device for collecting data from the user's analog activities and experiences;

[0103] Fig. 2 shows an exemplary arrangement of several devices for collecting data from the user’s analog activities and experiences;

[0104] Figure 3 shows an exemplary mobile arrangement of several devices for collecting data from the user's analog activities and experiences.

[0105] Detailed description of the implementation examples

[0106] Explanation of figures 1 to 3:

[0107] Figure 1 shows a mobile robot 10 used to collect, process, or forward data. The robot 10 is capable, for example, of observing a person within its range, generating data from it, and either processing this data itself or forwarding it. In addition to the person, the robot 10 also records other data from the environment, such as visual, acoustic, odor, air composition, temperature, wind speed, background noise, pressure, and humidity, for example, using appropriate sensor equipment.

[0108] For this purpose, the robot first has a mobile base 1 and a camera 12 suitable for recording a person's posture and physical reactions. Finally, the robot has a movable, in this case even rollable, head 13, which is equipped with a camera 14 and a microphone 15 for precisely recording and capturing all data of the observed user 17 (Figure 2), as well as what is happening within their field of view and what the user 17 can hear and / or see. In addition, not shown separately, the robot 10 has sensors throughout that generally detect the ambient conditions, pressure, temperature, lighting conditions, wind and / or smell, and generate corresponding data.

[0109] Figure 2 shows a typical use of a robot 10, as shown in Figure 1, wherein the arrows indicate that the lower camera 12 of the robot 10 is directed at the user 17 to record the body posture, movements, and / or gestures, wherein the double arrow means that the camera is directed at the user 17 and simultaneously collects data from the user 17. The upper camera 14 and the microphone 15 of the robot 10 are directed at an output device 16, which represents, for example, a monitor 16 at which the user 17 is currently looking. Via the camera 12, which records the body posture of the user 17, data can be collected about emotional reactions of the user 17 based on the digital experiences that the user 17 has on the screen 16, in connection with these digital experiences recorded via the camera 14.According to the embodiment here, the microphone 15 is directed towards the device 16, but there is of course also a microphone directed towards the camera 12, which observes and records the user 17 (not shown).

[0110] Figure 3 shows a 360° recording of the user 17, who in turn interacts with the device 16. Four robots 10, each of which records the reactions of the user 17 and the environment, as well as the experiences on the device 16, and simultaneously captures the data from both the device 16 and the user 17, thereby generating machine-readable data packets, are - as shown here - interconnected, for example wirelessly, and produce a real-world, all-round recording of the user 17 in his environment during his activity in connection with the device 16. The data from this all-round recording is collected, stored, synchronized and consolidated and used to train the system's AI to generate a digital personality clone.

[0111] The technology disclosed here for the first time makes it possible to generate a digital personality clone of any person who makes themselves available. A digital personality clone is a help in all areas of life and work. Professionally, the digital personality clone can make foreign experts available to the user immediately, within the country, and it can enable participation in meetings 24 / 7, multiple times, and around the world. Furthermore, it can portray empathy and emotions where otherwise only automated responses would occur. Finally, with increasing age, it can virtually provide previous physical, psychological, or other characteristics and cognitive abilities that would otherwise be limited or lost.

Claims

Patent claims 1. A system for generating a digital personality clone, comprising two or more recording devices, one or more computer-aided units that process the data from the recordings into AI-processable data, and one or more interfaces to neural networks for processing the input using artificial intelligence, the latter with or without access to the IoT. It is provided that the at least two recording devices (12, 14, and 15) are arranged with respect to the human user (17) in such a way that a first recording device (14, 15) is arranged to record the cognitively perceived environment of the human user as completely as possible. The recording device, following the user and / or imitating the user, follows the line of sight and records as much as possible of what is perceived and mentally processed by the human user with his or her seven senses, based on the sensors available.generates data therefrom and forwards this data in real time to a neural network with a Kl, and a second recording device (12) is focused on the human user (17) and records the reaction of the human user (17) to what he perceives from his cognitive environment in real time, generates data therefrom and forwards this data - like the first recording device (14, 15) - to the neural network with the Kl, and the processing by the Kl calculates a personality from this data, makes the data packet(s) with the calculated personality available for the creation of an avatar, and as a result of the processing of the data from the at least two recording devices (12, 14 and 15), a data packet results on which a digital personality clone of the human user (17) is based.

2. A system according to claim 1, having a plurality of subsystems, wherein at least one subsystem has one or more recording devices.

3. System according to one of claims 1 or 2, wherein a recording device is designed as a portable device, in particular one that can be worn on the body and / or on the clothing of the user (17).

4. System according to one of the preceding claims, which comprises one or more suitable interfaces for transmitting the data collected by recording devices to a neural network which is configured to run the CL.

5. System according to one of the preceding claims, which has one or more suitable interfaces to corresponding output devices for generating the outputs and / or displaying the results of the method for creating a digital personality clone.

6. System according to one of the preceding claims, wherein one or more physical subsystems are arranged on a small robot.

7. A method for generating a digital personality clone, comprising the following method steps: a) recording and data collection of the cognitively perceived environment as well as the digital and analogue activities and experiences of a human user (17), b) synchronous recording of the human user (17), how he experiences this cognitive environment and actions, how he reacts and / or what he perceives and feels during the digital and analogue activities and / or experiences, c) processing, intermediate storage and forwarding of the data obtained by a) and b) by one or more computer-assisted units, d) generation of one or more results of the computer-assisted processed input by a unit to create a data package representing a digital personality clone,e) Saving the result in the form of data packets and f) optionally refining and supplementing the data packets and / or results of the class processing with new training data from further recordings, recordings and / or inputs by repeating the first two process steps a) and b) and further processing by means of process steps c), d) and e).

8. The method according to claim 7, wherein a process step g) is provided for testing the digital personality clone generated by the process steps a) to f) in a game.

9. The method according to claim 8, wherein a process step h) is provided in which the user (17) manually corrects reactions and answers of his own digital personality clone.

10. Method according to claim 8 or 9, wherein a process step i) is provided in which the user (17) manually adds reactions and answers of his own digital personality clone.

11. Method according to one of claims 7 to 10, wherein in process step a) data from digital activities of the user are collected.

12. Method according to one of claims 7 to 11, wherein several subsystems (10) with several recording devices (12, 14 and 15) are used in combination for data collection.

13. A computer program product comprising program code means which cause a correspondingly configured processor to carry out a method according to any one of the preceding claims when the program code means are executed by the processor.

14. A computer-readable storage medium comprising a computer program product according to claim 13.

15. Use of the digital personality clone in human-machine communication.

Images