System and method for cloud-based assisted communication

Abstract

The invention discloses a system and method for cloud-based assisted communication, wherein the system (100) comprises: an electronic device having a processor and a non-transitory media (118); a cloud-based computer server includes a processor, a non-transitory media (118) for a cloud-based assisted communication (AAC) software program, a management control panel module (114), and a machine learning module (108), where the cloud-based AAC software program interacts with the non-transitory media, the management control panel module, and the machine learning module, a cloud-based AAC software program recommends a graphic card to create sentences and manage content, where the cloud-based AAC software program includes instructions for: user authentication; performing data retrieval; a machine learning module is used for recommending composition sentences of the graph card, and selection can be carried out according to sentence contexts; carrying out data serialization and anti-serialization; managing data; translating single characters and sentences; and a man-machine interaction interface (102) for providing optimal user interaction between the cloud-based AAC software programs.

Description

Technical Field

[0001] The present invention relates to an intelligent system and method for improving speech and language impaired individuals so that they can communicate effectively with others. In particular, the present invention relates to an assistive communication (AAC) device with artificial intelligence for personalized recommendation of AAC symbols and related multimedia content such as images and sounds. Background Art

[0002] AAC refers to tools and technology used to help individuals who have difficulty communicating using traditional speech and language. AAC can come in many forms, including devices that allow individuals to type or select words or phrases, communication boards with pictures or symbols, or mobile apps for symbolic exchange. The goal of AAC is to help people enhance communication and express themselves in a comfortable and effective way. AAC is often used by people with disabilities such as cerebral palsy, autism, or speech disorders, but it can also help individuals who have temporary communication difficulties due to injury or illness.

[0003] Symbol selection speed can be an issue for some AAC users. The time it takes to locate and select a symbol slows down the communication process for users, which can be frustrating and can limit their ability to communicate effectively in certain situations. Machine learning algorithms can be used to predict and recommend AAC symbols to increase the speed at which users can select symbols. However, this remains challenging for a variety of reasons. These include the fact that machine learning algorithms require large amounts of AAC communication data to learn from, the lack of pre-trained models in real AAC daily use environments, and the variability in user needs that makes representation of machine learning models important.

[0004] Take Patent No. 2020215128A1 issued by the World Intellectual Property Organization as an example, which is specifically aimed at providing assistance in reading books using AAC devices. It uses symbols to represent words and phrases in the book, and has components such as a user interface module, a symbol display controller, and a vocabulary database. The system provides configurability and may include voice and pronunciation data. It is designed to improve communication and language skills through symbol-based interaction. Although the system has a vocabulary database that stores book data with symbols and grows over time, due to the large number of words in the database, it may take a while for the user to find the correct symbol for the correct word, and the database cannot be accessed without an Internet connection.

[0005] Korean Patent No. 20230064143A discloses another technology with a machine learning implementation device, which uses a machine learning model to receive input words from the user, predict the next word, and then display the predicted word, and distinguish and display basic words from the default category. The technology also allows users to choose whether to display symbols with text and provides voice output options. The technology includes translation functions for English words and symbols. Although the technology has word prediction capabilities, its user interface is not suitable for other AAC devices other than computers, reducing the flexibility of this technology.

[0006] In addition to the above, U.S. Patent No. 2012330669A1 and Korean Patent No. 102309505B1 also implement machine learning models in their technologies, however, the models are trained for different purposes. The former automatically classifies text in the form of ontology and controls the operation of the display module, while the latter displays the text information data of the display module on the display screen separately. Since the machine learning models in these technologies have not been trained to predict symbols, users may expect that the speed of selecting appropriate symbols to convey their messages will be slower.

[0007] In light of the foregoing, there is an unmet need for systems and methods for implementing intelligent AAC symbol and content recommendations. Meeting these challenges requires innovative research and development of machine learning algorithms for AAC devices and improving the effectiveness and usability of machine learning-based AAC systems. The application of machine learning in AAC has the potential to significantly improve the lives of individuals with communication difficulties by providing more personalized and effective communication tools. Summary of the invention

[0008] An object of the present invention is to provide a centralized system and method for managing and analyzing data with a sharing function to facilitate access by multiple devices and / or administrators.

[0009] The present invention also aims to provide a machine learning module to quickly locate and select symbols, thereby providing word and content recommendations to achieve fast and effective communication.

[0010] Another object of the present invention is to provide a user-friendly interface suitable for use on a variety of devices.

[0011] Therefore, these purposes can be achieved by following the teachings of the present invention. The present invention relates to a cloud-based assistive communication (AAC) system including: an electronic device having a processor and non-transitory media; a cloud-based computer server including a processor, non-transitory media for a cloud-based AAC software program, a management control panel module, and a machine learning module, wherein the cloud-based AAC software program interacts with the non-transitory media, the management control panel module, and the machine learning module to recommend picture cards to create sentences and manage content, wherein the cloud-based AAC software program includes instructions for: user authentication; data retrieval; using sentences composed of picture cards recommended by the machine learning module, which users can select based on the context of the sentence; data serialization and deserialization; data management; word and sentence translation; and a human-computer interaction interface for providing optimal user interaction between cloud-based AAC software programs.

[0012] In addition, the present invention also relates to a cloud-based assistive communication (AAC) system, comprising: a cloud server receives a login request from a user using an electronic device through a human-computer interaction interface, wherein the login request includes user authentication information; authenticates the user based on the received user authentication information, and retrieves user-specific data, including user information and AAC content, including but not limited to preset card decks, in response to the login request; translates words associated with the card deck into appropriate picture cards, and generates input suitable for the user's electronic device, which combines the language, visual display and size specific to the device; transmits output as a response to the user's login request; allows the user to compose a sentence in a software program, wherein the user selects a card deck and selects a card deck from the selected card deck The method comprises the steps of: selecting a picture card and declaring that the sentence composition is completed; recording the assembled sentence in a non-transitory medium of a server, wherein the recorded data includes picture card information, command sequence and input timestamp; serializing and deserializing the assembled sentence into single words through an endpoint differentiation adapter (EPDA) and a presentation translator, using additional input information such as device type, input method and user group (if any); transmitting the serialized sentence to a machine learning module as feedback, generating artificial intelligence (AI) suggestions and updating the machine learning matrix based on the feedback; displaying the deserialized sentence back to the human-computer interaction interface at the same time with the help of the EPDA and UI / UX operation modules, so as to be displayed in the user device; and updating the suggestions for future user requests.

[0013] The machine learning module will be an independent AI learning instance. The system will provide AAC content and sentence records for training. The system will then (upon user request) request the module to provide AI suggestions or other responses based on the request. For simplicity and better performance, some related data / calculations can be stored in the system's database, including node-to-node counts and some preference parameters, and establish a machine learning matrix as a database model.

[0014] Another teaching of the present invention relates to a method for a cloud-based assistive communication (AAC) system, wherein translating words associated with a card set into appropriate picture cards further includes the step of introducing a new picture card into the appropriate picture card, the method including: determining whether the new picture card can be divided into at least one word, phrase or part with semantic meaning; searching for existing basic words in the system across non-transitory media, wherein if the basic word carries the semantics of the new picture card, the new picture card can be represented as a new representation of the basic word, wherein if no basic word matches the semantics of the new picture card, the new picture card becomes a new basic word; verifying whether any complex word contains a separate word or part with the same semantics and structure as the new basic word; and recording the new basic word as a new picture card in the non-transitory media of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The features of the present invention will be more readily understood and appreciated when the following detailed description is read in conjunction with the accompanying drawings of preferred embodiments of the present invention, in which:

[0016] Figure 1 An overview of augmentative and alternative communications (AAC) in the cloud is shown;

[0017] Figure 2 The management control panel module interface is shown;

[0018] Figure 3 An analysis of the processes occurring within the module is shown;

[0019] Figure 4 The sentence analysis process is shown;

[0020] Figure 5 The deck editing module and its functions are shown;

[0021] Figure 6 An overview of the interaction method between the user and the server is shown;

[0022] Figure 7 shows the process of adding a new card; and

[0023] Figure 8 Common user interface representations in different types of electronic devices are shown. DETAILED DESCRIPTION

[0024] As required, specific embodiments of the present invention are disclosed herein. However, it should be understood that the disclosed embodiments are only examples of the present invention, which can be implemented in a variety of different forms. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limitations, but only as the basis of the claims. It should be understood that the drawings and their detailed description are not used to limit the present invention to the specific forms disclosed herein. On the contrary, the present invention covers all modifications, equivalents and alternatives that fall within the scope defined by the claims. As used throughout this application, the word "may" means optional (i.e., it means possible), rather than mandatory (i.e., it means must). Similarly, the words "include" and "comprise" are meant to include but are not limited to. In addition, unless otherwise mentioned, the word "one" means "at least one" and the word "multiple" means one or more. When using abbreviations or technical terms, these refer to the generally accepted meanings known in the art.

[0025] The present invention teaches a system for cloud-based augmentative and alternative communication (AAC), comprising: an electronic device having a processor and non-transient media (118); a cloud-based computer server, comprising a processor, non-transient media (118) for a cloud-based AAC software program, a management control panel module (114), and a machine learning module (108), wherein the cloud-based AAC software program interacts with the non-transient media (118), the management control panel module (114), and the machine learning module 108 to recommend picture cards to create sentences and manage content, wherein the cloud-based AAC software program includes instructions for: user authentication; data retrieval; using sentence components recommended by the machine learning module (108), which the user can select based on the context of the sentence; data serialization and deserialization; data management; word and sentence translation; and a human-computer interaction interface (102) for providing optimal user interaction between cloud-based AAC software programs. Overview of the present invention Figure 1 shown.

[0026] Alternatively, non-transitory media 118 may be used interchangeably with the term "database."

[0027] refer to Figure 1 , the user layer provides individuals with a seamless and intuitive experience interacting with these devices, allowing for a wide range of communication and learning activities. Overall, the present invention provides individuals with a powerful and flexible tool to access and interact with human-machine interfaces in a variety of settings and environments. Typically, the possession or use of a device will be personal, and the data / information / history will also be private, so user authentication is required.

[0028] In a preferred embodiment of the present invention, the cloud-based AAC software program is hosted in an application ("App") module. More specifically, the present invention can be implemented in various software applications, systems, and platforms, and can include features such as voice recognition, gesture control, touch interface, and other technologies to enhance the user experience.

[0029] In a preferred embodiment of the present invention, the non-transient media (118) for the cloud-based AAC software program records multiple sets of data, the data sets are selected from the group including: AAC content (118b) records symbols and multimedia content in the form of picture cards, the multimedia content includes but is not limited to images and sounds; a card set (118d), wherein the AAC picture cards are grouped and classified according to appearance and order; user information (118a) including but not limited to preference attributes, wherein the preference attributes include but are not limited to groups, organizations and favorable cards; composed sentences (118c); and a machine learning matrix (118e) established based on node-to-node counts and at least one preferred parameter. The machine learning module (108) analyzes various data sources, including the user's communication history and communication preferences, to generate real-time recommendations of symbols and content that the user or user group wants to use in their conversations. This provides people with communication difficulties with an easier and more effective way to express themselves, thereby leading to better social and emotional outcomes.

[0030] In a preferred embodiment of the present invention, the non-transitory media (118) of the software program is a database that records and stores basic ("raw") data of application operations and calculations of machine learning algorithms, wherein each data entry in the non-transitory media 118 or the database of the cloud-based AAC software program has a unique identifier for data exchange within the software program.

[0031] In addition to the above, the AAC content (118b) will record the media format of the AAC tool that the user uses / will use. The typical unit format is the AAC card. For picture cards, the name / meaning and image of the picture card will be recorded. Optional information, such as the voice archive when reading and the source / producer of the picture card will be stored. On the other hand, since these picture cards may have similar or even identical semantics, the Word model is introduced. For a single word, it is regarded as a singleton carrying semantics within the system. It will have and only have one category attribute. Picture cards with semantic meaning will be associated with single words as one of multiple representation forms.

[0032] Categories can also be separate models to provide a more organized data structure. Sentences or picture card sequences written by the user will be stored. The original picture card sequence, the original input, including the command input timestamp, and the translated word sequence will be stored. For certain operational purposes, the pre-selected cards will be grouped and sorted according to their design to meet the user's communication needs. The appearance and order of the picture cards are recorded as card groups (118d). The card groups (118d) should be edited in the designated card group editing module (116) to facilitate management and easier use for users. Most of the above records can then be analyzed.

[0033] In a preferred embodiment of the present invention, the sentence structure in the software program includes: basic words, where each picture card is linked to a specific basic word, but a basic word can be represented by multiple picture cards; phrases that are semantically segmented into a sequence consisting only of recorded basic words, and the phrases are completely interchangeable with the corresponding basic word sequences without changing the meaning or sentence structure; complex words that are similar to phrases, but at least one separate part, word or phrase is not present or is not recorded, which may be due to a lack of representation of the separate part or the lack of semantics of the separate part itself, where the complex words are treated as basic words for analysis purposes; complements, where two basic words have opposite meanings and the same position in the sentence structure, so that exchanging complements can reverse the meaning of the sentence while retaining the sentence structure.

[0034] In a preferred embodiment of the present invention, the cloud-based AAC software program allows users or administrators to create groups to facilitate group communication among multiple users.

[0035] In a preferred embodiment of the present invention, the cloud-based AAC software program further includes: an adapter configured to serialize input and deserialize output into an adaptive format, wherein the adapter is an endpoint distinguishing adapter (EPDA) (106); a representation translator (110) configured to translate one or more representations of symbols and multimedia content in a card set (118d) into their corresponding meanings to construct a sentence and communicate the sentence to an electronic device; and an analysis module (112) configured to analyze recorded data, including but not limited to sentences created and transmitted between users or groups, cards and card sets (118d) recorded in a non-transient medium (118). While the EPDA (106) will serialize the data format, the representation translator (110) will specifically handle the translation of words and their representations, which will be introduced later in the database entry description. When using the translator, different inputs, such as different images of the same object, will be translated into the same (with multiple frequencies) input, while in terms of output, such as recommendations from a machine learning module, will be translated into representations that are consistent with the user, such as different users' favorite images of the same object.

[0036] In a preferred embodiment of the present invention, the EPDA (106) further includes an assistive feature that enables integration of the cloud-based AAC software program with an existing AAC software program.

[0037] In a preferred embodiment of the present invention, the human-computer interaction interface (102) receives and executes instructions, including: connecting an administrator or user's electronic device to a cloud-based server, including accessing a user authentication page; displaying visual effects on the electronic device based on deserialized data through a UI / UX operation module (104); and managing one or more sets of data recorded in a non-transient media (118) via a management control panel module (114). More specifically, the present invention relates to a user layer that enables individuals such as "students" to interact with a system through various human-computer interfaces. Administrators (administrators, such as "teachers") can also access the same system through the human-computer interaction interface (102).

[0038] In a preferred embodiment of the present invention, the UI / UX operation module (104) further includes auxiliary features, enabling adaptive display of user interfaces and user experiences in multiple devices, wherein the display includes design switching and image size to fit the user's electronic device, including but not limited to web design or mobile applications. More specifically, the personalization function introduces changes that need to be adopted, user interface (UI) and user experience (UX). The UI / UX operation module (104) will control UI and UX features according to the user's preferences, for example, switching to different designs to fit the user's electronic device, including web design for desktop computers and laptops and mobile applications for mobile devices.

[0039] In a preferred embodiment of the present invention, the administrator has access to the management control panel module (114), and the management control panel module (114) receives basic commands to create, read, update and delete CRUD or view data of any record, or basic CRUD instructions are displayed in the form of buttons or sequences.

[0040] The management control panel module (114) will collect and organize relevant information and execute most commands, such as CRUD, for administrators to view and manipulate data and records. Administrators should not be limited to having database knowledge, so the module should provide adaptation to human-computer interaction interface (102), such as web design or mobile application, to allow non-technical administrator users to control data without accessing database and other modules.

[0041] In a preferred embodiment of the present invention, the management control panel module (114) has limited access to the analysis module (112), wherein the access allowed is for viewing only and basic CRUD commands are not allowed.

[0042] More specifically, the model analysis can be provided by the sentence records in the analysis module (112). The data will be displayed in a page / section that should be able to be directed from a CRUD page of a record of a certain following model.

[0043] The function of the analysis module (112) is Figure 3 and further described below:

[0044] (a) User: All sentences uttered by users are obtained from the database. The cards and card groups (also called users) used in these sentences can be counted. Taking into account the selection time of each card, the average selection time of users can be estimated.

[0045] (b) Picture Cards: All sentences that include picture cards are retrieved from the database. From this, all users who have written sentences can be collected. The collected analytics include the total number of times a picture card is used, the ranking of all picture cards, the most frequently used connections / previous and next picture cards, the user ranking by picture card usage, and the average selection time when selecting a picture card.

[0046] (c) Card Group: All sentences including the card group will be obtained from the database, and the collection of cards and related users is similar. The total number of times used by users and the usage of cards in the card group can be collected. In addition, if the card group is set as a preset or / and prefix card group, all users and groups can be checked for counting.

[0047] (d) Group: All users in the group and the sentences composed by these users are obtained from the database. Similarly, the usage of picture cards and card decks by users in the group can be measured, as well as the activity and interest of users in making sentences using the card decks available to the group.

[0048] The system also provides sentence analysis and this Figure 4 The sentence analysis should include the following:

[0049] a) Basic information about the sentence. This includes the user name of the sentence selector / speaker / creator, the card group used, and the date and time the sentence was produced. Annotations can be made to clearly indicate whether the card group used is a prefix from the user / user group or general card group usage. User name masking can also be introduced if user privacy is a concern.

[0050] b) Time tracking information for sentences. Display the total time of sentence construction, the time of each command / input, and the time of picking each card. Since users can delete / deselect selected cards from unfinished sentences, while the command / input time is recorded and displayed normally, the time of picking cards afterwards should be adjusted. That is, deleted / unselected content should be considered "disappeared". Therefore, subsequent cards should treat the deletion command / input time as the starting time in the input time calculation, and this adjustment can occur multiple times if multiple deletion commands / inputs are valid.

[0051] c) Sentence analysis should be listed. Analysis can include but is not limited to sentence cards and total command / input length and average time per command / input and card. Estimated thinking time for starting sentence and confirmation time can also be included. Time can be calculated by recording the time between the first selected picture card, the system pre-defined starting point and the last selected picture card, and finally the send / end command / input.

[0052] In a preferred embodiment of the present invention, the management control panel module (114) has a graphical user interface (GUI), including: a data structure of multiple records arranged in sequence, including but not limited to lists, tables, buttons and selection boxes, optionally with paging, wherein each data structure is associated with a specific data type, including relevant descriptions and attributes of the data type, and the data of multiple records can be selected to execute at least one basic CRUD command or view when reset to an auxiliary interface associated with the selected record data; and a search interface, configured to facilitate record selection when implementing paging. Figure 2 An overview of the user interfaces available in the present invention is presented in .

[0053] Administrators can view, manage and edit all types of data stored in the database. The data structure and related descriptions of each type of data are as follows:

[0054] (a) Basic / General: For each data type, the management control panel module (114) should provide basic CRUD. The console should provide a link / redirection to an editor and will have an interface, typically a list or table, which will display all records, with or without paging. The administrator can view basic information for any existing record and edit (if applicable) any selected record, typically by redirecting to another page / section / interactive display. If paging is used, meaning that not all records are displayed / selectable without any further user action / command, a search mechanism may be included. When a new record is to be created, the functions and inputs required / able to be provided should be similar to those when an existing record is to be edited for management / editing purposes. Within the page / section / interaction for a particular record, the control panel should provide buttons / interfaces / command sequences to create, read, update and delete any record, including but not limited to lists, tables, buttons and selection boxes. For ads, the management control panel module (114) should provide a link / redirection to the corresponding data analysis page. Additional notes / descriptions for important practices will be mentioned accordingly.

[0055] (b) Category: For a category, all properties will be included for viewing and editing, including code / key and name / title. Please note that since the code / key is used as a key column for other database entries (usually for cards and words), if administrators are allowed to edit the code in the management control panel module (114), the handling of data changes must be carefully considered, as administrators may not be familiar with the data structure of the database / entire system. Setting cascade updates and deletes on related fields, performing automatic query searches and updates on related records (cards and sentences), and prohibiting editing on the management control panel module (114) in the first place are some example measures.

[0056] (c) Cards and words: For any word, all its card representations will be displayed. For example, the console should provide a direct link to redirect the administrator to the corresponding card's information page. On the other hand, a card can always refer to its word, also known as semantic origin. For example, this link will be used for / as a representation translation and will be referenced by machine learning algorithms.

[0057] (d) Decks: Although a record of a deck is kept, including the cards in the deck, the order / sequence, and other settings, the record should be implemented and displayed in an intuitive way, so typical interfaces such as text boxes and lists may not be appropriate.

[0058] (e) User: The groups to which the user belongs can be displayed, and links to corresponding information pages can be provided. Please note that the operation of joining / leaving a group should be performed in the group page, so the user page does not necessarily provide the function of joining / leaving a group. On the other hand, the user page can provide, for example, radio buttons to configure the user default and prefix card sets, which can usually also be set by the user himself in the user interface (also called application).

[0059] (f) Groups: There should be an interface where, for example, the administrator can add / remove users from a group with a multiple select list. Prefixes and default decks should be editable on this page with radio buttons and drop-down lists.

[0060] (g) Sentence: This section should generally only provide viewing capabilities in the control panel. Since any record is created purely by user input, administrators creating, editing, and deleting records in the console is generally harmful to data research, and no administrator should handle any database-level modifications for any unusual events in the console.

[0061] (h) Analysis: The console should provide links / redirections to different types of analysis, including but not limited to users, cards, decks (118d), and groups. Again, this data can only be viewed, and the console does not have create / edit / delete functions. The details of the page view will be introduced in the following section.

[0062] (i) Administrators: An administrator with higher permissions should be able to create and edit another user with lower permissions. Administrators with lower access levels should be prohibited from performing CRUD of the management model.

[0063] In a preferred embodiment of the present invention, basic CRUD commands of the recorded data can only be executed via an editor, wherein the recorded data is a deck and the editor is a deck editing module (116).

[0064] In a preferred embodiment of the present invention, the card set can be selected from a group of card sets (118d) used for sentence formation.

[0065] In a preferred embodiment of the present invention, the deck editing module (116) further includes: an interface for adding cards to a deck, incorporating user-friendly features such as multiple-select lists and searches to select and include previously unselected cards in a deck; an interface for removing cards from a deck, providing an intuitive method, including multiple-select lists and searches, to enable users to deselect and remove cards from a deck; a sorting mechanism for arranging selected cards within a deck, providing an interactive interface, such as swap buttons and drag-and-drop functionality, to facilitate customizing the order of cards within a deck; user-configurable card category display options, allowing users to select whether to display selected cards; The card group includes a user interface that displays the category associated with each card and adapts to various applications and display modes through presentation methods such as check boxes or radio buttons; user access management functions, including an interface with interactive components such as lists, search bars, and multiple selections, enabling users to define which users and groups can access the card group, while also providing the ability to exclude specific users within the defined criteria; fields and input interfaces for editing card group properties, including the card group name and other necessary records, such as generating and displaying QR codes to facilitate access to the card group; and card group management functions, including operations such as card group deletion and copying, as well as additional functions customized to meet specific application or system requirements. Figure 5 This further illustrates the point.

[0066] In addition to the above, a deck is a collection / combination of cards, usually pre-selected by the deck editing module (116), which will be presented / displayed / selectable for specific situations involving certain users, groups or locations. There are no restrictions / requirements on decks. The deck editing module (116) handles the database / matrix records of all decks (118d).

[0067] In a preferred embodiment of the present invention, basic CRUD commands in at least one of the recorded data are executable for use by a representation translator (110) and a machine learning module (108), wherein the recorded data is a picture card or a single word.

[0068] In a preferred embodiment of the present invention, the human-computer interaction interface (102) can be configured to include a text-to-speech function to perform reading of sentences for voice communication.

[0069] In a preferred embodiment of the present invention, the electronic device includes but is not limited to a dedicated AAC device, a personal computer, a tablet computer, a mobile phone, etc.

[0070] The present invention discloses a cloud-based assistive communication AAC system (602), comprising: a cloud server receiving a login request from a user using an electronic device through a human-computer interaction interface (102), wherein the login request includes user authentication information; authenticating the user based on the received user authentication information, and retrieving user-specific data, including user information (118a) and AAC content (118b), including but not limited to a preset card group, in response to the login request; translating words associated with the card group into appropriate picture cards, and generating input suitable for the user's electronic device, which combines the language, visual display and size specific to the device; transmitting output as a response to the user's login request; allowing the user to compose a sentence in a software program, wherein the user selects a card group, selects a picture card from the selected card group, and declares to complete the sentence compose; record the assembled sentence in a non-transient medium (118) on the server, wherein the recorded data includes picture card information, command sequence and input timestamp; serialize and deserialize the assembled sentence into single words through an endpoint differentiation adapter (EPDA) (106) and a presentation translator (110) using additional input information such as device type, input method and user group (if any); transmit the serialized sentence to a machine learning module (108) as feedback, generate artificial intelligence (AI) suggestions and update the machine learning matrix (118e) based on the feedback; display the deserialized sentence back to the human-computer interaction interface (102) for display in the user device under the facilitation of the EPDA (106) and the UI / UX operation module (104); and update the suggestions for future user requests. More specifically, the EPDA (106) acts as a gateway between the user layer and the system. The user's request to the server will be passed to the EPDA (106) along with the user's authentication and preferences. EPDA will read the data details of the payload / input, such as different attributes and corresponding formats, which vary depending on the type of device the user is using. It normalizes / serializes the input format. Similarly, the output of the server will also be deserialized for the receiving end to adopt. For example, when a user uses different devices to execute similar instructions, he / she will get similar but different displays and responses. EPDA will transform the input and output and send the payload and instructions to the UI / UX operation module (104). This is in Figure 6 , where the flow from the user level to the server level is summarized accordingly.

[0071] The general flow of sentence composition by users is further summarized below.

[0072] The user logs into the application using a device such as a PC or tablet. When the server receives the request and approves the authentication, the system retrieves the data to be sent back to the user, including user information (118a) and default card sets. For these data, such as card sets, the system first retrieves the required words and translates them into appropriate representations or image cards. The translated data is then placed in a payload suitable for the user device that issued the login request in order to provide the appropriate language, UI components, image size, etc.

[0073] Once the payload is complete, it is delivered to the user's device as a response to the login request. After a successful login, the user can begin making sentences on a certain page (usually the homepage or home page). The following steps will be performed:

[0074] i. The user selects the deck to use. If a preset deck is set, the default deck is selected and displayed first without the user having to select one.

[0075] ii. The cards in the deck will be displayed for the user to select. The user can select and deselect the cards to be added / removed from the sentence. Usually, this process should be done in the same deck to obtain a more structured data record, but multiple decks can be used depending on the purpose and design of the application. There should be a button / interface / input method to declare the end of sentence generation. After the input is accepted, the sentence will be confirmed to be generated.

[0076] iii. An interface / display should be presented to the user to notify the confirmation of sentence formation. Since the common display method is a row of picture cards with name / text meaning and images showing the sentence formation picture cards, the display can usually be used for other purposes along with the display of sentences / picture cards. Other functions such as text-to-speech functions can also be included to perform reading sentences for voice communication.

[0077] iv. The composed sentence will be sent to the server and stored in the system database. The user interface will serialize the data, including the picture card information, command sequence and command input time, and send the organized payload to the server.

[0078] When the server receives the payload, such as a POST request sent over the Internet, it performs the following procedures:

[0079] i. The received payload will be processed accordingly, for example, device type and user related information will be processed. Most of the raw data will be stored as history.

[0080] ii. For a sentence, after storing the actual sentence history consisting of the user, card group, and card, the card-to-word translation will be performed. This will require additional information previously in the payload, including but not limited to device type, input method, and user group, to identify and clarify which words or semantics are actually involved in the composition of the sentence represented by the card.

[0081] iii. As the sentences are normalized into words, the records will be sent / passed to the machine learning module 108. As feedback is received, the machine learning matrix (118e) will be updated.

[0082] iv. As the machine learning matrix (118e) is updated, the recommendations provided to future user requests / uses will be updated. At the same time, the machine learning module (108) will also be updated as it "learns" from the standardized sentence input.

[0083] In addition to the above, the user information (118a) includes but is not limited to preference attributes, wherein the preference attributes include but are not limited to groups, organizations, and favorable cards.

[0084] In a preferred embodiment of the present invention, when writing a sentence structure, the software program includes the following steps: linking specific basic words to their corresponding picture cards; splitting semantic phrases into a series of basic words; identifying complex words in a sentence and linking the complex words to their respective basic words; and exchanging complements to reverse the meaning of the sentence while retaining the sentence structure.

[0085] In a preferred embodiment of the present invention, in which a software program is used to process input and output, the processing includes the following steps: serializing the input using an adapter (106); translating one or more representations of the AAC content (118b) in a card set (118d); deserializing the output into an adaptive format and communicating the output to a user's electronic device; and analyzing the output and storing it in a non-transient medium (118).

[0086] In a preferred embodiment of the present invention, wherein the machine learning module (108) receives serialized sentences and generates artificial intelligence (AI) recommendations, the method includes: providing an independent AI learning instance configured to learn and adapt AAC content (118b) and sentence records for training; storing relevant data and calculations in a non-transient medium (118); establishing a machine learning matrix (118e) in the non-transient medium (118), wherein the machine learning matrix (118e) is used to store and organize data associated with the training of the machine learning module (108); receiving a request from a user initiated by a software program to generate AI recommendations using the machine learning module (108); analyzing the user's request to determine the appropriate context and scope for generating AI suggestions; and processing the request using the machine learning module (108), performing data retrieval and analysis using the machine learning matrix (118e), and applying machine learning techniques based on the learning patterns of the AAC content (118b) and sentence records to generate AI recommendations.

[0087] Figure 7 A UI for multiple devices is shown, and any representation of any device / display should include the following common features / interfaces / modules / components, including (1) buttons and / or other interactive components that allow a user to perform card selection, cancel the selection of a selected card, and confirm the selected card sequence as a sentence or record, (2) a method for displaying the selected card, the card to be canceled, and the confirmed card sequence, and (3) an interactive component for switching the card group (118d) and displaying the current display adapter group (118d).

[0088] The interface pattern / display / design containing the above mandatory components shall be, but not limited to, the following types:

[0089] (a) For desktop computers / laptops / TVs, the browser / application / user interface should be at least (640) pixels wide in landscape orientation and (960) pixels high in portrait orientation. For a single application supporting different modes / displays, this mode should include the most and all (where possible) and applicable user interactive components, including but not limited to a home button to return to the default / card selection home page and lock / unlock functionality to disable and unlock / enable card selection and / or card selection page swapping. Configuration / user information / settings pages should be accessible from buttons / links / interactive components and should include appropriate configuration modifiers, meaning that selection lists, checkboxes, radio buttons, text boxes and other appropriate input methods / components will appear accordingly.

[0090] (b) For eye tracking applications, which are designed to be able to accept eye tracking / eye gaze tracking as input and are therefore generally not limited to supporting eye tracking devices and / or cameras, functional buttons / interactive components should be relatively large in aspect ratio. Components should be adjusted to accommodate input from eye tracking devices, for example, treating eye gaze input within a small range as a "hold" input, to accommodate normal movement of the human eye / head and other appropriate rules / exceptions / accommodations. To avoid making it difficult / unclear to detect / receive input from the eye tracking system, designs should not include input receiving buttons / components near the borders, i.e. the sides of the screen / display area. Any input component should be at least (50) pixels with a border.

[0091] (c) For touch-enabled devices, including but not limited to smartphones, tablets, laptops, and smart TVs, adjustments / adaptations should be made for touch input, especially for AAC users who often cannot control their hands / physical input assistive devices. For example, there should be a mechanism to prevent / allow / translate multiple rapid clicks / taps in a small area due to uncontrolled violent hand movements. Another example is to accept / reinterpret the rules / algorithms for holding / relatively long touch screen without lifting fingers / physical touch devices for users who are unable to lift their hands, limbs, or related body parts.

[0092] In addition, the present invention teaches a method (802), wherein translating words associated with a card set into a suitable picture card further includes the step of introducing a new picture card into the suitable picture card, the method including: determining whether the new picture card can be divided into at least one word, phrase or part with semantic meaning; searching for existing basic words in the system across non-transient media (118), wherein if the basic word carries the semantic meaning of the new picture card, the new picture card can be represented as a new representation of the basic word, wherein if no basic word matches the semantic meaning of the new picture card, the new picture card becomes a new basic word; verifying whether any complex word contains a separate word or part with the same semantic meaning and structure as the new basic word; and recording the new basic word as a new picture card in the non-transient media (118) of the system. Figure 8 This further illustrates the point.

[0093] The above explanation of the present invention is not limited to the aforementioned embodiments and drawings, and it is obvious to those skilled in the art that various substitutions, modifications and changes may be made without departing from the scope of the present invention.

Claims

1. A cloud-based assistive communication (AAC) system (100), comprising: An electronic device having a processor and non-transitory media (118); A cloud-based computer server comprising a processor, a non-transient medium (118) for a cloud-based AAC software program, a management control panel module (114), and a machine learning module (108), wherein the cloud-based AAC software program interacts with the non-transient medium (118), the management control panel module (114), and the machine learning module (108) to recommend picture cards to create sentences and manage content, wherein The cloud-based AAC software program includes instructions for: User authentication; Data retrieval; Using the sentence components of the picture cards recommended by the machine learning module (108), the user can select according to the context of the sentence; Data serialization and deserialization; Data management; Word and sentence translation; and A human-computer interaction interface (102) is used to provide optimal user interaction between cloud-based AAC software programs.

2. The system of claim 1, wherein the cloud-based AAC software program (100) is hosted in an application ("App") module.

3. The system of claim 1, wherein the non-transitory media (118) for the cloud-based AAC software program records a plurality of sets of data, the sets of data being selected from the group consisting of: The AAC content (118b) records symbols and multimedia content in the form of picture cards, and the multimedia content includes but is not limited to images and sounds; A card set (118d) in which AAC picture cards are grouped and categorized according to appearance and order; User information (118a) includes but is not limited to preference attributes, wherein the preference attributes include but are not limited to groups, organizations, and favorable cards; to form sentences (118c); and A machine learning matrix (118e) is established based on the node-to-node counts and at least one preferred parameter.

4. The system of claim 1 or 2, wherein each data entry in the non-transitory media (118) of the cloud-based AAC software program has a unique identifier for data exchange within the software program.

5. The system of claim 1, wherein the sentence structure in the software program comprises: Base words, where each picture card is linked to a specific base word, but a base word can be represented by multiple picture cards; Phrases that are semantically segmented into sequences consisting only of recorded basic words and that are completely interchangeable with the corresponding sequences of basic words without changing the meaning or sentence structure; Complex words that are similar to phrases, but where at least one disjunct part, word, or phrase is absent or not recorded, either due to a lack of representation for the disjunct part or a lack of semantic meaning for the disjunct part itself, where the complex word is treated as a basic word for analysis purposes; Complements, where two base words have opposite meanings and the same position in the sentence structure, so that swapping the complements can reverse the meaning of the sentence while preserving the sentence structure.

6. The system of claim 1, wherein the cloud-based AAC software program allows a user or administrator to establish a group for group communication among multiple users.

7. The system of claim 1, 2 or 6, wherein the cloud-based AAC software program further comprises: an adapter configured to serialize input and deserialize output into a compatible format, wherein the adapter is an endpoint distinguishing adapter (EPDA) (106); a representation translator (110) configured to translate the symbols in the deck and one or more representations of the multimedia content into their corresponding meanings to construct a sentence and communicate the sentence to the electronic device; and An analysis module (112) is configured to analyze recorded data, including but not limited to sentences created and transmitted between users or groups, cards and card groups (118d) recorded in non-transitory media (118).

8. The system of claim 7, wherein the EPDA (106) further comprises assistive features that enable integration of a cloud-based AAC software program with an existing AAC software program.

9. The system according to claim 1 or 2, wherein the human-computer interaction interface (102) receives and executes instructions, the instructions comprising: Connecting an administrator's or user's electronic device to a cloud-based server, including access to a user authentication page; Displaying visual effects on the electronic device based on the deserialized data through the UI / UX operation module (104); and One or more sets of data recorded in non-transitory media (118) are managed via a management control panel module (114).

10. The system of claim 9, wherein the UI / UX operation module (104) further includes assistive features that enable adaptive display of user interfaces and user experiences in multiple devices, wherein the display includes designing switches and image sizes to fit a user's electronic device, including but not limited to web design or mobile applications.

11. The system according to claim 1, wherein the administrator has access to the management control panel module (114), and the management control panel module (114) receives basic commands to create, read, update and delete (CRUD) or view data of any record, or the basic CRUD instructions are displayed in the form of buttons or sequences.

12. The system of claim 7 or 11, wherein the management control panel module (114) has limited access to the analysis module (112), wherein the access allowed is for viewing only and does not allow basic CRUD commands.

13. The system according to claim 10, wherein the management control panel module (114) has a graphical user interface (GUI) comprising: A data structure of a plurality of records arranged in sequence, including but not limited to lists, tables, buttons and selection boxes, optionally with paging, wherein each data structure is associated with a specific data type, including relevant descriptions and attributes of the data type, and the data of the plurality of records can optionally execute at least one basic CRUD command or view when redirected to a secondary interface associated with the selected record data; and Search interface, configured to facilitate record selection when implementing paging.

14. The system according to claim 5 or 13, wherein basic CRUD commands of the recorded data can only be executed via an editor, wherein the recorded data is a deck and the editor is a deck editing module (116).

15. The system of claim 14, wherein the deck is selectable from a set of decks (118d) for sentence formation.

16. The system according to claim 14, wherein the deck editing module (116) further comprises: An interface for adding cards to a deck, incorporating user-friendly features such as multiple-select lists and searching to select and include previously unselected cards in a set; An interface for removing cards from a deck, providing intuitive methods, including a multi-select list and search, that allow users to deselect and remove cards from a deck; A sorting mechanism for arranging selected cards within a deck, providing an interactive interface, such as swap buttons and drag-and-drop functionality, to facilitate customizing the order of cards within a deck; User-configurable card category display options, allowing users to choose whether to display the category associated with each card, and presenting it in a checkbox or radio button format to suit various applications and display modes; User access management functionality, including an interface with interactive components such as lists, search columns, and multiple selections, enabling users to define which users and groups can access decks, while also providing the ability to exclude specific users within defined criteria; Fields and input interfaces for editing deck properties, including deck names and other necessary records, such as generating and displaying QR codes to facilitate access to decks; and Deck management functions, including operations such as deck deletion and copying, as well as additional functions customized to meet specific application or system requirements.

17. The system of claim 5 or 14 or 16, wherein basic CRUD commands in at least one of the recorded data are executable for use by a representation translator (110) and a machine learning module (108), wherein the recorded data is a picture card or a single word.

18. The system according to claim 9, wherein the human-computer interaction interface (102) is configurable to include a text-to-speech function to perform reading out sentences for voice communication.

19. The system of claim 1, wherein: The electronic device includes but is not limited to a dedicated AAC device, a personal computer, a tablet computer, a mobile phone, etc.

20. A method (602) for a cloud-based assistive communication (AAC) system, comprising: The cloud server receives a login request from a user using an electronic device through a human-computer interaction interface (102), wherein the login request includes user authentication information; authenticating the user based on the received user authentication information, and retrieving user-specific data, including user information (118a) and AAC content (118b), including but not limited to a preset deck, in response to a login request; Translating words associated with the deck into appropriate picture cards and generating input suitable for the user's electronic device incorporating language, visual display and size specific to the device; transmitting output as a response to a user login request; Allowing a user to compose a sentence within the software program, wherein the user selects a deck, selects a picture card from the selected deck, and declares that the sentence is complete; Recording the combined sentence in a non-transient medium (118) of the server, wherein the recorded data includes picture card information, command sequence and input timestamp; Serializing and deserializing the assembled sentences into words using additional input information such as device type, input method, and user group (if any) through an endpoint differentiation adapter (EPDA) (106) and a presentation translator (110); The serialized sentence is transmitted to the machine learning module (108) as feedback, and artificial intelligence (AI) suggestions are generated and the machine learning matrix (118e) is updated based on the feedback; the deserialized sentence is simultaneously displayed back to the human-computer interaction interface (102) for display in the user device under the promotion of the EPDA (106) and the UI / UX operation module (104); and Update suggestions for future user requests.

21. The method of claim 20, wherein when composing a sentence structure, the software program includes the following steps: Link specific basic words to their corresponding picture cards; Split semantic phrases into a series of basic words; identify complex words in sentences and link complex words with their respective basic words; and Swap complements to reverse sentence meaning while preserving sentence structure.

22. The method of claim 20, wherein a software program is used to process the input and output, the processing comprising the steps of: serializing the input using an adapter (106); translating one or more representations of the AAC content (118b) in the card set (118d); deserializing the output into an adaptable format and communicating the output to the user's electronic device; and The output is analyzed and stored in non-transitory media (118).

23. The method of claim 20, wherein the machine learning module receives serialized sentences and generates artificial intelligence (AI) recommendations, the method comprising: Providing an independent AI learning instance configured to learn and adapt AAC content (118b) and sentence records for training; storing relevant data and calculations in non-transitory media (118); Establishing a machine learning matrix (118e) in a non-transitory medium (118), wherein the machine learning matrix (118e) is used to store and organize data associated with training of the machine learning module (108); receiving a request from a user initiated by a software program to generate an AI recommendation using a machine learning module (108); Analyze user requests to determine the appropriate context and scope for generating AI recommendations; and The request is processed using a machine learning module (108), data retrieval and analysis is performed using a machine learning matrix (118e), and machine learning techniques are applied based on the learned patterns of the AAC content (118b) and sentence records to generate AI recommendations.

24. The method (802) of claim 20, wherein translating a word associated with a card set into a suitable picture card further comprises the step of introducing a new picture card into the suitable picture card, the method comprising: Determine whether the new picture card can be divided into at least one word, phrase or part with semantic meaning; The system searches for an existing base word across non-transitory media (118), wherein if the base word carries the semantics of the new picture card, the new picture card can be represented as a new representation of the base word, wherein if no base word matches the semantics of the new picture card, the new picture card becomes a new base word; Verify that any complex word contains separate words or parts that have the same meaning and structure as the new base word; and The new basic word is recorded as a new picture card in the non-transitory medium of the system (118).

Images