Methods and systems for predicting user emotions

Abstract

A method and system for predicting user (client or customer) sentiment associated with an internet speed test is described. In one embodiment, the system receives internet speed test data from an internet speed test performed on a data connection between a client device and a remote server. The system determines whether or not there is a user-provided sentiment score associated with the internet speed test data, and, depending on whether or not there is a user-provided sentiment score, uses the internet speed test data as input to a machine learning (ML) model to generate a predicted user sentiment score as the output of the ML model, the predicted user sentiment score relating to the overall sentiment of the user on the client device regarding the internet speed test data.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to methods and systems for predicting user sentiment associated with Internet speed testing. Other embodiments are also described.

Background Art

[0002] In recent years, with the popularization of media streaming devices such as computer tablets and smartphones, users stream a vast amount of data. For example, to stream a 4K movie, approximately 9 gigabytes (GB) per hour of the movie is required. As a result, the computing network to which the media streaming device connects to the Internet requires a wide bandwidth to handle the data. Bandwidth is related to the capacity of the network to transmit data. In particular, the bandwidth can be the maximum transmission rate (i.e., the maximum transmission rate) that the underlying network can achieve without loss (or with minimal loss). This is especially true when multiple computing devices are exchanging data via a given network.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The bandwidth of an internet connection (e.g., internet speed) can be measured using an internet speed test, which may be provided by a speed test provider. To perform a speed test, a client device (e.g., a desktop computer, smartphone, etc.) may request an internet speed test server to send as much data as possible to load the internet connection to its saturation point. After all the requested data has been transmitted to the client device, and it is determined that the saturation point has been "reached" or the timer has expired, the client device determines and records the bandwidth, for example, by measuring the rate at which data is received or by determining the time it takes to transmit the requested amount of data. The results of the speed test, which may include information such as download speed and / or upload speed, may be presented to the user of the client device through a graphical user interface (GUI) displayed on the client device's screen.

[0004] At the end of a speed test, when the results are displayed in the GUI, the speed test provider may offer the user the ability to provide feedback on their satisfaction with the speed test results (for example, through the user interface (UI) in the GUI). Such feedback can provide important insights into the customer's feelings about their internet speed. For example, if the results indicate that the customer's speed is slow (e.g., below a threshold), the customer may provide feedback indicating low satisfaction. This insight allows the network operator (provider) to proactively address customer concerns and optimize services to address low customer satisfaction. For example, the operator may upgrade the network infrastructure to improve network bandwidth or, based on the feedback, offer incentives such as discounts to the customer. This proactive approach to network operators can improve customer satisfaction, customer retention, and overall business success. However, generally, customers do not provide enough feedback for network operators to analyze and gain a comprehensive understanding of customer satisfaction. In fact, customer feedback is issued for only about 2% of the total number of speed tests performed. Therefore, it is necessary to predict (estimate) the customer's (user's) feelings about internet speed tests. [Means for solving the problem]

[0005] This disclosure relates to a method and system for predicting user sentiment of internet speed tests using a machine learning (ML) model. For example, the system may receive internet speed test data from an internet speed test performed on a data connection between a (first) client device and a remote server. The system may determine whether or not there is a user-provided sentiment score (or user feedback) associated with the internet speed test data. In particular, the system may determine whether or not the user of the client device has provided feedback, for example, through the selection of one or more UI items of a GUI displayed on the client device. In determining that there is no user-provided sentiment score associated with the internet speed test data, the system may use the test data as input to an ML model to generate a predicted user sentiment score as the output of the ML model, the predicted score relating to the overall sentiment of the user of the client device regarding the internet speed test data. In one embodiment, the ML model may be trained using training data which includes internet speed test data from several internet speed tests and user-provided sentiment scores associated with the internet speed test data. Internet speed test (and / or training) data may include at least one of the following: download speed, upload speed, download speed to upload speed ratio (DL / UL), jitter, latency, and the geographical region where the client device is located. Therefore, by utilizing machine learning techniques and analyzing various factors of internet speed test data, this disclosure may provide insights into user satisfaction levels that can be used to improve the overall user experience. [Effects of the Invention]

[0006] As described herein, an ML model may be trained to output a predicted user sentiment score based on internet speed test data. In one embodiment, the ML model may take into account the location of the client device when generating the predicted score. For example, users in different regions (e.g., different cities) may provide different feedback (e.g., on average) to similar or the same speed test results. In particular, the system may receive a second set of internet speed test data for a second internet speed test performed on a second data connection between a second client device and a remote server, where the second client device is in a second geographic region (e.g., in a different city than the first client device), and the second set of internet speed test data includes at least one identical piece of data, such as the DL / UL ratio, to the first set of internet speed test data for a speed test performed between the first client device and the remote server. The system uses the second set of internet speed test data as input to the ML model to generate a second predicted user sentiment score as output of the ML model, where the second predicted user sentiment score is different from the first predicted user sentiment score. This can be obtained for various reasons, such as the first region having faster internet speeds (on average) than the second region (for example, because customers in the first region have a better, upgraded network infrastructure than those in the second region). As a result, if speed tests for both users produce similar or identical results, the ML model may produce a lower predictive user sentiment score for users in the first region and a higher predictive user sentiment score for users in the second region.

[0007] In one embodiment, the disclosure may consider outlier client feedback. Generally, a customer providing feedback may experience worse test results than other customers in their geographical area. This may be due to network issues specific to that customer (e.g., network equipment failure). The predicted user sentiment score may be higher (or lower) than the average user-provided sentiment score of several internet speed tests performed on several other client devices located in the same area. As a result, the system may provide a more accurate user sentiment that may be higher than the user-provided sentiment.

[0008] In another embodiment, depending on whether the system determines that there are user-submitted sentiment scores associated with the internet speed test data, the system may train an ML model using the user-submitted sentiment scores and at least a portion of the internet speed test data. In some embodiments, the ML model may be at least one of a random forest, random search, neural network, and regression model. In another embodiment, the operations described herein may be performed by a remote server which may be operated by an internet speed test provider.

[0009] According to another embodiment of the present disclosure, there is a server including at least one processor and a memory having stored instructions that, when executed by at least one processor, causes the server to determine internet speed test data of an internet speed test performed for data connectivity with a client device, to determine whether the client device has provided feedback related to the internet speed test data, and, depending on whether the client device has not provided feedback, to use a machine learning (ML) model to estimate a predicted user sentiment score related to the user sentiment of the internet speed test data based on at least a portion of the internet speed test data, and to store the predicted user sentiment score in memory. In one embodiment, the server may perform at least some operations as described herein.

[0010] According to another embodiment of the present disclosure, there is a non-temporary machine-readable medium having stored instructions that, when executed by at least one processor of the electronic device, causes the electronic device to receive internet speed test data, without user feedback from the client device, of an internet speed test performed on a data connection between a client device and a remote server, and to use the internet speed test data as input to a machine learning (ML) model, and a predicted user sentiment score relating to the overall sentiment of the user of the client device regarding the internet speed test data as output to the ML model. In one embodiment, the non-temporary machine-readable medium has stored instructions that, when executed, cause the electronic device to perform at least a portion of the operations described herein.

[0011] In one embodiment, an ML model may be trained using historical internet speed test data, including user feedback from other client devices within the same geographical area as the client device. As a result, different ML models may be trained using feedback from client devices within these areas to predict sentiment in different regions. This could provide a more accurate representation of user sentiment among people in those regions.

[0012] The above summary does not constitute an exhaustive enumeration of all embodiments of the Disclosure. The Disclosure is assumed to include all systems and methods that can be realized from all suitable combinations of the various embodiments outlined above, as well as those disclosed in the following detailed description and specifically pointed out in the claims filed with the application. Such combinations have special advantages not specifically described in the above summary.

[0013] Embodiments of this disclosure are illustrated, not limited, in the accompanying drawings, and similar reference numerals in the drawings indicate similar elements. It should be noted that references in this disclosure to "some" or "one" embodiment of the disclosure do not necessarily refer to the same embodiment, but rather to at least one. Furthermore, the given drawings may be used to illustrate features of multiple embodiments of this disclosure, and not all elements in the drawings are required for the given embodiment. [Brief explanation of the drawing]

[0014] [Figure 1] Figure 1 is a block diagram illustrating a predictive user sentiment system in several embodiments. [Figure 2] Figure 2 is a flowchart of one embodiment of the process for predicting user sentiment in an internet speed test, based on several embodiments. [Figure 3] Figure 3 is an emotion graph illustrating the difference between user-provided emotions and predicted user emotions in one embodiment. [Modes for carrying out the invention]

[0015] With reference to the accompanying drawings, several embodiments of this disclosure are described herein. Unless otherwise expressly defined, the scope of this disclosure is not limited to the illustrated portions, which are intended solely for illustrative purposes. Furthermore, while numerous details are described, it should be understood that some embodiments of this disclosure can be realized without these details. In other instances, well-known circuits, structures, and techniques are not illustrated in detail so as not to obscure the understanding of this description.

[0016] References to “one embodiment” or “a particular embodiment” in the specification mean that certain features, structures, or characteristics described in conjunction with the embodiment may be included in at least one embodiment, but not all embodiments necessarily include those specific features, structures, or characteristics. Furthermore, such phrases do not necessarily refer to the same embodiment.

[0017] Figure 1 is a block diagram illustrating a user (customer or client) sentiment prediction system (hereinafter referred to as the "System") 10 in several embodiments. The System 10 includes a client (customer or user) device 13 which may be owned and / or operated by one or more users, a (remote) server 11, and a network (e.g., the Internet) 12. In one embodiment, the System may include more devices, such as having one or more servers or one or more client devices. In another embodiment, the System 10 may include one or more additional devices which may be communicatively coupled to the client device 13.

[0018] Server 11 and client device 13 can each be any type of electronic device capable of establishing a data connection over one or more networks to exchange data (for example, as data packets) in order to perform an internet speed test when a data connection is established. For example, a client device could be a tablet computer, a desktop computer, a mobile device (for example, a smartphone), a media playback device, etc. In another embodiment, a client device could be any type of network device such as a server, router, or hub. In one embodiment, the server may include a standalone electronic server or may include one or more servers. In another embodiment, the server could be any type of electronic device such as a desktop computer.

[0019] The client device 13 includes a network interface 81, a controller 82, memory 83, a speaker 86, a display 87, and an input device 88. In one embodiment, the client device may include more or fewer elements than those illustrated herein. For example, the client device may include one or more speakers, a display, and / or an input device. In another example, device 13 may not include a speaker, a display, and / or an input device. In one embodiment, elements of the client device may be part of the client device. For example, elements may be integrated into (or on) a housing (not shown) of the client device 13. In another embodiment, at least some elements may be separate electronic devices that can be communicatively coupled to the client device. For example, the display 87 may be a separate display connected to the client device and capable of receiving image data for display.

[0020] The network interface 81 may provide an interface for a client device 13 to communicate with an electronic device such as a server 11 via the network 12. For example, the network interface may be configured to establish a data connection (e.g., a communication link) with the server 11 (e.g., its network interface 14), and once established, to exchange digital data as described herein. In one embodiment, the network 12 may be any type of computer network, such as a wide area network (WAN) (e.g., the Internet) or a local area network (LAN), through which devices may exchange data with each other and / or with one or more other electronic devices. In another embodiment, the network may be a wireless network, such as a wireless local area network (WLAN) or a cellular network, for exchanging digital (e.g., test) data. With respect to the cellular network, device 13 may be configured to establish wireless (e.g., cellular) calls, where the cellular network may include one or more cell towers, which may be part of a communications network (e.g., a 4G Long-Term Evolution (LTE) network, a 5G network, etc.) that supports data transmission (and / or voice calls) for electronic devices such as mobile devices (e.g., smartphones). In one embodiment, device 13 may be configured to communicate with one or more devices via network 12 using any type of communications protocol, such as Transmission Control Protocol / Internet Protocol (TCP / IP), Quick User Datagram Protocol (UDP) Internet Connection (QUIC), etc.

[0021] In another embodiment, the device can be configured to wirelessly exchange data via other networks, such as a wireless personal area network (WPAN) connection. For example, the client device 13 can be configured to establish a wireless connection with another electronic device via a wireless communication protocol (e.g., the BLUETOOTH (registered trademark) protocol or any other wireless communication protocol). During the established wireless connection, the device can exchange (e.g., transmit and receive) data packets (e.g., Internet Protocol (IP) packets) with digital data.

[0022] The input device 88 can be any type of device configured to receive user input. For example, the device 88 can include a keyboard, one or more buttons, a mouse, etc. In another embodiment, the input device 88 can be part of the display 87, whereby the display can be a touch-sensitive display screen configured to receive user input through one or more user touches.

[0023] The controller 82 can be a dedicated processor (e.g., one or more processors), such as an application-specific integrated circuit (ASIC), a general-purpose microprocessor, a field-programmable gate array (FPGA), a digital signal controller, or a set of hardware logic structures (e.g., filters, arithmetic logic units, and dedicated state machines). The controller can be configured to perform one or more Internet speed test operations and / or network operations as described herein. Details of the operations performed by the controller are described herein.

[0024] Memory 83 can be any type of non - transient machine - readable storage medium, such as a read - only memory, a random - access memory, a CD - ROM, a DVD, a magnetic tape, an optical data storage device, a flash memory device, and a phase - change memory. One or more components are illustrated as being included within device 13, but can be part of a separate electronic device, such as a memory that is a separate data storage device.

[0025] As illustrated, memory 83 includes one or more client software programs 84 and an operating system 85. The operating system (OS) 85 can be a software component responsible for managing and coordinating the activities of device 13 and sharing resources (such as controller resources, memory, etc.). In one embodiment, the OS functions as a host for application programs (such as program 84) running on device 13. In one embodiment, the OS provides an interface to a hardware layer (such as a controller, memory, etc.) and can include one or more software drivers that communicate with the hardware layer. For example, the driver can receive and process data packets received via the hardware layer from one or more other devices communicatively coupled to the device. In one embodiment, the OS can include a kernel (or a part thereof) that provides an interface between one or more programs (such as can be executed by controller 82) and the hardware layer.

[0026] The client software program 84 can be any type of software application (which may include one or more instructions) that, when executed (for example, by the controller 82), causes the device 13 to communicate over the network 12 and perform one or more internet speed test operations. For example, the program may cause the client device (for example, via the network interface 81) to exchange messages with the server 11 over the network 12 using any type of communication protocol (e.g., Hypertext Transfer Protocol (HTTP), HTTPS, etc.) using any type of communication protocol such as TCP / IP, QUIC, etc. In one embodiment, the program 84 can be any application that can interact with a web-based application on which an internet speed test may be performed. In one embodiment, the software program may be an internet speed test application that, when executed, determines internet speed test data (e.g., one or more data connection characteristics) such as bandwidth by establishing a data connection with one or more devices. In particular, the software program may establish a TCP data connection (or QUIC connection) with the server 11, and once established, may send a request to start an internet speed test, and the server may send (e.g., test) data as described herein. In one embodiment, the program may be configured to access a web browser (which may be hosted, for example, by server 11), and the test may be performed via the web browser. Once test data has been sent (or time has elapsed), the client software program 84 may be configured to determine internet speed test data, which may include download speed, upload speed, download speed to upload speed ratio (DL / UL), jitter, latency, and connection characteristics such as the geographical region where the client device 13 is located.In one embodiment, the program may be configured to display a graphical user interface (GUI) on a display 87 that presents the internet speed test data results of an internet speed test. In another embodiment, the client software program may send at least a portion of the internet speed test data to a server 11, which may use this to train and / or test a user sentiment model 18. Details of the model are described herein.

[0027] In one embodiment, the client software program 84 may also be configured to display a UI item on the GUI that requests feedback regarding the results of an internet speed test. For example, the feedback may show (or include) a user-provided sentiment score associated with the internet speed test data. For example, the software program may request the user of the client device 13 to rate the speed test data within a certain numerical range (for example, between 0 and 5, where 0 indicates the user is not satisfied with the results and 5 indicates the user is satisfied with the results). In another embodiment, the GUI may request the user of the client device to indicate specific data points that the user is satisfied or dissatisfied with. In some embodiments, the client software program 84 may provide the user with the ability to include comments. When user feedback is received, the client software program 84 may be configured to send the feedback (user-provided sentiment score) to the server 11 via the network interface 81 from the client device. In one embodiment, the feedback may be sent to the server together with or separately from the internet speed test data.

[0028] Server 11 includes a network interface 14, a controller 15, and memory 16. In another embodiment, the server may include other elements such as one or more input devices. Memory 16 includes a server software program 17, a user sentiment model 18, internet speed test data 19, and a predicted user sentiment score 80. In one embodiment, the server may be a server of an internet speed test provider, which can be configured to perform internet speed tests on data connections with one or more client devices, such as device 13. In addition to performing speed tests, or instead of performing speed tests, the server may be configured to predict the user sentiment of the user of the client device performing the speed test. For example, server 11 may perform speed tests and / or predict user sentiment.

[0029] The server software program 17 can be any type of software program and, when executed by the controller 15, can communicate with one or more devices over the network and perform one or more internet speed test operations as described herein. The server program may be configured to perform an internet speed test using one or more client devices, and the server may send test data to the client devices so that the client devices can determine (estimate) the internet speed test data. The server 11 may be configured to receive internet speed test data 19 from the client device 13 and store the data in memory 16. In another embodiment, the server program 17 may perform any type of internet speed test technique so that the system 10 can benchmark the data connection with the client device 13. For example, when the server program 17 performs an internet speed test using a client device, it may be configured to measure at least some internet speed test data (e.g., upload speed).

[0030] The internet speed test data 19 may include one or more connection characteristics determined from internet speed tests performed by one or more client devices. For example, at least some internet speed test data 19 may be determined from internet speed tests performed between server 11 (and / or other internet speed test servers) and one or more client devices such as device 13. In another embodiment, server 11 may receive data from internet speed tests performed by other devices (for example, between other internet speed test servers and client devices). For example, speed test data 19 may be received by a client device from an internet speed test server that performed a speed test. In this case, once the client device completes the speed test, the device may send connection characteristics and / or feedback to one or more internet speed test servers, which may include server 11. In one embodiment, the internet speed test data 19 may be collected from several client devices across many geographical areas such as cities, states, or countries over a period of time (e.g., weeks, months, or years). In another embodiment, the geographical area may be any type of land area. For example, a geographical area could be one of several non-overlapping plots of land, such as a 500 square meter plot.

[0031] In one embodiment, the internet speed test data 19 may include at least some of the connectivity characteristics described herein, such as the DL / UL ratio. In one embodiment, the test data 19 may include (or may be associated with) user-provided sentiment of the client device to which the data 19 may be associated. In one embodiment, the test data 19 may be labeled data, which may be used to train (and / or test) a user sentiment model 18. In particular, the data 19 may be labeled based on associated user-provided sentiment. For example, if user sentiment is a score from 0 to 5, the data may be labeled with the associated numerical score provided by the user of the client device to which the data was generated. In another embodiment, the data may be labeled with one or more other labels, such as a geographical label indicating the geographical region where the associated client device is located, if the internet speed test data is determined. In another embodiment, the test data 19 may include connectivity characteristics with associated user-provided sentiment, and / or characteristics not associated with user-provided sentiment.

[0032] The user sentiment model 18 (hereinafter also referred to as the "model") may be a machine learning (ML) model that can be trained to predict user sentiment from the results of internet speed tests on one or more client devices. In particular, the model 18 may be trained using at least a portion of the internet speed test data 19 to predict user sentiment. For example, the model may utilize supervised learning techniques using labeled internet speed test data 19 to predict sentiment ratings (scores) without sentiment labels. In particular, the model may be trained to predict user sentiment for internet speed tests where users may not have provided any (or sufficient) feedback on the test, in order to gain a more comprehensive understanding of user (customer) satisfaction. In one embodiment, the model may be any type of machine learning model, such as a random forest, random search, neural network, or regression model. The model may be configured to output a predicted user sentiment score depending on one or more connection characteristics of the internet test data.

[0033] In one embodiment, model 18 may be trained to predict user sentiment based on the geographical location where the client device is located. Specifically, the model may be trained to predict sentiment considering regions such as cities, states, or countries. In this case, the model considers regions (e.g., as labels) by being trained using region-specific internet speed test data. For example, to train a model to predict sentiment for a given region, the system may collect previous internet speed test data for the given region and then use that data to train the model. As a result, the model may predict different sentiments for different regions. In one embodiment, internet speed test data collected with user-provided sentiment for a region where the provided sentiment is below average may be trained so that the predicted sentiment for that region is lower than the predicted sentiment for another region. As a result, the model may predict different sentiments for the same internet speed test data for different regions. For example, if the region is a city, user sentiment model 18 may be configured to predict that the internet speed test results for a client device in Los Angeles, California, will result in a predicted user sentiment of 3.5 for a first set of connectivity characteristics, such as a specific DL / UL ratio, as input to the model. To predict the sentiment of an internet speed test on another client device in Denver, Colorado, using at least some of the same connection characteristics, such as the same ratio, as input, the model may predict a user sentiment of 4. This is because even if the internet speed test results are the same, the sentiment may differ in different regions. Details of predicting different sentiments for similar results are described herein. In another embodiment, memory 16 may include multiple user sentiment models, each model being for a given geographical region.

[0034] In another embodiment, to provide more accurate predicted user sentiment, model 18 may be trained using multiple connectivity characteristics. Continuing the previous example, the model may predict different sentiments for two client devices that include at least a portion of the same internet test data, by having at least some different test data. For example, two internet speed tests may result in the same DL / UL but have different download and / or upload speeds. For instance, one client device might provide a DL / UL of 1.1 when DL=110 and UL=100, while another client device might provide the same DL / UL of 1.1 but have significantly different download / upload speeds, such as when DL=1.1 and UL=1. In this case, the model may use download and / or upload speeds as other inputs to account for these differences, resulting in different user sentiments for both devices because the first device's upload and download speeds are faster than the other device's.

[0035] In one embodiment, Model 18 may be based on historical internet speed test data. As described herein, a server may collect speed test data and / or user feedback over a period of time, such as several months or several years. As a result, Model 18 may be trained using this historical data to provide accurate predictions of user sentiment for a given geographical area. In another embodiment, Model 18 may be updated (adapted) to changes in internet speed test data (over a period of time). For example, if user sentiment improves, the model may be trained to adapt its predicted sentiment based on that improvement.

[0036] The predicted user sentiment score 80 may be the result of Model 18. As described herein, the score may be a numerical value (for example, a value between 0 and 5). As another example, the score may be one or more connection characteristics derived during an internet speed test, or may be related to them. For example, the score may indicate whether the user of the client device is satisfied or dissatisfied with a given characteristic, such as download speed. In another embodiment, the score 80 may include written comments from the user of the client device. In some embodiments, the score may be any type of indication that shows user sentiment related to the results of an internet speed test.

[0037] Figure 2 is a flowchart of one embodiment of a process 20 for predicting user sentiment for an internet speed test, according to several embodiments. In one embodiment, at least part of the operation may be performed by a system 10 (e.g., a server 11 of system 10). For example, at least some operations may be performed by a controller 15 (e.g., a server software program 17 that may be executed by the controller 15). The process 20 is initiated using the controller 15 and receives a request from a client device 13 to perform an internet speed test over the data connection between the client device and the server (block 21). As described herein, the user of the client device 13 may run a client software program 84 to perform the internet speed test. The program may include a GUI that can be displayed on the display 87 of the client device 13, which may include UI items that, in response to receiving user input via an input device 88, cause the client device to send a request to the server 11 over the network 12. The controller 15 may perform the internet speed test (block 22). As described herein, the controller 15 may instruct the server 11 to send test data to the client device, and the client device may use that data to generate internet speed test data.

[0038] The controller 15 receives internet speed test data from internet speed tests performed on the data connection (block 23). In particular, the controller 15 may determine test data for speed tests performed for the data connection between the client device and the server 11. In another embodiment, the controller 15 may receive data from speed tests performed between the client device and another (e.g., remote) server, which may be a separate (standalone) internet speed test server. The controller 15 determines whether or not there is a user-provided sentiment score associated with the internet speed test (determination block 24). Specifically, based on the received internet speed test data, the controller 15 may determine whether or not the client device has provided (e.g., sent) feedback that includes or may indicate user-provided sentiment. For example, when an internet speed test is performed, the client device may offer the user of the device an option to provide feedback. If feedback is provided, the client device may send the feedback to the server 11 along with the internet speed test data. On the other hand, if no feedback is provided (for example, if the client software program 84 does not receive feedback within a certain period after the speed test is performed, or if the user of the client device does not select a UI item to provide feedback), the client device 13 may transmit the internet speed test data without user-provided feedback (and / or an indication in the internet speed test data that user feedback has not been provided).

[0039] Depending on whether the controller determines that the internet speed test data includes user feedback from a client device (or that user feedback that may be associated with the received internet speed test data is provided by the client device), the controller 15 trains a user sentiment model 18 using the internet speed test data (block 25). In this case, the controller may label at least a portion of the received test data based on the user feedback, and then train (and / or test) the model using the labeled data. As a result, the system may train the model 18 using the received test data associated with the user feedback. In particular, the controller may receive test data that includes internet speed test data from several internet speed tests and includes feedback (user-provided sentiment scores) associated with the test data, and may train the model using the received test data (labeled based on user-provided sentiment).

[0040] However, if the client device 13 does not provide user feedback, the controller 15 uses the received internet speed test data as input to the model 18 to generate a predicted user sentiment score as the model's output (block 26). In one embodiment, the predicted user sentiment may be a prediction of the type (or amount) of sentiment the user of the client device would have expressed if user feedback on the internet speed test results had been provided. In one embodiment, the predicted user sentiment score may relate to the overall sentiment of the user of the client device regarding the internet speed test data. For example, the client software program may ask the user to rate their experience on a scale from 0 to 5, where 0 may indicate that the user sentiment regarding the test (results) is overall unsatisfied, and 5 may indicate that the user sentiment is overall satisfied.

[0041] In another embodiment, the predicted user sentiment score may be based on at least a portion of the internet speed test data. For example, a model may use one or more connection characteristics as input to predict user sentiment. In this case, the user sentiment may be (or associated with) those input characteristics. For example, if download speed is input to the model, the model may output user sentiment associated with download speed. In another embodiment, the controller may use other characteristics to predict sentiment. For example, if memory 16 contains different models for different regions, the controller 15 may be configured to extract a user sentiment model (trained to predict scores) for a given geographical region associated with a given geographical region associated with internet speed test data, and then use the model to predict sentiment as described herein. The controller 15 may store the predicted user sentiment score in memory 16 along with the stored score 80 (block 27). In one embodiment, the system may collect (store) the predicted user sentiment score 80 over a period of time and send the predicted score to one or more network operators that provide network services to client devices in order to provide insights into users (customers).

[0042] In one embodiment, at least part of the operation of process 20, such as blocks 21 and 22, may be optional. This could be the case if server 11 does not provide an internet speed test for the client device's data connection, but instead is performed by another remote server. Once the test is complete, the client device 13 (and / or the remote server) may send the internet speed test data and / or user feedback associated with the data.

[0043] In another embodiment, the system may perform at least some of these operations over a period of time to continuously train the user sentiment model 18 to learn and improve its predictions over time as more labeled data becomes available. As a result, the user sentiment model can adapt to and stay up-to-date with user sentiment (which may change over time in a given geographical area) to ensure that sentiment predictions remain accurate (for a given area).

[0044] As described herein, Model 18 can be trained to predict different user sentiment scores based on the geographical region associated with a client device. For example, Controller 15 may receive the same (or at least some similar) test data from different client devices located in different regions. The model may produce a higher predicted user sentiment score for one set of test data than a different user sentiment score predicted for another set of test data. This may be due to the fact that the test data associated with both regions may be labeled differently. For example, a region that produces a higher predicted user sentiment score may produce a higher user sentiment from other regions on which Model 18 can be trained.

[0045] In another embodiment, the user sentiment model 18 may provide a more accurate assessment of user sentiment than actual user-provided sentiment. As described herein, less than 2% of clients conducting internet speed tests provide user feedback. In such cases, users who did not achieve satisfactory results in the speed test (e.g., low download speeds) may be more likely to provide negative feedback than users who achieved satisfactory results (e.g., high download speeds). As a result, since most users who conduct internet speed tests and achieve good speed test results (e.g., users above a threshold) may not provide feedback (e.g., negative feedback), user-provided sentiment (e.g., in a given region) may be lower than the actual user sentiment (e.g., the average) of users conducting internet speed tests in that region. System 10 may train model 18 to account for (and adjust for) user-predicted sentiment so that it reflects actual user sentiment rather than low (or high) sentiment due to a small number of outlier feedbacks.

[0046] Figure 3 illustrates an example of an emotion graph 30 illustrating the difference between user-provided emotions and predicted user emotions in one embodiment. In particular, this graph 30 illustrates how the user emotion model 18 can predict a more accurate evaluation of user emotions that may differ (for example, on average) from user-provided emotions for a particular geographical region.

[0047] Graph 30 illustrates sentiment scores related to internet speed test data, such as download speed. The graph includes several user-provided sentiment scores and several predicted user sentiment scores. In particular, the internet speed test data in the graph may represent the average (or overall) representation of one or more measured connection characteristics from one or more internet speed tests. However, user-provided scores may be (or may be based on) scores provided by some of the users who performed the internet speed tests. As a result, the exemplified user-provided scores may not accurately reflect actual user sentiment, as they may be based on only a portion of the user-provided feedback (e.g., overall negative feedback).

[0048] Graph 30 includes a first user sentiment score 31a and a second user sentiment score 31b. Each user sentiment score may be associated with a specific graphic region and may be an average score of several user sentiment scores (based on several internet speed tests performed on several client devices located within each region). Each score may not accurately reflect user sentiment within each region based on internet speed test data. For example, the first score 31a may represent the average user sentiment for client devices in Los Angeles, California, and the second score 31b may represent the average user sentiment for client devices in Denver, Colorado. Clients in Los Angeles experience better internet speed test data (e.g., faster download speeds) than those in Denver, Colorado, but the first score 31a is lower than the user sentiment score 31b for Denver.

[0049] The discrepancy between the two scores can be based on various factors. For example, the majority of users who can provide feedback within Los Angeles (e.g., above the threshold) may be users who experienced worse internet speed test results than the average of all users in Los Angeles. Los Angeles users may provide worse feedback than Denver users (on average), even if they have better internet speed test results on average. On the other hand, a Denver user sentiment score of 3lb could be the result of some users providing higher positive feedback (e.g., higher numbers than those provided by the Los Angeles users who provided the feedback) than the overall user sentiment of users within Denver. This could be because the download speeds of users providing feedback (which may represent less than 50% of Denver's total population, or less than 50% of users who performed internet speed tests within Denver) are faster than the average download speed of users within Denver, resulting in a better sentiment score than would be expected for the average population.

[0050] Model 18 may take this outlier feedback into account to generate a predicted user sentiment score that reflects actual user sentiment, and the predicted user sentiment score may extend to a larger population than the population of user-provided sentiment. In one embodiment, Model 18 may be trained using historical internet speed test data, including user feedback from other client devices located in the same geographical area as the client device. In particular, Graph 30 illustrates two predicted user sentiment scores 32a, 32b associated with the regions of user-provided scores 31a, 31b. Specifically, the first predicted score 32a for a client device located in Los Angeles (which did not provide user feedback) is higher than the average of user-provided sentiment scores 31a, while the second predicted score 32b for a client device located in Denver is lower than the average of user-provided sentiment scores 31b. Both of these scores may be predicted using connection characteristics such as the download speeds that Server 11 receives from each of these devices, as described herein. In one embodiment, Model 18 may be able to predict more accurate user sentiment based on training data (e.g., across one or more regions) that can show higher user sentiment for the overall population if internet speed test data reflects better data connectivity (e.g., faster download speeds).

[0051] As described above, embodiments of the present disclosure may be a non-transient, machine-readable medium (such as a microelectronic memory) having stored instructions for programming one or more data processing components (collectively referred to herein as “processors”) to perform at least some of the operations described herein. In other embodiments, some of these operations may be performed by certain hardware components, including hardwired logic. Alternatively, these operations may be performed by any combination of programmed data processing components and fixed hardwired circuit components.

[0052] To assist readers of the Patent Office and any patents issued under this application in interpreting the attached claims, the applicant wishes to note that, unless the terms “means for” or “steps for” are expressly used in the claims, no part of the attached claims or elements of the claims is intended to apply U.S. Patent Law Title 35, Section 112(f).

[0053] While specific embodiments are described and illustrated in the accompanying drawings, various other variations can be conceived by those skilled in the art. Such embodiments are merely illustrative of the broader disclosure and not limiting it, and this disclosure should be understood as not being limited to the specific configurations and arrangements illustrated and described.

[0054] In some embodiments, the disclosure may include phrases such as, for example, "[Element A] and [Element B]." This phrase may refer to one or more of the elements. For example, "at least one of A and B" may refer to "A," "B," or "A and B." Specifically, "at least one of A and B" may refer to "at least one of A and at least one of B," or "at least one of either A or B." In some embodiments, the disclosure may include phrases such as, for example, [Element A], [Element B], and / or [Element C]. This phrase may refer to any of these elements or any combination thereof. For example, "A, B, and / or C" may refer to "A," "B," "C," "A and B," "A and C," "B and C," or "A, B, and C."

Claims

1. It is a method, Receiving internet speed test data from an internet speed test performed on the data connection between the client device and the remote server, To determine whether or not there is a user-provided sentiment score associated with the aforementioned internet speed test data, A method comprising: determining that there is no user-provided sentiment score, using the internet speed test data as input to a machine learning (ML) model, and generating a predicted user sentiment score as output to the ML model, wherein the predicted user sentiment score is related to the overall sentiment of the user of the client device with respect to the internet speed test data.

2. The method according to claim 1, wherein the internet speed test data comprises at least one of download speed, upload speed, download speed to upload speed ratio (DL / UL), jitter, latency, and the geographical region where the client device is located.

3. The geographical region is a first geographical region, the internet speed test is a first internet speed test, the data connection is a first data connection, the client device is a first client device, the predicted user sentiment score is a first predicted user sentiment score, the internet speed test data is a first set of internet speed test data, and the method further comprises The receiving of a second set of internet speed test data for a second internet speed test performed on a second data connection between a second client device and the remote server, wherein the second client device is located in a second geographical area, and the second set of internet speed test data has the same DL / UL ratio as the first set of speed test data. The method according to claim 2, comprising using the second set of internet speed test data as input to the ML model to generate a second predicted user sentiment score as output to the ML model, wherein the second predicted user sentiment score is different from the first predicted user sentiment score.

4. The method according to claim 1, wherein the internet speed test is performed while the client device is located within a geographical area, and the predicted user sentiment score is greater than or less than the average user sentiment score of multiple internet speed tests performed for multiple client devices located within the geographical area.

5. Receiving training data which includes internet speed test data from multiple internet speed tests and user-provided sentiment scores associated with the internet speed test data from the multiple internet speed tests, The method according to claim 1, further comprising training the ML model using the received training data.

6. The method according to claim 1, further comprising training the ML model using the user-provided sentiment score and at least a portion of the internet speed test data, in response to determining that there is a user-provided sentiment score associated with the internet speed test data.

7. The method according to claim 1, wherein the ML model is at least one of a random forest, a random search, a neural network, and a regression model.

8. The method according to claim 1, wherein the receiving, determining, and generating are performed by the remote server.

9. It is a server, At least one processor, When executed by the at least one processor, the server will be: Determine the internet speed test data from the internet speed test performed for data connection with the client device. The client device is made to determine whether or not it has provided feedback related to the internet speed test data. In response to determining that the client device did not provide feedback, a machine learning (ML) model is used to estimate a predicted user sentiment score based on at least a portion of the internet speed test data, wherein the predicted user sentiment score is estimated to be related to the user sentiment of the internet speed test data. A server comprising a memory containing stored instructions for storing the predicted user sentiment score in memory.

10. The server according to claim 9, wherein the internet speed test data comprises at least one of download speed, upload speed, download speed to upload speed (DL / UL), jitter, latency, and the geographical region where the client device is located.

11. The geographical region is the first geographical region, the internet speed test is the first internet speed test, the data connection is the first data connection, the client device is the first client device, the predicted user sentiment score is the first predicted user sentiment score, the internet speed test data is the first set of internet speed test data, and the memory further comprises Receiving a second set of internet speed test data for a second internet speed test performed on a second data connection with a second client device, wherein the second client device is located in a second geographical area, and the second set of internet speed test data has the same DL / UL ratio as the first set of speed test data, The server according to claim 10, which uses the second set of internet speed test data as input to the ML model to generate a second predicted user sentiment score as output of the ML model, wherein the second predicted user sentiment score is different from the first predicted user sentiment score.

12. The server according to claim 9, wherein the internet speed test is performed while the client device is located within a geographical area, and the predicted user sentiment score is greater than or less than the average user sentiment score of multiple internet speed tests performed for multiple client devices located within the geographical area.

13. The aforementioned memory further, Training data is received which includes internet speed test data from multiple internet speed tests and feedback associated with the internet speed test data from the multiple internet speed tests. The server according to claim 9, having instructions for training the ML model using the received training data.

14. The server according to claim 9, wherein the memory further has instructions for training the ML model using the user-provided sentiment score and at least a portion of the internet speed test data, in response to determining that there is feedback comprising a user-provided sentiment score.

15. The server according to claim 9, wherein the ML model is at least one of a random forest, a random search, a neural network, and a regression model.

16. A non-temporary machine-readable medium, when executed by at least one processor of an electronic device, with respect to the electronic device, Receiving internet speed test data from an internet speed test performed in a data connection between a client device and a remote server, wherein the internet speed test data does not include user feedback from the client device. The method involves using the internet speed test data as input to a machine learning (ML) model and a predicted user sentiment score as output to the ML model, wherein the predicted user sentiment score is a non-temporary machine-readable medium having stored instructions to be used relating to the overall sentiment of the user of the client device with respect to the internet speed test data.

17. The non-temporary machine-readable medium according to claim 16, wherein the internet speed test data comprises at least one of download speed, upload speed, download speed to upload speed ratio (DL / UL), jitter, latency, and the geographical region where the client device is located.

18. The geographical region is a first geographical region, the data connection is a first data connection, the client device is a first client device, the predicted user sentiment score is a first predicted user sentiment score, the internet speed test data is a first set of internet speed test data, and the non-temporary machine-readable medium is further, Receiving a second set of internet speed test data for a second internet speed test performed on a second data connection between a second client device and the remote server, wherein the second client device is located in a second geographical area, and the second set of internet speed test data has the same DL / UL ratio as the first set of speed test data, A non-temporary machine-readable medium according to claim 17, comprising instructions for generating a second predicted user sentiment score as output of the ML model, wherein the second predicted user sentiment score is different from the first predicted user sentiment score.

19. The non-temporary machine-readable medium according to claim 16, wherein the internet speed test is performed while the client device is located within a geographical area, and the predicted user sentiment score is greater than or less than the average user sentiment score of multiple internet speed tests performed for multiple client devices located within the geographical area.

20. The non-temporary machine-readable medium according to claim 16, wherein the ML model is trained using historical internet speed test data, including user feedback from other client devices located within the same geographical area as the client device.

Images