Method and system for predicting user emotion

Abstract

Methods and systems for predicting a user (client or customer) emotion associated with an internet speed test are described. In one embodiment, a system receives internet speed test data for an internet speed test performed over a data connection between a client device and a remote server. The system determines whether there is a user-provided emotion score associated with the internet speed test data, and in response to determining that there is no user-provided emotion score, generates a predicted user emotion score as an output of a machine learning (ML) model using the internet speed test data as an input to the ML model, wherein the predicted user emotion score is related to an overall emotion of a user of the client device relative to the Internet speed test data.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to methods and systems for predicting user sentiment associated with internet speed tests. Other embodiments are also described. BACKGROUND

[0002] In recent years, with the proliferation of media streaming devices such as computer tablets and smartphones, users stream a considerable amount of data. For example, to stream a 4K movie, approximately nine gigabytes (GB) of data is required per hour of movie. Accordingly, a computing network to which a media streaming device connects to the internet requires high bandwidth to handle the data, where bandwidth relates to the capacity of a network that can transmit data. In particular, bandwidth can be the maximum rate of sending (or maximum sending rate) that the underlying network can achieve without loss (or with minimal loss). This is especially true in cases where multiple computing devices exchange data across a given network. SUMMARY

[0003] Bandwidth of an internet connection (e.g., internet speed) can be measured by using an internet speed test, which can be provided by a speed test provider. To perform a speed test, a client device (e.g., a desktop computer, a smartphone, etc.) can request an internet speed test server to transfer as much data as possible in order to load the internet connection to its saturation point. For example, once considered to have “reached” the saturation point after transferring all the requested data to the client device, or a timer expires, the client device determines and records the bandwidth by measuring the rate of receiving data or by determining how long it takes to transfer the requested amount of data. The results of the speed test can include information such as download speed and / or upload speed, which can be presented to a user of the client device through a graphical user interface (GUI) displayed on a display of the client device.

[0004] At the end of the speed test, once the results are displayed within the GUI, the speed test provider can provide the user with the ability to submit feedback (e.g., through a user interface (UI) in the GUI) regarding their satisfaction with the speed test results. This feedback can provide key insights into customer sentiment regarding the customer’s internet speed. For example, when the results indicate that the customer’s speed is low (e.g., below a threshold), the customer can provide feedback indicating low satisfaction. This insight can enable the network operator (provider) to proactively address customer concerns and optimize its service in order to address low customer satisfaction. For example, the operator can upgrade network infrastructure to improve network bandwidth, or can offer incentives such as discounts to the customer based on the feedback. This proactive approach by the network operator can lead to improved customer satisfaction, retention, and overall business success. However, generally, customers do not provide enough feedback volume that the network operator can analyze to gain a comprehensive understanding of customer satisfaction. In fact, only about 2% of the total number of speed tests performed have customer feedback submitted. Therefore, there is a need to predict (estimate) customer (user) sentiment for internet speed tests.

[0005] The present disclosure relates to a method and system for predicting user sentiment for internet speed tests using a machine learning (ML) model. For example, the system can receive internet speed test data for an internet speed test performed over a data connection between a (first) client device and a remote server. The system can determine whether there is a user-provided sentiment score (or user feedback) associated with the internet speed test data. In particular, for example, the system can determine that a user of the client device has provided feedback by selecting one or more UI items of a GUI displayed on the client device. In response to determining that there is no user-provided sentiment score associated with the internet speed test data, the system can use the test data as input to a ML model, produce a predicted user sentiment score as output from the ML model, where the predicted score relates to the overall sentiment of the user of the client device regarding the internet speed test data. In one embodiment, the ML model can be trained using training data, which includes internet speed test data for a number of internet speed tests, and includes user-provided sentiment scores associated with the data for the internet speed tests. The internet speed test (and / or training) data can include at least one of download speed, upload speed, download speed to upload speed ratio (DL / UL), jitter, latency, and geographic region in which the client device is located. Thus, by leveraging machine learning techniques and analyzing various factors of the internet speed test data, the present disclosure provides insights into the level of user satisfaction, which can be used to improve the overall user experience.

[0006] As described herein, an ML model can be trained to output predicted user sentiment scores based on internet speed test data. In one embodiment, the ML model may consider the location of the client device when generating the predicted score. For example, users in different regions (e.g., different cities) may (e.g., on average) provide different feedback for similar or identical speed test results. Specifically, the system may receive a second set of internet speed test data for a second internet speed test performed on a second data connection between a second client device and a remote server, wherein the second client device is located in a second geographic region (e.g., in a different city than the first client device), and wherein the second set of internet speed test data includes at least one piece of data that is the same as the first set of internet speed measurements, such as the same DL / UL ratio, for the speed test performed between the first client device and the remote server. The system uses the second set of internet speed test data as input to the ML model and produces a second predicted user sentiment score as output of the ML model, wherein the second predicted user sentiment score differs from the first predicted user sentiment score. This may be due to various reasons, such as the first region may be used to indicate (e.g., on average) having higher internet speeds than the second region (e.g., because customers in the first region have better upgraded network infrastructure than those in the second region). Therefore, when two users produce similar or identical speed test results, the ML model may produce a lower predicted user sentiment score for the user in the first region, while producing a higher predicted user sentiment score for the user in the second region.

[0007] In one embodiment, this disclosure may consider anomalous client feedback. Generally, a customer providing feedback may have experienced worse test results than other customers within their geographic area. This could be due to network issues that may be specific to that customer (e.g., a faulty network device). The predicted user sentiment score may be greater than (or less than) the average user sentiment score provided for several internet speed tests performed on several other client devices located within the same reason. Therefore, the system can provide a more accurate user sentiment score than the user-provided sentiment score might be.

[0008] In another embodiment, in response to the system determining the existence of a user-provided sentiment score associated with internet speed test data, the system can train an ML model using at least some of the user-provided sentiment score and the internet speed test data. In some embodiments, the ML model can be at least one of a random forest, randomized search, neural network, and regression model. In another embodiment, the operations described herein can be performed by a remote server, which may be operated by an internet speed test provider.

[0009] According to another embodiment of this disclosure, a server includes at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the server to: determine internet speed test data for an internet speed test performed with a client device; determine whether the client device has provided feedback related to the internet speed test data; in response to determining that the client device has not provided feedback, estimate a predicted user sentiment score based on at least a portion of the internet speed test data using a machine learning (ML) model, wherein the predicted user sentiment score is related to the user sentiment of the internet speed test data; and store the predicted user sentiment score in the memory. In one embodiment, the server may perform at least some of the operations described herein.

[0010] According to another embodiment of this disclosure, a non-transitory machine-readable medium stores instructions that, when executed by at least one processor of an electronic device, cause the electronic device to: receive internet speed test data of an internet speed test performed on a data connection between a client device and a remote server, wherein the internet speed test data does not include user feedback from the client device; and use the internet speed test data as input to a machine learning (ML) model, and output a predicted user sentiment score as the ML model, wherein the predicted user sentiment score is correlated with the overall sentiment of the user on the client device regarding the internet speed test. In one embodiment, the non-transitory machine-readable medium stores instructions that, when executed, cause the electronic device to perform at least some of the operations described herein.

[0011] In one embodiment, past internet speed test data, including user feedback from other client devices located within the same geographic area as the client device, can be used to train an ML model. Therefore, feedback from client devices within these areas can be used to train different ML models to predict sentiment for different regions. This can provide a more accurate picture of user sentiment within that region.

[0012] The foregoing overview does not include an exhaustive list of all embodiments of this disclosure. It is contemplated that this disclosure includes all systems and methods that can be practiced according to all suitable combinations of the various embodiments outlined above, as well as those disclosed in the following detailed description and specifically pointed to in the claims filed with the application. Such combinations have specific advantages not specifically detailed in the foregoing overview. Attached Figure Description

[0013] Embodiments of this disclosure are illustrated by way of example rather than limitation in the accompanying drawings, wherein similar reference numerals indicate similar elements. It should be noted that references to “a” or “one” embodiment of this disclosure do not necessarily refer to the same embodiment, and they mean at least one. Furthermore, the given drawings may be used to illustrate features of more than one embodiment of this disclosure, and a given embodiment may not require all the elements in the drawings.

[0014] Figure 1 This is a block diagram illustrating a system for predicting user emotions according to some embodiments.

[0015] Figure 2 This is a flowchart of one embodiment of a process for predicting user sentiment during an internet speed test, based on some implementation examples.

[0016] Figure 3 The figure illustrates a sentiment graph according to one embodiment, showing the difference between the sentiment provided by the user and the predicted sentiment of the user. Detailed Implementation

[0017] Several embodiments of this disclosure will now be explained with reference to the accompanying drawings. Where embodiments of the embodiments described herein are not explicitly defined, the scope of this disclosure is not limited to the portions shown, which are intended for illustrative purposes only. Furthermore, while numerous details are set forth, it should be understood that some embodiments of this disclosure can be practiced without these details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description.

[0018] References to "an embodiment" or "an embodiment" in the specification mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment, but each embodiment may not necessarily include that particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment.

[0019] Figure 1 This is a block diagram illustrating a user (client or client-side) sentiment prediction system (hereinafter referred to as the "System") 10 according to some embodiments. System 10 includes a client (client or user) device 13 that can be owned and / or operated by one or more users, a (remote) server 11, and a network (e.g., the Internet) 12. In one embodiment, the System may include more devices, such as those having one or more servers or one or more client devices. In another embodiment, System 10 may include one or more additional devices that can be communicatively coupled to the client device 13.

[0020] Server 11 and client device 13 can each be any type of electronic device capable of establishing data connections via one or more networks to exchange data (e.g., as data packets) in order to perform internet speed tests on the data connections. For example, the client device can be a tablet computer, desktop computer, mobile device (e.g., smartphone), media playback device, etc. In another embodiment, the client device can be any type of network device, such as a server, router, hub, etc. In one embodiment, the server can include a standalone electronic server, or it can include one or more servers. In another embodiment, the server can be any type of electronic device, such as a desktop computer.

[0021] Client device 13 includes a network interface 81, a controller 82, a memory 83, a speaker 86, a display 87, and an input device 88. In one embodiment, the client device may include more or fewer elements as shown herein. For example, the client device may include one or more speakers, displays, and / or input devices. As another example, device 13 may not include speakers, displays, and / or input devices. In one embodiment, an element of the client device may be part of the client device. For example, the element may be integrated within (or on) a housing (not shown) of client device 13. In another embodiment, at least some elements may be separate electronic devices that can be communicatively coupled to the client device. For example, display 87 may be a separate display that can be connected to the client device to receive image data for display.

[0022] Network interface 81 can provide client device 13 with an interface to communicate with electronic devices (such as server 11) via network 12. For example, the network interface can be configured to establish a data connection (e.g., a communication link) with server 11 (e.g., its network interface 14), and once established, exchange digital data, as described herein. In one embodiment, network 12 can be any type of computer network, such as a wide area network (WAN) (e.g., the Internet), a local area network (LAN), etc., through which devices can exchange data with each other and / or with one or more other electronic devices. In another embodiment, the network can be a wireless network, such as a wireless local area network (WLAN), a cellular network, etc., for exchanging digital (e.g., test) data. Regarding the cellular network, device 13 can be configured to establish wireless (e.g., cellular) calls, wherein the cellular network may include one or more cell towers, which may be part of a communication network (e.g., 4G LTE network, 5G network, etc.) that supports data transmission (and / or voice calls) of electronic devices such as mobile devices (e.g., smartphones). In one embodiment, device 13 can be configured to communicate with one or more devices via network 12 using any type of communication protocol, such as Transmission Control Protocol / Internet Protocol (TCP / IP), Fast User Datagram Protocol (UDP) Internet Connection (QUIC), etc.

[0023] In another embodiment, the device can be configured to wirelessly exchange data via other networks, such as a Wireless Personal Area Network (WPAN) connection. For example, client device 13 can be configured to establish a wireless connection with another electronic device via a wireless communication protocol, such as Bluetooth or any other wireless communication protocol. During the established wireless connection, the device can exchange (e.g., transmit and receive) data packets (e.g., Internet Protocol (IP) packets) with digital data.

[0024] Input device 88 can be any type of device that can be arranged to receive user input. For example, device 88 may include a keyboard, one or more buttons, a mouse, etc. In another embodiment, input device 88 may be part of display 87, whereby the display may be a touch-sensitive display screen that can be arranged to receive user input by one or more user touches.

[0025] Controller 82 may be (or include) a dedicated processor (e.g., one or more processors), such as an application-specific integrated circuit (ASIC), a general-purpose microprocessor, a field-programmable gate array (FPGA), a digital signal controller, or a set of hardware logic structures (e.g., filters, arithmetic logic units, and dedicated state machines). The controller may be configured to perform one or more Internet speed test operations and / or network operations, as described herein. Further details regarding the operations performed by the controller are described herein.

[0026] Memory 83 can be any type of non-transitory machine-readable storage medium, such as read-only memory, random access memory, CD-ROM, DVD, magnetic tape, optical data storage device, flash memory device, and phase-change memory. Although illustrated as being included within device 13, one or more components can be part of a separate electronic device, such as memory being a separate data storage device.

[0027] As shown, memory 83 includes one or more client software programs 84 and an operating system 85. The operating system (OS) 85 may be a software component responsible for managing and coordinating activities and sharing resources of device 13 (e.g., controller resources, memory, etc.). In one embodiment, the OS acts as a host for applications (e.g., program 84) running on device 13. In one embodiment, the OS provides an interface to a hardware layer (e.g., controller, memory, etc.) and may include one or more software drivers that communicate with the hardware layer. For example, a driver may receive and process data packets received through the hardware layer from one or more other devices communicatively coupled to the device. In one embodiment, the OS may include a kernel (or a portion thereof) that provides an interface between one or more programs that can be executed by (e.g., controller 82) and the hardware layer.

[0028] Client software program 84 can be any type of software application (which may include one or more instructions) that, when executed (e.g., by controller 82), enables device 13 to communicate over network 12 and perform one or more internet speed test operations. For example, the program may enable client devices (e.g., via network interface 81) to exchange messages with server 11 over network 12 using any type of communication protocol (e.g., Hypertext Transfer Protocol (HTTP), HTTPS, etc.) using any type of communication protocol (e.g., Hypertext Transfer Protocol (HTTP), HTTPS, etc.). In one embodiment, program 84 can be any application capable of interacting with a web-based application that can perform internet speed tests. In one embodiment, the software program can be an internet speed test application that, when executed, determines internet speed test data (e.g., one or more data connection characteristics), such as bandwidth, by establishing a data connection with one or more devices. Specifically, the software program can establish a TCP data connection (or a QUIC connection) with server 11, and when established, can transmit a request to initiate an internet speed test, whereby the server can transmit (e.g., test) data as described herein. In one embodiment, the program may be configured to access a web browser (e.g., hosted by server 11) through which tests can be performed. Once the test data has been transmitted (or the time has expired), the client software program 84 may be configured to determine internet speed test data, which may include connectivity characteristics such as download speed, upload speed, download speed to upload speed ratio (DL / UL), jitter, latency, and the geographic region where the client device 13 is located. In one embodiment, the program may be configured to display a graphical user interface (GUI) on display 87, the GUI being arranged to present the internet speed test data results. In another embodiment, the client software program may transmit at least some of the internet speed test data to server 11, which may use this data to train and / or test a user sentiment model 18. Further details about this model are described herein.

[0029] In one embodiment, the client software program 84 may also be configured to display a UI item in the GUI requesting feedback on the internet speed test results. For example, the feedback may indicate (or include) a user-provided sentiment score associated with the internet speed test data. For instance, the software program may request the user of client device 13 to rate the speed test data within a numerical range (e.g., between 0 and 5, where 0 indicates dissatisfaction and 5 indicates satisfaction). In another embodiment, the GUI may request the user of the client device to indicate a specific data point indicating satisfaction or dissatisfaction. In some embodiments, the client software program 84 may provide the user with the ability to include comments. Once user feedback is received, the client software program 84 may be configured to cause the client device to transmit the feedback (user-provided sentiment score) to server 11 via network interface 81. In one embodiment, the feedback may be transmitted to the server along with or separately from the internet speed test data.

[0030] Server 11 includes a network interface 14, a controller 15, and a memory 16. In another embodiment, the server may include other elements, such as one or more input devices. Memory 16 includes server software program 17, user sentiment model 18, internet speed test data 19, and predicted user sentiment scores 80. In one embodiment, the server may be a server of an internet speed test provider, which may be configured to perform internet speed tests on data connections to one or more client devices, such as device 13. In conjunction with or instead of performing speed tests, the server may be configured to predict the user sentiment of the user on the client device performing the speed test. For example, server 11 may perform speed tests and / or predict user sentiment.

[0031] Server software program 17 can be any type of software program that, when executed by controller 15, can communicate with one or more devices over a network and can perform one or more internet speed test operations, as described herein. The server program can be configured to perform internet speed tests with one or more client devices, wherein the server can transmit test data to the client devices so that the client devices can determine (estimate) internet speed test data. Server 11 can be configured to receive internet speed test data 19 from client device 13 and store the data in memory 16. In another embodiment, server program 17 can perform any type of internet speed testing technique so that system 10 benchmarks the data connection with client device 13. For example, server program 17 can be configured to measure at least some internet speed test data (e.g., upload speed) when performing internet speed tests with client devices.

[0032] Internet speed test data 19 may include one or more connectivity characteristics determined from internet speed tests performed on one or more client devices. For example, at least some of the internet speed test data 19 may be determined from internet speed tests performed between server 11 (and / or other internet speed test servers) and one or more client devices (such as device 13). In another embodiment, server 11 may receive data from internet speed tests performed by other devices (e.g., between other internet speed test servers and client devices). For example, speed test data 19 may be received from an internet speed test server that has already performed speed tests with a client device. In this case, once the client device completes the speed test, the device may transmit connectivity characteristics and / or feedback to one or more internet speed test servers, which may include server 11. In one embodiment, internet speed test data 19 may have been compiled over a period of time (e.g., weeks, months, years) from several client devices in many geographic regions (such as cities, states, countries). In another embodiment, a geographic region may be any type of land area. For example, a geographic region may be one of several non-overlapping land areas, such as 500 square meters of land.

[0033] In one embodiment, internet speed test data 19 may include at least some of the connectivity characteristics described herein, such as the DL / UL ratio. In one embodiment, test data 19 may include (or be associated with) user-provided sentiment data from a client device, to which data 19 may be associated. In one embodiment, test data 19 may be labeled data that can be used to train (and / or test) a user sentiment model 18. Specifically, data 19 may be labeled based on associated user-provided sentiment data. For example, when user-provided sentiment data is a score between 0 and 5, the data may be labeled using associated numerical scores provided by users of the client devices from which the data was generated. In another embodiment, the data may be labeled with one or more other tags, such as geographic tags indicating the geographic region in which the associated client device is located when the internet speed test data is determined. In yet another embodiment, test data 19 may include connectivity characteristics associated with associated user-provided sentiment data and / or characteristics that may not be associated with user-provided sentiment data.

[0034] The user sentiment model 18 (hereinafter referred to as the "Model") can be a machine learning (ML) model that can be trained to predict user sentiment based on internet speed test results from one or more client devices. Specifically, Model 18 can be trained using at least some of the internet speed test data 19 to predict user sentiment. For example, the Model can utilize supervised learning techniques to predict sentiment ratings (scores) without sentiment labels using labeled internet speed test data 19. In particular, the Model can be trained to predict user sentiment during internet speed tests where users may not have provided any (or sufficient) test-related feedback to provide a more comprehensive understanding of user (customer) satisfaction. In one embodiment, the Model can be any type of machine learning model, such as a random forest, randomized search, neural network, or regression model. The Model can be configured to output a predicted user sentiment score in response to one or more connectivity features of the internet test data.

[0035] In one embodiment, model 18 can be trained to predict user sentiment based on the geographic region where the client device is located. Specifically, the model can be trained to predict sentiment by considering regions such as city, state, country, etc. In this case, the model is trained by using region-specific internet speed test data to consider (e.g., as labels) regions. For example, to train a model to predict sentiment for a given region, the system can compile previous internet test data for that region and then use that data to train the model. Thus, the model can predict different sentiments for different regions. In one embodiment, compiled internet speed test data with associated user sentiment for a region (where the provided sentiment is below average) can be used to train the model such that the predicted sentiment for that region is lower than the predicted sentiment for another region. Thus, the model can predict different sentiments for the same internet test data for different regions. For example, when the region is a city, user sentiment model 18 can be configured to predict that an internet speed test for a client device in Los Angeles, California results in a predicted user sentiment of 3.5 for a first set of connectivity characteristics, such as a specific ratio of DL / UL as input to the model. To predict the sentiment of an internet speed test on another client device in Denver, Colorado, using at least some of the same connectivity characteristics (such as the same ratio as the input), the model could predict a user sentiment of 4. This is possible when sentiments may differ across different regions, even if they may experience the same internet speed test results. Further details regarding different sentiments for predicting similar results are described herein. In another embodiment, memory 16 may include multiple user sentiment models, each designed for a given geographic region.

[0036] In another embodiment, multiple connectivity features can be used to train model 18 to provide more accurate predictions of user sentiment. Continuing the previous example, the model can predict different sentiments for two client devices that include at least some of the same internet test data, due to having at least some different test data. For example, two internet speed tests may produce the same DL / UL, but may have different download and / or upload speeds. For instance, if DL=110 and UL=100, one client device can provide a DL / UL of 1.1, and another client device can provide the same DL / UL of 1.1, but with drastically different download / upload speeds, such as DL=1.1 and UL=1. In this case, the model can use download and / or upload speeds as additional inputs to account for these differences, which would lead to different user sentiments towards the two devices because the first device has faster upload and download speeds than the other.

[0037] In one embodiment, model 18 can be based on historical internet speed test data. As described herein, the server can compile speed test data and / or user feedback over time (such as months, years, etc.). Therefore, this past data can be used to train model 18 to provide accurate predictions of user sentiment for a given geographic area. In another embodiment, model 18 can be updated (adapted) to changes in internet speed test data (over a period of time). For example, as user-reported sentiment improves, the model can be trained to adapt predicted sentiment based on that improvement.

[0038] The predicted user sentiment score 80 can be a result of model 18. As described herein, the score can be numerical (e.g., a value between 0 and 5). As another example, the score can be or relate to one or more connectivity characteristics derived during an internet speed test. For example, the score can indicate whether a user on a client device is satisfied or dissatisfied with a given characteristic, such as download speed. In another embodiment, the score 80 can include written comments from a user on a client device. In some embodiments, the score can be any type of indication of user sentiment related to the internet speed test results.

[0039] Figure 2This is a flowchart of one embodiment of a process 20 for predicting user sentiment for an internet speed test, according to some embodiments. In one embodiment, at least some of the operations may be performed by system 10 (e.g., server 11 of system 10). For example, at least some operations may be performed by controller 15 (e.g., server software program 17 that may be executed by controller 15). Process 20 begins with controller 15 receiving a request from client device 13 to perform an internet speed test on the data connection between the client device and the server (at block 21). As described herein, a user of client device 13 can execute client software program 84 for performing the internet speed test. This program may include a GUI that can be displayed on display 87 of client device 13, the GUI including UI items that can cause the client device to transmit a request to server 11 via network 12 in response to receiving user input via input device 88. Controller 15 can perform the internet speed test (at block 22). As described herein, controller 15 can cause server 11 to transmit test data to the client device, which the client device can use to generate internet speed test data.

[0040] Controller 15 receives internet speed data from an internet speed test performed on the data connection (at box 23). Specifically, controller 15 can determine test data for a speed test performed on the data connection between the client device and server 11. In another embodiment, controller 15 can receive data from a speed test performed between the client device and another (e.g., remote) server, which may be a separate (independent) internet speed test server. Controller 15 determines whether a user-provided sentiment score is associated with the internet speed test (at decision box 24). Specifically, controller 15 can determine, based on the received internet speed test data, whether the client device provided (e.g., transmitted) feedback that may include or indicate the sentiment provided by the user. For example, once the internet speed test has been performed, the client device can offer the user of the device the option to provide feedback. If feedback is provided, the client device can transmit the feedback along with the internet speed test data to server 11. Otherwise, if no feedback is provided (e.g., the client software program 84 does not receive feedback for a period of time after the speed test is performed, or the user of the client device does not select a UI item for providing feedback), the client device 13 may transmit the Internet speed test data without user feedback (and / or without an instruction to provide user feedback along with the Internet speed test data).

[0041] In response to determining that the internet speed test data does indeed include user feedback from client devices (or that the user feedback is provided by a client device that may be associated with the received internet speed test data), controller 15 trains user sentiment model 18 (in box 25) using the internet speed test data. In this case, the controller can label at least some of the received test data based on the user feedback, and can then use the labeled data to train (and / or test) the model. Thus, the system can use the received test data associated with the user feedback to train model 18. Specifically, the controller can receive test data that includes internet speed test data for several internet speed tests and includes feedback (user-provided sentiment scores) associated with the test data, and can use the received test data (based on user-provided sentiment labels) to train the model.

[0042] However, if client device 13 does not provide user feedback, controller 15 uses the received internet speed test data as input to model 18 to produce a predicted user sentiment score as the model's output (at box 26). In one embodiment, the predicted user sentiment may be a prediction of the type (or magnitude) of emotion that the user of the client device would express in the event of user feedback regarding the internet speed test results. In one embodiment, the predicted user sentiment score may be related to the user of the client device's overall sentiment regarding the internet speed test data. For example, the client software program may request the user to rate the user experience on a scale of 0 to 5, where 0 may indicate overall dissatisfaction with the test (result), and 5 may indicate overall satisfaction.

[0043] In another embodiment, the predicted user sentiment score may be based on at least a portion of the internet speed test data. For example, the model may use one or more connectivity features as input to predict user sentiment. In this case, user sentiment may be related to (or associated with) these input features. For example, when download speed is input into the model, the model may output user sentiment related to download speed. In another embodiment, the controller may use other features to predict sentiment. For example, if memory 16 includes different models for different regions, controller 15 may be configured to extract a user sentiment model (trained to predict scores) associated with a given geographic region associated with the internet speed test data, and then use that model to predict sentiment, as described herein. Controller 15 may store the predicted user sentiment score in memory 16, along with the stored score 80 (at box 27). In one embodiment, the system may compile (store) the predicted user sentiment score 80 over a period of time, and may transmit the predicted score to one or more network operators providing network services to client devices to provide user (customer) insights.

[0044] In one embodiment, at least some of the operations in process 20 may be optional, such as boxes 21 and 22. This could be a case where server 11 may not provide an internet speed test for the client device's data connection, but instead it could be performed by another remote server. Once the test is complete, client device 13 (and / or the remote server) may transmit the internet speed test data and / or any user feedback associated with the data.

[0045] In another embodiment, the system may perform at least some of these operations over a period of time to continuously train the user sentiment model 18 to learn and improve predictions as more labeled data becomes available. Thus, the user sentiment model can adapt to user sentiment (which may change over time in a given geographic area) and remain up-to-date to ensure that sentiment predictions remain accurate (for a given area).

[0046] As described herein, model 18 can be trained to predict different user sentiment scores based on the geographic region associated with the client device. For example, controller 15 can receive the same (or at least some similar) test data from different client devices located in different regions. The model can produce a predicted user sentiment score for a set of test data that is higher than another predicted user sentiment score for a different set of test data. This may be due to the fact that the test data associated with the two regions may be labeled differently. For example, the region that produces a higher predicted user sentiment score may provide a higher level of user-provided sentiment than users from other regions, which model 18 can utilize for training.

[0047] In another embodiment, user sentiment model 18 can provide a more accurate assessment of user sentiment than the sentiment actually provided by users. As described herein, less than 2% of clients conducting internet speed tests are likely to provide user feedback. In this case, users who receive less than satisfactory speed test results (e.g., low download speeds) are more likely to provide negative feedback than users who experience satisfactory results (e.g., higher download speeds). Therefore, the sentiment provided by users (e.g., for a given area) may be lower than the actual user sentiment (e.g., average) of users conducting internet speed tests in that area, because most users who receive good speed test results (e.g., above a user threshold) are unlikely to provide (e.g., negative) feedback. System 10 can train model 18 to take into account (and adjust) the predicted user sentiment so that it reflects actual user sentiment, rather than a lower (or higher) sentiment reflected only by a few anomalous feedbacks.

[0048] Figure 3 The figure illustrates a sentiment graph 30 according to one embodiment, showing the difference between user-provided sentiment and predicted user sentiment. In particular, for example, the figure 30 shows how user sentiment model 18 can predict a more accurate assessment of user sentiment for a specific geographic area, which may differ from the user-provided sentiment (e.g., on average).

[0049] Figure 30 shows sentiment scores for internet speed test data, such as download speed. The figure includes sentiment scores provided by several users and several predicted user sentiment scores. Specifically, the internet speed test data in the figure may represent an average (or overall) representation of the connectivity characteristics of one or more measurements from one or more internet speed tests. However, the user-provided scores may be based on scores provided by a subset of users who conducted the internet speed tests. Therefore, the user-provided scores shown may not accurately reflect the actual user sentiment, as the scores may be based on only a subset of users providing feedback (e.g., the overall sentiment may be negative).

[0050] Figure 30 includes a sentiment score 31a provided by a first user and a sentiment score 31b provided by a second user. Each user-provided sentiment score can be associated with a specific graphical region and can be an average of scores provided by several users (based on several internet speed tests performed on several client devices located within their respective regions). Based on internet speed test data, each score may not accurately reflect the sentiment of users within its respective region. For example, the first score 31a could represent the average sentiment provided by users on client devices in Los Angeles, California, and the second score 31b could represent the average sentiment provided by users on client devices in Denver, Colorado. Although clients in Los Angeles experienced better internet speed test data (e.g., faster download speeds) than those in Denver, Colorado, the first score 31a is lower than the sentiment score 31b provided by users in Denver.

[0051] The discrepancy between these two scores could be based on a variety of factors. For example, the vast majority of users providing feedback in Los Angeles (e.g., exceeding a threshold) might be those who experienced internet speed test results worse than the average for all users in Los Angeles. These users might provide worse feedback (on average) than users in Denver, even though the average internet speed test results for users in Los Angeles are better than those for users in Denver. On the other hand, the sentiment score 31b provided by users in Denver could be a result of some users providing positive feedback (e.g., values ​​higher than those provided by users in Los Angeles who provided feedback) exceeding the overall user sentiment in Denver. This could occur when users providing feedback (who might represent less than 50% of Denver's total population, or less than 50% of users in Denver who performed internet speed tests) have download speeds better than the average download speed for users in Denver, thus providing a higher-than-expected sentiment score for the average population.

[0052] Model 18 takes into account this anomalous feedback and can generate predicted user sentiment scores to reflect actual user sentiment, which may encompass a wider population than the sentiment provided by the user. In one embodiment, Model 18 can be trained using past internet speed test data, which includes user feedback from other client devices located in the same geographic area as the client device. Specifically, Figure 30 shows two predicted user sentiment scores 32a and 32b associated with the regions of user-provided scores 31a and 31b. Specifically, a first predicted score 32a for a client device located in Los Angeles (which did not provide user feedback) is greater than the average user-provided sentiment score 31a, while a second predicted score 32b for a client device located in Denver is less than the average user-provided sentiment score 31b. Both scores can be predicted using connectivity characteristics, such as the download speed from which server 11 receives data from each of these devices, as described herein. In one embodiment, Model 18 can predict more accurate user sentiment based on training data (e.g., across one or more regions) that indicates higher overall user sentiment when internet speed test data reflects better data connectivity (e.g., faster download speeds).

[0053] As explained above, one embodiment of this disclosure may be a non-transitory machine-readable medium (such as a microelectronic memory) having instructions stored thereon that program one or more data processing components (collectively referred to herein as a "processor") to perform at least some of the operations described herein. In other embodiments, some of these operations may be performed by specific hardware components containing hard-wired logic. Alternatively, these operations may be performed by any combination of programmable data processing components and fixed hard-wired circuit components.

[0054] In order to assist the Patent Office and any reader of any patent granted based on this application in interpreting its appended claims, the applicant wishes to indicate that, unless the phrases “means for…” or “steps for…” are expressly used in the claims, it is not intended to cause any appended claim or claim element to invoke 35 USC 112(f).

[0055] While certain embodiments are described and illustrated in the accompanying drawings, it should be understood that these embodiments are merely illustrative of a broad disclosure and not limiting, and that this disclosure is not limited to the specific constructions and arrangements shown and described, as various other modifications may be made by those skilled in the art.

[0056] In some embodiments, this disclosure may include language such as “at least one of [element A] and [element B]”. This language can refer to one or more of the elements. For example, “at least one of A and B” can refer to “A”, “B”, or “A and B”. Specifically, “at least one of A and B” can refer to “at least one of A and at least one of B”, or “at least one of A or B”. In some embodiments, this disclosure may include language such as “[element A], [element B], and / or [element C]”. This language can refer to any one of these elements or any combination thereof. For example, “A, B, and / or C” can refer to “A”, “B”, “C”, “A and B”, “A and C”, “B and C”, or “A, B, and C”.

Claims

1. A method comprising: Receive internet speed test data from an internet speed test performed on a data connection between a client device and a remote server; Determine whether there are user-provided sentiment scores associated with the internet speed test data; as well as In response to determining that no sentiment score was provided by the user, the internet speed test data was used as input to a machine learning (ML) model to generate a predicted user sentiment score as output of the ML model, wherein the predicted user sentiment score is related to the overall sentiment of the user on the client device regarding the internet speed test data.

2. The method according to claim 1, wherein the internet speed test data includes at least one of download speed, upload speed, download speed to upload speed ratio (DL / UL), jitter, latency, and the geographical region where the client device is located.

3. The method according to claim 2, wherein the geographical region is a first geographical region, the internet speed test is a first internet speed test, the data connection is a first data connection, the client device is a first client device, the predicted user sentiment score is a first predicted user sentiment score, and the internet speed test data is a first set of internet speed test data, wherein... The method further includes: Receive a second set of Internet speed test data for a second Internet speed test performed on a second data connection between a second client device and a remote server, wherein the second client device is located in a second geographical region, and wherein the second set of Internet speed test data includes the same DL / UL ratio as the first set of speed test data; Using a second set of internet speed test data as input to the ML model, a second predicted user sentiment score is generated as the output of the ML model, wherein the second predicted user sentiment score is different from the first predicted user sentiment score.

4. The method according to claim 1, wherein, When the client device is located within a geographic area, the internet speed test is performed, wherein the predicted user sentiment score is greater than or less than the average user sentiment score provided by multiple internet speed tests performed on multiple client devices located within the geographic area.

5. The method of claim 1, further comprising: Receive training data, which includes internet speed test data from multiple internet speed tests and user-provided sentiment scores associated with the internet speed test data from the multiple internet speed tests; as well as The received training data is used to train the ML model.

6. The method of claim 1, further comprising, in response to determining the existence of a user-provided sentiment score associated with the internet speed test data, training the ML model using at least some of the user-provided sentiment score and the internet speed test data.

7. The method of claim 1, wherein the ML model is at least one of random forest, randomized search, neural network, and regression model.

8. The method of claim 1, wherein the receiving, determining, and generating are performed by the remote server.

9. A server, comprising: At least one processor; as well as A memory containing instructions that, when executed by the at least one processor, cause the server to: Determine the internet speed test data for the internet speed test performed on the data connection with the client device; Determine whether the client device provides feedback related to the internet speed test data; In response to determining that the client device does not provide feedback, a machine learning (ML) model is used to estimate a predicted user sentiment score based on at least a portion of the internet speed test data, wherein the predicted user sentiment score is related to the user sentiment of the internet speed test data; as well as The predicted user sentiment score is stored in the memory.

10. The server of claim 9, wherein the Internet speed test data includes at least one of download speed, upload speed, download speed to upload speed ratio (DL / UL), jitter, latency, and the geographical region where the client device is located.

11. The server of claim 10, wherein the geographical region is a first geographical region, the internet speed test is a first internet speed test, the data connection is a first data connection, the client device is a first client device, the predicted user sentiment score is a first predicted user sentiment score, and the internet speed test data is a first set of internet speed test data, wherein, The memory further has the following instructions: Receive a second set of internet speed test data performed on a second data connection with a second client device, wherein the second client is located in a second geographical region, and wherein the second set of internet speed test data includes the same DL / UL ratio as the first set of speed test data; and Using a second set of internet speed test data as input to the ML model, a second predicted user sentiment score is generated as the output of the ML model, wherein the second predicted user sentiment score is different from the first predicted user sentiment score.

12. The server of claim 9, wherein the internet speed test is performed when the client device is located within a geographic area, wherein the predicted user sentiment score is greater than or less than the average user sentiment score provided by a plurality of internet speed tests performed on a plurality of client devices located within the geographic area.

13. The server of claim 9, wherein the memory has further instructions to: Receive training data, said training data including internet speed test data from multiple internet speed tests and including feedback associated with said internet speed test data; and The received training data is used to train the ML model.

14. The server of claim 9, wherein the memory has further instructions to train an ML model using at least some of the user-provided emotion score and the internet speed test data in response to determining the presence of feedback including a user-provided emotion score.

15. The server of claim 9, wherein the ML model is at least one of random forest, randomized search, neural network, and regression model.

16. A non-transitory machine-readable medium storing instructions that, when executed by at least one processor of an electronic device, cause the electronic device to: Receive internet speed data from an internet speed test performed on a data connection between a client device and a remote server, wherein the internet speed test data does not include user feedback from the client device; and The internet speed test data is used as input to a machine learning (ML) model, and the predicted user sentiment score is used as output to the ML model, wherein the predicted user sentiment score is related to the overall sentiment of the user of the client device relative to the internet speed test data.

17. The non-transitory machine-readable medium of claim 16, wherein the Internet speed test data includes at least one of download speed, upload speed, download speed to upload speed ratio (DL / UL), jitter, latency, and the geographical region where the client device is located.

18. The non-transitory machine-readable medium of claim 17, wherein the geographic region is a first geographic region, the internet speed test is a first internet speed test, the data connection is a first data connection, the client device is a first client device, the predicted user sentiment score is a first predicted user sentiment score, and the internet speed test data is a first set of internet speed test data, wherein... The non-transitory computer-readable medium further includes instructions to: Receive a second set of Internet speed test data for a second Internet speed test performed on a second data connection between a second client device and a remote server, wherein the second client is located in a second geographical region, and wherein the second set of Internet speed test data includes the same DL / UL ratio as the first set of speed test data; Using a second set of internet speed test data as input to the ML model, a second predicted user sentiment score is generated as the output of the ML model, wherein the second predicted user sentiment score is different from the first predicted user sentiment score.

19. The non-transitory machine-readable medium of claim 16, wherein the internet speed test is performed when the client device is located within a geographic area, wherein the predicted user sentiment score is greater than or less than the average user sentiment score provided by a plurality of internet speed tests performed on a plurality of client devices located within the geographic area.

20. The non-transitory machine-readable medium of claim 16, wherein the ML model is trained using past internet speed test data, the data including user feedback from other client devices located in the same geographic area as the client device.

Images